foreign
good morning again everyone welcome back
to the last session before lunch
um we're going to continue linking
python with other kinds of language with
a talk by David Bowe and David's a
software engineer and former University
tutor
um in his spare time David mentors the
drop bears a first first first
robotics competition team and helps to
maintain robot pie a project to bring
python to the FRC and I'm a little bit
worried about robotic drop bears so
we'll see if you've got them covered
um thanks very much David give him a
welcome
[Applause]
I feel like this is a bit of an
interesting duel to the the previous
talk because I'm about to do basically
the exact opposite sort of
um but let me set the scene a little bit
so perhaps you have a somewhat large C
plus library that you want to ship to
python users but you don't
want to put in a lot of work into doing
that
um or maybe you've written a a c plus
library for
perhaps for performance reasons and you
want an easy way to be able to use it
from python
um so this is the sort of problem that
we had uh it's robot Pi which is a
project to bring python to the first
robotics competition we have to deal
with some reasonably large libraries and
both open and closed source and this is
because they do actually have to
communicate with Hardware so it needs to
be able to call native code at some
point and to give you an idea of what
the first robotic competition is this is
teams of high school students with adult
mentors building 56 kilo robots to play
a sporty kind of game
um so you can see an example there
they get a new challenge each year to
try to keep it interesting
um so yeah let's get into it uh so
one option that you might have is Swig
or the simplified wrapper and interface
generator
um
this lets you uh
take some C or C plus plus and output a
bunch of bindings for
multiple different languages and this is
one of the the oldest ones that I could
find
um
and the fact that it works across so
many different languages is probably a
testament to how mature it is
and
they do intentionally try to keep it as
simple as possible so this is a very
simple example uh where you can take a
take your header file for your library
and you just you can just include it
um and you might want to ignore some
things maybe
um because they're convenient for
C C plus plus but it doesn't really make
sense for python
um
one of the other uh popular open source
projects is sheboken which is one of the
underlying libraries used by QT for
python or you if you've been around for
a while you've been using QT you might
know it as Pi side that
Pi side projects uh eventually became
adopted by the QT project and is now
like the quote-unquote official bindings
for QT
and they use some XML to configure
which kind of looks like this
uh which this is this is an example I
pulled straight from their docs it is a
little bit intimidating though so I'm
going to break it down a little bit and
so here we have our first class that
that we're binding to on the left hand
side is the XML the right hand side is
the C plus definition it's a declaration
for it sorry
um you can see we have two methods in
there
um but the XML only talks about one of
them so by default uh all methods are
exposed to python
and
here we have to if we say that we want
some special handling for this clone
method uh particularly we modify what it
calls argument index zero that's the
return value of of that method
and here we're actually saying that uh
the bindings should not uh destroy the
object and it should instead let C plus
plus deal with the memory management
because we get a pointer so so it
doesn't actually know
necessarily how to handle that
and then we have this other class
there's a whole bunch of more methods
but what's interesting in the the the
bindings here is that we also have a
property that we can Define so you can
just you know do truck dot arrival
message and instead of having to call
get a rival message and and set a rival
message to set it
um and and then we have a similar thing
with memory management
we
and argument index one actually refers
to the argument this time so
um
uh and this takes us to the
underlying library that we use uh we
chose to use which is pi bind 11. it's a
c plus header only Library so it's
pretty easy to pull into a c plus
project
and it lets you build a module uh and
you declare everything that you want to
expose to python in a c plus file
um and it does let you also
depend on classes from different Pi bind
11 modules that you've and that have
been built
um I believe shaboken does support this
as well but I haven't actually
use that all that much uh let me know if
if that's wrong but we'll see why this
is important a little bit later
um here is an example of Pi bindle of
what pi by 11 looks like
so on the left hand side I have my C
plus code to to generate to have these
bindings
um first we say that we're building an
a511 module and then next here is our
module name so so to import this module
from my python code I would you know say
import python EU example
and then M here is a variable name in
the C plus code to say this is the
module that I'm putting together so the
first thing that we have here is we
Define and add thing specifically the
where putting the ad
function that you can see on the right
hand side that declaration and we're
exposing that to python
and then we have this pet class
so we say I want a class called pet
we put that on the module and it
corresponds to the C plus plus pet class
that you can see
so you so we
this first line here with the the pi
pack in it
that corresponds to the Constructor here
and then we
bind the set name get name methods
fairly straightforward
and then we can also declare a property
just like you could in chibokin
so
this
kind of takes us into a journey that we
went through with a couple of our
earlier approaches to
to generate All That C plus plus code
that we would have to write otherwise
um
because
we have some fairly large libraries
right
um so the
first example that I I'll show here
is a involves a vendor library that
provides a c interface
so we ended up building a library that
can take a c plus header file
it pauses it using a pure python C plus
plus parser
there were good reasons for this
um
but this Library lets us
generate lit literally whatever we want
from a c plus header
so it takes in some python code and a
bit of yaml configuration
that's provided to that python code and
then puts it into a ginger to template
um
so
here's an example of what that sort of
looks like on the left I have some yaml
um
so each top level entry is each function
that we want to bind to
uh you'll you'll notice on the right
hand side each of these take this handle
pointer so we decide to abstract that
away and put it in a c plus so that we
don't have to deal with it
um and then you can see here's something
interesting
we provide we can set
some default values for for some of
these parameters and
you might notice that that's a default
value in the middle here that's because
in our python code we had set a
global default for for parameters called
timeout Ms to zero
this is what the
ginger 2 template looks like to generate
the C plus plus classes
um it's scary that if you actually want
to read the code there's a link there
but this is then the template for the pi
bind 11 code
um
feel like you probably get the idea so I
won't show you the pipe the python code
and that that first that first
preprocessors uh it so
but yeah
so
next example
uh we ended up getting a vendor Library
uh that's only exposed a an
object-oriented uh C plus plus interface
uh so luckily we already had all this
generation code from that previous
library that that we built
so we decided to reuse some of some of
it
um
so yeah similar sort of setup uh with
the same libraries but different python
code and and the yaml looks a little bit
different
so here we have this
uh C plus plus class at the top
and then we declare a few different
methods
um you uh you might notice if you look
at the right hand side there are two
methods with the exact same name
this is called overloading and C plus
plus
where you can
essentially uh
end up calling a different function
depending on what
parameters you pass in
um but in Python this kind of ends up
looking a little bit ugly so what we did
was we actually ignored this first
method and then we set the a default for
the second overload because
we dug into it and it turns out that the
first overload actually just delegates
to the second
I'll also note here you might notice
that these method names are Pascal case
so we turn them into camel case this is
because we already had a whole bunch of
existing libraries in robot Pi that were
all camel case
uh but this also lets students uh
easily copy stuff from java code because
there's a pretty healthy uh culture of
sharing code uh within FRC
um
I think you I don't think I need to show
any more code for for this hopefully you
get the idea now
um
so that takes us to what we have today
which is this sort of generalized build
system for
for generating these uh python these
these C plus plus bindings
so robot Pi build is a
fairly opinionated build system for a
python package so you use it to clear it
in your Pi Project tunnel
um
reads reads your C plus plus headers
generates some Pi by 11 code in the
background
and it will also package up your shared
libraries or dlls in Windows parlance
together with your python package when
you upload it to IPI
uh we use C types to load it at runtime
so it's it works cross-platform but
a consequence of us packaging them
together is this is that we explicitly
don't support binding to system
libraries so if you need to use
libraries that you have installed in
your system you
might be better to use Swig or shavokin
or right
Pi by an 11 code yourself I guess sorry
um
but yeah so so
we end up with
these data files that
uh describe what headers to look at in
this library that you're building
um for what names you expect so both
functions classes in arms things like
that and also any particular special
handling that you might need of
so
give you an overview of
what our dependency graph looks like
this is just the dependency graph for
the same Library as our previous example
just now
um you might notice there's
there's a whole bunch of different
yeah we've got diamond dependencies and
and there's also subclassing of
of classes from each of these different
uh libraries as well
so this is where the pi bind 11 support
for cross module class usage comes in
because
we've got
there's been a library that has to be
able to subclass a
an interface that that's uh on one of
our other libraries provides
so this is what the pi project.tamil
looks like so at the top uh we say that
we want to to download
uh
uh our vendor Library uh and there's and
yeah in FRC we've set up this uh
standard convention uh that's
maven-based uh for where to find
uh find libraries
and then we and then we start talking
about our actual C our actual Pi bind 11
wrapper
um and you can see there we have this
depends list
s uh other packages that we've built
using robot Pi build and this lets us
actually link bitten to those other
packages
and then at the bottom uh we have a list
of
of headers that we want to look at this
is just an excerpt so uh we're just
going to be focusing on on this example
here
this is the same class from the previous
example as well
um
so there is a little bit of new things
in here where we have this keep alive
thing so what this does is it's part of
hiveine 11 and it lets you bind the
lifetime of
of one object to another
so
in this case so so here we have
zero which is our return value and one
which is self so this is saying that
this PID controller thing that's
returned by this method make sure that
it doesn't outlive this
can Spark Max
so why else do you still have this PID
controller object
the cansbach max object would not be
garbage collected
and then at the bottom here we have a
bit of inline code so this lets you uh
shove
some C plus plus code into the pi Vine
11 module that's eventually generated
and this allows us for infinite
customization so here we actually add a
method to the python class that doesn't
exist on the C plus class
um
specifically here where
adding a Dunder wrapper method
just so that our users which are high
school students can more easily debug
their robot code right they don't care
about the memory address which is what
the default wrapper is
there are other
Pi bind 11 Auto generators out there
um
some of them are listed from the pi by
11 documentation which I have also
listed here I haven't had an opportunity
to try them out but
they do actually use clang and actual C
plus plus compiler rather than a C plus
plus parser that's been written in
Python uh
which does solve a lot of the edge cases
in
having to pass C plus plus because
there because
it's it's a lot harder to parse C plus
plus than you might think
okay
so you might be wondering then how do we
ship all these libraries to our users
right we basically just want them to be
able to pip install robot pi and things
work
so what we do is we build a bunch of
wheels in CI
we have a lot of repos so we created
this reusable workflow which is a GitHub
actions feature where you can say hey
this this CI pipeline
should just run this CI pipeline that's
that's defined in this other repo
you know we build for the three
big operating systems for all the
currently supported C python versions
so
that ends up being 15 wheels that we
upload to Pipi and we deploy these
whenever we push a git tag so that
that's part of the CI pipeline as well
is that
it checks whether
we've just pushed the git tag
and we also build a bunch of arm Wheels
as well and we have a private repo for
those this is because
the uh the main controller that's used
in
FRC is a bit of an unusual Linux
um platform
but
we also but yeah we do also Target
raspbian but because arm is so diverse
right like what uh like as general said
like what arm are you are you talking
about right
so we decide that we don't want to deal
with that uh when uploading to Pi Pi and
users have to explicitly opt in to
actually using our our Wheels
now
our Wheels do technically fail audit
wheel which is the the many Linux
auditor
um
and this is because of our cross package
Library dependencies
they do otherwise satisfy many Linux
as last time I checked
so and we just end up renaming the
wheels NCI just before we upload to Pi
Pi and
we haven't had complaints so
I I'm pretty sure it works
that's all I have
um
if you want to check out check this out
there's a link to the tower repo and our
docs we also have a matrix Channel
and
yeah
thank you
thank you very much David
we've got a little bit of time for
questions
I'm curious about how
um how many other
is it using other robot platforms or on
other
um uh C plus plus projects yet
so I do believe that one of our
co-maintainers the main maintainer
actually uh has used it at work
um I'm not entirely sure on the details
because work stuff I guess but yeah yeah
cool there's a question at the front uh
question I had was why you went down the
route of Auto
generating the pi bind uh code rather
than handwriting it that's a good
question yeah so
the libraries that we had to to uh
package are actually very large like
um
like we like in one of our libraries we
have like uh
I want to say like at least 50 different
classes
um
there's also the one of our
our packages is
um some maths very maths heavy
um and so there's a bunch of classes in
there and
we want to use C plus for performance
reasons there
um but then
actually yeah yeah that's a lot happened
cool any other questions
so I was just curious if you'd looked at
using uh CI build will to solve some of
those like Auto wheel compatibility
issues
it's a good question
um
I did have a brief look at it
um but
I'm not sure that CI build will support
uh the
uh Ubuntu versions that we have to
actually Target
um
this is a good question I'll have to
double check on that
cool
anyone else I'll just walk yep time for
one more
hello
um just wondering if your approach works
with very heavily templated libraries or
header only libraries yes excellent
question yes
um so uh a few of our libraries are
actually pretty template heavy and so we
do have explicit support for that in
robot pie build as well you do have to
manually list out all the the template
instantiations that you want but yes
very cool and please join me in thanking
David once again
[Applause]
thank you so much
[Applause]