[Music]
the last talk for this block. I wonder
if the tram ding is the same one as last
year's. Apparently, it was the wrong
ding for the tram on the PyCon AU logo
last year. So, trust a a conference full
of PyCon devs to to have a slightly
unhealthy obsession with trams.
For our last speaker for this block, we
have Russell, who is going to be talking
about taking wheels mobile. This looks
really interesting because I always
thought wheels were mobile.
Um, so give him the classic big warm
PyCon A welcome. Um, and I'll hand it
over to Russell.
Well, hi there. Uh, my name is Russell
Keith McGee. I am here today to show you
how you can generate binary wheels in
Python and more specifically how to
generate binary wheels for mobile
platforms like iOS and Android. First
off, who am I? I am from Maduk
Nungabuja, otherwise known as Perth
Western Australia. I am a member of the
CPython core team. I am a principal
engineer in the open source group at
Anaconda. Anaconda pays for me to attend
conferences like this one to talk to
people like you. So, I'd like to thank
them for that support. They also support
the Python open source ecosystem more
broadly by providing direct financial
support to organizations like Num Focus
and by employing developers to work
full-time on open source projects.
Projects like Jupiter and Number and the
open source project that I founded
Beware. For those who haven't come
across it before, Beware is a collection
of open source tools and libraries for
creating native user interfaces in
Python for desktop but also for iOS, for
Android, for single page web apps, and
for console. But today I'm not going to
be talking about beware, at least not
directly. For the last 10 months or so,
my co-orker Malcolm Smith and I have
been working on the tools in the broader
Python packaging ecosystem, adding
support for iOS and Android. This work
is not part of Beware itself. Beware
uses these tools, but since iOS and
Android are now officially tier three
supported platforms for CPython, any
other project that wants to target iOS
or Android can also use these tools. So
today I'm going to be talking about that
work about wheels, how you build them,
and more specifically how you build them
for iOS and Android. So what even is a
wheel? Well, a wheel is the
redistributable unit of code in the
Python ecosystem. It was standardized in
PEP 427. It's not a complex instrument.
It is a it is basically a zip file
containing Python code in the installed
directory layout along with some
metadata describing the package and
optionally some actual data for the
package to use. The contents of the
wheel is designed so that it can be
unpacked directly into the lo to a
location on your Python path. Usually
site packages, but not necessarily, and
then used without the need to run any
additional Python code or run a compiler
or anything like that. The wheel itself
has to follow a very specific naming
pattern. What's in a name? Well, an
identifier for the package, the package
version, and then the platform
applicability of the wheel. The package
identifier has to adhere to PET PET 508.
The version number has to adhere to PET
440. And that leaves platform
applicability in three parts. There's a
Python tag that describes which Python
implementation the code is compatible
with. PI3 for any Python 3 interpreter,
CP314 for a CPython 3.14 interpreter,
PP311 for a PI311 interpreter and so on.
Then there's the AI tag, the application
binary interface. This describes the
compatibility of any binary Python
components in the wheel. This will be
none if there's no binary compatibility
concerns. It could also be a specific
Python version that matches the Python
tag CP314. It could also describe a
variant. So something like CP314T
means the code requires a freethreaded
build of Python. Or the wheel could
declare ABI3 that it uses the stable AI
which is a subset of the Python AI that
is guaranteed stable across Python
versions. Lastly, there's the platform
tag. This is where you declare operating
system and architecture compatibility.
This can be a single platform like
Win332 or platform and architecture win
ARM 64 for example. It could include a
minimum version of an operating system
like macOS 11. It could describe a meta
architecture. Universal 2 describes a
wheel that will work on MacOss on both
ARM 64 and x8664.
Even the operating system can be a meta
tag. Linux isn't an operating or Linux
isn't an operating system by itself.
It's just a kernel. So the many Linux
and muscle Linux specifications describe
sets of system libraries that the wheel
can assume exists. As an end user, you
just pip install my package. Pip then
goes to Pippi, finds all the wheels that
match considering your operating system,
operating system version, CPU
architecture, Python version, Python
variant, and builds a list of candidate
wheels including binary wheels and pure
Python wheels, which uh which can
actually themselves be platform specific
if they need to be. Pips's resolver then
works out which wheel is the best match
and installs that wheel. But as a
package developer, that means that for
each release, you need to build wheels
for multiple operating systems and
operating system versions and meta
operating systems and architectures and
meta architectures and multiple Python
implementations and versions and APIs.
Pillow, for example, is the gold
standard library for image manipulation
in Python. And Pillow 11.3 ships 105
different binary wheels. That's 105
different binary configurations that
need to be configured, built, and
tested. So, how do you generate a
release with 105 different binary
artifacts? Well, this is where a tool
called CI build wheel comes to the
rescue. CI build wheel provides the
tooling to help you build a wheel in CI.
Uh, for the uninitiated, CI is
continuous integration. Uh, think GitHub
actions. Despite the name, it's actually
not CI specific. You can also install
and use it locally, but obviously it
does work really well as part of a CI
pipeline generating 105 binary artifacts
as part of a release. CI buildwhe
provides the tools to configure the
matrix of options as a matrix. To make
your package compatible with CI build,
you need to start with a package that is
compliant with PEP 517 and 518. That
means you need a piprotoml file and that
file declares a build system table with
a requires field and a build backend
field. Requires defines the set of tools
that must be in your environment to be
able to build a wheel for your package.
Build backend def specifies the path of
a module that will be installed by the
requires. That module has to adhere to a
specific interface which provides an
entry point to trigger a build. The
build system table defines the interface
that Python will use to build a wheel
for your Python code. It does not care
how that happens, only that the build
back end will define the entry point. In
practice, you'll likely need a bunch of
other configuration details. Uh, PEP 621
and 639 project configurations. You'll
likely also need to provide
configuration that's needed by your
chosen PEP 517 build backend. There are
a whole lot of PEP 517 compliant build
backends. Setup tools, Hatch, Flit,
Poetry, Matarin, Mison. Which one should
you use? That's entirely up to you. From
Python's perspective, it does not
matter. And that is the point of PEP
517.
But once a project aderes to those peps,
CI buildwhe can then do its magic. You
don't need any additional configuration
to use CI buildwhe. PIP install CI
buildwheel then run CI buildwhe. And by
default, CI buildwhe will run through
every currently supported Python version
for your current platform and use the
517 interface to build a wheel for each
currently supported Python version. At
present, that's at least six wheels. One
each for Python 3.10, 10, 311, 312, 313,
314, and one for 314 free thread it. On
MacOss, you have to have the official
CPython interpreter for each Python
version installed. If those interpreters
aren't available, you get an error
during the build process. On Linux, you
need to have Docker installed. The many
Linux and muscle Linux Docker images are
then used to host the builds. On
Windows, CIBU will automatically
download and cache uh Python
interpreters with Nugget as they're
needed.
Just call CI buildwhe is the simplest
possible invocation of CI buildwhe but
CI buildwhe is also very configurable.
You can configure CI buildwhe to skip
certain Python versions if you want or
to include pi builds or to skip and
include specific architectures. And
there are then multiple ways to define
those configurations. As an example,
macOS currently supports both x8664 and
ARM 64 architectures and there is a
universal 2 tag that supports both. All
three options can be compiled from a
single build machine. By default, CI
build will only build the current
architecture, but you can configure CI
build to build all the options. From the
command line, pass in- arches all. And
on a Mac, CI build will generate 18
wheels. Six for x8664, six for ARM 64,
and six that are universal, too. You can
also specify options in your pi
project.l file. If you add a tool.ci
buildw wheel section, you can specify
the architectures that you want to build
by default. So for example here you're
building all available architectures on
every platform. Those configurations can
also be platform specific by adding
subtables for each platform. So in the
second configuration we're saying we
want to build all architectures on macOS
but only the AMD 64 architectures on
Windows. Alternatively you can configure
CI build with environment variables. If
you define CIBW arches that value will
be handled at the same as if it had been
passed in as a command line argument or
specified in the configuration file. The
environment variable can also be
qualified to only apply to specific
platforms. So in a single environment,
you can define environment variables for
both MacOss and Windows. When the build
occurs on MacOss, the Mac OS version
will be on it. When the build occurs on
Windows, the Windows version will be on
it. And that's just one example of
Archers. CI build has dozens of options
for configuring exactly what versions,
interpreters, and architectures are
built. And every option can then be
defined on a platform specific basis.
By default, CI build will just build the
wheels. But any good CI process should
also be running the tests. So you can
tell CI build how to run the tests for
your project. Test command defines the
command line to run the test suite. Test
sources defines what subset of the
entire source tree is necessary for the
tests to run. Test requires defines
specific packages that need to be
installed to run the tests. Or you can
specify test extras to define an
optional dependency set or test groups
to use a PEP 735 dependency group
definition.
And these options can also be defined
using environment variables with CI
build test command for example or
platform specific environment variables
like CIBW test command macOS. If you set
this up and you run CI build wheel, not
only will you get a flood of wheels for
each version of the wheel that is built,
a clean test environment will be
created, the wheel will be installed and
the test suite will be executed.
Okay, so now we can pull all the pieces
together and actually use CI build wheel
in CI. I'm going to show GitHub actions
here, but the CI build docs have
examples for Azure pipelines, TravisCI,
CircleCI, GitLab CI, and Cirrusci. You
set up a workflow that on GitHub actions
runs on a push or a pull request. You
add a build wheels job that runs a build
matrix that's going to run on Linux,
Windows, and MacOss. What's in that
matrix job? Well, CI build publishes a
GitHub actions workflow that will do all
the heavy lifting of getting CI build
set up to execute. So, you only need
three steps. A checkout, so your code is
available. CI buildwhe and then saving
the build artifacts CI buildwheel
generated. That workflow will run the
project a piprotomal configuration for
all three desktop platforms on all
stable Python versions producing dozens
of wheels depending upon what
configuration is in your piprotoml file.
But the configuration doesn't have to be
entirely in piprotoml. This is why
environment variable configuration is
useful. Pyrotoml configurations will
apply to every user who downloads the
project. Environment variable
configuration lets you specify this
specific release configuration and do so
without the need to add complex
branching logic in your workflow. So you
set the variables that apply to the
platforms that need them and CI build
will do the rest. Okay, but can't you do
all this by yourself? Can't you just set
up a CI matrix over operating systems
and Python versions to spit out all the
wheels that you need by just calling,
you know, Python minus mus
build by yourself? Well, yes, you
probably could. However, that means
you're now responsible for a very
complex build matrix. CI build wheel
abstracts the complexity of everything
past the level of the operating system.
And in the case of Linux, that's a
non-trivial matrix because you've got
multiple many Linux and muscle Linux
targets that you need to make sure
you've got correct. Using CI buildwhe
means you need a top level operating
system matrix, but then CI buildwhe
manages everything else, including
automatic updates to your matrix. When
CI build 3.1 was released, Python 3.14
and 3.14 freethreaded support was
automatically added to every matrix by
default. And the only change you needed
to make as an end user was to update
your pinned CI build wheel version,
which is something that depend would
have done automatically.
But one of the biggest reasons to use CI
build wheel, are you sure you are
building your wheels correctly?
CI build wheel does two very important
things you probably aren't doing as part
of your handcrafted CI build wheel
pipeline. Firstly, tests are run using
the built wheel. If you build a wheel
and then run tests, it is probable you
are not actually using the wheel you
built to run your tests. You're using
your local source checkout and local
build artifacts. And that should be the
same unless it isn't. CI build does this
test isolation for you. So you know the
distribution artifact you're uploading
to Pippi is actually the thing you're
testing. Secondly, every wheel that is
produced is also audited and repaired as
part of the build process. If you build
a wheel on your computer, it is trivial
for you to accidentally link in a binary
that exists on your computer but won't
necessarily exist on the machine where
the wheel is ultimately going to be
used. CI build automatically audits
every wheel to make sure this doesn't
happen. And if it does, where it can, it
repairs and rewrites the library
dependencies so that the wheel is
guaranteed to be self-contained.
Now, both of those things are things you
could set up yourself, but in all
likelihood, you won't set up those
protections until after you've published
a wheel that's been broken. With CI
Build Wheel, you get all those
protections for free. It also means that
the complexity of setting up and
managing cross-platform compilation
environments for iOS and Android is
handled for you. From the perspective of
CI buildwhe, iOS and Android are just
another platform. If you've got CI build
configurations that work for desktop, CI
build will likely just work for iOS and
Android as well. To build for Android,
you need to have the Android SDK
installed with the Android home
environment variable set. Uh this can be
either a full Android Studio install or
just the command line tools. You need to
have a Java SDK installed with Java home
defined or at least Java C on your path.
Uh you need curl installed to manage
some downloads and you need patchel
installed to perform some of the postbu
auditing and fixes.
Any Android specific build details can
then be configured with the tool.ci
buildwhe.android table in your piper
autotoml or with the underscore android
suffix on any cipw environment variable.
There are two architectures that are
supported x8664 and arm 64v8a. uh 32-bit
Android is not supported because it's
not a tier three supported platform in
CPython itself. You can build Android
wheels for both architectures on any
Windows, Linux or Mac OS machine, but
only the local architecture will
actually be tested. Uh so if you're on
an x86 Windows machine, only the x86
wheel will actually be tested through
the simulator.
To build for iOS, you need to have Xcode
installed and you also need to install
the iOS SDK add-on uh inside that. iOS
specific build details are configured
with tool.cu.ios iOS table in your
pipro.toml or with the iOS suffix on any
CIBW environment variable. There are
three architectures you need to worry
about with iOS x8664 iPhone simulator,
ARM 64 iPhone simulator and ARM 64 iOS.
Uh iPhone OS iPhone simulator and iPhone
device binaries are not AI compatible.
They are separate targets producing
separate and incompatible wheels even if
they're running on the same CPU
architecture. You can only build iOS
wheels on MacOss because you need Xcode.
Uh, and only again the simulator will
will be tested. As with Android, only
the architecture that matches your
native architecture will be tested. So
on an x86 simulator, if you're on an
older laptop, an ARM 64 simulator on
newer machines.
The complexity of architectures on
mobile platforms means there's some
there's sometimes a need for
architecture specific configuration. Uh,
CI build can do that uh using the
overrides configuration syntax.
Overrides lets you define a pattern of
build targets that match and then
replace or augment settings when that
pattern matches. So for example, you
could specify as we have here that when
targeting any iOS simulator, the build
should redefine path in the build
environment or when targeting any iOS
ARM 64 platform, append an extra
requirement to the test requirements.
Now although these are IO iOS examples
here, the override syntax actually works
everywhere. So you could use it on
Windows, on Android, wherever you need
it.
Some extra details to keep in mind when
configuring tests for mobile platforms.
First, test command must be an a Python
entry point. Essentially, must start
with Python minus M. Android will also
allow Python minus C as an entry point.
On desktop, test command can be a shell
script or any other command line you
might want to invoke. That won't work on
mobile because you don't have a shell
environment. Android and iOS code is
tested using a test bed app, an iOS or
Android application whose sole purpose
is run the test suite. CPython includes
a test bed project for both iOS and
Android and CI build will manage the
process of cloning that test bed so you
can run your test suite as a specific
artifact. You also must define test
sources even if it's empty. Why is this?
Well, at runtime, the test bed app is
running on the iPhone simulator or on
the Android simulator, and it won't have
access to your local code checkout. So,
you need to specify what is going to be
copied into the test bed to be used at
runtime. Uh, on iOS, you also must
define XU build tools. Again, even if
it's empty, Xbuild tools specifies
native local tools like CMake or Ninja
that you want to be available in the
crossplatform build environment. The
wheels that are generated will specify
their OS version compatibility. Uh if
you want to see it at a different
version, you can export the iOS uh
iPhone OS deployment version or Android
API version uh API level environment
variables.
Once you're configured, call CI build.
Pass in the platform you want to build
for-platform iOS-platform Android. And
CI build will create isolated platform
crossplatform build environments. Invoke
a crossplatform build of your wheel. Set
up a test bed application. Copy your
wheel into the test bed. Copy your test
code and test requirements into that
test bed. Compile the test bed. Start a
simulator. Run the test bed. And rinse
and repeat for every supported Python
version.
CI configurations that support mobile
need to be a little bit more complicated
because you actually need to
differentiate between the build platform
and the target platform. To do that, you
either need to define more entries in
the build matrix and configure that
matrix. So there is one build on macOS
to build for macOS and one build on
macOS to build for iOS and so on or you
can split out individual architectures
as matrix targets. Alternatively, you
can specify a separate workflow for your
mobile, iOS, and Android wheels. That
keeps your desktop matrix relatively
simple, but at the cost of repeating
some of your CI configuration.
What platforms can you use to run tests?
Well, as of right now, you can run the
iOS tests on any uh macOS CI
environment. Unfortunately, any day now,
GitHub is going to start deprecating
macOS 13 runners, which is the last
GitHub configurations that run on x8664.
So, that'll lose your ability to test x8
x8664 wheels. As for Android, the
problem is that CI environments are
essentially containerized environments
and the Android emulator doesn't like
running in containerized environments.
Uh, at present, the GitHub actions or
GitHub actions on Linux is the only CI
platform that we know of that supports
running the Android emulator. And even
there, you have to do some preparation
steps to enable KVM virtualization. That
step shouldn't be necessary once Python
314 uh final is out, but at least for
now, you have to add a couple of extra
configuration lines for Android. uh
details are in the CI build wheel docs.
What do you need to do to pack two to a
package to make it compatible with
Python platforms? Well, the good news is
that most Python code and that includes
any Python 3 none any wheel will just
work. Python is ultimately a write once
run anywhere language and for the most
part that holds. The only place it falls
apart is when you start to do things
that are platform specific in code. The
first thing to look out for is any
references to cy.platform or os.name or
platform.system. If you've got code that
is making decisions based upon platform,
you're going to need to add logic
branches to accommodate iOS and Android
as possible responses from cy.platform
and platform.system.
The other big thing to watch out for is
uses of subprocess multipprocessing in
any other API that creates a subprocess
like OS.spawn and OS.fork. iOS and
Android don't do symmetric
multipprocessing. Essentially, the app
that is in the foreground is the only
app that is running at any given point
in time.
So you can't call subprocess to do
something in the background or offload
work to other CPUs by using
multipprocessing. If your package does
that, you'll either need to disable that
functionality or find a different way to
do that task, such as using threads
instead of using processes. And those
caveats also apply to your test suite.
If you've got tests that create
subprocesses, those tests won't work on
mobile. If you need to, you may need to
add additional platformbased skips to
your test suite to move around tests
that are trying to do subprocessbased
activity and so on. But my experience
has been that the Python code is not the
hard part of porting a package to
mobile. The hard part is the non-Python
code and the builds of that code. The
benefit of Pep 57 is that Python doesn't
have to care how to build your package.
It just calls the PEP 517 interface. The
downside is that package maintainers do
need to care. Whatever tool you're using
to build your package needs to be able
to target iOS and Android. Setup tools
has good support for iOS and Android.
The Python environment that is used to
run builds has been patched so that it
pretends to be the target platform. So,
it's relatively straightforward to add
extra code branches to add support for
mobile platforms. CPython even provides
shims to make it easier to configure iOS
compilers using naming schemes that
match the expectations of a lot of Unix
tooling. But in addition to configuring
the build of your package, you also need
to ensure that any binary dependencies
are available. For example, pillow is a
Python package. It's configured using
setup tools. That build almost worked
out of the box. But pillow also uses lib
png and libtiff and freype which are all
non-Python binary dependencies and they
all need to be compiled for each iOS and
Android architecture. And in addition,
when building for iOS, the dependencies
or the dependency libraries need to be
static. uh this is an unfortunate side
effect of how iOS loads libraries. The
compilation or the the complication is
that these dependencies might use
configure and they might use cmake. They
might use a custom pearl script to
manage their builds. If it's a configure
based project, you can usually pass in--
host and- build to configure for the
platform you want to build for and the
platform that you're building on. Uh
note that you have to use the GNU
preferred AR64 format for describing the
ARM 64 architecture because GNU uh you
can use AR64 Apple iOS for both the
device and the simulator as long as the
rest of your path configurations are
correct. If you do need to differentiate
the simulator, AR64 Apple iOS simulator
is the Apple specified tag. Um but
config.sub doesn't recognize that as a
valid option despite me submitting the
patch multiple times over the last 2
years. So you may need to patch
config.sub.
You may also be able to use d- disable
shared and or- enable static to force a
staticon build. CMake has native iOS
support, but it doesn't do a great job
of differentiating device and simulator
and it seems to expect the use of Xcode
projects a lot of the time. I found the
best approach is to lean on macOS
configurations, but pointing at the iOS
version numbers and system routes. Uh
you can use XC run to get you that
system route. Mison is a cross
compilation uh uh environment based upon
passing a cross file. You pass that in
as an argument and the values that are
provided in the cross file will be used
instead of any automatically detected
platform version values. If your
dependency doesn't use configure cmake
or mison uh good luck, have fun. It very
much depends. The biggest gotcha when
building third party libraries,
compilers are really good at identifying
CPU architectures. They are not good at
identifying platforms. If you are trying
to build an iOS binary wheel and you
have a MacOSS ARM 64 version of a
library on your path and you try to
build the iOS library, it's really easy
to accidentally include the macOS
version of the library in your iOS
wheel. For this reason, CI build
explicitly clears your path during
builds to only include paths that are
known to be appropriate for iOS. So you
may need to configure CI build wheel to
explicitly reallow certain paths if
you've got local binaries for iOS in
some location. All right. So you've
built your wheels, you've published them
to Pippi. How do you use them? Well, pip
allows you to just install iOS and
Android wheels. Pass in d-platform. You
can specify the wheel tag that you want
to satis satisfy with your downloads.
That will put iOS compatible wheels into
whatever virtual environment you're
currently using, which will be a local
MacOSS environment. So you might want to
specify- target to specify an output
directory. So only iOS content is in
that directory. Also at the moment at
least d-platform isn't transitive. If
you install an iOS version of my package
and my package needs to install other
binary requirements, pip will
unfortunately install the macOS version
of those downstream requirements. So you
may need to use- noep to prevent that
dependency training from happening. If
you're a UV user, well, support
literally landed last week for iOS and
Android as platform tags. Consult your
UV documentation for details. or if
you're a briefcase user, uh just add the
package to your apps requires and you're
done. All all that stuff is managed for
you. What's next when it comes to
packaging and mobile packaging in
particular? Well, one big issue we
haven't tackled yet are Python packages
that are built using Rust. Rust is a
compiler tool chain unto itself and it
has support for iOS and Android, but we
need to work out what modifications to
the Rust Pep 517 build backends like
matchin and setup tools rust are
required to support those platforms. Uh
we're also going to be looking at ways
to consolidate and standardize the way
that crossplatform builds actually work.
Uh crossplatform builds uh on Android
and iOS have very similar constraints,
but the CI build will implementations
take very slightly different approaches.
However, they share enough similarities
that there is an opportunity to
consolidate those similarities into a
shared codebase. I've already started
working on this with a project called
Crossbuild. Uh Crossbuild is a work on
progress. I've published cross build in
a sort of pre-alpha state so that I have
something to discuss uh in the coming
week at the CPython core team summit in
the UK. Uh that might lead to a new pep
enhancing pep 517 to add crossplatform
support. Uh but you can already use
cross build to build a virtual
environment that can fake being iOS
Android or in scripton and pip install
just works in that environment
installing platform or target
appropriate uh platforms uh packages and
all their dependencies including all the
uh the transitive issues. So, our call
to action, we need your help. Malcolm
and I are only two people and there is a
limit to how many projects on Pipi where
can we can port if we're the only people
doing it. But the tools now exist. You,
yes, you can support add support for
mobile platforms to the Python packages
that you use. If you visit
beware.org/mobile wheels, you will see
the 360 most popular packages on Pippi
by download categorized as to whether
they are mobile ready. Pick a package
that's in yellow. See if it uses CI
build. if it does add an iOS and Android
configuration as a pull request. Now,
ordinarily this would be where I say if
you're looking for something to sprint
on, this is one way to contribute and
I'll be around to help out. But
unfortunately, I won't be here for the
sprints because I'm on a plane tomorrow
to fly to the UK for the core team
sprint. But it's still a great place to
contribute. See how you go. If you'd
like to know more, the Beware project is
a great place to get started with Python
on mobile platforms. We've also got a
Discord plus discussion forums and
ticket trackers for the projects.
Details on the website beware.org or you
can go to the CI build CI build wheel
documentation directly. It's part of the
resources of the Python packaging
authority or the PIPA. Thank you all
very much.