[Music]
Hello everyone. We're going to get
started with our next session. Um so
today we've got Tennessee Leonberg and
they'll be presenting on PyF tools
machine learning for F system sciences.
Thanks
Joe.
And uh I I know it's already been done
earlier this morning, but just a quick
acknowledgement of of country as well,
just to add my support to those
statements. So, a little bit about me.
My name is Tennessee Luenberg. I've
worked at the Australian Bureau of
Meteorology for 22 years, mostly in the
research program as a data scientist and
software engineer. I'm the maintainer of
two open source packages, PI Earth
Tools, the subject of today's
conversation, and also scores. And my
main focus is the development of machine
learning models for now casting, which
is the time frame from right now out to
about 3 or 4 hours.
So I think for this audience, we'll add
a bit of a description of like, well,
what actually is Earth system science
and then we'll see some examples and
then kind of ease into what is the
application of machine learning to that
context. So earth system science models
essentially refer to modeling sort of
the processes and state of our kind of
natural environment, our planet, our
atmosphere. And traditionally they've
been process-based or rule-based systems
developed by
academics and researchers on the basis
of physical rules. So think force equals
mass times acceleration or the effect of
gravity on the atmosphere, things like
that.
uh they're amazing but they're also very
computationally expensive which is why
we have supercomputers uh which we don't
all have in our back pockets. So prior
to 2022 the general understanding is
that the complexity of that task would
have greatly exceeded any feasible
machine learning approach for modeling
the entire system anyway. However, since
then machine learning models have come
out and proved themselves very competent
at at the entire task. There's still a
scope to what they can do, but there's
really a change in the state of the
field. Some of these machine learning
models are entirely machine learning
based and there's no attempt to
explicitly uh measure simulate the
physical and chemical processes that are
occurring and it's the neural net doing
what the neural net does. uh whereas
others are hybrid and there will be like
a neural net aspect and also a
rules-based aspect and they're not all
neural networks but a lot of the big
ones that are very interesting are
neural networks. So let's sort of work
it through with it with this example. So
here's the example of a weather
forecast. I work at the weather bureau.
I kind of can't help making all of my
examples weather. Everyone understands
the weather, but the system can do a lot
more than the weather, right? And
there's a lot of pieces of information
in here. So where do these pieces of
information come from? So there's a
certain amount of human value add and
quality assurance to the way the weather
bureau makes them. But behind that are
these simulation models that drive the
prediction of these variables. So this
is an animation of a weather variable
evolving around the time in a
rules-based physics-based global model.
So that's an earth system model. Uh
that's earth system science and what
we're talking about is a neural network
that can do that.
So here is the code for a model that I
trained that does that to a greater or
lesser degree. Um and it produced that.
So basic proof by example that's the
kind of thing we're talking about and
that's what PI Earth tools is for. It is
for the construction of neural network
and other machine learning based models
which are capable of performing useful
earth system science simulations
to the level and standard uh again scope
and caveats apply. Um but this isn't the
sort of the deep dive on the science of
them roughly to the standard or indeed
in cases better than the standard of the
physical models.
So back in 2023, it was possible to
actually go and find all of the machine
learning models. So a colleague and I
went through this process uh of doing so
within a particular scope. So there's a
lot of caveats around what is and isn't
a weather and climate model and so
forth. But we we we found the scope and
we you can see in the timeline when it
is that machine learning really starts
sort of taking off in the field. And
since 2023,
it's just there's just been an absolute
explosion in the number of research, you
know, the amount of research findings in
this particular field and it would
probably no longer be really reasonable
to try to produce these diagrams now.
But a few things from this still very
much apply. So firstly uh fully
connected neural networks as opposed to
more advanced architectures are still
very much relevant in the field um
particularly for very specific process
problems and also we are talking about
things that don't just include neural
networks. But on the other hand, since
then, a great many of the most
significant results have come from the
application of transformers and graph
neural networks, especially to
predictive uh models and also diffusion
models which might be uh perhaps a
little more focused on the problem of
downscaling or super resolution as it
might be called from a more computer
vision perspective.
So this slide uh wasn't produced by me.
This was produced by Andy Brown, the
director of research at the European
Center for Medium-Range Weather
Forecasting. And this is where we have
specific examples of these models
performing to a greater st a higher
standard of accuracy than the physical
models. Uh again for specific metrics,
specific variables, various scopes and
caveats apply, but they very much held
their own in terms of the headline
statistics uh in some of these
variables. So it's very much not hype
and it's very much transforming the
field.
So let's talk about now that we've got
that sort of scene setting out of the
way Pythols itself and it you know how
it came to be and where it sits you know
in the field and what you might like to
know about it. So back in around 2022 um
you know we could see that change was
change was on the way and internally
started work on this tool for being able
to sort of deploy and run various
state-of-the-art published architectures
run them against our data sets etc. Um,
in 2020, March 2023, a framework called
Climate Learn was released. I believe
that's no longer being maintained, but
that's when that was released. Around
the middle of 2024, uh, a number of the
bureau's research partners, so the Med
Office in the UK, uh, NEWA, which is now
Earth Sciences New Zealand, and Access
NRI, which is a research group here in
Australia, joined as PY Earth
collaborators, and we sent the set the
intention of creating an open-source
framework so that we could share our
work and collaborate more effectively
across that partnership and with anyone
else who might like to join us. In
August 2024, a framework called Anamoy
was released by the ECMWF. This is a
very serious framework. It's it's ve got
very very compelling results coming out
of the models that come from that
framework. Uh in December of that year,
hot on the heels of one another. Um PI
Earth Tools was released as open source
shortly after the Miles Credit framework
was released. Miles Credit being by
Encar. So that's how we got where we got
and I'd just like to pause and
acknowledge uh the Pythols contributors
that have gotten us to this point. Um in
particular uh Harrison Cook who really
led who very much led the uh technical
development of the framework for the the
first couple of years and has now gone
to work at the European Center on the
Anamoy framework and is now helping
helping them in their journey but also
fully recognizing and valuing the
contribution of everyone else as well.
And of course it's open source so you
know your name could be here next year
if you like.
So what is PI Earth tools? So it has
source code and documentation. It is a
modular open-source Python framework
with modules for data. No one one line
no small task uh pipelines. So in this
case think less orchestration pipelines
and scheduling and think more
declarative pipelines of the
transformations that need to occur to
the data and it takes the philosophy
like a lot of models in a singular code
base may like closely couple the
pipeline and the model itself. uh PY
tools takes the perspective that these
things are should be better modularized
to allow more fine-rained research to
occur and more reuse of standardized
scientifically validated pipelines that
then data scientists can more easily
just sort of pick up and work with and
each each person sort of focus on their
core skills they're bringing to the
equation. So examples of what's in the
pipeline, reproing, resampling, reg
gridding, reagregating, normalizing,
denormalizing,
standardizing, adding, converting units
and and a great many things. Um, and all
of that ultimately turning itself into a
fairly basic data structure being a
Python iterable. And then that then
proceeds to the models. And then pio
tools also has a good number of uh
packages and uh utility in it for
working both with predefined models. So
if there's a published architecture that
we've been have a supported
implementation for, you can pipe new or
different data into a predefined model
architecture for people who want to
focus more on the pipelines or for
people who are more focused on the
modeling side of the equation. They can
define their own models. uh if
something's you know successful or they
want to publish it then they could you
know put it back into the into the main
package for wider reuse and a certain
amount of code uh on our road map coming
soon for evaluation and standard scorec
cards for model into comparison to
support the research journey.
So to recap PI Earth tools is for doing
this kind of thing.
So let's sort of pause and talk a little
bit about the data. So one of the things
that maybe differentiates the earth
system science world a little is that we
don't need to deal with as much say
conflicting user generated data as if
you were training an LLM but we do have
to deal with like a similarly large
volume of data. So you know we're
talking in the hundreds of terabytes to
pabytes range for the largest problems.
uh and even for the smaller problems,
it's still a decent decent whack of data
and it has to be brought together in a
great deal of different differentiated
formats um which is often specific to
Earth system science and I've always
really wanted to kind of hide all of
that behind an API. That's what you sort
of have to do to get data into a machine
learning model anyway. So this is a
mechanism to standardize in in a coded
way how to do that in a way that's a bit
more transferable. And these data
sources are not always sort of fully
documented for an ordinary user as well.
Often they you know require a certain
depth of science understand even
actually properly unpack and process the
the data itself.
So here are some examples of how PI
Earth tools might connect data through
to uh through to a machine learning
model. some of the the processing steps
that it goes through and we'll talk
about the processing in a little more
detail in a moment. But here we've got
uh three separate uh pipeline graphs.
The first one um essentially just being
a stick uh which connects the
weatherbench 2 based era 5 data set
through the various transformations um
that are required. So the steps here
being you need to do the dimension order
the variable sorting. So they'll often
be like latitude, longitude, height,
time. The various data s sets you want
to work with often have those in a
different order. So there's a certain
amount of just reshuffleling of data
that needs to occur. Certain amount of
data is 0 to 360. Some is minus 180 to
180. It's going to verifi vary by not
just the source, but like how the person
who archived your copy of that source
decided to do it at the point in time
they do it, which has persisted until
now. So there's a lot of
uh pre-thinking in the PI Earth tools
framework for the major sources of data
and sites that we optimize for. So that
that sort of takes that burden off the
user to some degree and they can use a
pre-named pipeline to sort of you know
get a bit of a head start on working
with the data. um temporal retrieval to
turn it into the appropriate sort of
sequence tosequence pattern that we're
talking about because there's often
there's there's a history to the
training the prior examples but then
there's the number of sequences that you
want to generate in your forecast period
of lead time as well. So that sort of
reshapes the data for presentation to
the network. Convert it to numpy, play
with the the dimensionality a bit more,
work out the zed scaling, standard
deviation, c it cuz that was a lot of
work. And then be ready to start, you
know, running epochs over the data. And
on the top right we have a similar
example but in this case we're trying to
this was is utilizing to produce uh the
uh El Nino index out of you know so the
right hand side goes down to a single
value which is that index value but on
the left we're preserving latitude and
longitude dimensionality in the input
data. So a lot of this a lot of this
sort of pipeline stuff is around how to
handle appropriate data transformations
and it's the source of a lot of
mistakes. So like a a recent example is
someone was working with hail data uh
sorry lightning data. Lightning strikes
are relatively sparse. They were
preserving latitude and longitude when
calculating their normalization factors.
There were a lot of missing data and nan
values. And so you know they hadn't
plotted they hadn't plotted it. They
weren't aware of it. These sorts of
things allow the scientists involved to
get in there and do things like inspect
normalization weights and understand
data data representations.
Quick plot, quickly identify the issue.
Let's just collapse it to a single
standard deviation rather than something
that's spatially varying. Happy days.
So what are the strengths of Pearth
tools? So we do take into account the
hardware consideration. So running one
of the global scale physical models is
not something you can do on an ordinary
laptop or computer. Running a global
machine learning neural networkbased
weather model is something you can do on
your own laptop or computer for many
laptops and computers. Training them is
something you might need a very high
power GPU for. More it's the data the
data volumes are very prohibitive but
training a low resolution one is not
prohibitive and can be done and we've
we've got that in our tutorials things
that you can do for yourself. So one of
the advantages I think is this can put
from a learning perspective the ability
to work with real world science into a
more consumer class of hardware or more
common commonly available source of
hardware and category that people are
going to be able to do more
experimentation with. So it's been run
on laptops, workstations and
supercomputers. It supports
standardization and repeatability.
It has it has lots of documentation and
we are it's a forever job to make it as
good as it can be but we're very much on
that journey and it's independent with
respect to the machine learning
architectures. So it can work with any
machine learning architecture or
framework. So the the three that we've
integrated it with thus far in the
tutorial examples uh would be XG Boost,
PyTorch and Apple MLX framework. But
there's no it will function you know
very happily together with with Jax or
TensorFlow or any other one that you
might might care to add. So that's
that's also a feature and that's why to
separate the pipelines from the models
because the the feed of the pipeline is
something that's typically going to be
numpy based or basic or a simple array
based very numerical. Having that as
something that can be done in a
standardized sort of xarray open-source
stack separate from the model itself
allows you to be more modular and
objective with regards to your choice of
of actual networking st neural network
stack. So when I say network here I mean
neural network not network cable
network.
So what are the motivations for
continuing to do pie earth tools and why
why is it of interest uh for others? So
much of machine learning research in
earth system science uses a a custom
stack. So there there's sort of a few
categories like there's there's there's
anoy which has a particular family of
models in it and it's you know it's
probably fair to say it's best of breed
for that particular family and class of
problems but there are equally as many
amazing models that are built in pure
pietorch or pure tensorflow or pure
jacks that don't connect to particularly
to one another. So, Pyear Tools, we aim
to be able to bundle more model
architectures, give people a leg up,
give people predefined architectures,
give them a starting point,
unbburden some of the interface into the
supercomputing facility and the
complexity of the data sources and move
a bit closer towards modular
reproducible science. So, that's sort of
the purpose. It's accessible uh by
being, you know, it's on Pippi, it's on
GitHub, you can just get it. There's no
overheads or barriers. and it will work
on your computer.
Um, flipping the equation, it allows you
to document and share your your workflow
back with the community effectively. So,
if people, you know, choose to adopt the
framework, they're going to be able to
collaborate with their partners on model
versions and model upgrades and
differentiation and pipeline versions
more effectively than if they were
gathered around specific models. So
that's the basic principle and it gives
people just a a way to give back as well
which is also a motivation for many. So
let's look at some things that you can
do for yourself. So we have four
specific tutorials which are focused on
requiring no more than a 4 GB GPU and an
amount of data that you can reasonably
just download over the network in real
time. So the the data volumes here top
out at kind of 10 gigabytes. Maybe you
don't even need more than three. and
many you know many workstations and
laptops would be able to do not just the
inference but the training. So that the
emphasis here is very much on putting
the the ability to train back into the
hands of as many people as possible with
as few barriers to entry as poss as
possible pardon me. Uh and we have a lot
a long and interesting tutorial gallery
full of lots of worked examples.
It's very easy to get involved. Um so
community engagement and community
contributions are warmly welcomed. Um
I'll be here for the sprints. So please
do consider coming along to the PyCon AU
sprints and there'll be some other
developers online who can help orient
you and get you started as well. run
through the tutorials and there's a few
other ideas and if you just want to keep
in touch with PI Earth tools and see
what's going on. Uh the access nRI
machine learning uh for climate and
weather working group uh is the place
where most of the conversation about the
ongoing development of PI Earth tools is
occurring at the moment. In terms of our
road map, probably the next major
objective is adding standardized
scorecards so that we can do far more
straightforward model into comparison
between the different architectures. Um,
and so that again students or people
starting new projects or or early career
people have ready access to see how what
they've trained stacks up against, you
know, baseline stats from other models.
um add an further range of benchmark
model implementations for more classes
of problems so that the framework is
open and of interest to more more
different researchers from different
backgrounds on different problems.
Bundling more kind of basic model
archetypes. So uh to me the difference
between an archetype and an architecture
an architecture is like this particular
network with this particular depth and
this particular size and this many
layers. So a lot of the choices are made
whereas an archetype is like a graph
neural network a CNN etc. So that we've
got more worked examples for some of the
the intervening steps between starting
you know a blank sheet of paper and a
functioning model and as always a bit of
refactoring and simplification.
So I'm not quite sure where that leaves
us for time but that has run me out of
slides. Um can I get a quick time check?
10 minutes. Well I think that's the
that's the goal isn't it?
Great. So, we can stop there for
questions. And if people don't have any
questions, I can fire up tutorials if I
can get the network working.
[Applause]
[Music]
Thank you, Tennessee. That was a really
awesome talk. It sounds like a really
neat tool. Um, so do we have any
questions?
Hello. Uh first off, thank you. Great
talk. Um so I'm a platform uh
engineering person. So I'm wondering if
there's any way you could uh say
something about how this is being made
available and how that's hosted for um
analysts and researchers to consume
within the bureau.
Sure. So the I mean in any particular
you you'll have to find someone to to
install it in any particular
organizational context. The main use of
this by the bureau and the community is
in research facilities that are for that
particular purpose because of most of
the focus being on training and
reproducibility at this stage. There is
I think very wisely a lot of scrutiny
being put on these before they would go
behind any kind of production service
and frankly we're not there yet. So it's
more so the the emphasis is on the
optimization at NCI. There is a
pre-built what's called a modules
environment which is that that would be
uh of use to that's fairly generally
open. I don't know exactly how open but
if you're a researcher in the field it's
I I don't see why you wouldn't be able
to access it there. But it's also pretty
straightforward to pip install. Like
there's a lot of packages, but other
than there being a lot of packages, like
I I run it on my machine, most of the
developers have work on run on both
their facility and an individual
instance on a laptop as well. It it
doesn't require much. So So it's it's
Python processes kind of beginning to
end. A bit of Dask, a bit of PyTorch.
You don't need like an an MLflow server
or this server or a database server or
anything like that. It can kind of go
where you want to take it.
Hey, um just interested in a sort of a
deep dive into the actual models the and
the machine learning side of that. Right
at the start you sort of talked about um
the co you showed a very dense slide of
code that was machine learning um
version of a particular model and I'm
wondering what are the data scientists
and the data modelers learning from the
machine learning that is kind of
producing a synthesis of those ideas.
Yeah that's that's a great question. I
don't know whether this dense slide oh
it stopped sharing the dense slide of
full of code is going to help an
enormous amount what people are learning
is that the machine learning models they
had a lot of let me reframe it in terms
of the questions they're asking okay so
one of the questions they were asking
are do they fail different or do they
fail the same uh do we have things like
model collapse occurring are they
effective outside of the training
envelope how do they handle rare and
extreme events How do they work in terms
of the probabilities of those events
occurring? And those are questions which
are kind of natural to ask from a
scientific perspective. And so a lot of
what they're doing is like you might
look at a uh I'm going to get the the
words on the tip of my tongue, but
essentially the spatial sensitivity map
of the model to the to the data. So
they're looking to find physical regions
that they can understand as a scientist
which are influencing and and resulting
in those model predictions. So and
whether the probability curve. So most
physical modeling now is what they call
ensemble modeling. So people there's an
increasing emphasis not so much on going
there will be exactly 20° of temperature
and more the odds of exceedence of a
variety of thresholds is such and such
and such. And they do that at the moment
through a a kind of initialization
process with carefully managed noise of
an of a fleet of ensemble members. and
they're reconsidering
how to do that in the light of things
like a diffusion process which is more
like a Gaussian noise model and less
like a physical noise model. And so
there's a lot of interplay between like
as as the physicists with those deep
understand and modelers with that sort
of deep physical understanding are
framing their understanding of the
machine learning models to try to
produce those similar outcomes. Um at
some level what they're finding is that
the machine learning models have quite
similar strengths and weaknesses to the
physical models. Uh they're trained
against something called reanalysis. A
reanalysis is a mixture of observations
but away from the observations. It's
model data. So it's a modeled estimate
in the sparse you know to to fill in the
sparse data. So to some degree the
models are trained on emulating a
physical model. There's a really
interesting line of research on direct
from observations training. Uh, which I
think is is really that a lot of the
focus of the cutting edge is really
moving into direct from observations.
That definitely gets me out of bed in
the morning. That's that's that's super
exciting. So, I'm not entirely sure if I
I just tried to give you a flavor. Yeah,
good play.
Thank you, Jennese. Um, great
presentation. just in the background of
this package, all the weather variables
are built into that. So, we don't need
to know where they come from. And and my
question goes to can we use things like
your that airport you showed that three
to four say wind. We could do wind
forecast. Can we use that then with
other products to say do fire modeling
in the short term? We've got a bush
fire. We then model what that wind's
happening with some topographic effects.
Can we predict that fire? Is that the
intent for a package like this?
Yeah, that's certainly the intent. So
let let's talk about this. Can we do it
like
not in production today right now in
easy grasp in principle using machine
learning with a datadriven approach in
research? Yes, definitely. So the the
intent there is to provide and the focus
again is still on training. So in a
production setting, you've got a whole
IT architecture to consider. In a
research setting, it's like can we make
the models to do that job effectively?
And I'm very interested in things like
whether we can take like a good
foundation weather model and then add a
field from another discipline or area
such as fire spread or some other pre,
you know, well well stored data set and
then learn those relationships without
having to do the full retrain.
up until now, as you can see, since
like, it's only just a small amount of
time that those models have gone from
kind of close par to the atmospheric
model to genuinely competitive
atmospheric modeling. I think that's
sort of we're on the cusp of all of
those sorts of changes and the the Yeah,
if you're interested, tap me and I'll
I'll send you a bunch of links. Yeah.
Okay, really quick question. This this
might be tangential to pi tools, but um
have machine learning models been used
to uh characterize uh the um the
statistical distribution and spread of
um of whether ensembles u models uh in
terms of in terms of trying to optimize
the number of ensemble members and
things like that.
Not in terms of optimizing physical
ensemble members. In a lot of ways, m
the machine learning has happened so
quickly, the physical models models have
only barely had time to react to this
change in the field. So they'll try to
work out how many machine learning
ensemble members to run. Um but that
they're quite cheap to run. So you can
more or less run just as many as you
want. So there's not the same kind of
constraint and people haven't really
sought to back connect to the physical
modeling world. There there are lots of
reasons to maybe do that like guard
rails, explainability, fallbacks, etc.
But um yeah, interesting times.
Oh, can everyone thank Tennessee again?
And we've got a mug for you.
Thank you very much.