foreign
experienced soft engineer he's a member
of our local python Meetup in Adelaide
and he's also going to talk about rusts
in his
um Titan conference or a rust for python
developers he's an experienced developer
I've seen some percentages of python in
the middle of Owen in the mid top in
Adelaide and it's really wonderful and
you I'm pretty sure you're going to
enjoy this one thank you Robin
cool thank you very much
all right thank you for attending yeah
my name's Owen and I wanted to talk
about some cool 3D visualization
packages in the python ecosystem
so Jump Right In so both the title and a
lot of the content for this talk
um I was inspired by our pycon us talk
in 2017 by Jake vanderplas he's a really
great developer and kind of developer
Advocate and he's produced a lot of
fantastic materials in the kind of
python data science and data
visualization kind of topics
um and in Jake's talk he put up this map
of all these visualization packages in
the python ecosystem and to very very
quickly recap when he did his talk he
kind of went around in a circle he
talked to matplotlib and all the
satellite packages around that and then
he taught JavaScript and some of the
really popular JavaScript python
integration
packages like your bouquet and plotly
and then he came back and he talked
about d3js which is a really popular
data visualization package in the
JavaScript ecosystem before finally
bringing it around to Old Tower which is
a visualization package that he's worked
on it's based on the grammar of graphics
and it's very cool
but even though Jake put about several
3D visualization packages on his map he
didn't really spend much time speaking
to them which is was fair enough he was
covering so much ground but that sort of
bottom left corner around the opengl as
well as some of the JavaScript libraries
for doing 3D visualizations from python
didn't get covered very much so this is
what I'm kind of calling the neglected
corner of Jake's map and this is kind of
the corner of the ecosystem that I
really want to drill into in this
presentation
I should mention that this is a slightly
polished up version of Jake's original
map that now lives on the pivist site
pivers if you're not familiar with it is
a website that tracks like that kind of
usage and popularity of lots of the
Python visualization packages um I
definitely recommend checking it out if
you haven't seen it before
all right so I'm going to talk text
stacks and packages really soon but
before we do that I want to think about
why we might want to use 3D
visualizations in Python and why we
might not want to use them what are the
potential issues so what are some
reasons for using visualizations well
one that's kind of obvious not
necessarily so great is just because
they look cool and they're engaging and
fun to watch
and I'm not saying that's a bad reason I
love it when people think my
visualizations look cool and want to
share them and engaging is is obviously
nice I want to hold on to people's
attention and for them to be interested
to look at a visualization but I think
we can have issues if that becomes the
primary reason for wanting to use 3D and
it can cause some problems
so the reason for this is when we're
working in data visualization we
normally want to convey out kind of a
clear message we want to give insights
one very kind of tell a story with the
data and hopefully that story isn't
ambiguous and it's very clear
and 3D rendering has certain properties
and if you use 3D rendering naively
sometimes the properties of the 3D
rendering can actually work against the
purposes of telling a clear message
so I want to go over that a little bit
first
so how can having depth and time in the
case of animations make visuals worse
and does a few reasons for this so one
of the first ones is when we're doing a
3D rendering we want to simulate both
light and Shadow or shading because 3D
objects don't look very 3D if they're
flat shaded we want the side facing the
virtual light source to be more lit up
the side facing away from the virtual
light source to be shaded and darker
but think about the process of shading
it that we're actually fading some of
the color and texture on that object to
Black so we're deliberately discarding
some of the color and texture
information on these objects as part of
implementing a lighting model which is a
bit funny to think about that you're
deliberately throwing away some of the
visual kind of definition and Clarity on
your objects but you have to if you want
to make them work 3D
other issues are that perspective can
distort your visualization and make some
details unclear so I guess a classic
example if you've ever seen like a 3D
pie chart and whichever slice of the pie
chart is closer to the virtual camera by
virtual perspective looks bigger than
what it is actually compared to the
other slices on the pie
and and likewise if our 3D scenes get
very large and obviously the things that
go further off into the distance we're
losing both kind of shape and color and
texture Precision on them they're
basically again too small to see clearly
they're getting fewer and fewer pixels
that they can render with as they
disappear into the distance so that's a
potential problem
a third one is the problem of occlusion
and occlusion is just a fancy word for
saying some object in the foreground is
blocking our view of some object in the
background so we can't see that
background object and and this kind of
3D bar plot I guess is a pretty good
example of that where we have tall
foreground bars and they're blocking our
view of the background bars we can't see
what they are
I guess I'll also quickly note that
occlusion isn't always a problem just of
what we classify as 3D visualizations
you even have kind of occlusion and
overdraw problems in 2D visualizations
if you've ever done a scatter plot in
matplotlib and whichever dots especially
if you use different colored dots the
ones that draw later overdraw the
earlier ones and you can sometimes tweak
around with the Z depth of items that
you draw in matplotlib but you can even
get that sort of occlusion and
foreground objects obscuring background
objects happening even there
and I guess lastly it's quite typical to
want to animate when you're doing 3D
visualizations if you have a 3D object
typically you at the very least want to
be able to animate a camera to move
around and see it from multiple sides
that's kind of part of the value of
having 380 but animation brings its own
problems so animation can become
distracting if you think like excessive
animated gifs blink and kind of tags
from 1990s web pages that sort of style
so it can be distracting also probably
worse if you animate too much motion
especially on a big screen that takes up
a big part of people's field of view you
can actually physically make people feel
uncomfortable so yes um an example of
this is that I used to work at a company
that did simulations of these large mine
type vehicles and as part of that
testing or just playing around sometimes
we throw these vehicles around and
that'd be somersaulting through the air
and we're watching this on big screens
that take up most of your field of view
from the fine and seeing like Horizons
and really aggressive motion and
movement in front of you makes a lot of
people very very uncomfortable when the
motion in your kind of visual field is
not consistent with your own sense of
balance and momentum so problems like
simulator sickness or VR sickness are
very real types of motion sickness and
lots of people experience them and that
can be an issue depending on how you do
3D animations
so now I've talked to a lot of reasons
why it's can be problematic I want to
look at some of the reasons where these
features of 3D rendering actually work
for us instead of against us
so the first example the one I'm showing
in that top left is so from a package
called ladybug and ladybug analyzes like
the solar environment around buildings
so in this case this is an example of
both lighting and Shadow where it's not
detracting from the message it like it
is the message that we're trying to
convey and that's kind of an appropriate
use
um also there are times and cases where
having animations and interaction over
time you know it does make sense so
quite often like for simple plots we'll
just put time on the x-axis but I guess
if we have reason to render stuff in 3D
we're already using up all our spatial
axes we can't map time across one of
them so it makes sense and there's some
events like Peak electricity usage or
Peak traffic convention we want to see
the lead up to an event the actual event
and kind of the aftermath or after
effects of that event
it's also talking about perspective that
sometimes perspective and the sense of
scale of objects does actually work for
us so I had one example that's not
specifically related to data
visualization but it is related to 3D
rendering mentioned that at working out
this same company that I mentioned
before that did the mine site vehicle
simulations they used to simulate these
really giant Vehicles so these huge Haul
trucks huge excavators these vehicles
weigh hundreds of tons and they're like
driving a four-story building around
that they're truly massive but when
trainers and when people try to you work
on these simulate these giant simulated
vehicles on the simulator with screens
that are just one to two meters away
from your head you often lost a lot of
that sense of scale because your eyes
like your head motion as you moved
around and you looked at the screens
your view
and your sense of scale was very tied to
the physical screens which were only a
few meters away so even though we're
simulating a giant vehicle it only felt
a few meters wide there might have been
other visual cues in the simulation like
trees down below you and small light
vehicles but you still didn't get that
sense of scale and then one day when
this was early on when the kind of the
Oculus and other VR headsets were
becoming popular we tried testing out on
users and suddenly that sense of scale
instantly came back you know if I'm in a
giant excavator I have to bend back and
look up high to see the bucket when it's
raised up I had to physically lean
forward and look down to see down where
I'm digging and when I moved my head
side to side with the VR headset on my
view at the front of the vehicle didn't
change very much so unlike the physical
screens when my the VR headset actually
helped with that because the size of my
virtual had relative to this virtual
giant machine was very very small and
moving around didn't change my field of
view very much so that sense of scale
um and perspective really worked in that
situation I guess I also mentioned that
sometimes we're visualizing things that
are inherently 3D in themselves so the
example on the bottom left where we're
looking at the three-dimensional
structure of our protein and
three-dimensional structure is pretty
important in this domain because it
affects how proteins interact and bind
with each other so I think this is a
good use case of using 3D rendering and
lastly so close to what I where I work
now I work at a company that does flood
mapping and this is an example where
occlusion can work for us because when
we're looking at flood and water raising
up and submerging Roads and buildings
and other valuable infrastructure that
water is now a
a clues in the stuff behind it you can
no longer see it so the occlusion
actually works for us in that case
because it shows us the impact of a
flood what's damaged what's submerged
what's inundated
anyway so hopefully that gives a bit of
an idea about when we might want to use
3D rendering and when we might not want
to use it so now I'll get on to the tech
stack and packages and so on so I've
done my own kind of mini-map of part of
the 3D visualization ecosystem in Python
down the bottom in rad I've drawn the
actual this is supposed to represent the
actual physical graphics processor unit
Hardware so your Nvidia cards or your
AMD cards or Intel cards and like and
I'm not going to go real that close to
the metal in this presentation
I guess one layer up from that in the
yellow we have a lot of the platform API
so DirectX for Windows metal for apple
and iOS
Vulcan from the Chronos group and some
of the older webgl and opengl one so
these are the ones that some of the apis
that abstract away some of the
differences between the actual graphics
card hardware and it's getting still
lower level than what would actually
touch in Python but we're starting to
get closer
in the blue level I'm showing some game
engines and some higher level Graphics
apis some of these do have python
bindings which I'll speak to in a bit
and at the very top level in the green I
have the
selection of python 3D visualization
packages and I which I will speak to
later on and I should mention different
especially at that blue level in the
green level I'm only showing a very
small subset of what's actually out
there just a few kind of Representative
examples in reality both that blue and
green layer are much much wider than
what I could ever fit on one slide
so yeah I'm going to start walking up
this Tech stack and speaking to it a bit
I guess I'll start with opengl
um this is a little bit of a lie because
there was stuff before there was 3D
rendering before opengl but opengl is
kind of it's quite old and it's still
around still popular still used so I'll
I'll talk to that for a little bit so
opengl was started by a company called
um called silicon Graphics back in the
1990s it was something I looked at doing
my undergrad at University so that was
in you know at Sea and Northern GL and
doing really basic 3D rendering and
opengl has this concept as do many other
Graphics apis of a pipeline so you come
up with your three-dimensional Vector
geometries you can add light sources and
light material properties to these
objects as well as a viewing perspective
or a virtual camera and you put them in
the front of the pipeline and there's a
sequence of steps that it goes to
manipulating these geometries before
ultimately rasterizing them and turning
them into the rendered image from the
Viewpoint of the camera
and those steps you you can look them up
if you're interested it's changed over
time the earlier versions of opengl had
very rigid steps that you could tweak
the parameters on a bit you might be
able to choose the resolution of the
ultimate raster image you might be able
to tweak the field of view angles that
this virtual camera had but you couldn't
change much in the pipeline and over
time these pipelines became more
customizable and configurable you know
the Advent of shaders which are C or
assembly like languages that would let
you customize different stages of these
pipelines so you would have like vertex
and geometry shaders for tweaking how
these objects are transformed and warped
before relative to the Viewpoint of the
camera and you'd have things like our
fragment shaders which would let you
customize the way these things got tax
rasterized in the end and the ultimate
pixel colors that you would see
opengl is now managed by a group called
The Chronos organization
it's been featured Frozen for at least
five or six years I guess the main
reason is that it hasn't really kept up
with modern GPU hardware and
architecture so this Chronos
organization
um
would prefer you to go into a new lower
level
Graphics API called Vulcan which I'll
speak to in a little bit
um all that said though opengl is still
very heavily used and it underlines a
lot of the 3D visualization packages
that we have access to in Python so so
we are in this bit of this transitionary
period where we're still using opengl a
lot but all the new work and all the new
advancements are happening in different
platform apis
so still kind of riffing on the opengl
theme a little bit later on in the kind
of life cycle of opengl there was opengl
embedded systems or opengl ES this was a
subset of opengl to add support for like
mobile devices so tablets and phones to
my understanding this is still used on
your Android devices today I believe iOS
devices used to use this before they
more recently switched over to their
metal API
and the version of opengl embedded
systems or of NGL es2 later became kind
of adapted and rebranded as webgl so now
it was going to give you Hardware
accelerated 3D rendering support for
browsers and webgl is what underlies a
lot of the browser-based data
visualization packages that we use in
Python today
and lastly just kind of wrapping up some
of the Native platform specific apis I
spoke to them briefly so we metal is the
new API for newer Apple devices and iOS
devices Vulcan is supposed to be the
like successor to opengl but it's a much
more lower level Graphics API the more
opengl is and DirectX which has been
around on windows for a very long time
and as I'm after DirectX 12 at present
I'll say that a lot of game engines do
tend to stay up to date with these
platform apis they care about squeezing
as much performance as they can out of
the gpus but not so many visualization
packages have kept up to date with these
platform apis
there's um
one library that I I know of
specifically
um it's actually a C plus plus Library
dear IM GUI and there are python
bindings to this Library so as the name
implies it's it's for creating guise or
Windows systems buttons and icons and
elements and widgets but it also has a
subset of its API for doing kind of
conventional plotting like your Scatter
Plots line plots and that and it has
Bindings that hook into all these
platform specific apis so it's very very
performant so if you ever say need to do
a scatter plot with a million data
points and animate and update that
really fast many many times a second
that's potentially an interesting one to
look at so the the name of one of the
most popular python bindings for that is
called deer Pi GUI
but I didn't include that on the map
because as far as I know it doesn't have
any support for 3D rendering
I guess one of the most high level
libraries that I'll talk about related
to 3D rendering is this visualization
toolkit it was a c plus Library it was a
higher level abstraction over opengl it
has a number of python binding packages
that let you hook in and use it from
python so there's Pi Vista mayavi veto
and more
um and then we'll just quickly mention
that Pi Vista is a personal favorite of
mine I'll speak to her a little bit more
later on but if you're ever interested
in getting into 3D rendering
um using python that would be one of my
recommendations to try first just
because it has an excellent set of
documentation and examples as well as it
supports many many use cases from doing
real-time rendering to doing offline
rendering to images and videos as well
as it you know works as a standalone
script or inside a jupyter notebook so
it's really versatile and the
documentation is great so it's a
personal favorite of mine
a couple more visualization packages
that are quite closely tied and coupled
to open gel are this pie and Glam pie
yeah I mentioned the quiet low level
these two packages I believe they're
quite interrelated they've shared some
of the same developers they have some in
the same examples and that Gunplay
itself seems to have gone a bit stale I
haven't seen much activity in its repo
for the last couple of years this play
on the other hand is very actively
maintained and developed and I know
visplay is used by certain people in the
earth observation community so it's it's
the one that I've used a little bit
before as well
um and I know with this part two they're
talking about adding Vulcan support and
maybe some of those other platforms
specific apis so that one looks quite
interesting to me
in terms of um for quite a long time
there have been game engines out for
decades and this game engines quite
often have support like python Bindings
that let you hook into them and use them
for rendering so obviously this is not
your traditional data visualization but
if you do need to render
three-dimensional scenes in a world that
people can interact with preferably real
time that these offer potentially
interesting opportunities so one by
relatively famous open source game
Engineers that ogre game engine I'm not
sure if I'll get that acronym quite
right but something like object oriented
Graphics rendering engine
um it had python bindings as a plug-in
and I believe it's engine natively
supports python now so you can check out
packages for that if you're interested
and another one that I think has huge
potential and is really interesting is
that Unreal Engine and I don't know if
you've ever checked out like images or
videos from Unreal Engine 5 now I think
they're up to version 5.2 but it's an
amazing games engine that can do
photorealistic graphics and has all
sorts of you know really cool tooling
and lighting effects and support you
know it's so powerful people can even
use it for you know rendering like movie
quality animations and they do
um you cannot use python in real time
when the engine itself is running and
people are interacting with it but it's
scene editor that lets you compose the
three-dimensional scenes and the camera
and the elements within those scenes has
had a python API for a long time I have
seen some kind of funny old demos of
people doing like rendering matplot lib
plots and having them ultimate
ultimately added as textures onto 3D
models in unreal but I think there's it
has so much more potential than that and
I'd be really interested to see more
happen in that space
um another
potential way of doing 3D rendering in
Python is looking at blender so blender
is uh quite probably the best known open
source 3D Ray Tracer Ray tracers if
you're not familiar with them this is
for offline rendering they typically
have much more sophisticated lighting
models than what your real-time
renderers do and as the the name kind of
implies Ray tracers model light rays
that can both reflect off and refract
through multiple 3D geometries in your
scene so you can get these really
realistic looking lighting effects and
lens effects and so on and water
properties and blender has had a python
API for a very long time that lets you
automate its rendering as well as build
plugins and extensions with it so that's
another one that's potentially
interesting if you wanted to just
offline rendering to still images or to
videos
via python
and I guess there's a whole ecosystem of
3D rendering options if you're working
in Jupiter as well as this also includes
some web front-end stuff that has
Jupiter Integrations and a lot of these
packages that I'll mention here are
built on webgl
um so I've already spoken to Pi Vista
that's the image shown in the top left
um it's had a Jupiter extension for a
while
um there's IPI volume which lets you do
a bunch of interesting 3D rendering
um in Jupiter one thing that's
particularly cool about this ipy volume
packages it's it's one of the few
packages that lets you programmatically
from the python side save the um offline
videos or screenshots of your renders so
most of these browser-based data
visualization packages they may have
buttons on the JavaScript side to let
you save a screenshot of what you're
currently seeing but traditionally
that's been quite complicated to do from
the python side because it's hard for
your python script to know what state
the browser is in whether it's finished
rendering one of the frames of your
animation or not but I probably iPad
volume does a bunch of clever tricks to
make that work so I have used it before
and you can use it to you know save
still images to save animated gifs to
save MP4 video files so it's very cool
for that I think and it's a fairly
straightforward API
um there's also k3d Jupiter which is
built on webgl I think it might have
bindings to that vtk as well I'm not so
familiar with that but they do have a
bunch of very cool demos
there's Pi 3js so for those who are not
familiar 3js is uh is a higher level
Graphics API in JavaScript built over
webgl kind of more intended for games
development and real-time simulations
and that and it has abstractions for
dealing with camera motions and and
animations and reading characters and
that so not traditional data
visualization but if you want to do
something that's more of an interactive
3D World in a browser then that might be
a potential interest to you and yeah Pi
3js is just a very level kind of low
level
python it Maps very directly to the 3Gs
functions anyway
and just a couple more quickly we have
Pi DAC which is the python bindings for
the deck GL JavaScript library if you
haven't seen this before so this does
some very cool geospatial animations and
effects it works with Factor tiles but
allows you to render 3D meshes and
structures over them this is just one
demo that's the UK plot with the
hexagons over the UK
and lastly I'll mention yeah plotly
which is one of the biggest and best
known
sort of javascript-based visualization
packages also has a subset of its API
that does 3D plots
probably more on the traditional side
but it lets you gives you controls to
Pivot around them and zoom them just
like you can most of the 2D plotly plots
cool so I'm almost at the end so for any
further reading watching listening that
you might be interested in I mentioned
that I'm a huge fan of Jake van
implass's materials and talks so that
was the talk that he gave six years ago
now at pycon us the python visualization
landscape which was the inspiration for
my own talk despite it being six years
old it still holds up really really well
pretty much all the visualization
packages he speaks to back then are
still alive and well and heavily used
these days so that's really great and
another one of his talks that I
particularly liked was how to think
about data visualization and this is
less Tech focused but it's more about
how do you compose good data
visualizations and and how do you use
elements of visualizations the in the
most effective way to convey your
message and to present the data as
neatly as possible so I think they're
great for a short kind of online ebook
or reading there's the python in openg
jail for scientific visualization
believe this is by the same author of
that glum player package once again it's
a few years old but it still covers a
lot of this 3D Concepts that I've spoken
about and Dives a bit deeper into them
if you're interested in getting into the
weeds a little bit more with them
um yeah I've mentioned heaps of pretty
cool looking 3D visualization packages
but as I said as a starting point I
would probably recommend checking out
that pie Vista first if you're new to
this domain as I think that has one of
the best kind of onboarding experiences
for new developers and because it just
covers such a breath for possible use
cases for you and it's the last thing I
mentioned that's not specific not to do
anything particularly with data
visualization but if you're interested
in looking at shaders and what's
possible with them check out the Shader
toy website so this site is a huge
collection of these different Shader
functions and they can do these amazing
3D renders that update dynamically in
your browser and um it's pretty cool
ecosystem
so yeah I'm pretty much at the end so
just to summarize if you are thinking
about doing 3D rendering and
visualization make sure the properties
of 3D rendering suit the sort of message
and data that you want to convey so that
the features of it are working for you
not against you and out of all that big
selection of packages I've shown you
some things to think about when you go
to choose one or other of them is think
about whether they support doing
pre-rendered visuals if that's your aim
to like include them in a presentation
or a talk or whether you want to have
interactive 3D visualizations because
that's a different set of libraries also
think about your deployment story
whether you're deploying to a desktop
application or whether it has to be
browser-based and lastly think about the
kind of scale and performance
requirements if you need to render a
scene that's particularly realistic or
just has very complicated 3D geometry
you might be inclined to look more at
those packages that type closer to the
the newer 3D platform apis
and that's it thank you very much for
listening if there's time I'm happy to
take questions
thank you Warren very good we don't have
time for questions but we do have the
Discord Channel you can pop some
questions there and then if you can log
in and answer the questions and
obviously you can hang around in in the
aisles and ask questions so and thank
you and here's a Morgan uh thank you
note for the conference all right so
we'll be back in 10 minutes
[Applause]