foreign
welcome back everyone to All Things data
uh we've got in this uh block we have
two talks before we jump into
introducing the next talk just a
reminder that we have a Discord Channel
there's some really good conversations
about the talks and and links to open
source uh packages and things that
people talking about so Jump On In say
hello we've got people on the online uh
tuning in remotely as well joining in
there so uh to kick us off uh we've got
Alex Ware and I'm just going to read a
little intro here uh Alex is a software
engineer working for geoscape Australia
and he's based in Canberra Australia she
previously spent several years working
as a data engineer in the Australian
Public Service she's a co-organizer of
the Canberra python user group and a
co-organizer of the upcoming Django
girls Canberra Workshop she's passionate
about Big Data clean code and supporting
those with a marginalized experience of
gender in Tech so uh with that um please
big round of applause for uh
uh Alex Ware who's going to be talking
about an introduction to Pi Spark
oops
hello hopefully this microphone is
working okay uh so today I'm going to be
giving an introduction to Pi spark in my
talk I'm going to be trying to answer
some questions such as what actually is
pi spark can it really solve all of my
data problems and possibly most
important are you sure I can't just use
pandas instead
uh we sort of covered the who I am a
little bit so software engineer at
geoscope Australia working predominantly
with geospatial data uh the very quick
call out for my employers we our claim
to fame is Gina from the national
address file but we also do a whole
bunch of other geospatial products I
know there's been a bit of discussion
about geospatial stuff today so if
you're interested in things like data
around property cadasta buildings roads
solar trees maybe check us out uh
previously in the public service won't
talk about that too much other than
that's where I first started using pi
spark and the workshop has actually
happened now it happened a week ago and
it was a lot of fun and thank you to
Django girls for supporting us in
running that
okay the really important disclaimer is
that after that slide is I don't talk
for any current or former employees I am
here entirely sort of presenting my own
opinions and very importantly I don't
have any like major or minor or any
links to Apache I do not speak for them
in any way I am just a hobbyist who
played around with the library and got
talked into giving a talk at pycon
um as happens to the best of us
cool so as part of this talk I'm going
to be giving a couple of code examples
to do that I needed some data and very
fortunately Brisbane City Council
actually releases a bunch of information
about the library checkups that have
happened over a three day period each
month they've been doing this since back
in about the start of 2020 so there's a
fair bit of data there
um and I am going to use some of that
for my presentation
so
let's start with pandas hopefully this
is all pretty familiar so far I actually
it's very bright up here so I can't see
faces too well
um but yeah basic example we're going to
read in some data from July of this year
we're going to select some columns or
fields and we're going to look at those
rows
so far so good
we can do the same thing with pricebach
now you might be looking at this code
example and saying hey Alex well what's
that thing going on with the spark
session I'm going to go that's a great
question I'm not going to answer it yet
but if you look at the two lines
underneath it very similar to pandas so
we read in the same data select the
columns and we're going to show it
in a very similar output
maybe we want to do something a little
bit more interesting with our data we
want to pull out some statistics I
actually don't write pandas very often
so if I've committed any cardinal sins
please forgive me but hopefully it looks
mostly how it's meant to look in terms
of getting out a row count maybe looking
at the different languages how many
different languages of books been
checked out in and what's the breakdown
by sort of the different age categories
uh juvenile is the most popular for this
life across the Brisbane libraries but
interestingly like the adult category is
pretty much up there as well
and possibly more interesting let's look
at the pi spark version so we can see
pretty much most of the functions do
what they say on the tin we've got our
account at the top which will tell us
our rows in our data frame we can select
and filter down our data frame to get
those languages or this distinct we can
do a group by an account on those age
categories so at this point it should be
at least becoming a little bit obvious
that like if pandas is python kind of
pretending to be art of some extent
this part of the pi Spock library is
python pretending to be SQL which I
quite like I I find that quite intuitive
there's an element of you're able to
make some assumptions about what you
should be able to do based on your
knowledge of SQL and you can translate
that across into python
cool
let's go with one more scenario so in
this case we have we're going to try and
group any checkouts that occur within
five seconds of each other in the same
Library so I'm going to make this
assumption if it's happening close
together in time then I can say that the
same person did it now obviously you
might have different checkout machines
in the same Library it's not perfect but
it's going to let us play around with
the data a little bit
and we've got another code example now
don't worry too much if you can't really
read this this isn't really one that I'm
going to like go through in a lot of
detail this is more just about like
proving that I wrote the code
um that it is possible also just it's a
little bit of my love letter to window
functions because I think they're
fantastic and I love using them and I
liked that I got to use them in this
example but I will move on pretty
quickly but feel free if you are curious
about this or any other element to come
find me afterwards and I'm also going to
be putting a lot of this up on GitHub so
you can find it later
but
if we run this we can find the checkout
with the low group with the largest
amount of checkouts and we can go have a
look at it
and it's this one which might be a
little bit hard to read but I love this
because I get to imagine some kid had
just like the best 68 seconds of their
life as they borrowed out like
everything ever like there is Wings of
Fire Miles Morales uh half of ando's
back catalog and I mean really like why
would we work with data except to find
things like this
um my personal favorite I don't know if
you know how fastbacks work but
basically it's a category in the library
where you get exactly a week to read
that book and you are not allowed to
renew it the degree of optimism I get to
imagine that this kid has about their
week
um
like
I love that so much
um cool but back to the point of the
talk
what should I use
who doesn't love a benchmark
okay so luckily I found one someone made
before me so you can go check them out
I'm
these report this evaluation actually
happened back in 2021 all of these
libraries have changed to some degree
since then so this is really just meant
to be a very like look they're all
relatively similar on small data sets
um
but if we go to a larger data set
and this is an important point of uh one
of the reasons you might be interested
in learning more about Pi spark might
have less to do with your like innate
interest in learning about a new topic
and it might be that your current tools
have stopped working for your data set
because it got too big
so that's kind of a big selling point
and maybe you're at this point you're
like okay fair enough I'm curious I want
to learn a bit more
let's go have a look under the hood
and this is the count function and we
see that when we call the count function
on a data frame it immediately calls
something else and asks it for the count
and that thing has a little J prefixing
it
which is making us think like maybe
under the hood there's something
happening that's not totally pythonic
and we go to but we go to a collect
and a collect is where we say hey I want
this data frame as a list of Records a
list is that's very like that has to
that is in Python this has to be far
more python
um
you know this will be great
uh
yes who doesn't love seeing sockets and
pickling in their python Library
yeah
so that's
I mean I'm told it works and a lot of
smarter the people than me have worked
on this and I believe it works and I try
not to think too hard about this exact
moment when daughter is being thrown
back and forth in the worst game of
catch ever
um but yeah I as far as I'm aware it
works we get the data out in Python it
does appear to be that a Java is
entirely hiding under my library
slightly concerning
and at this point I have to come clean
and say that I have actually come to a
python conference
ostensibly to talk about a python
library that is just like the smallest
amount of python wrapped around a
scholar Library so I'm sorry but you can
for the most part pretend The Scholar
doesn't exist except when it throws
errors that a massive Java stack traces
which is slightly ominous from your
python code
um I like this image because uh it's
just this the star on the end with the
tiny little bit of python wrapped around
it because I think it is very
illustrative of what is going on in this
Library
but this does raise the obvious question
why why do we need a massive thing of
Scala underneath our python right what
could possibly be offering me that would
justify this
uh
and this could Spark
it's basically what if we had functional
programming on data and we distributed
it and wouldn't that be great and fine
and it would never cause any problems
for anyone and it's fantastic
cool sorry
distribution
um now important to touch on at this
point in the presentation all of the
code examples everything I've been kind
of talking to up to this point that's
the driver program we can see that
lovely spark context kind of dodged a
little bit earlier
um in terms of if you just want to get
started you just want to have a play
around you're not too concerned about
sort of dealing with that out of memory
like really running a massive data set
you can kind of get to this point and
just have a play around you don't need
to worry about the next part yet you can
do a pip install maybe play around with
a couple of java settings but you can
open up a notebook and be running pretty
quickly and on your sort of smaller
Library checkout data set
um
but let's say you are interested in this
idea if I want to I want to throw some
worker nodes on my problem I want to
really have the benefit of being able to
work with that larger data set and not
spend three days waiting for my code to
run and great we're going to stick a
cluster manager in the middle so we have
our Lively diagram and it's going to
look something like this in terms of a
dashboard this one's pretty empty but
it's just there for illustration
and basically the promise Pi spark is
making to us is if you write this Java
program and
you write all the code using the library
that you've been given
it will handle all of the thinking about
how we decide what data goes on which
node and which node is doing which task
and how the processing is going to
happen
and whether you believe the library
is entirely up to you but
I will admit it does make life a lot
easier sometimes
but if you don't totally believe Library
when it says hey you don't never need to
worry about it we might want to know a
little bit more about what's happening
under the hood and to understand that
we have to go back to mapreduce which is
the thing that came before Spark
um and so I'm going to move through this
very very quickly but it's just a kind
of a nice background
mapreduce start with some input data
splits the input data across the
different nodes applies a map to it
shuffles the data across the different
nodes applied to reduce to it and
outputs it's lovely it does exactly what
it says on the tin makes it very simple
and easy to remember super useful in the
sense that it solves a problem around
how do we split out the data and with
our processing
downsides are that if you want to do
more complex things you have to start
stringing a lot of these together those
shuffling steps are going to take a
while the input and output steps are
going to start to come for you after a
while so eventually you are going to
really see some issues with the runtime
of your programs
but we all promise spark was going to do
it better
so what does spark do differently
so spark takes this idea of things like
the map and the reducer those
Transformations and says what if they
were lazy
so we can just as we're building them
all up we can build this lovely big
beautiful evaluation plan of all the
different Transformations we're going to
want to do
and then
so it's going to be a lovely big
directed acyclic graph
uh
and then we can start to lay it out and
we can start to optimize across that so
we can break it out into stages a stage
is any kind of processing we can do
before we have to shuffle the data
so maybe we've got the paralyzer filter
the map they can all happen with the
layout of the data across one node and
it
um and it's sort of moving this around
and reorganizing this under the hood I
think of it a little bit like the
relationship of SQL to relational
algebra I don't know if that's a totally
accurate sort of parallel but it's
useful for me in my head it's where
we've got this we've got a stage two
where because we're going to reduce by
key we need to shuffle the data relative
to the key that we're reducing on
um maybe we have to shuffle again for
our stage three because we're going to
do a join we're joining on some
different fields potentially
um yeah we have this lovely optimized
graph that Pi Spock is producing for us
there is one thing that might come up
again depends how sooner or later
whether you will hit this you might get
an out of memory error at some point and
that's going to be really annoying
because you're going to be like hey I
just started using this library because
it promised it would solve all of my
memory issues and I would never have to
worry about that ever again and yes uh
the problem is is that this out of
memory issue is actually that the plan
got too big
and so you either need to simplify the
plan or allocate more memory in config
it's a very easy fix there's a bunch of
stack Overflow questions and answers
about it it's just a really weird thing
to look up so I like to flag it early
before you spend half your half a day
trying to figure out what just went
wrong and why your program is
complaining at you
cool
so we've got our lovely Transformations
we've got all those stages of processing
but they're all lazy so at some point
we're going to have to throw some
actions into the mix which is when we
actually want the result
so these are going to be eager they're
going to force the evaluation of the
plan
and you might be sitting there thinking
great I got you
I'm going to have my lovely plan I'm
going to do my stages one two three of
processing and then I'm going to do my
first action which is going to be
account because I want to know how many
rows of data my output's going to have
I'm going to have my shirt because I
want to have a couple of examples of
what my output looks like and then I'm
going to you know save out save as table
save a safety whatever I'm going to do
and that's
so yes
but your processing might end up looking
more like this
because for all of the optimization
that's going to happen on the
transformation side
spark is going to be really eager as
soon as it hits an action so it's going
to do all the processing when you ask it
for account and it'll give you that
count result
and then it will potentially do all of
that processing again
and show your results
and then it'll do all of that person
using a third time
and then it'll save out the data
so if you're programmed are running
really slow you're feeling like they're
running maybe three to four times longer
than they probably should take
this might be what's happening
so again in terms of introductory stuff
I'm not going to sort of go hugely in
depth into memory management
but it's worth flagging that it's
something that you will need to think
about sort of sooner or later and it the
sooner is going to be when stuff starts
taking away longer than you're thinking
but you when you think it should
um but definitely
you need to be thinking about this to
avoid that sort of multiple levels of
processing across the stages
so
as an example I found this stack
Overflow question I love it because it's
someone basically has asked hey I put
some checkpoints in and uh I have all
these skip stages and is that helping my
performance
and the answer is yes
um so we can kind of and we can break
this down and we can see uh the concepts
we've been talking about up to this
point so each row is an action that has
happened it might be a little bit hard
to see but we can see in that second
from the right column the different
stages of processing and especially near
that top they've got sort of seven
stages of processing happening but
they've got 11 skipped and that's
because they're holding the data that's
been processed up to that point in
memory so they've told spot like hey I
want you to reuse this bit don't forget
about it the second it's gone
um and right in the file we've got the
tasks which is the count of like the
individual tasks that happen across all
of the worker nodes across all of the
stages
so that number tends to be pretty big
cool
um so in terms of what I want to cover
oh yeah there was going to be a whole
version of this where we just ranted
about Java stack traces in my Python
program for ages but full credit to the
people working on postmark is that the
error messages have gotten substantially
better uh over the last few years so
there is substantially less ranting from
me uh on that front uh so shout out to
them uh in terms of otherwise because uh
we have the actions of being
lazy and then the eager evaluation
um
depending on the complexity of your era
it's pretty likely that you're an error
in your plan is going to happen or be
thrown at the point of evaluation so at
that count at that show
but it could be quite higher up in the
fair bit higher up in the plan and so it
is worth looking at
um not too scary but just I have had a
colleague who haven't initially had a
heap of programming experience come in
from SQL and they've been doing a really
good job of like figuring out where the
error was happening and trying to put in
statements around it and trying to
explore it and just not realizing that
it was substantially higher up because
of the way the lazy and eager interact
with each other
um the other fun gotcha is uh because
our code is going to look more like the
thing on the left in terms of you know
we've written out the stages of the
Transformations that we want and then
the actions
um
but the processing is potentially with
that duplication is if you have
non-deterministic data coming through so
for example if you are grabbing a
thousand rows from your table and you
don't really care which thousand rows
you grab at the start you could
potentially end up in a situation where
the rows that are fed into that show are
different to the rows that are fed into
that safe and again memory management is
going to potentially save you there in
terms of saying no I actually just I
want to save the processing at this
point and use it from there
that most confusingly comes up when you
start joining a data frame to itself and
you aren't getting any results from the
join because it each side of that data
frame is taking its own source data so
that can be a bit of an odd one to track
down again and if you sort of shows in
they'll get different results as well uh
cool the other way to handle that is to
just put like an order by or be more
selective in terms of how you if you are
doing a subset of data
um cool
so
what should I use
um this is going to depend really
heavily on you and I unfortunately can't
recommend uh one way or the other there
are a bunch of tools out there but
things you might want to consider how
much data do you have uh what does your
current code base look like
um so in my current job we work with a
lot of geospatial data Pi spark doesn't
have good geospatial capability there is
Apache Sedona which just came out of
incubation in I think March of this year
if anyone here knows anything about it
please come and find me I am very very
curious to learn more about what is
going on in that space
um
other questions how much time do you
want to spend on infrastructure that can
also be substituted for how much money
how much headaches what's the capability
of the people in your team do you want
to ask someone else buy a different
product uh I'm sure there are people at
this conference who would probably very
willingly sell you some stuff uh who is
working on your code and any personal
preferences I actually just really like
using pi spark as just like noodling
around on my laptop for small projects I
find it a little bit more intuitive than
pandas but that's going to be a total
personal preference thing
um
in conclusion
what is pi Spark
oh yep
hopefully I've answered that at least a
little bit uh
can it solve all of my data problems
kind of but you also get some really fun
new ones
um
and are you sure I can't just use pandas
instead
so they they did a bit of a maintenance
which they did actually bring out a
pandas API a while back I haven't
personally used it so I can't give you
any advice on how well it works but it
is there so if you don't like the look
of the code I've been showing you there
is a whole other section of the library
that you can definitely check out and
have a look at
uh cool and then the one other thing I
wanted to cover uh sweet so if you have
sat through to this far and you're like
yep that's that's pretty good that's
pretty informative
um but Alex like I I really want to know
more about this and I really want to
know from someone who actually like
really knows what they're talking about
um I highly recommend checking out
Holden Corral uh she actually gave a
presentation on Pi spark at pycon
Australia back in 2017 so that's on
YouTube
um but she's given a bunch of
presentations about Pi spark I watched
many of them when preparing for this
talk uh so I'm definitely very grateful
for that
um she's written a bunch of books on
spark so if you have an employer that
enjoys buying textbooks that's worth
checking out
um but yeah otherwise I think that's
everything
sweet
I totally lost track of time on that one
no that was great thank you so much for
that talk um I think the problem of what
you know what I've run out of machine
for memory for analyzing my data and uh
what what to do is a perennial one which
I think it's always good to have some
good guidance on what to do and it was a
really good um practically sort of
demystifying uh Pi spark so thank you so
much
um uh and I'm going to just forgotten
my
gift we have here for you
thank you very much if you have
questions for Alex please pop them in
the Discord or come and chat to Alex uh
later on um and so with that can we have
a big round of applause
[Applause]