[Music]
good morning I am Adam um that is my dog
Stanley who listens to me ramble a lot
about the sorts of things I'm going to
be talking about for the next 25 minutes
or so and he only falls asleep about
half of the time um in the past I've
been a PHP core contributor I worked at
New Relic for several years on
instrumenting various runtime languages
I got a film degree somewhat by accident
um and today I work at the rust
foundation on ecosystem security and
with a special Focus around Supply
chains at the moment and I'm also a
member of the rust projects crates.io
team which is the right grust version of
PPI this talk is sort of related to that
work but also sort of not and I'm pretty
much not going to mention rust again for
the entire talk because it's not really
anything language
specific um the title for those who
haven't somehow uh gotten thoroughly
sick of it refers to this
XKCD um and we're just going to
basically dig right into uh that little
project or at least try and find it so
most of you have probably heard this
story already um feel free to look at
your phones or your laptops for a couple
of minutes while I summarize it if you
already know all this but I'm just going
to quickly put us all on the same page
so supply chain security has been
obviously a buzzword for you know at
least a couple of years maybe maybe a
few now um but it really got thrown into
sharp focus in uh February this year um
as a result of this project XZ um which
it's really nice to be in Australia and
be able to say Zed uh instead of North
America it's a compression standard um
like gzip Zed standard lots of others um
that's been in widespread use for
several years um quite a few dros uses
like their their core their core
compression uh format the core of it is
an open source project called exed utils
that provides a C library to handle
compression and decompression and
binaries to do on the command line
python has wrapped this in the standard
Library since uh 3.3 which was quite a
while ago um so it's you know it's part
of python as well so for most of its
history XZ has been a single man project
written and maintained by a guy called
lass cin whose name I'm sure I just
mangled cuz he's Danish um like a lot of
Open Source maintainers the amount of
time that he's been able to devote to uh
his project over the years has varied a
lot right um and so a couple of years
ago he got a handful of not very
friendly mailing list posts um like this
one um on a very otherwise quiet mailing
list uh basically insinuating that he
wasn't doing a good enough job of
triaging bugs or responding to feature
requests um and I'm just going to quote
the last couple of sentences of this you
ignore the many patches bit rotting away
on this mailing list right now you choke
your repo why wait until 540 to change
maintainer why delay what your repo
needs this is actually one of the friend
having been an open source maintainer
for many years off and on this is
actually one of the friendlier messages
that you sometimes get um but it's still
kind of not nice right like you're not
going to walk away feeling good about
yourself I also just want to point out
that this appears to have been um crap
basically like as far as I can tell it
wasn't falling behind he just there
wasn't a ton to do
so but around the same time a savior
appeared uh a new contributor called
Gan um Gia provided several bug fixes
was apparently very helpful uh both on
and off the mailing list and partly most
importantly they weren't a driveby
contributor right they stuck around um
they were around for a couple of years
and eventually lass promoted J 2
co-maintainer status and J released xed
utils
5.6.0 in February this year
that release included a back door most
of the back door was not in the git
repository for xed UTS but it was only
in the source tabbles it is depressingly
common in the SeaWorld for your source
tabbles to your releases to not be
reproducible from the repository easily
because they often include pre-compiled
artifacts because autocon is terrible um
mostly this is to prevent people from
having to be exposed to the full horror
um side to the aside um this is actually
obviously not XZ this is work I did in
the rust ecosystem earlier this year to
basically try and look at where the
packages match their repositories lots
of every ecosystem is doing this kind of
work python definitely is as well I
think probably the next two speakers in
this room probably know of efforts that
are going on uh right now to do this but
this is this is happening everywhere the
exact details of the back door don't
really matter it was theoretically
really bad it would have allowed
essentially arbitrary SSH access to any
system that it was targeting
what does matter is it was implemented
just just sloppily enough that a guy
called Andres fre who was doing
unrelated testing work for a drro found
it a few weeks later critically before
most dros had shipped xed 5.6 um by the
way I use Arch it had already shipped
xed 5.6 but conveniently the back door
didn't work so that was good we got
super lucky on this if this had been
more competently implemented I don't
think it would have been
um we we're just we're lucky that it was
kind of not super well
done and yet in some respects it was
super Well Done Right like somebody put
in like two years of effort to be a
maintainer on this project and actual
effort and if you find me when I'm
drinking beer later I can tell you about
the betting pool uh that I'm aware of
for who was actually responsible for
this and who the favorites are um open
source can give many eyes and make bugs
shallow but I don't think this would
have been caught quickly in most cases I
think we got as said very lucky which
then scares me because how many of these
Exist Elsewhere so that's the background
um there
were a couple of things that were
interesting which I've already touched
on one of them was It was kind of hidden
because it wasn't in the repo and the
second one is that it wasn't it wasn't
an attack that was being caused by a
maintainer like security PE Dr language
and Dro security people tend to have
tended historically to worry more about
what happens if a maintainer gets their
like password compromised or something
like that like this is why tofa is a big
thing why you know ubo has a you know
thriving business
Etc this was someone who'd spent
multiple years gaining trust to get into
a position where they could do this the
call was coming from inside the
house but the thing that I also keep
coming back to on this is like it's not
the inside baseball security side of
this it's like how this happened if
you'd ask me in January what do I know
about xed I could have probably told you
it was I could have told you it was
compression standard I could have told
you it was a bit better than bzip not
quite as good as Zed standard you know
something along those lines I didn't
know who maintained it I've never heard
of lass Colin before that I don't I've
never heard of Gan it's one of those
libraries that gets installed by like
some package that's part of the base set
of packages on you know most Linux drro
and Mac OS and so on you know it's it's
something that's implicitly being part
of python for almost a decade um I'd
actually like to think I pay attention
to this stuff but I missed this I didn't
know this was like I didn't Noti this
was a oneman project from a guy who was
getting heat on a mailing list um about
how he wasn't doing a good enough job so
it's literal metaphorical
Nebraska so like most of you all I hope
um I would like the things I depend upon
to be stable and secure some
dependencies are stable because they
literally never change but that's
actually pretty rare like even xed which
the format hasn't changed for over a
decade still had a little bit of
development going on right you know
people invent new dros and build systems
and all sorts of stuff so there's still
going to be
releases so my contention which I don't
think is controversial is that projects
are more likely to be stable and secure
if they're properly supported we're kind
of in maslo territory here right like I
don't want an open source maintainer
having to worry more about you know say
how or food as their primary concerns
and that's before we even start thinking
about like parts of the world where
those are actually higher up the pyramid
because your home is getting blown
up
so that's the rub how do we find people
where they are worried about those low
levels of the pyramid and where they
could use some help and how can we what
is going on in metaphor and possibly
literal
Nebraska so how do we find them how do
we find what they need and how do we
maybe help them um so I'm going to talk
about this I said a little bit from a
python perspective um you know I only
have you know 25ish minutes and also you
know python um but nothing I'm talking
about is exclusive 21 ecosystem I'm
going to guess most of us in this room
even if your job is theoretically just
python you probably do more than just
python right like when I've had jobs are
python I'm still you're still writing in
my case it was usually JavaScript like
to to do front ending stuff so you've
probably got dependencies in multiple
EOS systems multiple languages and this
applies to all of them so let's dig a
little bit into a specific project so
this project is called zulip it is best
summarized as a fully open- Source slack
alternative uh the Russ project uses it
as a uh as its primary method of
communication and conveniently it's
written in Python with jeno as the
framework so like most python projects
we have a requirements file in the repo
uh it's not a p project. tomal um it
might be now actually it was a few
months ago I looked at this um and it's
even locked which is nice so when I'm
working on a project I usually hopefully
know my direct dependencies but I
definitely don't know the transitive
dependencies right like I that's not a
layer I tend to look at so but
conveniently we do have them all laid
out so let's let's dig into that um I
used a tool called sift to generate an
sbom uh software build of materials from
the requirements uh the reason for doing
that in theory you could go straight
from the requirements file and dig
through through the pii API and figure
out like what repo matches and things
like that just don't like you know it's
it's a lot easier if you have if you use
a tool that already exists for this
there are zulip has 301 unique python
dependencies which I'd actually say is
on the low side honestly um it's a
different language but crates.io which I
work on uh has almost 500 rust
dependencies and exactly
1,999 JavaScript
dependencies I assume a bunch of them
are like pad and things like that but
sure so I can use this data to basically
get any other metadata I'm interested in
but here's the thing I don't have the
time or the energy to look at like 300
projects in depth right like we've got
to filter this down somehow I can look
at five GitHub repos but 300's kind of
pushing it I assume at some point my you
know if it's a if it's a paid thing my
boss would probably like me to do
something more productive and if it's an
unpaid thing I'd probably just like to
do something more productive
and as we work through this it's
important to remember this is still
incomplete right like CFT doesn't jump
to the c dependencies or the rust
dependencies if using something like
cryptography that those python packages
have and that's before we even start
thinking about base packages in the
container image your operating system
kernel there are limits to how deep you
want to go right like it's probably
impossible to directly help the
sustainability of say the people in in
you know Taiwan working at the chip Fab
but you definitely at least want to be
conscious of this as you start digging
down okay so there's how we evaluate
this is honestly tricky so there are
scorecards out there like the op ssf
scorecard that attempt to capture
various metrics and quantify them um you
can access this information on depths.
which Nikki ringland who I just met will
be presenting in the very next slot um
but these aren't necessarily the right
or only tools that are available for the
job although the raw data is itself
useful
so one thing I just kind of want to
touch on here is the current state of
security School cards doesn't
necessarily capture all of the risks
that you might be interested in like if
you think back to the the profile of
again X Zed as an
example you know XZ like I said I I
would argue XZ was not a terribly
unhealthy project on the face of it like
you know maybe there were patch of bit
rting on the mailing list but I'm not
really convinced that
were
so it's important a lot of these
scorecards currently because they're
coming out of a security space which is
also where I work they're very good at
capturing security risks so you know are
they published securely are the
artifacts reproducible do they match
their sources Etc but it's harder to
capture some of the more human aspects
of this now to give op ssf they're due
they're actually trying pretty hard like
I'll just go back to that for a second
like you know right at at the top
there's code reviewed there's maintained
like you know these are things are
trying to capture like these are scores
they're trying to capture are people
basically do people care are people able
to care about this package um
but that's still it's hard to quantify
and I'm it's still you know there there
were two categories there and there are
several more that go into this score
that are secur more security
focused security is important I should
probably say this because my job title
includes the word security
but as I said earlier my contention is
for security to happen you also need
sustainability I don't think you can
have the security without people in good
places
so trust me when I've maintained open
source projects I already know if I
don't have enough hours in the day I
know I don't have enough hours in the
day or reviewers or testing so it's
crucial to sort of understand where the
problem spaces are
because it might be more complicated
than just throwing money at the problem
and I'll come back to this in a bit
although also if you have the ability to
throw money at the problem please throw
money at the problem um you know
maintainers would appreciate that but
you've also got to do it in the right
way because just throwing angry issues
at maintainers saying you need to do
this extra security thing because you
know the scorecard says you need to is
actually not super helpful um the person
at the other end probably cares um on
some level they might say they don't
because that's a defensive re reaction
but they probably also have their own
things they need to be focusing on so
again I'll come back to this a bit at
the end but figuring out how to provide
support is important so okay so what
health factors can we look at little are
more that we can try and quantify to
some extent and I said some of these you
know some of these are on deps dodev and
the and the op ssf
scorecard um issue count obviously uh
how many issues being open but the the
actually important one is actually how
many issues are being closed or assigned
or you know triaged essentially how much
is actually getting dealt with PO
requests essentially the same thing how
many things are getting reviewed how
many things are getting closed how many
things are at least getting you know a
comment something that's not from you
know the latest llm hype bot that's
trying to analyze
it number of maintainers is a tricky one
because small projects the healthy
number often is actually one um but you
know if you're looking at a bigger at a
bigger project you know if there's
100 poor requests being opened a month
and there's one maintainer that's
probably an issue that ratio is probably
a problem recent activity level again
this one can be misleading because if
it's a stable project that's doing a
stable thing maybe there's not a lot of
recent activity CU it doesn't need to be
but it's saying you have to be conscious
of and this is kind of related I don't
mean diversity in the in the Dei sense
here I just mean how many commit authors
are there and this is related to the
maintainer one but again the ratios can
kind of matter here right if there's you
know 100 committers and one
maintainer maybe that maintainer needs
some
help and again you know just to
reiterate the point there are different
shapes of projects like what XZ looks
like is very different to what Django
looks like which is very different to
you know pick a project what what
Twisted looks like for
example
so it's important to balance that level
of scale and the level of recency right
you know how how active does the project
need to be um if there's you know if if
every po request is getting closed but
it's getting addressed then well okay
maybe it's just a singular maintainer
who has a very strong Vision that may or
may not affect your choice of whether
you want to continue maintaining dep
depending on their project but it's
valid so I threw together some some
basic code to take that s bomb and just
like throw out the raw numbers the op
ssf has tools for this as well like the
specifics of this don't really matter
I'm going to zoom in on a couple of
these I've redacted the names um
basically because I don't actually I
don't not I think anybody in this room
would do this this is something I put in
more for American audiences but like I
don't want anyone going on a crusade
right like you know just because just
because a particular python project has
bad numbers doesn't mean they that you
need to go like be a knight in shining
armor
immediately or as I another way I put it
a couple of times is do not go start
ganning these
dependencies so what did I
find hello lights must be 10 minutes
minutes a mixture of things which is why
the the human element is important so
again redacted the the names so here's a
package that accept sponsorship has a
lower commit rate recently so you can
see like the percentage of expected
commits is sort of lower than the
historical average and haven't closed an
issue in the in the last they haven't
closed an issue in the last month um
well they've closed some issues in the
last
month they've also historically had a
lot of contributors but you know there's
not a ton of activity right now so on
the bright side they accept sponsorship
both the repo and the organization that
owns this so this is one where if I was
in a position to you know offer some
some support or ideally convince the
people I work for if it's a work project
to offer some support then this would
probably be a good candidate right like
this would be worth digging into
further um here's another example of a
package that is probably fine they don't
take sponsorship they've got lots of
contributors they have
they're active they've got a lower open
PR and issue rate they're closing issues
they're efficient they don't accept
sponsorship and I'm kind of jealous a
little bit to be honest so they're
they're probably
fine and as a final example here are a
couple of dependencies from the same
organization and repo they're
technically different packages um again
I sort of look at this and I think maybe
they could you know maybe they could use
a little bit of help they do have a good
um they do have a good commit rate
they've got plenty of contributors but
they do accept sponsorship and they're
not closing issues at the rate that you
would probably hope so again since they
they accept sponsorship maybe that's an
option right you know maybe you can kick
them some some some cold hard
cash but again I just want to reiterate
at this stage we're still really just
identifying potential candidates for
support at some
point you kind of have to actually
engage a human element of this or you
know an llm maybe I
guess you know spend half an hour an
hour just looking at the recent issues
in PRS skim the chat backlog and so on
because if you've already narrowed it
down to like you know a few then you've
got the time to go do this because the
need for support doesn't always manifest
as overtly as like somebody putting up
the flag saying they want sponsorship on
GitHub right um I know of a a rust team
for example right now that's struggling
with lack of uh maintainer time and yes
we're we're doing something about that
but you wouldn't know that if you
weren't actually on their chat Channel
because from the outside most of those
metrics actually look fine it's just
that they're clearly approaching burnout
so you don't know that just based on a
scorecard necessarily it helps you
narrow it down and helps you think about
what's truly fundamental to whatever
you're working on but it doesn't get you
all the way to I should actually support
this or not support
this all right so we've got a we've got
a short
list we found some projects that might
need some help what can we do okay if
you have a budget and you can give money
and and they want money give them money
like this is this is not rocket science
um obviously it's late 2024 um money is
not as abundant as it once was alas um
but you know if you work at a company
that depends on open source software and
you don't have a budget to basically
support your
dependencies start finding out if you
could add one like ask people I know
multiple people at multiple companies
who basically
got this by just going and asking in
somebody in a position of power just
being like we never thought about this
like this is an existential thing if
you're building on top of an open source
stack and everybody's building on top of
an open source stack there are also
Clearing Houses like um tidelift
ecosystem foundations hi um that may
provide donation opportunities but since
the point of this was to find
underserved dependencies in your
dependency graph chances are they
probably have the same sort of blind
spots you might have because they're
probably using the same scorecard
fundamentally another option I think is
often underused in the corporate world
is in kind donation um donating 5 or 10%
of a of an ic's time to help with
project tasks is actually huge for a lot
of projects and maintainers I know
personally for things I maintain I'm
really time poor so that's actually the
kind of help I would generally like more
I mean I also like money but you know
practically the time's actually more
effective project tasks here don't mean
necessarily writing code um I have a
sneaking suspicion the most useful thing
I ever did in open source was triaging
the bug fire hose for PHP for a couple
of years um triage documentation
Community Management code reviews design
these are all things that an individual
can help with plus there a huge bonus if
you say have someone go do code
review having someone who knows a bit
about that project even if it's
transitive and five levels down
independency graph might be really
useful if you get a if you get a we
weird bug most of us probably have
stories of like oh that one time that
this bug was caused by Solaris having a
weird get time of day issue um yeah like
these that's the kind of domain
expertise that you can't necessarily buy
but you can definitely invest
in um if you are at a larger Place
sponsoring infrastructure can often be a
useful way of helping projects um open
source projects of a certain size
usually get pretty good support from
large infrastructure uh providers um
definitely a little more challenging
that used to be again I have stories
probably shouldn't tell them on stage on
video um but smaller projects often
don't get that and it's often not
covering everything so getting credits
or donations to cover infrastructure
Services Etc can really
help but the most important thing to do
is actually just to engage go to the
person who go to the maintainer of the
project ask them what they need um send
them an email I don't think I know too
many maintainers I'm going to guess it's
about 1 or 2% of the population
who would do anything other than be
extremely grateful that you asked you
might not get an answer for a while
they're probably busy but they
definitely want to know but if you're
going to do that you need to be in a
position to follow through empty
promises don't help and they just waste
people's time
so you know provide the help that you
are offering
basically so the punchline the talk is I
really lied with the title um I was a
liar yesterday when I did my lightning
talk on multiple levels and now I'm a
liar today quantification doesn't
actually get you everything it just gets
you the starting point to make the
judgments that you need to make to judge
what to support and what might need help
um I just want to S you know the point
of talk really is that we can take
things that we already have lock files s
bombs um The Dirty Little Secret of s
bombs is that lock files are s bombs
really but anyway um use them to find
the components that are the Nebraska in
the dependency graph and then go out and
figure out what level of support you can
provide and even as an individual that
might just as I said it might just be
donating some time helping with code
review helping with documentation
something like that because at the end
this is a human problem like almost
every problem in Tech well a bunch of
them in llm problems but most of them
are human problems which you need some
level of intuition and experience with
with open source and to do that you have
to show up and
help um thank you I intend to share the
code eventually but dev.d shares most of
this anyway so this is not as important
as it used to be um but if I can SP you
the horrors of figuring how to piece
together the GitHub graph and rest apis
I'm happy to because I had to do it
professionally for a couple of years um
I think we started slightly late I can
probably take one or two questions if
there are any uh but again thank you if
you like the talk I'm Adam and if you
didn't like the talk I'm Tom
Eastman thank you so much um and as the
token for appreciation
this is my awesome thank you thanks