[Music]
Hello everyone. Uh welcome back to talks
for this afternoon. Uh I have the
absolute pleasure of introducing Caleb
Brown who will be talking to us all
about how archives can be very
vulnerable.
Thanks.
All right.
Uh, I'm glad you could all join me after
lunch. I hope you can not fall asleep.
Um, so as you've already heard, I'm
Caleb. Um, I work at Google as a
software engineer. Um, and I work on
open source security. Uh, I also do a
bit of work on deps.dev. Uh, and some of
my teammates will actually be talking
next. Um, I'm also a maintainer on
something called the OpenSSF malicious
packages project. Um, which means I get
to see lots of reports about malware all
the time. And so my background in Python
and my work in security got me wondering
a while back.
How secure are the technologies our
ecosystems are built on? I'm talking
about the fundamental technologies, the
nuts and bolts. For me, my interest has
been focused on packaging. So source
discs, wheels, that sort of thing.
So, having spent a fair amount of time
poking around Python packaging, I was
intrigued when I read a report earlier
this year that used tar files and sim
links to compromise a system. After
reading that report, I immediately
wondered if Python's tar files were
vulnerable to the same sim link based
attacks in tar files. And long story
short, I found um three vulnerabilities
um in the the tar file library in
Python. Um, and these were fixed uh
earlier this year in in this version of
Python along with two other tar file
CVES as well. Um, but today I really
want to focus on this one here.
Um, particularly because this
vulnerability is the most serious. This
vulnerability allows a malicious tar
file to write anywhere on the file
system. Well, at least anywhere with
privileges to do that. And while I
explain how this particular
vulnerability works, I I would like to
discuss, I guess, some general security
concepts and ideas I was reminded of
while I was discovering these this this
vulnerability and the others.
So to get started, we need to go over
some background just so we're all on the
same page.
Um, I'm sure most of you are familiar
with TAR files, but I'll cover it
briefly. Um, TAR stands for tape
archive. Um, it's really old. Um, and
it's original intent was for backing up
your system to a tape file. Um, and so
it's kind of intended for this
sequential operation. Uh, and it doesn't
do compression by itself. It's just, uh,
usually you add some compression on
later. So you might see a tar file
followed by agz orbz 2. Um, that's the
compression added afterwards. And it can
store stupid things as well as normal
stuff like you expect like files and
directories. But it also stores device
nodes and fifo. What? Um, so that's tar
files. Uh, the other thing we should all
be on the same page about is sim links.
Um, if you're not from a Unix
background, um, these are kind of like
shortcuts. Um, so a symbolic link is
essentially a file. Uh, it stores a path
to some other place on the file system
and we call this the target. And for a
lot of file system operations, it just
behaves like the target. So if the
target is a file, you can open it, you
can read to it, you can write to it. If
it's a directory, you can list the
contents. So that's sim links. And the
last thing we need to introduce are path
traversal vulnerabilities. U so in a
path traversal vulnerability, you've got
a file system. There are stuff in your
file system you really don't want people
to touch. There's stuff there that
that's where you expect things to
happen. That's here. Um so when you are
doing operations, you kind of want
everything to stay in that circle. Um,
but a path traversal vulnerability is
when an attacker can manipulate the
input, usually a file name, in such a
way that it can get out of that circle
and access something else in the file
system it wasn't meant to do. Now, if
you're from a web background, you might
be familiar with this sort of um pattern
where you have a query parameter and
someone's just added a whole bunch of
dots. This is like the most simplest
thing. Um there are a lot more complex
complex ways of injecting the path
traversal. Um but then you can see that
it just gets passed straight into open
and if this Django snippet was run it
would dump your entire password file
into the web response assuming you had
permission to read it. Uh archive
vulnerabilities are kind of similar. Um
there's just a file name and a bunch of
content and it's user supplied. Um, and
so when you if you had a tar file with a
a file name like that, it would extract
and it would output some authorized keys
into your SSH directory and give the
attacker access to your host um
potentially the root account. Now,
archive vulnerabilities uh that are path
traversal vulnerabilities um has a term
that's commonly used called zip slip. Uh
this was coined in 2018 by Sneak. Um,
it's sometimes called tar tar slip for
for tar files, but they're basically the
same type of vulnerability. Uh, but for
Python and the tar file library, uh, its
history of path traversal
vulnerabilities goes back a lot further.
Uh, so in 2007, a CVE, a vulnerability
report, um, was created about path
traversal in the Python tar file
library. Um,
so not much happened for a while. Um and
then in 2023 uh with PEP703 it
introduced the concept of filters to
Tarfile so that you could make the
extraction uh safer.
Uh and it it provides two filters that
we care about. Uh the first is the tar
filter. Uh and its job is basically to
behave like the command line the tool
the tar tool. Um and so the main thing
it does is uh stop you using absolute
paths and it tries to avoid u path
traversal for writing. So if you give it
a destination all the rights should stay
in that location. Uh the other filter is
the data filter and it is the like it
does everything the tar filter does but
it's stricter. Um it also tries to avoid
having links inside a destination that
point somewhere else. Um and uh it does
other things like only lets you extract
like files, sim links but not device
nodes. Uh and it does sensible things
like setting permissions and stuff on
directories. And this is going to be the
default next month when um the next
version of Python is released. Uh along
with this PEP, they also added um some
more changes to the API so you could use
them. Um you can go into tar file and
you can add um your filters.
Um you can even add them if you're using
it from the command line. Um so these
changes got me thinking about um these
filter changes got me thinking about
security retrofits.
Um and first big point here is that
retrofitting security is hard.
Um it's hard because usually you're
doing it because you're forced to. It's
in reaction to a vulnerability or a
compromise. It's also hard because
usually you're adding security to a
place where there's existing code and
there's existing technical debt and
you've got to manage that. And after
you've added your security fix or you've
hardened the security, the complexity
and the technical debt is usually a bit
higher than it was before.
And then after all that, you've got to
deal with like maybe you've had to
change something in a way where you need
to migrate systems or um get users to
change to a new process.
Retrofitting security is hard
and when you do it, please avoid
shortcuts because you are going to do
it. Um, but avoid shortcuts. Fast fixes
have their place. If you have a critical
vulnerability, you're leaking private
user data. Um, do what you need to do to
get that closed.
But otherwise, it's really important to
approach security hardening carefully.
What are the tradeoffs when you're
what's with your solution? What are you
gaining? What are you losing? Um, you
it's good to avoid unintentional
consequences.
It's very easy for an intended security
fix actually make or introduce new
vulnerabilities as well.
And you also want to take your time and
be careful because you want to limit
complexity. Um, complexity is the enemy
of security. And in the long term, if
you take shortcuts, it only makes the
challenges of retrofitting security
harder.
But retrofitting security well is worth
the effort.
If we look at the tar file filters, um
the secure default is is almost here. It
lands next month. Uh and this secure
default for anybody who's using Python
314 is the default. It means that
they're all protected um by this change.
Also, the change is well considered.
it's only a small API change. Um, and
it's optional. If you're using it,
you're using it to opt out. And the last
thing, and probably most importantly, is
that it's a centralized implementation
that is strong and well tested. Um, if
we didn't have this, you'd be forcing
developers to all go and do it their own
way. Uh, which risks them doing it wrong
and for everyone having different
vulnerabilities all over the place.
But I'm getting ahead of myself. Um, we
still haven't got to our vulnerability
yet. And I'm sure that's why you're
here. So, let's go back to these
filters. Um,
the vulnerability that I'm interested in
is focused on breaking this. We want to
be able to write files anywhere,
anywhere we want on the file system.
So, the vulnerability I found does allow
this. Let's jump into tar file
implementation in Python. I promise you
it's not going to be like super
detailed. You'll be able to hopefully
follow along. Um so first of all here is
the extract all function in the tar file
module. Um the and it's I've simplified
it. I've got rid of code you don't need
to worry about. Um it ensures the first
thing it does is it ensures that the
filter function is actually a function.
So you can pass in a string or a
callable or none and it will make that
sure that's something you can call and
then it goes through each member in the
tar file one by one in the order that
they appear in the file
and then for each of the members it will
call this function get extract tar info
and that basically just applies the
filter to the each member and then after
that it extracts the single member.
So what are the filters doing when it um
calls that? Uh this is a simplified
version of the filter. Um and this is
specifically like the top part of the
function. And this bit of code is only
about making sure that the file that
we're writing or the directory we're
creating is inside the destination path.
And you see there it's got this real
path call which kind of gets rid of
traversal and links and stuff. And
there's another one. And there's this
check to make sure that they both start
with the destination prefix. Um, and you
might be wondering, this code looks
pretty normal.
Can't see much wrong with it. And I when
I was looking through this code, I
passed over this many times and I
thought it was perfectly secure as well.
But I started wondering,
Real Path is doing a lot of heavy
lifting here. What What does Real Path
actually do under the hood?
Maybe maybe we can find a bug and bypass
these checks. So let's jump into real
path. We're going down the layers here
um into more and more detail. Um so real
path has a basic process um where
firstly if the link is the path is
relative it makes it absolute by
sticking the current working directory
on the front. Then it goes through each
of the path element. It splits on the
slash and then slowly builds up from
left to right the path checking each one
as it goes or processing each one as it
goes. So A then slashb then slash C all
the way. So as it does that it does this
very basic uh set of things. Firstly if
it's a dot or a dot dot it deals with
that. If it's a dot it'll get rid of it.
If it's a dot dot which is a parent
reversal it gets rid of the dot dot and
the one just to its left. Uh if it isn't
one of those, it'll call LSTA. Uh lstat
is an operating like a wrapper around a
system call that returns data about the
path.
Um and it's specifically used to check
if it's a sim link. And if it is a sim
link, then it calls read link and
replaces the element in the path with
the contents of that link. And it does
that over and over again until it's
done. So, this all looks pretty normal,
but then we start to see interesting
things. This is not working. Oh, now you
decide to um Sorry, it's giving me like
USB warnings. Um, so we look at the
error handling
and the first thing I noticed was that
the default for real path is to be
strict equals false. So errors are
ignored by default which means that if
the lstat call that checks if it's a
link fails um and it could fail for a
bunch of reasons um then all of the rain
remaining path elements that it hasn't
processed they essentially get stuck
onto the bits that it has processed.
Let's exploit this. I think we can do
this. Um all right let's introduce our
our favorite thing of the day which is
path max. We're now getting even lower
into operating system. path max is a
constant um and it's a constant that
applies to operating system calls. So
this is when you're calling into the
kernel um there's this limit that says
how many bytes of a path you're allowed
to pass to it. This is important to
remember. It's not a file system limit.
You can make as deep a directory
structure as you want. Um I think um as
long as you don't run out of iodes or
disk space um and so its main job is to
stop you breaking the operating system.
And here are the the amounts that each
operating system has as its limits.
We're not going to use that limit
because that's kind of big to fit on a
slide. Um,
so we're going to just assume we have a
path max of about 30 cuz uh that's what
my drawings kind of worked out to be.
Um, so assume we've got a uh a buffer
here where we're building our path.
We'll just stick the destination at the
front. That's going to be there. Uh, and
then let's do something funny. We're
going to add a bunch of directories one
by one. Um, and this is important later
when we try and uh traverse out. And
then at the end, let's add a sim link
that exceeds the length of path max. Um,
and and this link is just something that
takes us all the way back to the top
where destination is. So if we pass the
green part and the red part into lstat,
this is what happens. Um, the first one
passes happily. returns the information
about that directory, but the second one
doesn't. It throws an error and by
default, real path swallows that error.
So, what happens? We're kind of stepping
out now. We'll go back to have a look at
real path and have a look what it does.
So, if you pass that long link with a
sim link at the end that exceeds path
max into real path, what it returns is
the unexpanded sim link.
But what it should have returned is
either an error or slashest which is
kind of a problem. So this is a problem.
Any path element that is overlapping or
after pathmax never gets expanded.
So this behavior of real path
has me thinking about abstractions.
Um, abstractions are really good. Uh,
they're about uh in computing they're
about providing access to something and
a high about hiding details and
complexity so you don't have to deal
with all the minutiae. Um, and in the
end, an abstraction allows you to think
at a more high level about what you're
doing. Abstraction is powerful.
Computing is built on layers and layers
of abstraction. It's awesome.
Like who has written machine code or
assembly recently?
Is it two? What about anyone like
writing a raw Ethernet frame to a
network interface card? No. Does anyone
know what those terms mean? Yeah,
there's a few of you. Okay, good. Um,
this is why abstractions are awesome.
You don't have to think about all that
stuff. Um, programming languages are an
abstraction. Operating systems are an
abstraction. Even the tar file library
is an abstraction over a collection of
bytes in a file.
But abstractions do have a weakness.
They can make security harder and they
do this in a variety of ways. And these
are the four that I came up with. They
they make the problem simpler than it
should be. Uh and this is often the case
with IO. Um, they can bury critical
security flaws that you should be made
aware of but don't know. Uh, they can
have default behaviors that aren't
secure.
And the last one is harder to explain.
It's they are leaky. Uh, there are side
channels. Um, spectre and meltdown or
rowhammer are examples of this type of
abstraction problem. Um, so if we look
at real path, it has a couple of these.
The first is obviously the insecure
default. Strict equal equals false is an
insecure default.
It ignores any error that is thrown
inside it.
But we can't simply just change the
default to true
because of oversimplification. If we did
that, it's all or nothing. You don't get
the nuance behavior. It forces that onto
the user of that function which is not
great.
The name too long error is not the same
as it doesn't exists.
So different errors need different
responses.
And in fact, one of the main fixes to
this vulnerability was introducing a new
uh behavior for strict in real path
called allow missing. So all errors are
thrown in this case except if the lstat
call fails with a missing uh file not
found.
you're not seeing my RAM scripts. Um so
abstraction presents a challenge on one
hand we so it it hides a bunch of
details which um is both its strength
and weakness. It allows higher level
thinking. It may lets us build bigger
more capable systems but it also hides
important security related details and
issues. So, when building secure
systems, it's important to consider our
abstractions.
All right, let's Oh, man, I'm running
out of time. I thought I had more. Um,
so we're back here with our leaky
abstraction real path. We already saw
this. Let's turn it into an exploit.
Here we go. Uh, this is our tar file.
The and I'll, uh, the first thing we do
is we create that directory. The next
thing we do is we have to create a sim
link to the bottom of that directory.
And then we create use that sim link we
create to create the really long link.
And I we need this actually because you
know how that path is too long. We can't
just create the sim link or the creation
of the sim link will return the same
error. So we use the sim link to kind of
trick it into creating the thing that
exceeds the length. Uh and then we have
this escape sim link. And this is really
just to make things nice. And then we
have our whoop our payload at the end
which is to overwrite the authorized
keys. And what happens is the uh input.
So this is the file at the bottom and
this is kind of reversing those sim
links. Uh the escape is actually uh this
is what it means. So we have this really
long path and that really long link
remember is these um this directory
traversal which takes us all the way
down to the root. But because of the bug
uh because of this pathmax bug what
happens instead is that real path
returns this
um but in reality it's this
and because real path returns this
target path in our filter is set to the
output of real path and this check in
the comment path at the end of the
filter passes and we now can point
target path anywhere we want. We can
write anywhere in the file system. We
have permissions to our arbitrary right
is achieved. Um
so uh you might be thinking like why do
we even have sim links in tar files?
Well this got me also thinking about sim
links and incentives. Um so you have I'm
going to simplify the world down to just
a library user and a library maintainer.
And statistically the maintainer is just
one person uh in their spare time. Um
and the user is kind of interested in
these things. They they want to ship
features. They don't want to break their
stuff. They want to keep it working.
They want their boss to be happy. But a
maintainer, our our statistical
maintainer is more interested in this
this these sorts of things. The
maintainer has a day job. They have a
personal life. They might have a family.
They might have health problems. They
might be caring for elderly people. Who
knows? They just want to resolve their
issues quickly and keep their users
happy.
And at the end of the day, the user only
cares about a limited feature set, but
the maintainer in wanting to keep their
users happy is interested in building a
general purpose library. So sim links
are in tar because it's a general
purpose library.
So what can we take away with that from
this? Uh you need to be wary of
incentives. Um, for a user, be wary of
large feature sets.
Disable things you don't need.
Um, be careful of things you're opted
into without being able to control. For
a maintainer, think about building for
default secure and making stuff that
isn't core optional. Uh, and and make it
optional, but also make it optin. And I
really like Django's approach to this in
some ways where you can pick and choose
the bits that you you want. So, beware
of different incentives.
All right, I've got a minute or two
left. I think we've got time for a demo.
Uh, hopefully this all works. I only I
had multiple demos, but I'll do this one
first. So, this is my We've got a file
system here. We have two directories.
This is where we're going to extract the
file into destination. Do not touch is
the thing we don't want to touch. We
have our very important document in
here. Uh, if I open it, it says on my
screen, not yours. Uh, let me drag this
over. Says hello world. Okay, this is
great. Um, but in our demo directory
here, uh, we have this tar file,
pop.tar. I'm going to I'm going to list
the contents of this tar file.
Uh, it's very long. Um, and because this
is the like a Macbased operating system,
we need really long paths. So, there's a
whole bunch of these directories and sim
links and stuff going on. But, we can
see at the bottom what we like some of
the stuff we're going to do. Um, those
flag links are one of them creates a
hard link, which I won't get into, and
the another changes its contents. Um, so
what I'm going to do is I will try and
find my copy and paste command on the
other screen
because I don't trust myself. Oh, just
to prove that I'm using a vulnerable
version.
We got this. I had to use UV because
like um you going to
All right. This is a vulnerable version,
so it's not one of the ones that's fixed
fixed. I'm using UV because it lets me
pick the Python I want to use. Um, all
right. Now, we're going to Oh, wait. I'm
not in destination file. We'll go into
the destination.
And now I run my pock. And the first
thing you'll notice is not a lot
happened there. But if you look in our
do not touch directory, a couple of
things happened. Uh, there's a new file
there. Uh, and the important file has
uh, uh, let's have a look at it. It's
opening up my screen. Uh let's drag it
across. Oh, this one now says that. Um
and this one uh created a new file as
well. So there we go. Uh it works. Uh
which is good. Um
you can't see the cooler demo where I I
created a quick website using uh Gemini
and uh you can upload tasks to it. It's
fun. Uh all right, let's just jump back
into the presentation. So wrapping up um
quickly while I've still got a minute.
Um, I hope you've enjoyed me talking
about breaking things. Uh, finding
vulnerabilities is a roller coaster ride
and um, but I also hope you take away
some new ideas about security and
retrofits, abstraction and incentives.
Uh, and uh, like this is important
because attacks are growing particularly
in open source. I don't know if you've
heard about fishing and npm. Um and
regulations are increasing as well and
new laws like the Australian cyber res
security act. These are mouthfuls and
the EU cyber resiliency act. Um these
are acts that change the landscape for
software engineers in terms of what we
are required to be doing. So tackling
cyber security is more important now
than ever. Uh and finally upgrade your
Python. Uh you can check uh download
stats and like 42% of pe people
downloading no not people clients
downloading from Pippy are like from
vulnerable versions. Um and if you're
interested in metrics like this stay for
the next talk cuz um they'll talk more
about that. Um yeah that's me.
[Applause]
Thank you, Caleb, for the wonderful
talk. Uh, I do have a my speaker gift
for you.
Thank you very much.
A year 2022 month. Uh, we might have
time for one question if anyone has a
question.
If if you want if there's a question,
I'll happily answer it, but I also have
like more demo. So, if you want to see
more demo,
for another,
but anyway, uh, questions probably
better. Um,
sorry. Um, have there been
vulnerabilities found in Python's own
um, you know, legacy type file formats
like uh, uh, pickle or shelf? And uh now
what would you suggest in in terms of
going about opening a pickle or a shelf
from 10 years ago or 20 years ago?
What sort of file format? Sorry. I
pickle or shelf.
Pickle or shelf? I'm not familiar with
shelf. Uh pickle does have its
well-known problems particularly in um
the AI space and sharing models and that
sort of thing. Um so pickles are a well
understood problem and I think there are
mitigations to deal with that. I'm
unfamiliar with the uh shelf example. Um
yeah, but I'm sure there are
vulnerabilities if they're old and
unused.
Uh that's all the time we have for
questions. Uh please join me again
thanking Caleb's talk. Thank
[Applause]