foreign
[Music]
so many people here
okay so let me introduce
our speaker
Rohan McClure so Rohan is a first year
grad at IBM mostly hacking on the Linux
kernel
the Linux kernel strived to be bad work
compatible for user space
today
Rohan will talk about the time that he
accidentally broke that promise
please welcome Rohan
[Applause]
two apologies before I begin I was awake
in Canberra at four at 4am this morning
and so I may be a bit loopy the other
apology is for presenting a talk about
the Linux kernel from an Apple laptop
but um thankfully for your sake I save
you the second hand stress of watching
me X rander so um
I'd like to begin today with an
acknowledgment of country I begin today
by acknowledging the Warren Jerry Roy
warrang people of the colon Nation
traditional custodians of the land on
which We Gather today and I pay my
respects to their Elders past and
present I extend that respect to
Aboriginal and Torres Strait Islander
peoples here today
hi my name is Rohan thanks for coming to
my talk so for the last year in a bit
I've been hacking on the colonel as well
as openssl and uh just in my first patch
series I managed to really max out on
the number of people I could irritate
um so uh I've made my first patch series
and a month after it's upstreamed I get
this sort of feedback
this breaks power PC 32-bit
uh
probably breaks every Sysco with a
64-bit argument all right great great
um
sorts of damage exactly well a bit of an
introduction to the kernel perhaps it
needs no product no introduction but
I'll give a a brief one at a thousand
feet
there sets of sort of dichotomy between
kernel space which is privileged and
user space which is basically
unprivileged and essentially two
security rings in this sort of
Monolithic kernel
um and so user space relies heavily on
the Kernel then for its Process
Management for its file system access
for just general access to resources
Etc and so with that comes a number of
software contracts and so if you were
let's say trying to break all of user
space that is all
unprivileged programs on your system
what might you do well you might
actually violate these contracts
um for those of us unfamiliar with the
idea of a software contract essentially
if you've programmed it all you've used
some sort of API an API stands for an
application programming interface and
critically I'm going to draw on this
distinction later
it's a sort of contract on source code
it's usually not difficult to consume
someone else's Library when you have
tooling which supports you doing so
that is there's a common ABI which I'll
get to in a moment so that you can call
library code provided the parameters
which the documentation if there is
documentation or examples if there are
examples or code if there are is code
um you can usually call that Library
um subject to its specification and an
API implies that that library is
intended to be stable to some extent and
it has exported functions and you can
call it without risk of personal injury
at some rate but uh contrast that
however to an ABI an application binary
interface
what keeps programs which were compiled
yesterday or just as easily 20 years ago
what keeps them running on Modern
systems well that's that's an
application binary interface and these
are extremely prescriptive this covers
stuff like calling Convention as well as
the focus of this talk is on the syscall
interface so I'll get to that in a
moment but um ABI is extremely
prescriptive is what's worth knowing
it's down to exact register allocation
and stack layout
um
uh with an API you can call your library
and as much as your tool chain has
support for that perhaps a foreign
function interface with a bi it
prescribes exactly how binaries look and
how the kernel must respond so there are
actually a number of apis it's not
always just user space to Kernel
um usually when it comes to API the base
right is a what's your object format
which you're compiling to
um if you're compiling code for modern
Linux that's almost always elf the
executable and linkable format
on top of Elf you get further uh uh
further specifications which come from
your choice of architecture the machine
which you're compiling for architecture
is sorry ABI by the way is not ISO
instruction set
ABI defines things like calling
convention for example and then even
operating system can further
constrain
ABI that is how libraries and their
consumers operate but um
the particular level of ABI we're going
to talk about is What's called the
kernel to user space API and that one is
stable intended to be stable there's
another ABI in the kernel which is the
kernel to modules ABI but that's not the
one I broke so we're going to talk about
this one
here
so about the Linux kernel to user space
API this particular bit of the API is
about the sort of tools which arbitrary
binaries on a Linux system should be
able to rely upon existing on their
system
so they include things like the vdso
that's the virtual Dynamic shared object
it's a library which is always mapped
into process memory whenever you load
any sort of Elf file
as well as that this Cisco interface
I'm going to talk about more about those
in a moment the video so actually uh
exists to circumvent some of the
performance ramifications of CIS calls
which almost always imply a sort of
context switch
and so Cisco system calls have been
opted out so that your Lipsy can can
choose to call things via the video so
if appropriate
um
as well as that there are also file
systems so Unix participates in a sort
of sorry Linux participates in a sort of
Unix file based ideology and so
access to your csfs procfs and the
little the little known config FS are
also deemed to be Abi this talk is going
to focus on ciscalls however so
um what actually is a Cisco I think at a
thousand feet again the way I describe
it is a sort of pseudo function call
that is uh when it comes to control flow
the the same pattern exists so your your
code will eventually yield when you
initiate assist call
when the syscall should terminate
hopefully
um and then uh execution should be
resumed at your calling process and so
Linux is a bit interesting I'll talk
more about this in a moment but this is
um
this is the sort of code which the
kernel is expected to support right so I
don't know how familiar you are with
power assembly I walk you walk uh this
through with you
um so uh as per the calling convention
we're going to specify in r0 what sort
of Cisco we want to do clearly we want
to do an exit first parameter is zero
exit with status zero says call
um we have two Cisco instructions on
power
um your Lipsy will choose the most
recent one
uh
in short this Cisco instruction is a way
for unprivileged code
to cause the kernel to facilitate some
sort of critical function call
and so nearly always the way this is
implemented in Hardware is by means of
an interrupt this Cisco is a synchronous
interrupt
in general your interrupt Hardware works
for events which can fire at any time
but this is an interrupt instigated by
code
and that interrupt Handler is expected
to fulfill the contract of whatever
ciscal you just called
a really critical Cisco is fork for
example and this is just to motivate how
how privileged how much reflection on
program data the assist calls end up
invoking so this is a process in
threading controls this call and it's
being abstracted out by libsee so I can
just call the fork Fork function
um and so Fork has to look at the
entirety of your process state
and it what it does is it it duplicates
that thread
um that is it duplicates its memory
mappings and it keeps the same
instruction pointer in the resultant
thread and so when the threads wake up
they both resume execution after the
call site and you'll notice that we take
a return value bid
um that's our process ID it's zero if
we're Frankenstein's monster it's the
name of Frankenstein's monster otherwise
and you you branch on that condition
figure out whether am I the child thread
or am I or am I the parent
um and so ciscals yeah they're a sort of
function they or rather they have a
function like kind of interface but
clearly but interestingly the the
function calling ABI and elf is not
actually doesn't 100 line up with um
with uh Cisco calling conventions but
clearly this is a very privileged
operation which I
uh well a few more bits of a side
uh before we cut to the chase so Linux
is a bit interesting
um so uh above you can correct me on the
Nuance of this later but above other
free and open source operating systems
Linux has a particularly High commitment
to keep a stable Cisco interface that's
because you don't usually as a
programmer invoke sys calls directly you
might use libsy and uh Linux doesn't
doesn't
doesn't in principle prefer a a
canonical lepsy implementation and so
the Cisco interface is deemed to be it's
deemed to be authoritative
um certainly your proprietary operating
systems avoid syscalls at all for user
space so there's lib system on on Mac OS
Cisco interface is not deemed to be
stable
um and then you have nt.dll on Windows
um but yeah Linux has this kind of
extreme attention to detail when it
comes to ABI it's best sum up summed up
by this one particular Linus quote which
is of course in all caps we do not break
user space and uh so that's what I did
all right well
all right um so uh so we have our
interrupt code and um because the kernel
is essentially just this stitch together
big old massive elf binary stitched
together by link scripts
we have these symbols in here for the
actual body of a Cisco our interrupt
code is going to call this body their
inter interrupts are handled by kernel
um and so they used to have uh this sort
of calling convention however to give a
little bit more freedom to how we write
our interrupt code we decided to use
some ugly and horrible macros
to change the convo calling convention
to look like this you don't need to know
what a PT regs is it comes from p-trace
I can't tell you any more of the
entomology
etymology Dragon
um there
um it's it's actual in actual fact just
a structure which represents the
registers of the caller of the Cisco
because it's an interrupt right so we're
we're there's a entirely new context
between user user space code which
called us and kernel code we keep that
context and we're expected to save and
restore it accordingly hence a context
switch
um and so we want to change that calling
convention uh
stay cleared for a bit of motivation on
why we do that exactly but the the kind
of crown jewels is uh I've been inducted
onto the hardening team and uh so we
decided we'd do a hardening and thought
of that included
um uh we wanted to essentially uh
sanitize whatever
um
uh whatever continuity there is between
user space code and kernel code and so
whenever user space can directly or
directly enough in invoke an interrupt
of any sort we don't want the
architected register state of that user
code in any way influencing the
speculative runtime behavior of the
kernel and so what do we do we have to
save even more registers for this
mitigation
we sanitize those registers we set them
to zero and then we restore those
registers before we return
and if it's not obvious to you that
there's an exploit kind of in the
waiting here then um that's fine it
wasn't obvious to me either
um but let's just sort of dive into the
activity of Kernel hardening right so
there's been a sort of seismic shift in
the world of security research with
Spectra and meltdown which is to say
that
um user space may not be able to elect
leak secrets from the kernel directly
but it might be able to do so in an
implied fashion through timing attacks
and these timing attacks become
increasingly available with an awareness
of of the reality which is that
microarchitectures are necessarily
speculative on on super scalar processes
and um
and so what we're doing is uh is
hardening measures right we're just
trying to produce isolation between user
code and kernel code and so I can't
actually guarantee you that I I've
foreseen exploits per se
but um it's a bit tedious to just chase
after exploits when they become
discovered right you'd rather Harden the
kernel against against the prospect if
people gluing together info leaks and
gadgets and one day creating an exploit
and so a bit of a bit of a of a red flag
which alerts you to to the fact that
this might need hardening is that there
is shared architect at State uh that is
the registers of the calling code are
um are in fact retained although cleared
pretty quickly by by kernel code
if um by the way if there's a if there's
an architectural bug we've got bigger
problems so uh we're on the micro
architectural level here
um okay so a quick aside as to how Cisco
wrappers are implemented in the kernel I
promise to not show you too much code
um so there's this Cisco Define for
macro and so this kind of creates the uh
signature the the Prototype what's it
called and see the the prototype for
um uh for the phallicate F allocates uh
Cisco Handler
um and it's going to prepare this CIS
underscore prefix to the beginning
because no one should be directly
calling it by name
um and so if you're going to create your
own Cisco if you're mucking about the
kernel you'll Define this is called like
such and then you'll add a number in a
in a table to say when r0 is this number
in kernel please call this so it's cool
for me
what we're going to do is uh in order to
get a new calling convention see Cisco
handlers uh like any good C program or
would we're going to write an ugly macro
um and uh that ugly macro better
implement the calling convention
perfectly with no bugs ever for
our Cisco Handler and so clearly this is
this is where my bug came from
um as a bit of an aside so I work on on
Power Systems Linux on power
we have our own subsystem arch power PC
and uh there are about 12 million lines
of code in the kernel uh not including
modules not including user space
libraries tests Etc about four million
of those lines is architecture specific
and about half a million of those lines
sit in Arch power PC so it's a big
subsystem
why is it so big well
you could hint to a few things I suppose
for comparison arm has spun off its
32-bit and 64-bit subsystems
um and so we're like x86 in that respect
we they're both under a common
subdirectory but unlike x86 where by
Indian
so you end up with this kind of horrible
Matrix of kind of platforms running on
Arch powerpc
and sometimes the same code Services all
four of these platforms
Ibis are fiddly enough and so with four
sub-architectures of sorts uh yeah
necessarily there there are going to be
at least four abis I need to implement
an API is Right are meant to be
implemented by Tool chains because
they're fiddly and not buy crappy macros
so how did I do in my patch series my
first patch series on the Kernel
um well there's a little babby kernel
developer I had to touch a lot of
um untouched Legacy code
and uh I started off with six patches my
first revision and uh if you don't know
in the kernel we um
we do our communication over email which
is insane to me
and uh once you send off saying hi I'm a
young and not yet cynical kernel
developer and I would like to modernize
your stuff you get a laundry list right
of things which you should fix and so uh
by the time I I submitted part of my
patches it was revision six and uh I had
25 patches on the list which was
ridiculous most of them got into a
kernel six one but the actual hardening
component got into six two because again
if you went to Russell's talk everyone
will complain if you make your CPS like
two percent slower so
um
yeah like where I really
um hit a wall is with this 32-bit
calling conventions I I really
um did work hard to to make these work
did I miss the slide no all right I
really did work hard to make these work
but there are just some cursed
um uh special cases so here's a here's a
relatively normal ciscal CIS F advise
64. where does the 64 come from well
this lofty you kind of you know if you
know that's always 64-bit irrespective
of of what tool chain you're using
um I think that's a posix thing
um and uh
yeah well some architectures actually
want you to implement this one too
um this is this is where it gets bit
cursed you notice that the advice
parameter actually changes its position
this is probably the most attention
you've ever paid to this fact but um the
reason why it does that is because
there's a sort of soft rule of no more
than six parameters to ciscals because
calling conventions on all these
architectures do register bound so it's
called most of them architectures do
registered Bound syscalls for the first
six at least
um and so that's it's kind of common
um and uh
see 32-bit machines still need to be
able to send these uh these two 64-bit
params that's the the lofties I'm
pointing to
how do you sand a 64-bit parameter with
32-bit arguments uh registers well you
obviously split it in two and so that
comes to six and why did we have to move
advice to the back
well it's because these architectures in
their function calling conventions
specify that these registered pairs
which unite to become a single 64-bit
values is register pairs better be odd
even in their in their indexing all
right so
um yeah so some 32-bit args kind of
expect the the Sysco handlers to look a
bit like that
and then the order of uh Lo and hi in
each case is going to be dependent on
endiness
um
yeah well all of these are implemented
not in generic code they're implemented
in power PC and they were last edited in
2005. so so I had to clean up a whole
bunch of them they weren't even using
the macro I showed earlier so yeah a lot
of fix up I won't believe with the
details and then there was this little
gem all right well so it turns out
because this is a Cisco right and we
don't break user space this was ABI for
a while this was this was
I couldn't change this without much
arguing so in short there was this crazy
um like power PC 32-bit only uh Cisco
where um it would check whether the
first parameter is a pointer or Not by
saying is it bigger than a prohibitively
large number of file descriptors and um
if so it would arbitrarily decide use
these old semantics if not use these new
ones and that's nuts so I after quite a
bit of arguing and then resuscitating
all the conversations on the mailing
list and saying no look we really were
going in this direction just people
didn't have time
um we should just replace this with
something sensible and so I
finally cut out this thing and replaced
it with say select
besides the way I'd set things up you
couldn't reliably recurse this call
handlers so this kind of had to go
anyway details
all right so how did we do with
performance right so we're having to
save more registers in the way and that
we hope to sanitize all of the users
registers
and then eventually restore them and so
saving to the stack restoring from the
stack it's going to be hella slow right
oh sorry sanitizing more registers
clearly that's more work on every
contact switch
and it turns out it made things faster
um I made context which is faster
accidentally
um and so I've hence been summarily um
removed from the hardening team because
that goes against our our secondary goal
primary goal
just making context switches slower
um it turns out that I made them 14
faster you can ask me in the questions
how
uh and uh even with the mitigation
kicked in contact switches are still 5.6
faster so that's nice
all right so you didn't come here to
hear me waffle about cescals you came to
see me break things so
um
right well this is the feedback I got
and um after a little bit of a frantic
slurry of trying to figure out what was
going wrong this is what I figured out
what my macro was doing if you
um don't like debugging macros then uh
good
um so so I my macro spits out these
three symbols
um just in various stages of trying to
unpack this args parameter
and uh
you don't need to see all of them
because uh right here you can probably
see the problem
if you're facilitating a 32-bit calling
convention by the way you should be able
to run 32-bit programs on a 64-bit
machine there needs to be compatibility
um
if you're running 32-bit my macro
basically
it cuts out here it sees that this is a
uh for arity Sysco and so it only emits
the first four registers so it just it
breaks calling convention and I knew
about this right I just I just didn't
know that there were any ciscals left in
the kernel after I had done all my work
that were expected to run on both 32 and
64.
and so the rest of this is all a wash
um but yeah that's some cursed macro
and uh yeah just to kind of spell it out
a little bit further
uh we only pass in four registers uh the
the last two are five and R6 should be
parts of offset
of the offset parameter but instead
they're interpreted as a random bit of
offset and around a bit of Len and so
this this is what the macro should spit
out but it doesn't
which is not good and uh yeah so the the
exact part of offset and Lan which are
represented by each each of R5 and R6 uh
basically that's sort of random but um
but not quite random that's determined
by our endian-ness which brings us to
the next bug I wrote
um and uh there's nothing really deep
here but I thought I would give back to
the community and um I would Define this
compat ARG u64 dual macro there's a
subsystem in the kernel you may be
interested to know called ASM generic
and uh azim Sans stands for assembly and
it's you know it's like generic assembly
um essentially it's bits of macros
um which uh all architectures could
benefit from there being common code and
people override their own macros and so
I overload these macros
so that everyone can have a
compatibility version of f allocate
everyone who chooses to
uh and that compat are q64 Jewel that's
going to split offset into a low portion
and a high portion and then the order of
those low portion and high portion is
going to be determined by endiness well
I got that wrong I got that flipped
around the other side and my patch
series went through six revisions and
somehow no one caught that
um
so so I really was just hitting high
schoolers here and I had in fact bricked
risk five 32-bit and power PC 32-bit at
this stage
um a prominent member of the colonel
community
made a comment about this see the
blunders which I'd made right they were
they were great
um in nature but not great enough to
topple two-haul architectures because um
no one's really running these Upstream
kernels but a certain prominent member
of the community decided to chime in
saying you know is it just me or did
this break every architecture did did
this grad just manage to destroy
everything
um and people just didn't scream because
32-bit code has become so rare
um thankfully it was it was just risk
five and powerpc so no one noticed but
um but
um
but you know every cloud has a silver
lining and a guy I got my first shout
out from Linus
so uh by means of conclusion essentially
yeah I'm a first year grad
um I go to work on the Linux kernel
which supports the vast majority of of
web servers which is kind of amazing and
it's just kind of amazing how how easily
someone can throw their hat into the
ring and work on
um essentially just infrastructure which
which everyone depends upon
and um essentially my ability to
contribute as much as I have is really
owed greatly to the rigorous uh
reviewing this is not a joke by the way
these these these tests uh the people
really did boot kernels on 32-bit and uh
it turns out they they didn't actually
corrupt enough to not boot
um which is delightful so it was one of
those kind of wicked problems
um but they certainly were corrupting
um and really anyway my ability to get
this far is very much I'm thankful for
for the testers the reviewers and the
maintainers who makes this possible
essentially what I want to say is that
persistence seems to be a sort of key
skill for kernel developers
um and that if you if you're someone who
enjoys low level programming and you're
willing to go through a mailing list to
go through multiple lines of reviews you
you really can actually contribute to
some pretty significant software
projects
and uh there's always work it seems
manually manipulating registers
and but if you have this sort of
meticulous attention to detail yeah you
can you can contribute on most things so
I hope
in hearing this talk uh this encourages
you to contribute to the projects which
interest you
um
and I hope that your first patch series
is not quite this Coast
um
any questions yep
[Applause]
so how did you speed up context
yeah yeah good one so uh should I show
some kernel code for this one that might
take a little a little while go ahead
okay uh all right I closed my editor
um okay well in in short
uh
there's been a number of reforms to
replace what is a whole bunch of
assembly so we we we use uh gnu
assembler to
write our interrupt handlers at the
exact region in virtual memory where
intra handlers are expected and then we
just have this we have a number of
trampolines which eventually call into C
code and then the rest of everything is
performed in in C code in short when I
changed calling conventions I know what
I'll do
I'll just show this slide
yeah so when I change calling
conventions
for the assist call handlers I was also
able to call it change the calling
Convention of some of these uh C
functions which facilitate these
interrupts
and that might not sound like a lot but
um in principle these Cisco handlers can
take up to six arguments so that's six
of the scratch registers uh consumed and
so what I did is I just freed up
register allocation in short so uh it it
uh decreased the the total stack
utilization and it turns out that yeah I
got more than 10 speed up which is kind
of phenomenal there actually not at all
what I expected
um yeah there's another part to that but
I've gone any other questions
hey
um sort of a two-parter like uh how much
background did you have like in kernel
development or like
um looking at this kind of low-level
development before you got stuck into
this work and secondly why was risk five
impacted along with powerpc good one
okay uh
uh yes so what background did I have I
primarily got into low-level programming
actually by just writing simulations
um I've touched a bit of Fortran but
thankfully mostly in C
um and uh so yeah just working with low
level libraries like
um MPI uh and openmp and whatnot yeah
and I just discovered that I I just like
tuning optimizing things
um which is cool uh
yeah
um and then later on I got into Colonel
Dev I took a class at Uni
um that kind of interested me to read a
whole bunch of Kernel code but didn't
have any projects immediately which I
necessarily wanted to write in the
kernel
um and so yeah I really did come in
blind at some rate in terms of the the
context of what I was looking at and um
actually this was a this is a really
good first task working on uh Cisco
handlers
in that uh essentially so the The Cisco
interfaces I imagine probably the
largest user to to core kernel
interface and so uh the user user code
needs to request resources of of all
sorts and so it should eventually touch
all subsystems in the core kernel and so
yeah I got to read quite a bit of code
that way so it's a good induction
um and then your second question how did
I break risk five as well
um let's go back to that slide because I
don't think I explained it super well
um
in short I um I influented this compat
ARG u64 Jewel macro that was already
implemented and being consumed by risk
five and um uh
I opted in powerpc to using it so we
could remove some of our own custom
Cisco handlers which were basically the
same
uh but critically I I exchanged their um
working version of that Macro for a one
which broke
uh yeah I um I misplaced a um
yeah if deaf big Indian
um turns out not all architectures have
a if def little Indian which is which is
weird and more architectures are likely
to have a a macro called defined config
big endian then config little ending
which is weird because more
architectures I imagine a little Indian
than and whatever
fun
yes indeed well I suppose if Little
Indians the default then
anyway
whatever but um in short I uh
I I realize this that oh if I have if
config little endian I'm gonna break a
whole bunch of architectures and so I
decided let's not use that so I decided
to use
if and death big end in except I missed
the N is down the end and so did my
reviewers apparently so
kind of been that important yeah
um
any more questions
so I have a fully non non-technical
question as a newcomer to the Linux
Community what if anything do you think
should be different about the Linux
development culture to get more people
like yourselves who are newcomers to
involved and working Upstream yeah I
would say that I mean the grads are
today are natives right in in at very
least GitHub
um I mean sorry I think I'm only
touching on concerns which are
I think pretty broadly felt within even
within the kernel Community who've
solidified their workflows now for for
some years but um yeah I think the key
thing is um uh the way people contribute
uh new grads will be more familiar of
course with with merge requests
um or pull requests or if you're on
GitHub
um and uh the way CI Works kernel CI is
a talk um happening in this conference
about it
um Colonel CI uh has quite a quite a
long history there are some really good
examples of it with um uh so for example
the zero day CI Which Intel does is
um has been very helpful
um but ultimately you know we built up
this Matrix right of the sub
architectures present present on on
powerpc that's really the tip of the
iceberg because there are any number of
modules and configs of any sort so
essentially the The Matrix of things
which you might want to test become
enormous
um
yeah yeah there are there are some very
good talks from previous lcas on on the
state of testing but yeah it's
fundamentally it's it's testing and how
do you contribute
um setting up your your mail client to
be able to contribute is it's kind of
yeah
um non-trivial
um and so in just decreasing that that
barrier to entry
um yeah I'd say barrier to entry yep
just thank you
[Applause]