foreign
[Music]
Peter is a senior software engineer at
Jump
um tradings core engineering division
focusing on the Linux kernel and device
driver development uh in embedded
systems today Peter will uh discuss the
changes being made to the Linux kernel
to handle Hardware interrupts on
non-traditional Hardware
thank you very much
you guys hear me okay all right
so
um like you said I wanted to give this
talk today to talk about interrupt
balancing it's something that I do kind
of outside of my day job
for fun
and there have been some changes since
the last time I talked about it in New
Zealand
particularly the year-round changes that
arm and risk five and some other
architectures have been driving so
um so first and foremost
um I am not going to attempt to
pronounce
um the names because I don't want to
offend anyone but I did want to
acknowledge
um the the native folks um in this land
and uh want to extend extend that
respect
um
and I also I would really like to
mention the everything open 2023
conference the organizers the volunteers
everyone that's working to bring this
back just like it was mentioned this
morning it's really exciting to be back
in person and actually see people in
person get to talk to people have people
throw questions at me that I don't know
how to answer
so it's really really nice to be back in
person and it's also a really big honor
for the program committee to have
invited me to come and speak so thank
you very much to all of you who made
this
go and hopefully make this go well for
the rest of the time
all right so
first and foremost why do I hope that
you're here
so hopefully that first bullet you do
know that this is the interrupt
balancing talk right so
um hopefully it is
um what is interrupt balancing why is it
important why is it something that we
have something that does this in Linux
systems
and also get some background as to why
has this only been typically an x86
Focus
right it's really been something that
has targeted those systems it really
hasn't been either a thought for other
architectures or a need and if that was
still the same case today then I
wouldn't be here talking to you about
this so
um
and so yeah understanding why some of
these other instruction set
architectures are driving changes and
kind of causing some some
issues that have to be addressed in
order to make this a more robust
subsystem
and I I figured a really good place to
start is like I mentioned at LCA 2019 I
gave a talk on what exactly is interrupt
balancing so it's called demystifying
interrupt balancing
um so I wanted to kind of recap a little
bit just so that people are kind of up
to speed as to what exactly is this
thing how does it work
um but it'll be a very condensed version
I do have a link in the end of this
presentation to that talk in case you
are interested in actually knowing a
little bit more about that so
um and then I wanted to kind of end with
what does our current roadmap look like
in the internet balancing space
um you know we do have some other things
that are still on the roadmap to finish
um I'm sure that that will change as
more and more platforms come online but
we'll get to that in a little bit
I also wanted to mention
um we'll try to encourage questions to
be at the end just so that we can get
the microphone around but if there is
something that I talk about that you
have no idea what I said I'd rather you
guys let me know so I can address it
please feel free to shout the question
out I'll try to repeat it because if you
don't know something on like slide five
and then it doesn't get explained until
slide 23 then you're lost for what the
18 slides or so
so I'd prefer not to do that
okay so brief who am I
um I'm a kernel developer who maintain
some subsystems I've been working in the
kernel for about 17 years
through various companies most notably
for the length of time in Intel my
background primarily is in the network
stack so I've worked on
the core stack protocols also the net
Dev layer implemented multi-queue
support in the kernel many years ago
worked on device drivers and
specifically more for more importantly
for this talk is the scalability of
network devices let me see if I can turn
on a little laser pointer thing there we
go
um and so this scalability push is what
was really got me into the world of irq
balancing so when we have devices that
can generate massive amounts of i o you
have lots and lots of cores in a system
it's really really bad when all of those
interrupts from all of those cues and
everything fall on one CPU right that
that kind of defeats the whole purpose
so
that was how I got into this and then my
day job when I'm not out fighting crime
with hierarchy balance I do kernel and
Driver work in the high frequency
trading space at Jump
Okay so
long time ago feels like a long time ago
since uh pre-covered but at LCA 2019 in
Christchurch we had this presentation
talked about what hierarchy balance is
how it works why it exists
um
and really what it kind of came down to
was what are the real challenges that
hierarchy balance has and I'll have a
couple more slides here in a bit I'm
trying to highlight some of the
challenges but namely it's trying to
take a bunch of disjoint information so
information about where is your card
physically plugged in like which
pneumonode is it plugged into so that
means which PCI Express slot it's
plugged into to which CPU socket it's
plugged into that's one aspect and then
the other aspect is where is your
application running which core is it on
which you know pneumonode is it on and
then you have this pesky thing called
the device driver that has an interrupt
that's tied to a queue and maybe you
have multiple cues and where are those
interrupts pinned in terms of their
affinity and is trying to take all of
this disjoint information from the
kernel that's exposed in various ways
and then trying to make a smart decision
as to how to balance that interrupt
and then you have this other pesky thing
of there are other devices in the system
that also have interrupts and they're
also getting routed to CPUs so it's it's
really a very hard problem to solve to
make everyone happy and so usually
anyone that has IR key balance running
on their system knows they're not happy
and many often times that they kill it
and then they run a script that manually
affinitizes interrupts but there are
ways that we have actually extended irq
balance to try to solve some of that
um and much of that detail is actually
in that Christchurch implementation
um
namely policy scripts where you can
actually inject some user control into
ireqbalance saying I want to balance
these interrupts myself I know how
they're supposed to be laid out and then
just do whatever you want with the other
system level interrupts so there is some
flexibility there that we talked about
um but the main focus has been on x86
right
so think multi-core systems that are
like more Enterprise Cloud big
workstations lots of i o big network
cards you know 10 25 40 100 Gig cards
um it was really found mostly in the x86
world right this is really the bread and
butter there
and this last part I want you to kind of
file this one in the back your mind
interrupt Discovery in terms of figuring
out from the system at the PCI level and
even in some non-pci level situations
has typically been through acpi right so
we can go and query you know the device
firmware through the PCI systems and we
can figure out where interrupts are how
many do we have all of that is very
standard and kind of Auto discoverable
and I noticed that my speaker notes here
are super tiny
so I'm trying to think of what's coming
up next without speaking to it before I
actually get there
okay
so let's try to recap some of those
those challenges that I kind of brought
up so I already mentioned that they um
that there's all these unrelated sources
so one is see State and P state so
different processor state or different
socket States in terms of power right
one of the things that our key balance
tries to take into account is did a
certain CPU core drop into a like a
deeper sea state it wants to be turned
off for whatever reason
you don't want to route and interrupt to
that core because you're going to
immediately wake it up and
if you're also concerned about
performance then you're going to have a
latency hit waiting for that core to
spin back up and blah blah blah so
so we try to get that information out of
um out of sisfs and some of the CPU
Governor stuff of the scaling bits
CPU load right this is one of those like
how do you measure CPU load is it you
know wait time is it user time system
time so we take some aspects of CPU load
how busy is this CPU
um and also part of that is how many
interrupts are actually firing on that
um on that particular core
ah I just figured out how to make this
bigger excellent
um so cash locality this is something
else that it tries to take into account
right so if I am trying to balance out
to a core
and if I'm on a system that has a
certain hierarchy say like an L1 is you
know per CPU and then an L2 is maybe
shared across you know very tight cores
and then we have an LLC or an L3 cache
that I'm trying to land these things
kind of in that same domain so I have to
take that topology into account as well
pneuma locality so if you are on a
multi-socket system
you know you want to make sure that the
interrupts for say
um like a Nick plugged into network
interface card plugged into a slot in
pneuma node zero you don't want to Route
all of its interrupts to pneuma node one
unless you have a really really darn
good reason to do that
um now so there's a new one we'll talk
about it a little bit more later this is
actually since 2019 there's a new
thermal
integration where we take some thermal
characteristics and also make decisions
based on do we want to route an
interrupt to a CPU that's overheating
the answer to that is no we don't want
to do that
um so we try to steer interrupts towards
our way from CPUs based on all of these
above sources of information now the
problem is uh one thing that I don't
have up here and it's still an issue and
I kind of alluded to it on one of the
previous slides is application Level
stuff we have no insight into what the
scheduler is doing how it's dropping
applications down on CPU cores we have
no idea what they're doing so this is
really like a best effort at the
platform level to make it look like
everything is happy and interrupts are
firing in a spread out way
and if anyone like I had mentioned and I
heard some laughs so I have a feeling
that people have run into this that
doesn't translate into that your
applications actually work better when
this is in in play without any knowledge
about what's going on
okay so I promised some crudely drawn
pictures
um so consider this uh let's see this
you know two CPU systems so two two
pneumonodes will say our first four
physical CPUs are in one socket the
other are in the other socket we have
two DDR controllers we have an i o
controller
we'll just call it PCI Express for now
um
that we have a like a high speed you
know super high-tech card down here
um has a physical PCI connection and
then I'm trying to help illustrate like
this is like worst case scenario kind of
thing and this is what
um is like the bane of irq balance right
so I've got a Nick
um I have a driver running on it in the
kernel and let's say the kernel driver
happens to allocate memory for its data
structures and its buffers its dma
regions off of the other pneuma node at
this point we've already lost like in
terms of the performance battle because
every memory access is going to go
across either the qpi or the infinity
Fabric and that just sucks
so let's make it suck even more so now
I've got an application maybe running up
on what CPU one or no I'm sorry in cpu2
and maybe the interrupt for this is
actually routed to CPU one and so the
application is actually going to be
woken up by CPU one's interrupt
um
the exit of the interrupt and then if
the application has its memory mallicked
off of pneuma node zeros so you can see
how this is starting to get very very
bad right this this is bad don't don't
do this
um
so ideally what we're trying to do with
irq balance and this is again where we
don't have as much insight as we need to
do some of these things but this is
ideally what we want we have an
application running on a CPU let's say
cpu5
I have a Nick that is on that same
pneumonode it has its memory allocated
from that pneuma node I have an
application that's also running its
buffers on that pneuma node this is all
trying to keep memory bandwidth from
having to burn qpi or Infinity fabric so
Intel AMD
um resources going between the CPUs so
your latency on memory accesses is lower
and then I want that same interrupt that
is servicing the data coming out of the
Nic to go to that same CPU that that
application is already running on this
is the ideal situation you wanted a very
vertical pillar
because this also maintains cache
locality right so that applications
buffers are sitting in cpu5's Cache you
want to wake up that application and
then not have to immediately take a
cache Miss block you know take the
exception page fault swap it in and then
at that point again if you're worried
about performance then you've already
lost
right so this sound reasonable
everyone's still kind of awake all right
good
I'm really glad that this wasn't uh like
after lunch right after because that's
usually the food coma hits so
it's I know this is like edge of the
seat topics here so all right so um what
are some other things uh that are worth
noting
um irq balance uh only balances Hardware
interrupts uh not software interrupts so
this is something that I think some
people forget and this was actually a
question that Paul McKinney who just
spoke prior to this in a different room
um Paul actually asked me this in
Christchurch and I thought about it I
was like ah crap you're right
um but our key balance does influence
how soft irqs can run right so one of
the main uses for like heavy duty
software interrupt for data processing
is in the networking context or the
networking core
so all of networking processing these
days is done out of software EQ context
basically it's polling it's known as
nappy how many of you have heard of
nappy
okay good good so it's still known as
the new API even though it's what 18
years old
um but it's effectively we take a
hardware interrupt and this is where our
key balance comes into play we take a
hardware interrupt on a particular
particular CPU
and the nappy model is to not do any
work in the Nic in processing uh
incoming cues incoming data and we
immediately disable the interrupt on
that CPU and we enter software soft irq
polling with nappy and then the kernel
goes ahead and decides when to start
Doling out work well the CPU that we
actually go into software EQ context is
the CPU that that software soft irq
interrupt will fire out of the kernel so
if we take the interrupt on cpu5 and we
go into nappy context the software
accuse fire on cpu5
but at that point it's now out of irq
balance's ability to migrate it to a
different CPU the only way is nappy to
not have any work to do falls back in
Hardware interrupt context we move
things away to a different CPU and then
we start the process over again
um
yeah so I already already mentioned that
hierarchy balance isn't application
aware to prevent bad bad decisions from
being made
um
it tries to balance the system right so
we're trying to balance
um between CPU load we're trying to
balance between you know one one CPU in
particular isn't taking you know 10x
um the number of interrupts from other
CPUs
so that's really its goal
like I said some applications
performance can still suffer and more
often than not they do
um
and then I had mentioned these policy
scripts this is a way that you can feed
a script to hierarchy balance Has anyone
used policy scripts in here
even know what they are
it's actually not a terrible option to
irq balance but it is so I'm sure that
everyone has seen the venerable set Iraq
affinity.sh script out there that you
kill hierarchy balance then you run said
irq affinity and then set all of your
interrupts well this is basically taking
that logic for said IQ Affinity allowing
you to generate a script
in a certain format that you hand Iraq
balance and say apply this when you're
actually doing these other things unless
there are some certain things like
thermal events or like CPU hot plug then
that you can't balance onto a socket
that doesn't exist anymore
so I would encourage people to
uh take a peek at that
um
now this is one other point that I want
to make
and this is actually something I I have
contended and advocated for this for a
number of years this last point of
letting drivers control their Iraq
Affinity in fact I had a patch set many
many years ago
that was met with open arms and
wonderful conversation
um that was a pun
um
it was kind of dead in the water and um
this is actually if you have ever seen
in looking at how hierarchy balance
works with the irq Affinity hint exposed
in in proc that was the alternative to
this the the Upstream maintainers said
no you can't have a driver go ahead and
set its own irq Affinity we don't want
to expose that kind of policy level
stuff in the kernel
um I Heard key balance should do it so
we went through and exposed a different
thing in the interrupt core up to user
space allowing a driver to say I want
this Affinity to be this give it a CPU
map and then Iraq balance decides to
either use it or throw it away
um I thought about this a long time and
I always contended that drivers had a
better idea of where am I allocating my
memory uh from what pneumonodes I should
have a better you know I in terms of the
driver should have a better view on the
world as to where I want these things to
land
um
I don't think that's the case I still
think I think these days that whether we
balance in the driver or from hierarchy
balance we're going to get the same
result
um driver can still do a really good job
of managing its own interrupts but that
can totally screw everything else on the
system so
I'll I'll go ahead and uh
admit that that was uh probably not the
right direction so
okay
so what you're probably here to talk
hear about is what are these uh to ices
and what is their role in kind of
driving some of the RQ balance changes
so arm and risk five
obviously arm has been around for quite
quite a long time risk five
as well but it has certainly heated up
quite a bit on the last few years in the
Linux space itself I've also got some
links I'll talk about them in a bit on
some talks from Drew festini who's very
very active he's one of the risk 5
ambassadors he's a good friend of mine
I've linked to a lot of the talks on
risk five and Linux in general
um
so
when we talk about some arm and risk 5
socs they do have similar challenges
with hierarchy balance in terms of
um they themselves have similar
challenges with hierarchy balance so I
wanted to make sure that
um I call this out just for the sake of
less complexity in the slides
that each Isa has similar issues but
they're presented differently and so
specifically around interrupts the way
that an arm system actually exposes
interrupts through like the ppis gispi
type interrupts is very differently
presented than the MCAS from the risk 5
side but the net effect is the same
right so so they present things to the
kernel the kernel implements them down
in the system level and you know how it
discovers things through device tree and
then irq balance then has the same view
of the world and it breaks in the same
way on both of them
surprises prize
um so just uh you know if I'm saying
like arm I mean our men risk five right
okay so why haven't we seen these
problems up until recently well
hobby boards have had typically really
really wimpy i o right they didn't need
to balance interrupts well one hobby
boards either had socs that might have
had one CPU which I didn't call this out
in this presentation but recently we did
have a patch come into hierarchy balance
to support hierarchy balance on single
CPU systems
which means more to detect that we're on
a single CPU system and don't decide to
run
but so really
um The Hobby boards you know the
Raspberry Pi's
um you know the Beagle bones of the
world really just didn't have anything
compelling to worry about the interrupts
from generating generating enough load
that they actually caused a problem
you know typically
um these things had like a one gigabit
Nick one gigabit Ethernet Nick
one gigabit Knicks on those like even
the real text you know very very um
inexpensive ones still had enough
Hardware offloads in the chip that they
could do things like TCP segmentation
offload or checksum offload and it gave
it plenty of offload capabilities for
one single CPU core for like an armor
risk five to be able to drive gigabit
um
to a lot of these one gigabit nics were
also hung off of like the USB bus
internally so there's no way you're ever
going to get to one gigabit on these
things anyways
um so it just wasn't a problem
and then really the other things that
people bought these boards for was all
of these other you know gpios the I
scored C the Spy interfaces and these
either don't have an interrupt or
you don't care about this interrupt it's
going to fire and you can get to it in
milliseconds later and it's not going to
be a big deal
whereas like a 40 gig or 100 Gig Nick
you got to get there in like you know a
few microseconds at you know the latest
before you overrun packet buffers
okay so no real need to to balance
interrupts well enter
today why why are we here
um these SOC boards are getting more
interesting
um uh some of them there was a recent
one
um It's actually an active discussion
right now
um on the Iraqi balance mailing lists on
some wireless routers that have some 10
gig ports
um well 10 gig definitely needs more
than one CPU even in x86 to drive at
least on the receive side
so um we're now getting into where we
actually have to spread some load
um new Wi-Fi protocols Wi-Fi five Wi-Fi
six are definitely causing
um issues
nvme support so high speed disk traffic
is coming in
we also have laptops right so people
that have like an M1 or an M2 MacBook in
the room or somewhere here that are
running Linux you know Linus famously
runs Linux on a on a MacBook with an arm
chip right that has an nvme disk that
has Wi-Fi six that has the ability
through Thunderbolt to run 40 gigabits
so we are starting to see some things
that are demanding that kind of um that
kind of load and then the other aspect
is the server side right so the server
side has always been very disjoint
um but with people like ampere you know
cavem with all the thunderx parts
um
you know Amazon with their graviton
ampere with ultra just looked up Ultra
has 128 cores per socket that's that's
pretty dense and you have a peripheral
that's trying to load up all of those if
you have you know thousands of network
cues attached to an ultra with 128 cores
and you dropped all the interrupts onto
CPU zero that really sucks
right so this is something that we now
have to take pretty seriously
okay
okay so the architecture differences can
be subtle but they're enough to break
things right
um the interrupt types are different
from x86 right so we have these ppis for
example the private peripheral
interrupts
um they get presented to Linux
differently than say some of the other
like system level interrupts in x86 and
surprise surprise they don't get handled
by req balance at all or we handle them
improperly
um
so this is another part and this is
where I said like file away the acpi
thing in your head
um device tree and how device tree
actually presents things and how things
can get named and how they actually get
sorted into the tree
can cause a problem with layout or
naming naming has actually been the
bigger problem
or some other bsp difference in how The
Blob is actually presented to the kernel
device
um
Discovery is much different between arm
with arm and risk 5 than it is in x86
and so in terms of the like the
topologies and the trees and what is
exposed to us to our key balance from
the kernel is very different
um we also could have different platform
drivers that just don't exist
um right there's the x86 platform
drivers there's the arm platform drivers
they have their own class of stuff
so much more interesting stuff what
sorts of issues and I'm just watching
the time here
okay so these are actual issues that
came up
um over the last few years it's just a
sample but um handling platform
interrupts so so the architecture
differences can be subtle but they can
be painful and so this was the first
um kind of I'll say like important fix
um
so this commit came in
um
so hung actually works for Huawei and
I'll mention a number of times here in
this presentation actually
so basically what happened was the the
PPI interrupts these private peripheral
interrupts and arm get presented to proc
interrupts proc interrupts is the
exposure of all of the interrupts and
how they are laid out across the CPUs
that is one source of where we scrape
information with irq balance to see what
what is the kernel doing with interrupts
where are they at
um and so what what really ended up
happening was so here on x86 and kind of
trimmed down we have you know these
system level interrupts that are for CPU
like local timer irq work
etc etc tlb shootdowns these are
interrupts that hierarchy balance
ignores because these are per CPU you
don't screw with them they they just
kind of hang out there
well in arm we have similar ones these
ipis
um in a processor interrupts these are
ppis these are also things that we don't
need to mess with the Affinity well irq
balance grabbed them threw them into the
irq balancing database and attempted to
do stuff with it and that was bad so
um you know seems like a very simple
thing but it was something that we had
never run into before because no one
cared and so someone all of a sudden had
a board that they cared and it became an
issue so we got a fix that was great
okay
interrupt classification this is one
that I don't think is going to be put to
bed
um
yet I think we're going to see more
manifestations of this
as more and more systems come online
but one of the things that irq balance
tries to do is as it's going through and
and parsing all of the interrupt
information and how interrupts are
actually presented by devices which
there is no standard right so if I do
request error queue in the kernel and I
give it an interrupt name
I could name it anything I want more or
less
but there are certain things that
certain devices export certain
interrupts in certain ways that
hierarchy balance traditionally on x86
could figure out what exactly is this
device and what kind of interrupt is it
so the types
um
you know so we have type Legacy we have
MSI msix
gigabit Ethernet
yeah we didn't we didn't plan ahead on
that one
um eth scuzzy which also covers ATA
Serial ATA nvme so again we didn't
really plan ahead on that one uh virtual
event so if it's like something
interrupt firing off of like a virtual
function routed to maybe to a virtual
machine and then the venerable irq other
we couldn't detect what it was
well see maybe you can see where this is
going
um
so we had another
um actually this was
yes this was also from Huawei
we had a fix to guess our arm irq hints
and
um so what happened was we were going
through and we were trying to match
something we matched it once we went
through the loop again said we still
don't know what this is and then it got
thrown into the irq other and so it was
getting handled like a system level
interrupt and not an Ethernet level
interrupt
um so this was a this was becoming a
problem
and
then we had another one that was very
similar and this one came in from TI
um and so again I think that you might
be sensing a trend of companies that are
actually like more and more interested
in RM socs and they're getting bigger
um so this one was basically another one
where we were parsing some of the things
and we weren't guessing the right class
based on how they were being presented
so we got some fixes in for that
which took us to a third one
and this really goes to like the device
tree level thing we got foiled by a
string
um
and this is not going to be limited to
Armory risk five it just happened to be
that we didn't run into this before in
x86 because typically it's acpi based
there's now device tree support and
there's more
momentum behind that for supporting the
vice Tree on x86 so I suspect that we
will see something like this but
something really simple and stupid as to
the way that we parsed it out there was
an extra space and so when irq balance
went looking for this platform driver
couldn't find it because there was a
space in the name
or we parsed it out that way so
oops
and then
um this is the last one I'll share and
this is a simple CI CD issue where if we
remember two slides ago
um there was a fix to
fix the guess check out your Q Affinity
uh and uh I'm sorry get Iraq class and
try to classify the interrupts the fix
went in and we happily merged it and
then we shortly after had a fix that
says hey you broke everything else
that's non-arm it won't even compile
so this really highlights uh like a
platform issue right on on the RQ
balance side this is now maintained by
three people
in our spare time we don't have formal
CI CD to do you know pipeline builds and
testing on multiple architectures this
is a gap this is something that we need
to address
so why all the issues all of a sudden
right so
arguably hierarchy balance was kind of
in maintenance mode we were not really
doing a lot with it it just worked I
mean for varying degrees of how you want
to say it works it worked
um but uh really the arm stuff that was
coming in arm is getting bigger
um
and this is something that I'll make the
statement I'm sure that I'll get some
some hate mail from it but um our arm
has struggled right over the years with
a fractured ecosystem
um
so slightly different CPU designs you
know vendor a wants to put in you know
floating Point vendor B wants to put in
like this instruction vendor C wants to
not put that instruction in it's put an
immense amount of pressure on the tool
chains right so the compilers if I'm
using like a TI you know beaglebone blah
blah blah oh did you get GCC version
blah blah blah blah Dash blah blah blah
blah oh no you have the not blah blah
blah you have the other blah then yeah
that's not going to work so that's been
something that has really hurt arm over
the years
and and there's been a lot of
improvements to fix this right I'm not
saying that this is an unsolved issue
um but also the bsp support right so you
get a bsp to support a certain board
and everyone has a variant of a board
and then you have a different bsp blog
for that and the kernel has to deal with
that that that's been a maintenance
issue right
um so hobby boards have really you know
kind of made this a challenging issue
risk five is not immune to this
um
yeah so same vendor different cores
um look at that risk five not immune to
this I should remember what I wrote on
the slides
so why all of a sudden did this become
an issue um so server CPUs so ampere
came in ampere hired a huge open source
team they're a great group of folks um I
know quite a few of them and one of
their first jobs that they really wanted
to focus on was cleaning up the
ecosystem for arm-based CPUs
very very big task I think they've done
a fantastic job helping like Shepherd
that along
arm desktop chips have become more
powerful and
um
yeah so I mentioned this active thread
around open wrt this is actually the
thing with the um got it thank you this
is the thing with the 10 gig ports and
Wi-Fi six issues
um
so what really happened is all of these
things started showing up and then they
started running into problems with
hierarchy balance because they kind of
needed it to help the system level stuff
and they found out everything's broken
that's why this all of a sudden caused
some renewed interest
so
um that's a really great thing so one of
the really nice things to come out of
this was Pung Zhao from Huawei who
authored a number of those patches that
I that I showed on previous slides kind
of came in and said hey between Neil
Horman and myself who come maintained
this said what do you think about having
me as well we're like that sounds great
um
yeah
um he's been very very active
it's been been a really nice addition
because then this is also uh some really
strong arm expertise coming in and
helping out with the direction of the of
the project
um and what ended up happening is as
more and more people started like
messing with this and they're like hey
there's this hierarchy balance UI thing
too and
um hey I'll fire it up and holy crap it
doesn't work brakes so we got some
patches so as people were starting to
come into this from the arm and the risk
five ecosystems they started realizing
that there was a lot of stuff that was
wrong and so all of a sudden this
renewed interest started and people
started looking even deeper
um we also added improved support for
PCI hot plug so if you hot plug a device
you want to remove the interrupts from
The Balancing database that was
something that wasn't there before
thermal events so this actually came in
from Intel
um
so they actually hooked to Thermal D to
basically feed information to ireq
balance saying the CPU went offline
because or the CPU is having problems
because of thermal issues please route
interrupts away
um so the limitation here and this is a
call to action is that this is only
available on x8664 right now but you
know not to fall back on Paul's thing
before uh today patches are welcome
um but this is I think a good place that
we could probably improve on in the arm
and risk 5 space
okay so looking to the Future what does
the road map look like
um RQ balance I think still has a lot of
work to better
have better flexibility for device tree
defined resources
right so this is this is something that
has been baked into the RT balance core
is we can go through the acpi mechanisms
that are in the kernel having something
that we can actually hook into device
tree and be a little bit more aware of
how things are laid out
because one thing that we have not run
into yet but I do fear this is msix
resources so these are things that are
typically dynamically allocated on a on
a Nick or or an nvme drive on some kind
of peripheral these things get Auto
discovered and then we try to bring them
up well these things are actually tied
to a device through a device tree should
we should we be balancing them I don't
know so we'll I guess time will tell
um
yeah the the
um risk 5 platform interrupts that are
similar to the arm PPI SPI SGI things
you know we need to comprehend that for
risk five
cxl devices so um if any of you are
familiar with compute express link it's
a bus architecture thing that's
basically on top of physical PCI Express
um just started started showing up in
Sapphire Rapids and AMD Genoa
um
there's a whole class of things and I
kind of also do a lot of stuff with the
cxl Consortium so if you have no idea
what type 2 type 1 type 2 type 3 devices
in cxl are see me outside I'll be happy
to give you a brain dump on cxl but it's
basically
um like an i o memory and cash level
protocol we have no idea how to deal
with that
and the arm server proliferation
so Linux development I mentioned that
Drew fustini has a lot of talks on the
state of risk five they're just marching
along really fast I have a feeling that
we're going to start seeing some like
public server class risk 5 CPUs here not
too long it's probably going to put some
pressure on hierarchy balance
um
so here's our GitHub project page please
go ahead and feel free to poke around
there that's where the code lives there
is an old PR for CI CD that just kind of
rotted
um you know we need to kind of resurrect
that and get it going please feel free
to contribute always happy to
bring in some people
um
so recap if you fell asleep
balancing is still hard it's hard to get
it right
um but arm and risk five
uh getting into the into the mix is
really helping us out
um we're back to pretty much active
development but we have a lot of work to
do
um and so just
I promise there are some some links here
in the presentation they'll be posted
later but uh yeah go ahead and put me on
the spot and see if I can not answer
something that you guys have appreciate
your time
does anyone have any questions
oh boy here we go
um do you talk to the device tree people
much about putting things in the device
tree to help you know what sort of
interrupts are do we talk to the device
tree people no that they love
standardizing things these days so I'm
sure they'd love some um some more
standardization you know if it had
helped um it could be absolutely uh used
to be a lot better at being useful no
that's a that's a pretty thing that's a
great idea I you know we
haven't really thought about that but
that's a really great idea especially
with x86 now kind of getting more into
the formal device tree area
so the question is if the x86 device
tree stuff is server class chips the
short answer is yes
um really for like Appliance level stuff
so red hat is looking to add device tree
support for x86 into relate into
pre-boot and all that stuff
yeah yeah device tree did exist for like
uh what's the The Minnow board
um
x86 based hobby board and um
yeah yeah AMD yep yep
other other questions
I will be here all day today I
unfortunately have to go back to the
States tomorrow so if you need me grab
me
um wandering around or contact me
through
through whatever contact info is
actually exposed to the site so
all right well thank you for your time
enjoy the rest of the conference
[Applause]