MODERATOR: Hello, and welcome back to the Platypus Hall. Alan Green is next.
He is a software engineer and in his spare time Alan likes to pay with sill
scope and do soldering. >> Thank you, Kaitlyn.
I am coming from a leafy side of the hills. I would like to knowledge the people of the land and
the traditional custodians the land. I am excited to present the CFU-playground. It builds
machine learning accelerators. Python is key to making it all work and making it fun. Myself,
my co-author and a number of others have been building the CFU-playground for the past year.
It uses FPGA and ML which is machine learning.
You can think of an FPGA as hardware but with software defined configuration.
We will examine what machine learning is and how models are run specifically in the context
of small computers and then look at FPGA and how they can accelerate arithmetic. We will
take a high level overview of RISC-V and how we extend them with function units.
We will tie it back together with an example of accelerating ML on FPGAs.
I have tried to pitch the talk so anyone with experience gets talk
but if you find yourself lost, don't worry. A new topic will be coming along.
Running machine learning models on FPGAs is unusual. We had a project where it was necessary
to run ML models at low power, quickly and using a completely -- completely Open Source stack.
CPUs are power hungry for applications. Since CPUs are too large, we looked at microcontrollers with
embedded digital signal processors or DSP. Many met the power budget but all the ones we found had
a closed source component. A secret C++ compiler, a model preprocessor or a block to link against
and were not suitable. While a micro controller without a DSP can be low power and open it can't
also at the same time be fast enough. We thought we might build our own ML engine out of an FPGA.
My co-author suggested using custom function units as a way to reduce complexity. The idea
is we would implement our own low power micro controller power design and add custom function
units to make the ML evaluation fast while doing all this with completely open source toolchain.
Once we decided to build our own ML accelerators with custom function units it became clear we
needed a way to experiment with the CFU and that's how the CFU-playground came about.
This let's programmers build ML accelerators that are fast, low power and open source
on FPGAs and as a bonus in Python.
The CFU-playground CFU-playground runs on real hardware. This is one board we use.
We use this because it is reasonably obtainable,
capable and has an open source chain and a USB port to communicate with there host computer.
What goes on the FPGA? The FPGA is a blank slate why -- we start. We know we need a CPU
that needs memory so we have memory too. To communicate with the host,
we at a serial periphery bort. The exact kind of CFU we build depends are largely on the model we
want to accelerate. Did I mention open source? I will out -- call out the Python framework we use.
Running which uses the model. Let's look at training first. In training, one takes a bunch of
labeled examples which is fancy way of saying when you see this example, keep the label as an answer.
You give it examples for all different kinds of output you expect. For something like image
recognition, you would need many labeled examples. The more the better. After a lot of computation,
maybe hundreds or thousands of hours, we have a model that knows it is hot dogs.
We can take this model and use it I will run the model and get the result.
What does it mean to run the model and how does it actually work?
The model specifies a series of operations. To run the model, we set the input data, then run
each of the operations one after the other to get a result. Data in, operations, result out -- easy!
An important type of operation is 2D convolution.
Let's look at the Tensorflow source code and see how these two deconvolutions work.
The code is in C++. It is about 30 lines. It will look complicated but in less than
two minutes you will understand everything we need to know about this code. The first thing
to notice are the seven levels deep nested loops which we can illustrate with a friendly rainbow.
Let's focus on the innermost loop since that is where the code spends almost all of its time.
The inner loop does three things. It fetches something called import from memory,
then fetches something called filter, and then performs a simple calculation and adds
an off set to the input, multiplies and adds a result to an accumilator.
The code spends a lot of time multiplying and adding. Let's look at the accumilator.
It is reset before the loop, after the loop, there is a complicated process to compress
from large 32 bit to 8 bit, and finally the output data is stored in the output data array.
The rest of the code is concerned with making sure that we get the write indices
into the input data, filter data and output data arrays at the right time.
The most important thing to learn from this is that an ML evaluation is mostly a matter
of running this one-line calculation millions of times with the right inputs.
That was a whirlwind tour of machine learning, 2D accumulation and
next session we will look at FPGAs and how they are configured.
Before this presentation, many of you probably already know that this is an acronym chip and
stands for field programable gate array. FPGAs are relatively slow by hardware standards
but incredibly fast by software standards. We build up the gateway
in Python with the nMigen library. You can use verify log but that is not Python.
The FPGA contains many different kinds of logic building blocks. Here are three that are
important. The look' table, the flip-flop and the multiplier. The look up table can be programmed
to calculate any function that has up to four 1-bit inputs and one 1-bit output. For example,
if we program it to always output 0 unless all four inputs are 1 then we have an and function.
If it is programmed to output one unless all inputs are zero then we have an or function.
Arbitarily functions of 4 bit can be programmed. We can use a large number of boots. They function
continuousally and any change will be reflected in the output after a just a few nanoseconds.
To sync the outputs we use flip-flops and their job is to remember a single bit.
If enabled, it takes the input and sends it to the output on the next clock cycle.
If it is not enabled it ignores the input and shows the same output value. A single bit can
be useful but to store multi-bit numbers we can gain flip-flops together into registers.
And finally, the third kind of logic block mentioned is the
multiplier and it does exactly what you would expect -- it multiplies numbers.
The question is given large number of these types of logical building blocks, how would
we use them to implement a single function that did an addition and a multiplication?
The picture on the right shows one way to do it with lots and multipliers.
A and B are inputs, A is added to a constant offset and A and B are multiplied together.
The adders are made of look up tables and we use the adder block because multiplication is harder.
We express this like this. The null function begins by creating a 16-bit number object
and assigning it to R. The n.d.com plus equal notation generates the domain
meaning it happens continuously without flip-flops to sync.
Let's make a bit more Python code here. Our next step is to have four add multipliers.
FPGAs are great at having many copies of the same logic running at the same time.
Our configuration language, Python, is great at making many copies of the same object.
On the right, we have inputs capital A and B which are 32 bits each.
We slice up the 32 inputs into 4 values and for each of the four slices we perform the same
additional multiplication and add everything together to get a single result number.
Looking at the Python code on the left, we have the null function from the previous slide
and added a new function which is null 4 because it does four of them.
This uses generation to slice 32 umput kitchen -- input and use one for each
function. Then it specifies the results be added together.
Looking at -- at this point, we think it might be convenient to add the result of null 4 to the
acomilator, we make a register and connect on the right so the result is added to the existing value
of the accumulator. To do this in N region we declare a 32 bit value named ac and use n.d.sync.
The value on the next clock cycle is whatever value there is now plus the result of the nul4.
That's quite a bit. I hope you see that nMigen is a relatively friendly way for Python programmers
to specify FPGA configuration. Back to the outline, we will move on from FPGAs to how a
RISC-V processor can be extended with custom unit function. Here is the architecture diagram again.
We are interested in the interface between the CPU and custom function unit.
To illustrate how the interface works we will begin with a standard RISC-V add instruction.
This is the assembly instruction in the form a programmer would use
as the comment hopefully points out it adds register 4 to 7 and puts the result
into register 24. Of course, the CPU doesn't read the instruction or the comment. It only
understands the 32 bit instruction word. Let's look at those 32 bits.
The first thing the RISC-V CPU looks at is the right-most seven bits forming the up code.
In this case, 011, 0011 means this instruction is going to do something
with the registers and the arithmetic logic unit or ALU. It tells the code
and CPU how it should interpret the rest of the 32 bit instruction word.
The RS1 and 2 fields specify which registers are sources or input that need to be routed
to the ALU. Register 7 and 4 are read from the register file in this case.
Func7 and 3 tell the ALU what operation to perform. They tell the ALU to add in this case.
Rd specifies the registration and that is sent to 24.
Two values coming from the register file, go to the ALU that performs the calculation
and the result is placed back into the register file. Let's look at how the
custom function unit is used. The RISC-V instruction set has two reserved opcodes.
We use one to tell the CPU to process data using the CFU.
Once the CPU sees the up code it understand how to interpret the remainder of the instruction.
Rs1 and 2 specify the source registers and the CPU routes them to the CFU instead of the ALU.
Funct7 and 3 tell the operation and rd specifies where the output should be sent.
With the CFU instruction, a program can tell there is five CPU to take two 32 bits from registers,
send them to processing and put a 32 bit result back into the register.
To allow the CFU to be used from a C program, the CFU-playground provides a macro that translates
into a single CFU machine instruction. The example at the bottom of the slide shows C code and the
assembler it is translated into. As you can see the translation is pretty much one-to-one.
So that was an overview of how a CFU works and how to use it from a C program.
Now we are going to put it all together and accelerate Tensorflow like 2D convolutions.
Here is our system on chip again. To accelerate 2D convolutions
we will need to build a custom function unit that can assist with the 2D convolution
and modify the Tensorflow implementation of 2D convolution to take advantage of the CFU.
Here is the schematic for the gateway we defined 10 minutes ago. It needs a couple tweaks.
Having a fixed off set might work but it is a little crude.
We will replace that with a register and some way to set the content. We will add that reset too.
This is enough to be useful. Now, here is the image in code to implement all of this.
There is a little more broiler plate that surrounds this code
but I omitted it to get it into one slide. Here are the important parts.
We begin by defining off set and ak as 9 and 32 bit sign numbers.
Next are mul and mul4 functions working the same way as before.
The next sends a done signal after the status is received or the functions we
perform take a single cycle. This line ensures the CFU always outputs the accumulator value.
With the m.f means whenever. Whenever this is true, the contained logic sen abled, otherwise
it is disabled. Whenever start is received and funct7 is 0, the first input value in 0 will
be set into the offset register. If funct7 is 1 while status is true the accumulator is set to 0.
If funct is 2 while status is true we perform it on the values and add it to the accumulator.
That's all the gateway we need. The Python image and syntax makes it pretty compact.
So remember this code. We are going to use a new gate way to accelerate the inner loop.
The first thing to note is the original code that retrieves the values and this allows for
data and arrays. At the top of the file, we hash to find a macro that wraps the CFU macro
and use it here. These are the main changes we need to off set and reset the cume -- accumulator.
This shows how it works for the example model.
The overall model evaluation time went from 147 million cycles to 8 million. If we look
at the 2D convolution we managed from 95 million cycles to 33 million cycles. Not bad for 20 lines
of Python and another 15-20 lines or C++. In this talk, we looked at how tiny ML models run.
They are doing millions of adds and multipliers. We looked at what FPGAs are and how to build
gateways that add those. We looked at how a CPU interacts with the custom function unit
and put it all together to build an accelerator that makes running an ML model faster.
There are plenty of aspects of the CFU-playground we didn't cover.
We put a lot of effort into the tutorials and guides for beginners, people who do them
tell them they are helpful and fun, so please, try them out and raise bugs to give feedback.
Unit testing gateway is more fun than debugging on hardware.
I am a bit sad I didn't have time to talk about the iterative baby steps approach to
building gateway that is easier and more fun than trying to build and integrate giant components.
There are many people to thank their part in making CFU play grund a reality and I would
like to call out a couple. First, Tim Callahan put together the structure of the playground
so the rest of us can have fun writing. I would like to call out the work of Rachel and Joey
who started off knowing very little about FPGAs and implemented in the end whole accelerators.
Joey made a step-by-step for getting started and I used that heavily in this talk.
If you would like to learn more about FPGAs or understanding how ML models are evaluated
or building ML accelerators with open source tools here is how to get started.
The tutorial on read the docs the best. The code is on Gubernatorial -- GitHub.
Feel free reach out. >> We have great questions we can get straight
into them. Fantastic. What are you building? >> We are building a thing which may or may not
work. If it works, it may appear in a Chromebook. >> Great. Next up is is the support for non- -- or
the interesting Chinese FPGA and the lattice
targeting - urban source toolchain? >> We really want our product to be
open source. We are going out of our way to support open source toolchains. The
lattice toolchain is great. These are all good. >> Thanks a lot. Next question was as a person
with curiosity but no hardware knowledge what's a good entry point into edge or tiny ML? Projects or
references? Thanks for the beautiful explanations. >> The CFU-playground is a great way to get into
it. [Laughter] The Tensorflow for micro controller site has pretty good instructions if you want to
get started. I recommend a number of micro controller products and that's not a bad way
to get into it. The CFU-playground if you want to get ahold of like an FPGA development board,
we make it super easy to get into it and start messing around with the C code.
>> And I think some conferences give FPGA boards as a particular badge for the
conference. That can often be quite good. >> Yeah, if you have a development board,
or a conference badge you would like to see the CFU-playground on,
please get in touch. That would be fun. >> You mentioned that you were are using
a reserved RISC-V is this space reserved for custom lock codes? Or future use by standardized
opcodes? >> Good question.
No, the RISC-V specification reserves these two opcodes for implementers. They are never going
to be used for RISC-V instructions. There are other reserve spaces for RISC-V instructions.
But these are specifically so that implementers like the CFU-playground can try out new things.
It is one of the really great things about RISC-V. You have this giant RISC-V GCC toolchain that
already works and you can add just the little bit you need to make it better and do what you want.
>> One more question. If there are more you might have time to type it in. Is it easy to write
functional tests for components in nMigen? >> Yeah, advanced with nMigen.
>> I had to ask. >> Yeah, you would have to. Do
you run it through Python or hardware simulators? >> We did both. We definitely find the most bugs
with Python unit tests. Python unit tests import unit tests and subclass unit tests
and then nMigen has a simulator and you can poke at the simulator and check it is giving you
the results back that you want. The test cases are really fun. I don't
know if any other environment that let's you write some gateway and test it and fix
it and run it again in less than 30 seconds. >> That's pretty good. We have a flip side.
As a person with lots of hardware experience but no ML experience where is a good
starting point for that? >> I would have to think about
that. Please come and talk to me afterwards. >> We have the hallway track for that.
One last question is why do you need four times mul times 8?
>> Right. This all comes down to the technology your ML model is using. There is a
lot of research being put into 8-bit models at the moment and the good thing about 8-bit models
is you can move the data around four times as fast because there is four times less of it.
That's why we use are using 8-bit mods models. >> Thank you so much for your talk, Alan. This
was really good. >> Thank you.
>> Thank you to everybody. >> If you want to continue the
conversation, we have a long break coming on. You can continue to use this hall or the hall
text chat that's related to it. During the lunch break, if you have got some spare time,
and you are interested, the lightning talk submission close at 1:00 p.m. Australia time which
is in half an hour. If you think you can come up with a 5-minute talk on anything interesting,
doesn't have to be Python related. Thank you, mickey, my cat meowed. You can find the links
and it has all the social events happening in the next day. Thank you