foreign
welcome back everyone uh we're gonna be
hearing from Felipe who is
a programmer involved in everything from
satellite Communications to machine
learning and we'll be learning from this
well-versed programmer about learning to
sketch with differential differentiable
rendering
all right
thank you
all right so let's do this
um we are going to be learning to sketch
actually the computer is going to be
learning to sketch
um and there are a few things about
sketching there are interesting for a
computer one of them is that it's not a
very computer-like so let's get into it
um first question is can actually
computers can computers actually sketch
and turn this out yes and they are very
good at it actually you are too good at
it so let's move the gopost and
let's ask can computer sketch badly now
now that computers are not having a good
time let's actually show what this looks
like so if you have a picture like this
this is what I want uh none of this
high quality stuff
um
all right so let's get into how you
actually do this and you do it with
machine learning
um and machine learning is something
like this you have a bunch of numbers
you have something there in the middle
and you have a bunch of numbers in the
other end and the thing in the middle
that's actually neural networks but we
don't like neural networks here so we
are going to switch that over for
something
that I'm going to talk about later and
in the output we also don't like a bunch
of numbers we are trying to sketch here
so that's going to be a sketch
all right and so it's something like
this in the output I put a very bad
scatter V Line there that's something
like what you want and the thing in the
middle that's just a function that Maps
a bunch of numbers
to sketches
now how does machine learning actually
work it works by
say that you have the numbers you use
those as knobs and then you turn them a
little and the thing in the output is
going to change
all right
so that's basically how machine learning
works I'm leaving out a very major part
but we're going to talk about that later
another thing that we need is rendering
so I said you have some thing that
generated sketches and that thing is a
function but what is that actually
so sketches are usually made of a bunch
of lines and if you want to render a
line you basically select two points and
you say I want to draw a line from one
point to the other and then you check
every single of those queries on the
screen and if they intersect with the
line
you fill them up all right so that's how
you draw lines with a computer very nice
but what is this business about being
differentiable
that's where the interesting things
start to come in I said that I left out
a very major part about machine learning
and this is that part now the thing is
when you generate those sketches here
for example
function sketches
you actually want the computer to learn
how to make a good sketch or in our case
a bad sketch
so you need to subtract whatever
generating from whatever you actually
want to generate and when we say like
subtract you may notice I put a question
mark between parentheses there because
it's not very clear what you mean like
subtract sketches what is that actually
so basically we are going to represent
every sketch is a big bunch of numbers
and then you're going to subtract each
one of those numbers from one sketch to
the other so that's that's what we mean
um so for example let's say that you
have this blue line and the blue line is
what you want to draw but your little uh
rendering engine is actually creating
the lining red so you need to figure out
what you need to change to make that
little lining red go to the position of
the line in blue
and how I do that is basically you need
to figure out a way of understanding how
far away they are the issue here is that
like if you just subtract those two
lines there is no overlap right so there
is no way of knowing how incorrect am I
in here because basically I'm 100
incorrect the line is not overlapping
what do we do we do there we need a way
of tracing back the influences of each
pixel in the way of saying okay I
created this lining red but I want to
create the line in blue but how far away
am I from the blue line
and it's basically just like Minesweeper
and the idea here is that if you are
incorrect
we shouldn't know by how much and that's
kind of the information that you get in
mindsweeper and the way that we do it
here is basically if you want to
approach some wine instead of just
saying this is the line that I want you
also say this is one pixel away from the
Line This is two pixels away from the
line
etc etc etc
and to be a bit more visual let's say
that this little blob is a pixel that
you generated and the big gap in the
middle that's the line that you want to
generate now if you generate things like
this the little pixel has no idea where
to go because as it sees the world it's
just a flat surface
so what you do is you basically
create pixel rides so you want to put
pixels in places where they can actually
find lines and find where they want to
go by creating this smooth
um slide like surface
all right so that's what differentiable
means you are actually instead of
creating things like this you are
creating things like these because
pixels like them
all right so you have the one to you
want it versus actual now you know which
direction you need to move and you can
map all of that back to the input space
all right
so let's get more into rendering again
um you have this thing called bezier
curves and they are essentially we don't
like straight things here so we are
going to bend them
um all right so instead of having
straight lines now you're going to have
curves
very nice and you can see there is a
middle point there instead of just
having these starting end points and
that's called a control point so now if
you want to represent that curve you
have six numbers two of them are the
start point two of them on the end point
and two of them are the little gray
point in there
all right so that's bezier curves
um there's a bit of math there uh but
it's basically just a set of equations
that we can use we don't care too much
about what exactly they're saying
um and we can jump into code now
all right
we are going to use pytorch uh we are
also going to use byters lightning and
you're going to use Jupiter lab
all right so to start with we want to
draw those lines
and basically we are going to translate
that set of equations that we had into
python code
and in machine learning we like to do
things in big batches because you want
to do everything the GPU which is much
faster than the CPU because it has a
bunch of small computers all bunched
together so to do computation the GPU
you essentially do the same computation
in all of those little computers and if
you see those two first lines there
where it says curves and then colon
colon that's basically adding Dimensions
so we can do a bunch of those at the
same time
um and then we have more code to
actually generate those lines in here we
are basically generating a grid so we
can generate those curves and the way
that you generate a grid is just your
generative line and then from that line
you use something called the quotation
product which basically turns lines into
squares sort of
um
and then you use a lot more math to
actually compute those lines and the
distances to those lines so all of this
is basically about doing this thing here
where instead of just having like the
red pixels there you actually have the
dim red and the even dimmer uh red in
there
right so we generate that and the lines
that actually do that distance
calculation are those around the
are there basically you are calculating
the distance to start point of the line
and then every single point in the line
and then the distance to the end point
and then combining all of those together
and what does that actually look like
because now you're talking too much
about code so it's not to bring it's
nice to bring it back this is what it
looks like up there you have those six
points that I said represent curves
and down here you have the actual
rendering of that so basically just
drawing lines on the screen very nice
and now we get into the actual machine
learning part so so far what we have
done is you have built something that
can do on the screen but instead of
drawing the screen the usual way it
draws on the screen with the little
pixel rights
um
so to build something uh with pytorch
essentially we use those modules which
are classes and we are going to call
create one that's called the rasterizer
and all that it's doing is calling that
code that I just showed to generate a
sketch so basically you have this bunch
of control points which are like the
input points for the curves and then you
generate a final sketch from that
then I'm going to create another class
called a sketch and that's going to be
the model the thing that learns how to
draw these sketches
and all that it's going to do is create
a variable called rasterizer and it's
just going to call the previous code
this thing
um
and
save that in the rasterizer variable and
Define It Forward pass which is
basically how do we use this thing to
generate some kind of output so if you
remember I showed like you're basically
mapping a bunch of numbers a bunch of
other numbers this is the thing that
does that so basically our mapping like
this bunch of points into this image or
sketch if you will
um and then how do you actually train
this thing so here we use a little trick
just to make sure that we don't generate
points that are outside of the screen
um but we use something called a loss
function to actually calculate the
difference between what we generated and
what we wanted to generate
and for loss functions there are a bunch
of different ways that you can uh
calculate differences the simplest way
is just what I said initially like you
just have this bunch of numbers and then
you subtract them and you use that as a
difference but nowadays there are ways
that are more interesting and they're
more natural of generating those
differences and there are basically
things like using neural networks to
actually calculate something that looks
like is the same or looks like it's
different
as opposed to just numbers without any
meaning and what that makes is it
creates different functions that are
actually kind of like what people see so
if you have two images that you look
like look at them any and they are very
similar that function is going to say
okay the difference is zero and if
they're very different that function is
just going to say okay the difference is
very high or one for example
um and I also snaked something in here
uh Sobel differences because those are
interesting for sketches
um the super operator is something that
you use to basically calculate outlines
from images which is something that
makes a lot of sense when you are
drawing sketches because sketches have a
lot of outlines
and then you could combine all of those
for example but the idea here is that
you have a bunch of ways that you can
generate those loss functions and it's
very important to get this right because
that's going to be what teaches
the machine or the computer how they are
going to draw those sketches
all right and once once we have all of
that in we can just train the model and
training in here is not going to be
something where like you get thousands
and thousands or millions of images and
you train the model with that but it's
going to be something where you get the
single image and then you try to make
the model output something that looks
like that image in that thing because
it's composed of a bunch of lines and
because we created this loss function
um that tends to create something that
looks like sketches is going to generate
something that looks like sketches so
for example if we have this picture
once you run that we are going to
generate something like this
which is properly bad and exactly what I
wanted in this case
um
if we input something like this
perfect
all right
if we input something like this all
right it's getting too good I think
maybe not that one but yeah a bunch of
different examples and the interesting
here is that you're not just capturing
like okay does this thing that's being
generated look exactly like the picture
it doesn't but it captures a lot of the
essence of the picture which is what I
want in the first place you actually
want the reason that I did not want like
the good sketches is because I think
it's way more interesting to capture
the essence of a picture and that's
essentially what a sketch is as opposed
to just capturing what the picture is
and that's the reason that it's not
super straightforward to do all of that
you need this differentiable rendering
business
um to actually be able to get through
that
all right and all of that
um is also partially a better version of
all of this that I have done here is
this thing called clipasso and it
basically uses a model that's very good
at telling like this image looks like
this other image but they're not the
same and that's the clip model and they
use that together read a lot of this
differentiable rendering thing to
actually generate sketches that capture
a lot of like what what what you need to
actually tell that for example this is a
flamingo and then they have like I don't
know five
lines in there and then you know that's
a flamingo so
that's uh I also have got the their
GitHub there you can take a look at how
they did that
all right and that's it
questions
[Applause]
that was super interesting
um I think I saw in some of the Python
there it was 128 lines was what you were
working with yes yep so you basically
create where is that I think yep yeah
you basically the big Vector that's
composed of other vectors that each one
of those has like six numbers represent
one of those curves and then like you
say a number of those to actually
generate the sketches so in this case
yep 128 did you um did you do many
experimentations with drastically
dropping or increasing the amount of
lines yes yep in this specific case
because the loss functions that I use
are very simple it tends to do better
with more lines so it creates something
that resembles more of the initial image
with more lines but if you use a better
loss function which is the case in here
then you can go down like to five lines
and still generate something that looks
like the initial image and that's
exactly what they're demonstrating here
like the same algorithm different number
of lines
so it's the only difference between your
approach and the one from this paper is
the loss function itself
mostly yes there are a few other things
that you can do to improve the quality
for example uh in here what we are doing
is initially like the initial condition
the initial set of lines they are
completely random
but one thing that you can do to
actually improve this is Select hotspots
so basically use any function like
for example even the syllable operator
you could use that to select areas of
interest and then you put the lines
initially close to those areas and then
the results are going to be quite a bit
better than if you don't do that so yeah
like in general it's the same approach
different plus functions but there are a
few other things that you can do like
that generally improve quality in this
light ways cool thank you
um how expensive or how slow is it to
run the neural network
um trainer rather than right
yeah so it's quite a bit more expensive
you usually are using a pre-trained
neural network in those cases so it's
not going to be as expensive as like
training the network because like the
the you are just using it as a loss
function not really training it
um so it's not really much more
expensive but because the model is so
simple it's basically just like
generating curves when you add any
neural network the performance drops a
lot so for example uh I did like 10
epochs for all of those and they took on
the CPU on the Mac it takes a few
minutes I would say two minutes but like
if you put a function that uses a neural
network it goes like to maybe four so it
doubles the time
yeah so I've noticed those um all the
curves they're only curved in One
Direction is it possible to deal with
mult uh curved
occurs a curve that goes in multiple
directions like like an s yep I'm
following you um so there is a paper uh
I did not put it in here but I can try
to find it later
um there is a paper that exactly does
that like what's the difference if you
have straight lines if you have curved
lines if you have lines that can curve
in multiple places
um and what they found is like the
visual difference is very small because
what you can do is for example a is a
line that Curves in let's say six places
you can just use six lines that curve
right so just by bumping that number of
Toto Strokes you can do sort of the same
thing and it's in general way less
expensive to use this curves that's just
like curve in a single place
um a question from the Discord is uh how
long did it take to train in total and
what kind of Hardware are you using
right so yeah as I said like
for those examples couple minutes hitch
because like there is no training the
model you are training on a specific
image like you just want oh I want the
model to generate a specific image so I
want to like
approximate all of those parameters that
generate the curves I want to
approximately adjust those based on this
specific image but yeah a couple minutes
on the max CPU it's not even used using
the GPU in here
um some of these sketches
remind me of the sort of sketches you'll
see in an art gallery underneath the
actual painting maybe I should sell them
no no but I guess the point that I'm
making is that um could you use this
technique as like a first approximation
to of an image and then use once you've
got these boundaries use use those
boundaries to sort of do a next
iteration of some sort of image
recognition
you can do one of the things that I'm
quite interested in is the uses that
this could have for animation because
one of like the highest costs that you
have in animation is actually creating
in between frames so you have someone
that draws like the keyframes and then
someone has to go and actually render
each one of those and if you want to
apply techniques like this to that
problem the thing is like you want to
generate a bunch of sketches to train
other models right so if you have things
like this maybe you could use even like
as a first approximation to generate
something else or you could use them to
generate training sets that you can then
use to create models for example that
help people with like generating in
between frames and animations so yeah
definitely something that's possible and
it's actually an active area of research
uh like this thing came out last year so
people are actively working like what
can you use this for
when you sketch you use different
pressures with your pen is that also a
possibility that you can like
differently it is yeah
um so in here as you guys could see like
I'm trying to go for the simplest
approach like I'm not using multiple
curves that can bend in multiple places
I'm using like the simplest version of a
curve I'm also not using colors and I'm
not using different pressures but one
thing that you can do is just like in
your definition of a curve let me just
find that real quick in your definition
of a curve you could just say I want an
extra parameter here that's going to be
the initial pressure and the final
pressure for example and that's going to
improve like the kind of throws that you
can do a lot and you could also say I
want three extra parameters uh for
representing the caller or you can go
crazy with this like you just like the
set of firmers that are training the
only thing that you have to be mindful
of is like when you're actually writing
the thing that takes those curves engine
generates like the final image you have
to be mindful that they are actually
differentiable but otherwise yeah you
could go crazy with the kind of
parameters that you're taking
sorry can you say it again I think that
the question was are there loss
functions that support that already
yeah yeah yeah yes yep in this case like
the last functions that I use here it
supports color image images but I'm
converting everything to grayscale and
in terms of like different pressures
um that would be supported out of the
box because you are just comparing like
two images so if you have a stroke that
initially for example has a
either a
higher brightness or a bigger stroke for
example depending on the way that you
want to represent like the pressure
um
When comparing the pixels the pixels
don't care right it's just the way that
we are generating those pixels
hi thanks a lot for the talk um my
question is during training did you ever
encounter situations where the gradient
descent algorithm did not converge and
you just end up with a bunch of random
squiggly lines that say the
not really
um because like the model is so simple
and the gradient in this case are very
simple as well
um but they only converge up to a point
right so you cannot improve the quality
much further than what you're seeing
here like with those models specifically
um because like this is a maximum
conversions that you're actually going
to get but yeah I did not have any
issues with actually getting into
conversion
I've got a question over here
here have any more Q
it's the little pixel going down the
road
I need to do more of those they're great
yeah
yeah I'm so happy
um and one actual question as well is
there any way of converting that into 3D
or would that be an entirely different
ballpark
that's an interesting question
um the concept of differentiable
rendering exists in 3D so instead of
extracting like a sketch or extracting
like the set of parameters for a curves
you're actually extracting for example
the meshes that you could use to render
that image and there's a lot of research
on that but yeah I don't know much about
that but yeah it's possible
uh are there any like open source
packages for helping people get started
with this
um
for rendering I believe there is
um
in that same paper there are reference
that I did not really put in here but I
do believe they have an open source
version of like their rendering approach
and also for the loss functions there
are packages uh one of them I'm using
here is
this package and it basically provides
uh perceptual loss functions that you
could use out of the box
all right let's give a round of applause
for Felipe