foreign
flag that we have at Curtis Maloney and
then Evan Brumley straight after this
one too and then we break an early break
for lunch for us
all right so we have Curtis Maloney with
HTTP request an unsung hero of Django
[Applause]
yay it's showing up okay hi everyone
um I was goaded into this talk by
somebody on Twitter I will just say that
up front
I'm funky Bob on IRC where I've been
helping people over the last 20 years
with their various coding problems
um I've been using python since the late
90s started on version 1.4
Believe it or else
16 years in the Django Community I was
invited to join the Django core team in
2014 there is no core team anymore but
that's another story I'd like to thank
my employer Burns Red
for uh helping me be here today
all right now
through my time with using Django
early on I had realized that the
framework needed to do two things there
were two aspirations that any framework
should really be
aiming for
in making our lives better the first is
to protect her office from the tedious
things so in Django we have the orm so
that we don't have to deal with raw SQL
we have templates so we don't have to
deal with it a lot of stuff is there so
you don't have to do that thing you do
all the time and it doesn't have to
consume all of your work the other thing
is to protect developers from the
dangerous things these are the mistakes
we often make the little corner cases we
often forget or the security of mistakes
the crop up all too often
the goal of this talk is pretty simple
it is not
to have everyone familiar with
absolutely every aspect of what's in the
request object
the goal of this talk is to show you
just how much tedium and danger and how
many sharp edges that Django is
protecting you from just as this one
object and you may not even realize it a
lot of the time
a lot of this comes down to the dry
concept the don't repeat yourself which
I often hear being kind of misunderstood
and misrepresented it is essentially you
only Define something once
you only Define it once
that might be a meaningful value to
which we give a name so that we can
always reference it or that might be how
to test or perform some function
but the benefit of defining it once
means that we can correct it in one
place and it's corrected everywhere
another major factor of this is that
ideally
from the Zen of python everybody reads
this don't we
there should be one and preferably only
one obvious way to do something
now let's just brief overview of what an
HTTP request looks like and I've just
seen a typo
um
you have the headers in blue here the
request path what's being requested and
then the content of requests this is
actually a sample post that I copied
from somewhere else just to see what's
in it so in an HTTP request object
we reflect this and there are three main
groups of functionality
there's handling headers
there's handling URLs and paths
and then there's content handling and
these are the three sections I'm going
to address
so
first with headers well Django makes
that really easy to access we have this
nice cached object called HTTP headers
which holds all of our headers so when
you you don't have to dig through the
meta object to find any of it
now Django also here is protecting us
from a lot of tedious things
so for instance if anybody's worked with
Django for a long time you might
remember having to dig into the meta
object to find a lot of details of stuff
that you might have learned along the
way
that some headers have an HTTP prefix
and some do not so Django remembers this
and deals with it for you so you don't
have to remember that
additionally
HTTP specifies using hyphens between
words whiskey supports using underscores
so again Django is protecting us from
this common mistake so we don't have to
remember either way
has actually got when it parses the
header name it replaces all the
underscores with hyphens and it
remembers to remove
the prefix only for the ones they have
it
and when you ask for a header it's also
very forgiving in allowing you to use
underscores or hyphens it'll work either
way
[Applause]
next
handling the content type header every
almost all requests will come with a
content type header and if we get one
it can give us a lot of useful details
so we can see here it's telling us this
is an HTML content and in fact
oh that hasn't formatted well and in
fact it's it's utf-8 encoded or it might
be a multi-part form data and it needs a
particular boundary
so we need a way to grab that
information and parse it out and
fortunately this is all done for us
additionally
we have a function for parsing the
headers additionally Django will go and
say hey if you told us what the
character set is if you told us the
encoding we'll go and find it and
remember what that encoding is we're
going to need that later on
this used to use the CGI pars header
function from the standard Library which
was one of my earliest comic commits to
Django but that whole package has been
deprecated so it's on the way out
the accept header tells is the client
telling the server I would like in order
of preference
these different content types I'm happy
to deal with them so again Django has a
class to help parse all of that out and
hand them back to you in a state that's
much easier for you to use you don't
have to remember how to parse that
header you don't have to remember how to
interpret it
but even better than this
it exposes all of those as cached
property for you to access as a list and
then gives you a nice convenient
function to say hey are they okay with
me giving them this response type
none of these are very complex functions
but all of them are ones you could
subtly get wrong if you weren't paying
attention every time so here they are
so next is the host header
which is
how do we know
who the client thinks they're talking to
when multi-site hosting came along this
was essential now it's just natural and
standard for everywhere
we learned this by asking well what was
the server name that we that we believe
is our name or what was the host header
that the Reliance sent in their request
or
actually maybe we're behind a proxy and
so the proxy is telling us
who their request who they were accessed
as and though who we're forwarding for
so we've got to be able to resolve all
of this
you might forget that sometimes you
might do try and take a shortcut
sometimes and then the system changes
and you don't keep up so
Django takes care of that for us it has
a thing and says okay if you're using X
forwarded for then we'll use it if it's
present and if that's not there we'll
use host and if not there we'll use our
own server name and then we'll add the
ports if it's non-standing
all these little corner cases it's just
protecting you from
oh and this self is secure down at the
bottom there which we saw earlier is a
trivial little function but again test
it once write it once don't redefine it
going further with the host option
get host now protects you from the
dangerous things we've seen the tedious
protection but there's also dangerous
what if that's not the host we think we
are what if that's a host name that we
don't think we should be answering to
so Django includes the functionality for
the allowed hosts list which was a known
attack back in the mid to early 20s
2000s
and if it doesn't match and if it's not
a valid List It protects you from that
Danger
our next step is URL buildings
what was the path that the user actually
asked for
well we've got two functions for getting
that which is subtly different in ways
that are detailed but the ones to do
with the full path the ones to do with
the path that chango's handling in case
we're on a subpath and so on
so under the hood these are both calling
this function get full path which is
doing actually a surprising amount of
work for what you'd think would be a
very simple thing get me the path
anybody here remember the late 90s with
ie and all of its path attacks and and
uh
exploits
no well yep that's a couple of people as
old as me
um it was a common problem and it was a
problem that I had turned up again and
again and again because this solution
wasn't centralized
so here Django is going through and it's
making sure that your iPath is escaped
properly it's making sure there's a
slash if we think there should always be
a slash if there's a query string we're
going to include that and put that in
properly with a question mark and
dealing with all of those problems again
remembering to do every step that we
might forget to do sometimes
next
game is it HTTP is it https is it
something else are we getting confused
well there's a request dot scheme there
okay
there's a lot hidden behind that of
course much like host there might be
proxies in the way there might be other
things that changing how we resolve this
so again Django takes the best way to do
this wraps it up in a function and
protects us from that tedium
and you can see here if we're told to
and you can turn on and off different
options and different features but it
will make sure that we get one of those
two answers and if not
the last step there
is a hook so that the localized
implementation is depending on whether
you're using whiskey asgi or some other
actual transport host or interface layer
can hook in and say well this is how I
think it should be implemented as it
turns out whiskey and asgi have subtly
different ways of doing this
in whiskey we get it from the
environment that's posted us in asgi
we ask the scope
and then otherwise we fall back to the
default
next is absolute URLs so sometimes you
want to include the entire host name the
entire scheme all of the way so that you
could for instance embed it in an email
so on so again a lot of tedium a lot of
steps that you might forget Django
remembers them first it calls get full
path like we saw before which makes sure
that everything's encoded properly and
constructed properly then it actually
parses what it got out of that
so they can make sure it fills in the
scheme and the host name or net lock as
it calls it in case they haven't been
provided
make sure to clean up and resolve all
the relative pathing for dot dot slash
and so on and finally
iri to URI encoding because we can't
necessarily give people International
codes we have to encode it properly for
consumption on the web
and just a taste of how fiddly this can
get
this is one part of the function that's
doing it and it's remembering to tick
does the path start with Slash does it
have you know do we have a Celine do we
have net lock do we have dot dot in
there do we have dot slash in there and
so on or all these little corner cases
that be very easy to forget
okay next we have content handling
we've got reading the whole raw body or
maybe iterating rows of data or reading
and parsing the content or handling the
past content like form data and files
and so on all of that is stuff that
needs to be taken care of all of that
would be tedious if you had to remember
how to do it every time
handling the body the request object
works like a file so you can access it
with read read line read lines you can
iterate it to get lines of data out all
of that is wrapped up for you and
additionally there's request.body which
if you ask for that grabs all of the
requests as a single data blob byte
string so that you can just do whatever
you like with it anyone who uses drf
this is how it gets the data so that it
can parse the Json
now this is all wrapped inside a
property which has a more defensive
stuff inside there so that it'll try to
protect you from things like if you read
a few lines of data and then do an
operation that's supposed to operate on
the whole body that's a mistake almost
certainly so it catches that and warns
you it will raise an error it also
enforces the maximum data upload size so
that if somebody sends you a very very
large post it will spool it out to disk
or handle it some other way so that it
doesn't exhaust the memory and can't be
used as an attack Vector protecting you
from the dangerous things
our next step is content encoding as we
saw earlier the client can tell us about
the content type and we need to make
sure that we
parse all of the content in the body
according to that content type
so when we receive it and when we set
that content type we have to rethink
about how we're parsing the get and and
body the post data accordingly so when
you set the encoding Django remembers
to clear out the if if it's paused to
get in post data clear it out so the
next time you access it will be
re-parsed with the correct encoding
so next we have handling of post data
this will be parsing posted data and it
is pretty common practice there's a
standard function for it in J in Python
but Django does it a little differently
to extract the files out for you
but also we've got to make sure what
what if the body was sent to us with the
wrong content type we don't want to be
treating every post like its form data
when it isn't and there's two different
ways of encoding form data so it's all
right Django's got you covered
when it has to load the post data and
the files
it goes and checks is this actually a
post
if it's not a post don't do it we'll
just put in some empty values and move
on with life
next it'll check
have we actually started reading the
content already
then that's a mistake and we're going to
flag that and raise an error
finally we'll go and say okay what
format is it is it a multi-part form
data yep then we've got to go parse it
have we already read all of the body
then we'll go and use that deposit have
we already done this again saving you
from those things you might forget later
on when the system changes
and finally the other content types in
case they handled one is it X form URL
encoded which means it looks just like a
query string and the last case is
we don't know what to do with it it's a
content type we don't understand so
we'll just give you an empty content
and also I mentioned file handling so
who wants to remember how to pass files
out of the form
who wants to handle that
just do it once do it right what about
if somebody sends you an enormous file
again let's put a limit there let's have
Django's handles go I can handle that if
it's really big I'll spool it out to
disk and you'll never know the
difference
so to handle that we have upload
handlers
you can ask for a list of upload
handlers if you never touch them they
don't happen a lot of the things in
requests are lazy so that we're not
doing work if we're never going to need
it
if the upload handlers have been not
been set then we'll go and run through
and initialize them which does a whole
bunch of let's get what you specified in
the settings file and go and ask all of
them to get ready
But ultimately you're still in control
you can override the default set of
upload handlers for your view if you
have a special need
and the last part of this section which
somehow has a lot quicker to talk about
this time when I tell you than when I
did this run last time is handling
cookies
does everybody remember all the
subtleties of how to pause a cookie
correct
so again Django takes care of it for you
it's a tedious thing it's a thing you
might make mistakes with it's the thing
that we don't want to have to deal with
we just want to get the answers that we
want so here it is it's all taken care
of for you
there's actually a cooking handle
handling Library built into python but
Django does a bit more on top of that
just to make life a little more a little
safer and finally Django includes a
bunch of signed cookie handling
functionality
why would you need signed cookies well I
mean cookies are stored in the browser
for a start so who means your trusting
information from the client is cookies
have a lot of rules
about how the browser handles them who
they should show them to you know is it
HTTP only is it only in Secure contacts
is it only to be used when talking to
the same site should we allow JavaScript
to read it
so you know there's a lot of rules there
that's great
Miss Koreans don't play by the rules so
just because the browser might do all
the right things doesn't mean that the
client is talking to you is a browser
so as a result
Django includes
get signed cookie
cat sign cookie
does exactly what it says it will take
the cookie that you've asked for and
then make sure that the content in there
has been signed with the secret key that
you keep in your setup so you're able to
set a cookie with content that the
client can see as much as they want but
if they modify it you can reject it
immediately out of hand
okay and as I said this went a lot
quicker than last time I did this last
time it went over 30 minutes
whereas now I am done
that was surprisingly quick
I'm guessing it's the nervous energy
speeding me up
you see with being done early now you
can field questions now I can feel it
questions oh wow
have we got 10 minutes of questions
excellent not Russell yeah
hello test
um you mentioned uh Jagger does a little
more with cookies than the standard
Library just interested well anything
interesting in there that you're doing
that the standard Library doesn't it
does mention a few interesting quirks
and bugs uh cited in
um
Mozilla Dev Network as for why it it
handles things a little quercally it's
basically again to protect you from
dangerous mistakes around the world of
not every client is implementing their
cookies correctly in fact if if anybody
has been around you long enough you'll
know that in the early days cookies were
not a standardized thing at all Netscape
introduced them
there were wild wild west concept of how
they were encoded in format and what the
different fields meant and then they
only got standardized a lot later on and
so some of those are still a bit quirky
and it's protecting you from that I do
recommend by the way as an adjunct to
that that everybody have a look through
the HTTP request content because it is
really quite easy to read in most cases
and very well commented as you saw in
some of those bits
does anybody else have a question oh
fantastic
hey um
what other are you able to draw any
comparisons to other HTTP requests
implementations that are in the python
ecosystem like I guess flask would have
one there's like Starlet things like
that are they also are they working at
the same layer are they similar things I
have not delved deeply enough to see
what's going on I did mean to include a
little story at the beginning of this of
what really got me interested in this
which was some years ago I built as an
experiment to learn new you know
upcoming Technologies on the web I built
a little web chat service
in Django originally and then I moved it
to being a raw whiskey service and on
top of that I started learning that oh
I've got to deal with encoding oh I've
got to deal with breaking this on oh
I've got to deal with that and going
back to Django and going how did you
deal with it how did you I didn't know
you even did that or how did you deal
with that and that's what got me on
along the line so I know that flask
Builds on top of workflow I believe
who specializes in being exactly that I
have not looked at how a lot of the
other platforms are dealing with but I
expect that you'll find a lot of
similarity style it if I remember
correctly was built by somebody who came
from a Django background it's built by
Tom Christie yeah yeah so I'm sure
there'd be a lot of similar Concepts
involved and a lot of the same
philosophy in the background of let's
protect you from that tedium and from
that Danger
I think I saw some hands over that way
you know any other questions hands up
for Curtis oh a lovely speaker
Katie getting her steps in uh so Django
has these layers has these extra apis to
protect you from all the nasty edge
cases and Corner cases of the spec
are there any legitimate uses for going
directly to the metal either if what's
the decision process of deciding no hang
on I don't want to use Django as
processing here I actually need need to
be dealing with the war request manually
managing it myself
um I would expect from my experience
that those are very very Corner cases
I've seen as a particular use case where
uh for instance the CEO csrf processing
was tripping content reading
and we didn't want it to so we had to go
and sort of sidestep a few things and
say stop doing that let us take care of
this
um that was one
um you would have to be I would expect
in a particularly unusual scenario of
your own engineering before any of these
things really start to get heavily in
the way but again Django lets you do it
if you don't want to process that as the
body don't access request.post or
request.files and it won't try to parse
the content if you want to read the
content yourself you can and even if you
do access
request.body all of that content is held
in the string i o object for you for
later
so that you can still get there if you
want to request parse the headers
yourself if you want to do all of that
yourself it's still there it's still
held it's just a layer on top rather
augmenting rather than obstructing your
access to all of the raw request
information
and just a reminder if you're joining us
online please just use a Discord thread
and type in any questions we can ask
them here in real life but right now
I'll pass the mic to another question
just on site called East Curtis um are
they something that is opaque within the
browser or is it possible to use a sign
Cookie From Within JavaScript perhaps
uh no from what I can see it's it's
simply signed with a Mac so it's your
content with a Mac added on so you
should be able to still read it and
parse it
from the client side
but yeah if you modify it that's what
they're detecting
that's an interesting concept
you could stop using jots and start
using signed cookies instead there's one
behind Judy
highly available microphones
um I could look this up in like two
minutes but since we have time I'd like
to ask since uh HTTP request headers are
case insensitive
do I have to call Dot lower to compare
them to or does shanko for the for the
header Keys yeah I will go back to the
code there because it actually does
thank you
[Music]
if you look here
the last line of I can't highlight it
easily the last line of that first Paz
had a name function it calls title case
on it so that will normalize all of the
the casing
so the first letter of each word is is
capitalized and all the rest are
lowercased
um excellent
but that does mean that looking up
Mateen might uh
might feel differently about it
yes so it normalizes all of them to
title case to keep it consistent and
standardized
question up the back there okay great I
think we've got four more minutes so
we'll fight for it Katie's Katie's
rushing no no hold on one second I
thought I was going to fight for it just
for the online people
when should I use Django and when should
we consider one of the other python web
Frameworks
oh the Eternal loaded question
um
just quickly because I mean I've been
using I've been using Django for so long
that Django is just what I reach for
um I experience of of other Frameworks
such as flask is flask can be much
lighter weight because you start with
almost nothing built in
but that tends to lead to down the track
you construct you add on all the bits
that you want in the end but that means
the next project that you go to may
choose to use different tools to answer
the same solution so it's not your
skills in a flask project aren't
necessarily as fungible as on a Django
project Starlet is built from the ground
up to be asynchronous so if you want to
be asynchronous and go the modern path
that that's asynchronous native
everywhere and because it was from
somebody with a strong Django background
now going to uh let's revisit the world
it's entirely possible that they've
stepped past some of the the clumsier
bits that we may not want to continue
hanging on to and they've been able to
do it clean it really comes down to I
find what you're most familiar with and
uh if you have specialized needs
which one will answer those those needs
I don't
feel that any one of them is necessarily
particularly the best solution in any
case or how to choose that depends
mostly on on Personal Taste and and
Community or um ecosystem you might find
that there's a flask plug-in that does
the thing you need or you might find
that this is a style add-on that answers
your questions and in that case that's
probably going to be a better choice
I think we have time for just one more
quick one
this one part of me I'm not doing my job
properly it's fine
I just uh it's a continuation of my
previous question uh so you said uh it
was capitalizing and I kept looking for
capitalizing on there and I didn't see
it but it would ID did sees that class
HTTP headers inherits from case and
sensitive mapping so I'm assuming in a
in a in a second there is where it
normalizes and yeah okay you see how the
there's a inheritance from case
insensitive mapping so I'm assuming HTTP
headers is addiction is a special
dictionary yes sorry regular lookups are
case insensitive that so it doesn't
matter uh what Casey is there it is
that's in the slide I just didn't say it
this time yes I think that I think
that's Sasha action actual answer right
so it normalizes the headers as it's
parsing them and then gives you a case
insensitive lookup when you ask for them
again taking away that tedium
protecting you from the tedious stuff
okay is that all can we give um one more
round of applause to Curtis
[Applause]
foreign
[Applause]