foreign
giving a talk on the complexity of
simplicity
uh Chris used to run pycon IU and um
has passed those rains over well as he
has moved to the U.S he is the director
of the Python software Foundation a
contributor to pants which we're all
wearing of course wait what
and um yes probably the person one of
the people at the conference needs the
least amount of introduction because you
all know him so well please help me give
a warm welcome to Chris oh no an
audience
I was fearing this
um hi everyone
um this is another one of my talks about
how pythonisters think about software
engineering I've been giving talks in
this vague direction for like five years
now
um and because I've been doing this for
five years things are now starting to
get scarily abstract which I apologize
uh there is less code in this talk than
my talk tomorrow will have
um unfortunately uh for those of you
seen other talks where I make old
versions of python do weird things I'm
not doing any of that today I I
apologize
um but you're here anyway um thank you
so much for coming along
Shall We Begin
so here is a value of pythons simple is
better than complex and what better way
to demonstrate Python's Simplicity than
with the first python code that most
people write
let's talk about print
uh print it is the first function I see
introduced in most introductory python
textbooks understanding print is very
very simple you call it unlike on this
slide you normally give it an argument
and then some text gets displayed to the
screen uh wonderful but what does print
do who here knows what print does
print does a lot of error checking
here's the actual implementation of
print it's taken from the very source of
C python the function is called built-in
print impul which is complexity that you
didn't need to know about that's the
function that actually gets called it's
quite long there is a lot of error
checking here is the line somewhere in
it that actually takes an argument
writes it to what will usually be
standard out
and the bit that's really important is
this bit here the bit that says Pi file
underscore right object
which itself has a whole world of
implementation details to to worry about
and you're looking at this and what's
actually important about this is not
none of none of what you're reading
the important thing is that this is C
and this is a python conference and so
for you and me as python people
understanding the C API is this going to
add an undesirable amount of complexity
to this talk
um
but I'd like to take a moment to point
out here that this is already complexity
that's inherent in doing something very
very simple in Python rather than just
executing some code that's written in
Python we are executing C code python
needs to allow you to run C functions
and to keep that overall Simplicity it
needs to prevent uh present these C
functions to the rest of the language
runtime in exactly the same way that
presents your python functions and the
CIA API that's needed to facilitate all
that is complex you saw all that error
handling but this is a python talk so
let's talk about python
print to a python developer has a
complex interface with complex behaviors
that really depend on how you specify
these keyword arguments at the end and
that's complexity that I don't want to
touch so we're going to narrow the scope
even more let's talk about printing just
one thing just a single argument
so what we're going to do is we're going
to Define a variable and then we're
going to print the value of that
variable and if we set the value of our
variable to a string like Hello World
um
this is very simple uh how would print
work well we can we can iterate over
each of the characters in The String we
can write each character to a file stand
it out in sequence
um it doesn't actually work like this
under the hood you saw that earlier
there's some complexity that involves
block i o which becomes more efficient
if you transfer in larger blocks across
the the API boundary but event there's
eventually there's something that's just
going to go through every single
character and put them onto some sort of
buffer somewhere so there's already a
whole bunch of complex conceptual
complexity here but it's it's not too
bad
as a python person what if we want to
print an integer
well this code is going to it's going to
throw an index error because you can't
iterate over an in object this code this
code won't work
um
so what you need to do is you need you
need to stringify the the parameter that
we pass in to print
and under the hood what stringification
will look something like this you you
dereference the type of the object and
you call the dunder string method that's
on that type pass in the parameter that
was part
and with that you know you can print
anything you can print anything recently
but in reality this this stringify code
is probably provided us to the C library
but if we're not going to use that and
to deal with this complexity we're going
to write our own string of fire which
which needs a bunch of knowledge of
ASCII and stuff and
that's before you even get to that
implementation of this put C function I
didn't even talk about which is
underpinned by a whole bunch of
sophisticated software and Hardware
Concepts that'll eventually give you the
ability to see the text that will you
asked to be displayed on your screen
[Applause]
nobody sets out to write complex code
there's two things that you should have
learned from that discussion of print
here the first is that we should all
want to interact with the simplest
interface possible and the second is
that even the simplest thing you can
teach someone about programming hides a
lot of complexity under the hood I I
don't think that anyone would disagree
that the best most maintainable code is
the simplest code that you can write
under the constraints that we have to
operate in
with software it exists because there is
complexity inherent in the world and if
we can model that complexity then then
we help people navigate that complexity
and by helping people navigate
complexity if we do it right things
become simpler for them now looking at
print is is instructive it only appears
uh simple whatever that is because we
wrote building blocks and with each tool
you're from the LCD elements in our
projector here right up to the python
interpreter itself more complexity gets
modeled and simplified for us which is
leading me to some indisputable facts
here the first is that complexity exists
that it will be experienced by somebody
on a project of some description it
could be an end user it could be you as
the programmer and on a systems design
level complexity is either modeled or
ignored
and more importantly that Simplicity
usually belies some level of complexity
on some level within a system
put another way anyone who is
experiencing Simplicity is doing so
because complexity has been dealt with
in some sort of way
so with that in mind
I used to think about how to build the
simplest possible solution to a problem
which which raises the question of what
we might think is simple and I don't
think this is a good question anymore I
I don't think it scales well
um instead if we accept that Simplicity
will usually belie some sort of
complexity within a system then the
questions we should be asking is who
should be experiencing Simplicity within
a system
which parts of our system should be
simple
and where should Simplicity be
experienced
and the main thing that I want to convey
is that the answers to these questions
are in flux python is an evolving
ecosystem and as new problems arise and
new tools make different problem domains
easier to navigate our Focus changes and
our Focus will continue to change
um so the first idea that I want to
explore is the idea of internalized
versus externalized complexity which is
one where python has really changed its
tune over the last 10 13 15 years
thereabouts there used to be this idea
that python would come with everything
that you needed to solve most problems
you might ever encounter out of the box
and this idea was prevalent back around
the time the Zen of python was actually
written
is actually less relevant for these days
for a number of reasons that's but not
particularly important right now
um who's seen this thing
right firstly like you know ignore this
a joke at the expense of pearl
um this line was written as a joke
there's still enough people uh in the
python Community who take this seriously
um and so python had this idea that if
if it made the big decisions for us
about how you'd solve your problems then
everyone would have the same version of
python and you could read another
person's Python's code and that would be
great and everybody could understand
everyone's code so we're going to talk
about a thing called URL lib2
um
so that that that two there that two
there
um that's entirely down to python not
addressing the complexity of name
spacing in the standard Library which
they fixed in Python 3 they called it
urlab dot request and and that
introduced complexity and reading
documentation which I'm not going to get
into but that that's the thing as well
um eurolib 2 was built around the idea
that your code could be simpler if you
could just drop URL handling code into
python code that dealt with other
interfaces and so it did all this in
what would be a relatively familiar way
to you uh your request objects they kind
of Quack Like files there's an
object-oriented interface for installing
handlers for more complex use cases and
you could install these handers for
authentication and redirect handling
either with or without affecting Global
state
and so in doing so it let you do a
reasonable job of modeling the
complexity of working with perfectly
specified HTTP but how does that look in
practice so URL Ed 2 has a thing called
authentication helpers
um what you do is you create a password
manager to associate some credentials
with specific URLs that's this this HTTP
password blah blah blah blah blah blah
thing
um
and you can create an authentication
Handler that supplies these login
credentials and then you create an
opener that knows how to open URLs with
these auth credentials and you use that
opener to open the URL and then you
treat the response like a file and you
hand them into json.load which I presume
you've you've imported somewhere in this
in this code before
this this code becomes visibly simpler
if you choose for URL lib2 to not manage
the complexity of authentication for you
as long as you know how to write this
this header you can make a request with
that header and with this you're forced
to make a trade-off
you either get simple code and you model
the complexity yourself or you have the
complexity handled for you but you spend
a lot of time rigging up this reusable
authentication handling infrastructure
and so the alternative is that we use a
different Library like requests the
behavior and authentication is a lot
more declarative requests you say you
want better off you make a request using
those declarations and as a bonus you
don't need to think about file handling
because it Imports Json for you and this
is obviously much simpler code to
interact with an authenticated HTTP API
you can make use of the libraries work
to model the complexity of
authentication and you get short code
you don't need to make these trade-offs
within your code but what's particularly
interesting about this is that
requests chose to re-implement something
that existed in the standard library and
then it choose it chose not to be in the
standard Library itself
this was a very odd choice at the time
but in retrospect this is how things
tend to get done in Python these days by
being out of the standard Library
request was able to adapt as a broader
web communities changed their attitudes
changed particularly around things like
security and it shows to model some
complexity of interacting with HTTP
services for end users and as a result
the code that you need to write to
interact with requests is much much
simpler than interacting with URL lib2
um
so the question that that arises from
this is what do you get for that gaining
Simplicity is this absolute
and the answer is no this this gain is
not absolute you now you need to model
dependencies you need to have a
requirements.txt you need to write
instructions on on which
requirements.txt file to load with Pip
you need to manage execution
environments you need to keep things up
to date and so there's now added
complexity in maintaining a project
so which is the right choice here or if
you're making the choice you use
requests or urlab 2 in 2011 the choice
might not have been immediately clear I
think people were still thinking about
whether using virtual amp was the right
thing to do or not and this was still
fairly early in the trend of HTTP apis
where you know everything is an API now
it wasn't the case back then so maybe if
your application needed to hit a number
of services that look like GitHub and
regular you might do that
I mean if you are fetching packages of a
Debian mirror or something like that
that doesn't require authentication you
you are a lib2 might have been simpler
for you and so you'd have you'd have a
bit more first party code wherever you
needed to hit a weird API but your
overall code base would be smaller with
urlib too and you wouldn't need to track
external dependencies and the
alternative would be to have less code
but more complexity in terms of the
environment that you need to manage
I think in retrospect this idea of less
first party code and letting your
dependencies do more of the work is an
approach that is one
and this is not because absolutely
everything was in place 10 years ago
it's because we saw this as kind of a
good way forward and we built tools to
help us manage complexity that arose
from deciding to manage external
dependencies and configuration we have
things like GitHub and and git more
generally to manage our configuration
management we have reasonable tooling
for managing execution environments and
we have a quite good package index
and so now we're in a place where our
definition of Simplicity has actually
changed for an identical problem a
modern python code base is going to be
smaller than one that we wrote 10 to 15
years ago to solve the same problem but
the trade-off is that we have an
increase in complexity of configuration
management so we have more dependencies
but we have more rigid environmental
requirements
but those dependencies they model
complexity themselves now we have
configuration tools they modal
complexity and so for the modern python
developer it seems that we get
Simplicity as a virtue of other people
managing complexity for us and just as a
thought like if simple is better than
complex well what does it say about how
good the code that we're using to manage
our environmental complexity is
so the second idea I want to talk about
is one that's less conclusively answered
in Python which is the idea of how we go
about managing complexity in a code base
and whether the tools that we have
actually present us uh present us with
Simplicity and specifically I'm going to
talk about quality assurance which is
the set of tools that we use to make
sure that our programs do what we say
where we'll do what they're meant to do
but first let's talk about duct typing
so duct typing is an idea that two
objects are interchangeable if you can
use them in the same place and not get
interface related errors if you're a
type theorist you would call it
structural polymorphism I'm not so I
won't
um Python's use of duct typing is a very
very simple type system both to
implement and to understand firstly
there's no syntax in the language that
describes the types that exist on an
interface just have functions and
variables this is simpler than Java or C
plus plus which require you to enumerate
the types that an interface expects and
then on the other hand there's no
automatic coercion so the language
doesn't do complex Behavior to make data
behave in different contexts this is
simple as a JavaScript or Perl or PHP
this forces some complexity in terms of
managing types onto you as a developer
um so when faced with some Library code
you may ask yourself well how do you
make sure that your library code is
correct and that normally comes down to
two questions which is what does your
code do and where should your code be
used
so let's talk about this first one or
how do we solve this what does your code
do problem from in terms of quality
assurance perspective
um it's it's to do things you know test
you know do your numeric inputs work do
your strings work do your valid
non-numeric inputs work
things that you write with tests
so automated tests help us manage
complexity of verifying Behavior by
allowing us to automatically run code
that will check the code that we're
verifying under the test works and to go
back to this talking about computer
science terminology stuff I was doing
earlier
um what we're doing is describing a set
of things called invariance and
invariant is basically a specification
of some behavior that a system should
hold given a set of circumstances if an
invariant stops holding then either the
behavior is wrong or the specification
is wrong and the more invariants you
describe the narrower the set of
behaviors your system can display
which means that the more invariants you
have specified
the more your complexity is managed
and a unit test is a way to specify an
invariant an invariant the circumstances
of this data that you provide to the
function under test and the invariant is
a set of assertions that are specified
about the result
and this is really familiar to us it
gives us a simple code-based structure
to reason with which is that
functionality lives in the
implementation files you have and in
variants are specified in test files
but what if we think about our simple
framework for quality insurance well
we've only really looked at one of these
two questions the question of what our
code actually does if we're thinking
about these and giving them fancy names
we can classify these two questions into
functional styling variants which are
the things that specify how the thing
works and structural ones which is how a
piece of software fits into a system
around it and the tool that we have for
describing functional invariants I don't
think are very good at describing
structural invariance we can describe
where things will work but we can't
really exhaustively describe where
things won't work and in Python for the
longest time we kind of solve this
problem with documentation
on the other hand
we have types now and types are great at
specifying structural invariants and
recent versions of python can describe
types for most reasonable python code
as well as providing structural and
variance you also don't need to write as
many tests so if you can't specify the
wrong type you don't need to enumerate
all the cases where you might have a
type error you can't specify the wrong
type so you don't need to test those
things but I'm not here to evangelize
types to you I think they're great but
they still aren't universally popular
and they're really really good reasons
for that firstly types are a tool for
managing complexity that really only
occurs in large code bases and it's not
always clear when you've crossed the
line of structural complexity that
arises where you can no longer fit
things entirely within your head I I
don't even think it's always clear the
code base will ever achieve the level of
structural complexity where you need to
have typing in place
and generally it can be hard to judge
when types are necessary and once you do
you kind of have to learn to use an
entirely new system which itself can be
a complex thing
and so you end up with a trade-off you
can either have a simpler toolbox for
managing quality assurance or you can
have a simpler way to specify the
structure of your code base
and so you can either have more tests
and more documentation or you can make
your interfaces harder to read and
harder to understand
and so there isn't actually a correct
answer here yet I don't think I think
both approaches provide some sort of
Simplicity depending on what sort of
problem you're trying to solve but they
both leave complexity elsewhere in a
project depending on how big it gets
so I'd like to finish up with a
discussion of evolution of python itself
because a common complaint I hear is
that that python is becoming more
complex with every single version and I
think it's worth taking some time to
examine whether that is the case
laughs
so python was for a long time a
completely static language for a lot of
us Python 2.7 was around for a very long
time and that's kind of the time when a
lot of the growth of of python actually
happened and was only really around
python 3.5 that the language started to
become stable for people who needed to
do python who needed to use Python 3 for
things other than data science and the
language is kind of been adding on
language features very regularly with
every single major release of python
um but if you're used to python not
changing on you for seven whole years
then maybe you'd be asking yourself well
what happened to this idea of of simple
being better than complex
um to which I would respond with print
because python has always been a
language that kind of tries to model
complexity so that users can write
simpler code and one thing that happens
as more people from more diverse use
cases decide to use Python to solve
their problems is that we discover more
cases where python is just not simple to
use yet
so python just models more complexity
for us
most of us really when we first learned
python only really picked up a subset of
python as a language and likewise as new
features emerge we'll only ever need a
new a subset of these new features as
somebody who who writes apis for things
I don't need a matrix multiplication
operator
but then again a data analyst is
probably never going to need to write an
async await Co routine in their Jupiter
notebook
um I was at kiwi pycon last year and I
gave a talk there and I tried to make a
point that Python's success was its
willingness to support use cases from a
diverse set of disciplines another talk
called the when of python argued that
you can very easily use a subset of
Python's features and never need to
think about the more complex features
until you actually need to use them and
they even proposed a model for it it's
called The when of python I don't
entirely agree with it but I think it's
a good start and you should look it up
and either adopt it or criticize it
basically go to last year's kiwi pycon
uh there are videos
that's the secret like this this
complexity is reasonably self-contained
and it's optional until you need it and
this stands in contrast to needlessly
complex languages like Java or even
necessarily complex languages like rust
all right we we as programmers want to
continue to model more things and then
the question we have to ask ourselves is
well do we want to have a simple
language or do we want to write simpler
code with the language that we end up
using
um right so now I ended at the end of
the examples section of this talk and
we're going to enter into the realm of
commentary again
um so when we started
we had this proposition that that simple
is better than complex and we went
through a bunch of examples that show
that when you are writing python
somebody is going to have to model
complexity for us and in the world of
python we're really lucky that the
language we use chooses to Model A lot
of complexity for us and it provides
tools that can model new and unexpected
forms of complexity
but I haven't really spoken about code
that that we write ourselves which
brings me to this thing that I said at
the top of the talk which is that
complexity exists
it would be better if complexity didn't
exist but it does and
as developers we can choose to contain
that complexity or we can choose to do
not contain that complexity
so what I hope I've conveyed today is
that we should be thinking about who
should be experiencing Simplicity and
who should experience complexity you
know we work in narrow domains as
developers and so even as if we're
working in a team on the same project
different people focus on different
things
with each of us having a pretty good
understanding of of the stuff that we
work on ourselves and so it's probably
best if we mod our own domains
complexity so it's not complicated for
other people
and this is why you have a giant pile of
C code that does a whole bunch of stuff
and then present a coherent interface
that is safe to give to new programmers
is literally the first thing we teach
them
like that's the point our code has end
users whether they're consumers or
whether their other software developers
on your own team your code is almost
always going to be part of some bigger
system and it will exist because it's
modeling some sort of complexity better
than a human could your code has
dependencies each of these dependencies
from the python interpreter to the
packages your dependencies depend upon
model some sort of complexity so you
don't have to otherwise it just wouldn't
be useful for you and it wouldn't need
to exist and like maybe your own code is
somebody else's dependency chain
we've seen earlier in this talk that the
things that may be simplest they can
change as we get newer tooling that
solves problems more repeatedly or if
people's opinions about what is good
changes
we've seen that what's simplest may
depend on a trade-off between more
upfront work and deferring decisions
until later I for me a guiding principle
is that Simplicity can be a gift that
you give to others if if simple is
really better than complex then leaving
your end users with a simpler experience
in managing your own complexity is
probably better because that means that
you're the only person dealing with this
dealing with complexity that you have to
deal with also I lied
this is a slide that suggests that you
should definitely use types if only at
your interface typeset your interface
give your users less complexity to need
to worry about because you've told them
what things will never work with your
code
choose to model complexly for your end
users and annotate your interfaces with
types if nothing else
more generally like simplify things
appropriately because I think this is
what you should take away from this talk
the idea that simple interfaces are a
thing that we should strive for and
complex ones are not but if you simplify
things in the wrong place
you tend to remove so much complexity
for us to be useless and inactionable
and you get talks like this one
foreign
[Applause]
okay apparently we have time for
questions so jacinta is going to hand a
microphone somewhere and we'll see if
anybody has a question
you can ask the obvious one as well
yes Tom
what's the obvious question
uh do you have your laptop out yeah you
just type import this
into a python interpreter okay yeah and
just read what it says there to yourself
um
I'll just give you a moment
have you have you found it
okay well do you want to ask what the
question is then
well I guess now the question is um
how do you know if it's simple if you're
Dutch
and that wasn't the question I'm sorry
does anyone else know what the question
I've prepared for is
foreign
what is the question you prepared for
no no you don't get that sorry
thanks everyone
um is complex better than complicated
I'm not answering that
[Laughter]
all right thanks everyone for coming
along
[Applause]