foreign
it's my pleasure to introduce Evan Evan
is a software engineer whose passions
lie in improving the developer
experience by reducing human error
enhancing code health and optimizing
workflows when he's not thinking about
security informatics or giving talks
about as many projects you'll find him
numbing on subway cookies or chasing
bunny rabbits please put your uh an
Applause for Evan
thank you
um cool hello everyone and welcome to my
story about refactoring
but what is refactoring well it's the
process of improving horrendous code
like this
to a beautiful code like this
all while ensuring that code still runs
tests still pass functionality is kept
the same and the user doesn't suspect
the thing
this could be done by doing things like
changing formatting
renaming variables
updating the structure
or using different algorithms and
tick-tactic sugar
and today I'll be talking about the
story of refactoring a side project of
mine that solves annotates and validates
the minimum Paths of this train game
well I'm such a nerd um anyways why am I
refactoring it well
I was a bit stuck and I wanted some help
from others
not only that but it was hard for me to
easily make changes and if it was hard
for me to understand well it would have
been even harder for someone else
okay so how did I get here
well often I have an idea that I want to
implement
and python is a great choice for doing
so quickly
but as I added code and more code and
more code
it quickly turns from a small script to
an unmaintainable project
do you want to see
no I I won't get you to read the entire
code because you don't have to
in the same way that you can often just
skim an entire article to get an idea of
what it's about
like if we skim through this code
uh
you see that uh
could you say that you have a reasonable
idea of what it does where to start
what it's made of what's important how
it works
it's quite long
um
yeah my answer to that is no
um but when I look at where I'm at now I
can easily Hazard that the project has
something to do with graphs and
annotations and mapping and stats and so
on
why was it so messy how did it become
like this well because no one is perfect
and can know what all the future
requirements will be or know all the
syntactic sugar of a language
or no identify when to use design
principles and patterns
or have the time to make all these
changes up front since with any piece of
code you can try your best to avoid
these problems by having an idea for
what to look out for
but you need to stop step back and take
a look at what you're doing
okay so if we do that what can we do
well the first and easiest step is to
use a formatter
it lets us take messy long condensed and
inconsistently written blocks like this
which you can see are gone off the
screen
and automatically turn them into this
which is much more human readable
and even if I blur the example to
simulate skimming it
it's probably a lot easier to tell that
the formatted case we're actually
filtering a list
now in terms of formatters there's a
couple you can choose from so you can
spend a couple of hours researching each
one taking a look at what they do
no wait why do you really want to do
that I mean if your goal is to learn
sure
but a good methodology to take away here
right from the beginning is one about
pragmatism
because one of the biggest qualms of
refactoring is the amount of time it can
take
it's important to realize when something
is good enough
so if you're trying to be pragmatic just
pick the first thing you find
I chose to use black it's opinionated
but that doesn't matter to me all that
matters is that my code is always
formatted to be readable and consistent
because in future trusting the formatter
to automatically fix things and not
having to debate with myself or others
is what will save time
and if I do find issues then it's very
easy to swap out later because you can
just reformat all my code at once again
but even formatted code can still have
issues format is going to tell you about
code smells like problematic naming and
like single characters spelling mistakes
like Celsius and Fahrenheit
unused code like this last line
or duplicated code
but I'll enter like rough wheel lenses
are great because they can look at all
your code as you write it and raise
issues for fixing that you might not be
aware of
there's a whole list of them and you
don't even have to follow all of them
you can silence the ones you don't like
or make your own
but they'll often guide you into good
patterns that you might otherwise have
not considered
like this morning I got about my
function have too many arguments
first it made me think that's silly what
what's wrong with that
but when I took a step back and thought
about how to address it it made me
realize there was a better way
if we took a look at the beginning we'll
see that there's a bunch of code
and some of that code loads some State
into multiple variables and then we take
those variables and we pass them into
our function with many many arguments
and then we pass those verbs again
individually into other functions
and while this was better than just
having everything as Global variables it
was so messy
instead a better idea was to
compartmentalize everything by putting
these functions as part of a class
for example by introducing a data
holding class
that lets me section off creating
storing and loading the data elsewhere
so where I was previously passing around
all these arguments
now when I call my function I only need
to pass around the starter object making
my code much simpler what do you think
is this better
is it the best
no why
well let's take a look at this distance
checking function
you'll see that we iterate over the
distance list and get the port name and
some distances and then we iterate over
those distances
to get the cine name and the expected
distance
and then we calculate the actual
distance
and then
we do the check to see if it expected
value but this isn't very intuitive
what might make more intuitive sense
is to go through every pair of nodes
that we want to check the distance of
and then have a method from an object
that calculates the actual distance
and another that retrieves the expected
distance
and this is only great for readability
but it's also great for abstraction
because now all these functions make it
very easy to change the underlying
functionality
for example if you wanted to change and
make optimizations to the distance
calculation to use a faster algorithm we
can just change that within the function
but
instead of passing around because
instead of passing around an object's
data what we really want is an interface
to access particular kinds of data
showing the usefulness of Designing
around requirements
but what I had really done was apply
programming principles
known as interface segregation and
dependency inversion
which together both dictate that
functionality should depend on
abstractions not the data generated by
some unknown entity
and if you're curious for more these
both come from a set of five called
solid
but if you're not aware of these
principles or you find them easy to
forget then I think a better methodology
that I found useful is always telling
myself
there's got to be a better way
it might take some thinking research or
time to find what's available to make
code simpler better and shorter but I
would have never got in there if I
always think if I wasn't always thinking
about how to do better
and now that we've got a way to find
issues in our code we we need a way to
easily address them
and I found the easiest way to do that
is to use an integrated development
environment like pycharm
ideas like pycharm are quite powerful
letting you out of the box easily do
refactoring without any setup
I mean look it has a refactoring menu
Isn't that great and some of the things
that I found quite helpful were the
ability to rename variables everywhere
within my project
or fields and Method throughout the
entire project
it can also extract repeated blocks of
code into a function
uh or change the signature of a
function's definition and updating for
everywhere that it gets called
it can also list usages and letting you
quickly navigate between them
or it also lets you move files between
folders and updates their usages
accordingly so now you don't have to
change all your Imports
and it also shows you documentation on
hovering or you can even give you more
extensive things
and look I know what you might be
thinking
amazing lightweight editors like vim and
vs code do that too and yes as someone
has used both I'm sure it could
but after I put after I had put on my
pragmatic hat on I thought to myself is
it really worth all the effort it would
take to set up when all of this comes
out of the box for free
now something I forgot to mention that
an IDE can also do is help you with
types
unlike other languages when we're
writing small Python scripts it might
not be worthwhile to have types but as
our project grows
uh type kinting can become super helpful
for quite a few reasons one of these is
spotting inevitable human errors
like here we write our code to
explicitly output a list
but Network sex.paths returns a
generator
so for example if we save it as a
variable here
and then iterate over at once
we'll get the expected output
but because it's a generator when we
iterate over it again our output will
still be the same
even though we might expect something
different from a list
but with the right tools errors like
this can be highlighted for us
another case is when we're unpacking an
ester data type
here there's nothing
but if we do if you use some type
annotations now we can auto complete
with strings
okay so now that we know what types are
important let's go use them
whatever you might say you have so many
functions
that's going to be so much work for your
large project and you're right
but we have technology and we can
automate finding in adding in types for
us
with a tool called monkey type
where for the following function
can produce stops like this
and even update your existing code so
you don't have to edit it yourself
of course we don't need an IDE to
benefit from type painting but once
typing is all set up it lets us run type
checking tools like my pi to ensure the
continuous checking for errors and types
and I forgot to mention again
typing thing is also really helpful for
documentation
take this function with no typing for
example
we can make some guesses at what most of
these should be
for example we could guess that names
are probably strings
and since we're using them to Index this
paths argument parse is probably a
dictionary
that stores a list since we're iterating
over the output
and the function says it's returning a
set and we have this dictionary
comprehension so we can make a guess at
that as well
but with our context we're still missing
a lot of
um we're still missing a lot of things
we have no clue what positions from Edge
returns or what directions set from Edge
returns and therefore we have to make
guesses at why a connection path is a
dictionary
in fact there's no reason why pods
couldn't be a list
if for example the cities and ports were
ins
but if I properly fill in these types
then there's a much better idea that
connection path is actually a dictionary
mapping between positions and directions
and paths uh wait let me format that uh
cool
is actually a 2d map of names to
positions
and if I do some formatting to Alias the
types
then now it's a lot more clearer that
um it's a lot more clearer that this
Tuple intent is a position and
if I you know become a bit excessive I
could set aliases for Strings as well
make it much more clear that you know
we're returning a set of directions or
we're taking in locations
but what it does mean is that if they
need to become classes then we can
easily support that
for example if we change position to be
a class
and with that maybe it's worth asking
how documented is your code
I don't just mean the number of comments
and dog strings you've written
or self-documenting type hints what I
mean is does it speak for itself through
its naming and structure like how easy
is it for someone other than you to pick
up and understand your code with no
context
if we take a look at what my repo would
have looked like at the beginning
here's the GitHub page with no
description
and
nothing at the bottom no documentation
or a readme four ways to start so we
probably have to read some code
but it's not clear which of all of these
files are that code
and even once we do find the file it's
thousands of lines long
and there is no main function to signify
where to start
if this was an actual Repro I probably
would have given up my hopes of using it
or contributing to it by now which goes
to show
the easier your code is to pick up and
understand with no context the easier it
will be for others to answer their
questions or contribute
now in comparison we could look at what
we ended up with
here's the main wrapper with only a few
files
and if we scroll down we'll find the
readme
which says that the entry point is the
main file in the directory so if we go
to that directory we can easily find the
main file
and then if we go to the bottom of that
file
we'll find the main method and then
we'll see that the function being run
comes from this helper
so then if you go to the helper that we
easily found in a different file
we can easily follow what comes next
after update map and then continue
throughout the whole code
in fact if we look at the entire
structure as a whole you can see how
these classes and packages
make it much easier to reason about the
different components of the project
not only that but it Embraces another
important programming principle
this time the single responsibility
principle which says that everything
should only have one responsibility
you can see the impact if we compare the
main function from when I first started
you can't see it but it's actually a
quite long
um
actually let's go on a tangent while I'm
here you might have seen the extensive
use of uh named arguments for example in
world map
which might seem a bit redundant
but
that's because I have an utmost
adoration for Python's beautiful
function argument system let me explain
why
if we're using another language and we
want to set up rotation to default to
zero then we often need a function
override
but since python has default arguments
there's no need for these chained
overrides
and often though they'll use Builders to
resolve this issue
that let us optionally add this data
but yet another issue lies due to the
nature of the required ordered arguments
and that is that there's no knowing
whether arguments are set up correctly
in this case you might have not noticed
but it should actually be width and
height not height and width
python resolves this with named
arguments and thus also does away with
builders meaning that not only are your
functions self-documenting by having
constant arguments labeled for you
but now your ordering is redundant you
can set it up however you like
or more importantly your code is now
always future proofed against errors
under cases where arguments are inserted
or ordering is changed
and if this opened your eyes I'll be at
the Sprints if you want to contribute to
automatically finding and applying these
guardrails and enhancements
anyways where were we
um yeah this main function was just too
long
and if you looked at the files you can
see how
what like what are these what is
graphing and graph what is the
difference
if we take a look graph is actually
storing two data classes one being quite
long
again you can't see it here
but it has quite a bit of logic
when instead we could split out them to
be uh their own single files
this might be of concern given the extra
Imports and files to jump around but
that's a trade-off to consider depending
on how long is your class
okay now that we've started to update
our code what else can we do
another good principle we can follow is
dry or don't repeat yourself
it's important for so many reasons here
we have some code that repeatedly
calculates distances
we could refactor it by first
introducing a function for abstraction
and then using that function to provide
consistency
and then before finally introducing a
loop
now our code is cleaner and safer
providing safety through reassurance
that everything is running in the same
way
for example did you notice that there
was a bug in the first example
at the end it should actually be p3.y
minus p2.y
so dry allows us to be more efficient
because now we don't need to copy or
rewrite code to calculate distance
or when we want to make changes like
fixing that bug or in this case changing
the method of square root we only need
to do it in one place and can ensure
that a bug fix or functionality is used
everywhere very quickly
but where did I apply in my code
one example is this validation code here
that was being used in two places
our first thought might be to create a
utils class but this is often not a good
idea as it creates a new dependency that
incentivizes us to add more small
functionality that there isn't a good
context for
instead this could be moved to be inside
validations
and rename to validate with
or maybe what we should really do is ask
ourselves do we really need this
maybe it would be better to instead
create a grid class that ensures all
rows of the same width or maybe we
should ask ourselves if there's a better
way and remove the need for this
entirely to instill to instead deal with
the issue when it appears
another place we could remove repeated
code is during the annotations
in my map annotations are what colors
the path between a port and a city
and there are grouped annotations for
both ports and cities however even
though these are very similar and are
done by the same mechanism we need to
implement this slight difference since
ports are annotated through nesting
but conceptually this is just like
recursion with a base case and it made
me think there must be a better way
and so I ingeniously thought of this
which creates a nested list of
specialized functions to be called with
another generic function that could be
used for both cities and ports and thus
they duplicate our code
did you get that
no great because I quickly realized that
I didn't either
and even though I learned that a while I
could give myself a pat on the back for
This brilliant ingenuity
what I really learned is that I should
give myself a kick in the bum for this
added complexity because sometimes using
neat coding tricks to simplify code is
not the best idea
if it was all our code would be golfed
to be as short as possible and that's
just not readable
so it's often worth considering will the
method of our refactoring really help or
more importantly in bringing up another
principle are we keeping it simple
stupid
our aim should always be to simplify a
code and remove what might not be needed
not introduce added complexity
eventually I settled in a class like
this
that could be inherited and override to
provide the extensibility that I was
looking for
but was that extensibility that I needed
maybe it's worth Consulting another
principle yagni or you ain't gonna need
it
because we should we really be
refactoring for things that don't exist
thus I'll still always have a feeling
that there's got to be a better way
okay but how can we be confident that we
can make all these changes while keeping
functionality the same
well isn't that what tests are for in
fact having good tests is likely one of
the most important things to have while
refactoring because if we have tests to
lean on then we can make all the changes
that we want run our tests and if they
pass then we're good to go
tests aren't always the easiest thing to
set up though especially under
complexity
but even having very basic tests where a
program's correctness might even be
manually verified is better than nothing
since once that's done it can be a
starting point into automating it using
integration tests
since with lots of refactoring unit
tests may require too many changes
for example this data class
takes in some file names
and create some state which can be
easily tested by an integration test
as in our test we can just create the
file to test against
but it may become really hard to test
the functions of a class like this as
different files will be needed to create
to be created for every test not only
that but it's not clear what's inside
these files
if you haven't noticed this is because
it violates the dependency inversion
principle
in a way we can fix that is to instead
change our class to be composed of this
data
and then use a cloth method that will
take in these file names
and create the data needed
to pass into the class
now in our code we can continue to
create data using the file names
but in our tests we can specify that the
data that we want to be used which makes
things much clearer
and now while tests might ensure that
your program stays the same they don't
really ensure that your program doesn't
slow down
it's often not obvious when code can be
slow especially with python and more so
when we're trying to ensure that we're
not spending time in premature
optimization
a profiler like scaling can be helpful
in analyzing the runtime of our code to
determine where there's inefficiencies
in Computing or memory
for example this was the output I got
when I started to notice my code started
to be slow and I couldn't really figure
out what change I did to cause it
and here we can see that this is
actually caused by the creation of
coordinates
because what I had done was set up my
code in such a way to compute the values
of a field on the classes creation
but the edge nodes aren't always needed
instead I could move this field to a
function instead only being run when
needed
but maybe I wanted this as a property
since it's easier to read and implies
it's a value that shouldn't change
well we can do that with a property
built-in decorator which lets us return
a function's output as a field
but a value shouldn't always be
recomputed if it's not going to return
the same thing
this is where the func tools.cash
decorator comes in which saves the
output of a function so that we don't
have to recompute it
this is great for values that we access
many times but only want to compute once
we could use both to solve this issue
however it's worth noting that it can
lead to memory leaks as the cache May
retain instance references preventing
garbage collection
the solution is to use the cached
property decorator which does both in
one go
and easily one of my favorite Python
built-ins and one that resolves so many
lintel warnings too
all right that's our profiling and
optimization done
but I wonder whether my refactoring
story would be fast or slow if I
profiled it
how much time do I have left
okay well that was quite a fun story
you might be wondering though what about
the profit did I get the help that I
needed
yes I did but there's still plenty of
more to figure out
and while you could say that there is
profit and saving time in the future
I think the real profit though was
everything that I learned along the way
if you wanted to profit yourself I've
put up some resources here that you can
reference later
I'll leave this on for a couple seconds
um
pool
can also talk to me later
um but yes I want to say thanks to my
partner and all my friends for
supporting me and thanks to all of you
for listening
I would love to hear any feedback that
you have I'm always trying to improve
and you can find me online at ecohillas
or GitHub if you wanted to take a look
at my code
however I won't claim that my code is
perfect because I think there's always a
better way
you can also find me in person right
here as I think we still have some time
for questions thank you
yep I think we've got time for questions
I'll put the QR code up as well
um all right um I think that was that
was a great talk cute
but I do have a questions in in an ideal
world where you go into a project
there's a whole bunch of tests
um 100 coverage for example then you can
ensure that no changes or your
refactoring doesn't accidentally change
any functionality but there are cases
where you go in and there's no test what
would you do in that case you ensure
that you don't accidents accidentally
change things yeah that's really hard
um
so when when I started I basically had
this tool that I would run and I didn't
have any tests because I just wanted
things to work
um and what I did was I just
would run it and then visually check it
um we'll run it and really visually
check it and I will do that over and
over and uh I think you need to have
some kind of test you need to have some
kind of method
um even if it's just like
create an instance of what the output
should be and then check that it's the
same
um
and then at some point once you've
started refactoring and changing all
that you can then
start to introduce better tests that you
can make sure that all this is the same
in terms of existing code bases I'm not
100 sure
um
I guess look at the entry point and the
exit point and see what you're trying to
compare it's probably a topic for
another talk on testing
um yeah does that answer your question
yeah thank you awesome thank you
hi this is a question about typing and
documentation so uh when using say
Django
um it's hard to do you use my pie
because you know for my pie to work it
has to run all the code and that's
really hard with Django
uh so typing is not that useful because
you have there's all these libraries
that enable using Wi-Fi but they're a
pain so typing isn't that useful in
Django is there um a format of sort of
Doc strings that you've uh that you're
aware of that uh are a good replacement
for typing yes
um so
uh you don't actually need to use mypai
to tell you whether things are correct
or not and you don't actually need to
put in the typing you can also just put
comments that's originally how typing
was done
um there's a certain syntax I don't
remember
you can also just use doc strings as you
mentioned to
um
you can use doc strings to just make
your own system or use an existing
system because really like the point if
you have typing it's nice because then
you can validate against everything
um
but you don't really need that
validation you what you really need is
some way to
you know you might be working at
something and then you want to check
okay what
um what is this function outputting
um if you just go to that function's
definition and then look at the doc
string
provided this developer has like put
something there for you even that's a
really good start and then over time
when things get better you can start
adding more typing Etc
does that help uh yeah like so my
problem is like uh there's no pep for uh
doc strings where it says how to define
arguments right okay uh
um
I have a feeling that the there should
be
um I
I'm not sure I think I've seen a lintel
warning where it's something like ensure
that it tells you what it returns I have
I think I have seen functions that tell
you like texting these arguments returns
this it might not be a type definition
but it might just be like a wordy
definition at the least
um it's a it's a hard question yeah it's
a hard problem and would you ever prefer
uh sometimes when um the typing is a
complex thing there's a dictionary of
lists of sets of whatever these
sometimes prefer to just omit that and
just write a dog string describing what
what it is and adding context to why
yeah that's what you want in this
function yeah I had that case pop up
when I um
was importing like Json and that Json
would be converted to dictionary and it
was like oh this massive really long
dictionary like could be a string or it
could be um
well
uh it could be a string or it could be a
um
an into etc etc
um
I think in that case I just opted to
remove it but another thing you could do
is to just
um save that as an alias somewhere
um and just say Okay this return to
Jason now Jason is formatted like this
and then call it a day yeah
yeah thanks thank you
thanks Evan for a great talk here's a
token of our appreciation
thank you very much