[Music]
but before that we have Sam Sam is a
professional software developer amateur
rocket scientist and
astronomer they love python Jango cats
working on their personal software and
Hardware projects everything space
playing games of all kinds and tinkering
with 3D printers please welcome
Sam so yeah this is a bit of a
postmortem so it's going to be
interesting
um so yeah I'm Sam uh I'm a freelance
software developer and consultant uh
I've been working on some space robots s
a startup and recently started a podcast
as well uh the rest of the contact info
is on the slide just you know wasting
time so anyone who wants to take a photo
of it has a chance but it'll be there
later so to do a postm modov project we
kind of have to start with what the
point of the project was what was the
aim and in this particular case it was
just a simple project it was a regular
run-of-the-mill business SAS for a very
specific business vertical nothing
complicated no plans to become a unicorn
and change the world and become
billionaires just regular business and
it was going to be done with MJS and D
Jango with Django rest framework now for
anyone who's not familiar with MJS it
has a number of really beneficial
features for this kind of plug andplay
functionality uh in the fact that it
actually can base everything off the
Json API and you can get a nice just
plug andplay kind of functionality from
it and other than just the usual man of
business logic getting put in the back
end and the front end where it needed to
be this was just straight up a
JavaScript Spar and a Json API back end
nothing fancy and that's where you start
from your basic concept digging in how
you actually going to put it together
well for starting point there was some
thirdparty app compatibility that was
going to be used we had the Django rest
framework we had Jango polymorphism
because when we did the model and schema
work we realized we had some
polymorphism that we needed to account
for and there were a few other minor
libraries along the way that you know we
all necessary parts of making an entire
Django app if you aren't going to sit
there and write it all from scratch half
the benefit of using a framework is the
ecosystem now I mentioned that it's
compatibility well that's because a big
chunk of this was decided to be built as
a reusable app now it does make sense
why but it it has consequences
now it's a multi-tenant app since it's a
SAS there are going to be multiple
customers so each one is going to need
to have their data kept separate now
there's going to be background
processing there's going to be the usual
celery for background jobs there might
be some occasional Chron jobs nothing
fancy just straightforward good
oldfashioned business logic
stuff now I mentioned the multi tency
and I mentioned building it as a
reusable app this is what makes the
drive on this because when you dig into
the actual process and when the
customers use cases were explained what
we noticed was okay there is a overlap
between the customers so some of the
customers have multiple location and
those multiple locations need to have
commonality things can be moved from one
to the other some of the things are
shared between them but only some of
them have that so there's a mix between
single operators and multiple operators
and groups of operators and it needs to
be able to handle all of this in order
to be useful for the actual customer
base and that's where it gets
complicated and it becomes a plan to
make that tency information that
management of the multi-tenancy into
reusable
piece how hard can it
be
and this is the part where I have to put
out a carat the code you were about to
see is old unmaintained not free but it
is open source and feel free to ask a
question about that later if you want
and I going to do the horrible thing of
confronting code that I wrote over half
a decade
ago because nothing is quite so humbling
as realizing how stupid you used to be
and and I've left it in wats and all
there are todos there are comments it's
it's got all of it so we get to the
implementation part we start off with a
fairly straightforward reusable app I'm
just going to make a poetry project
great off we go real easy then we get to
the basic models after you do your
Django scaffolding don't think anyone
needs to see eight Django commands at
this point so we've got a tenant model
you can't really track tencies without
having a tenant it's straightforward
model we've got a name we've got an ID
we've got a slug for short URL stuff and
we've got some Json fields to keep track
of various bits of metadata that don't
really have a plan yet because like any
business SAS requirements may change you
need to be a bit more flexible and with
that we've then got the relationships
because obviously if we're going to
match users to tencies we need to
actually track that relationship so
there is a tenant relationship piece
here which is using one of the mixins
you'll see soon and it's tracking the
permissions so it's got the permission
attached to these objects and that lets
us work out whether someone is allowed
to see things inside of a tendency or
not and then we've got a model manager
because at the heart of this kind of
multi- tendency is keeping track of
which kinds of data should come out of
the big everything in the same pile
schema other kinds of multi- tendency
let you just put separate databases and
keep things separate that way that
wasn't going to be practical here
because of the way things needed to be
moved between on a regular basis so the
plan was keep it all in the one schema
and the only way to work with that is to
do your SQL queries the right way so we
have our query sets and in this case
you'll notice that at the top here it's
actually doing a return none so if we're
not actually getting a tenant it returns
none because it shouldn't actually give
information when there's no tency there
so the default query path when this is
invoked will filter for a tenant and
this is what we use inside of the ACT
base model after we get through the
other one because there's a multi-tenant
one as well and the only difference here
is that because there's a many to many
for these models the query is slightly
different and when we use them inside
the models it is very much what you'd
think it's just a straightforward
abstract based model mix in and it's
using the single tenant model manager
there the multi-tenant one uses the
multi-tenant model mix in and it
obviously has to keep track of the
original manager because every now and
then you need your Escape patch but by
default you want it to be doing things
the way you plan on it doing them and
there's a bit of a save logic down there
because obviously you don't want to save
something that belongs to a tenant
without actually having a tenant to save
it to and that's the difference between
these two actually because on the
multi-tenant one with a many to many you
actually have to save it differently
because it can't exist you have it has
to exist before you save it to the other
object with the Min to mini table so
that's slight difference between the two
they're otherwise the same really um
then we have obviously because we've
made some classes we're going to have
some detail views you know we're going
to have to expose that data to our front
end nothing fancy here you know I think
everyone's seen a detail you before now
the middleware is where it starts to get
fun because we're going to actually have
to keep track of the tenant as we run
through the entire request response
cycle the request has come in for a
tenants data which tenant and do I want
to try and modify every single call to
pass it down the stack
no so the only really good place to put
that is a thread local which you'll see
in a second because when we put it in a
thread local we can actually get that
tenant information wherever we need to
access
it and using it this way kind of gives
us that escape hatch because we don't
want to need lots of extra refactoring
every time we change something about
this that straightforward thread local
just bypasses the
complexity although when came time to
actually get it working with
async it became a pain because this this
code never worked with async at all and
as mentioned earlier it needed to
support a couple of third party apps and
then we have the duplicates here it had
to be marked as a duplicate to stop my
IDE complaining because it is
functionally just a duplicate but
because of how polymorphic works you
need to actually subass from its model
manager and create basically a duplicate
for this kind of work it doesn't the way
it does polymorphism is its core model
manager behaviors so you kind of have to
extend them and you can't really
substitute that and kind of have a
universal one so there's a whole
duplicate copy also base model has to be
duplicated and we've got the
multi-tenant one as well and the actual
multi- based model there's a lot of
repetition in this kind of code but as
you flesh it out it does all make a lot
of sense and then we had the Jango rest
framework support with their permissions
framework serializers for the tenant
data so that the front end can actually
get that data it's all very
straightforward business stuff and again
permissions permissions permissions and
another serializer and another
serializer and we see the permissions
are where it really becomes a key thing
here because when you think about the
kind of purpose of this functionality is
to separate data between customers
naturally you need to make sure that
your permissions are structured that way
because obviously the permission to
access the data for one tenant is not
necessarily going to be completely
decoupled from the permission to access
the data of another tenant because if
you're say a manager in one office but
you're only allowed access to read data
for a different office that's going to
be a different permission structure to
say being a manager that owns all of
them and having full access to
everything so there's a bit of a complex
relationship that needs to be handled in
these permissions which is why that's
not the whole of it it's actually this
much so the permissions really kind of
give an idea where the real core of this
app's purpose is it it is designed to
manage and control this the
straightforward business logic of those
previous ones really comes down to the
nuts and bolts of how do I make sure
that I'm handling the customer data the
correct way for the users requesting
stuff in the app and consequently when
you actually have all that noisy code
you kind of want to you want to make
your life easier you have a lot of
helpers for this kind of stuff you know
things like a bypass mode because if
you're running a management command you
don't want all the queries in the entire
thing complaining because you didn't
give the management command a tenant you
want to just have that code run on
everything hence we start making little
shortcuts here um there's actually a
nice little thing there at the bottom
there talking about Sentinels there's
actually currently a Pepin proposal for
adding Sentinels as a built-in Python
language feature I think they're awesome
it's kind of one of those little things
you don't appreciate until you need it
now in this particular case and enom
would have done the job but enom weren't
a language feature fully integrated at
the time this was actually built so it's
a case where Sentinels were the
appropriate solution there
now it's another talk for someone else
to do but Sentinels are quite a nice
feature um there are more tenant helpers
you know things like requesting which
tenant came in based on HTTP headers you
know all the sort of standard SAS logic
stuff it's you know the kind of thing
you write lots of these for uh but once
you're done writing it all we get to the
testing and and this is where the uh the
problems begin because management had a
particular philosophy around this choice
to make it reusable it's it's we put it
to the side so that we can test it
really well and I not kidding I actually
did an entire talk about the stuff I
learned to do the testing for this app
because it was that extensive it was
very extensive you know cross database
testing you know we're running the local
tests in sqlite but is it actually going
to work the same when we run against
postgress they wanted to make sure so we
kept building tests I even ended up
making a better hypothesis test because
uh look. net.org is nice but you know
what if we break paing on a do you know
NL you know somehow someone manages to
you know mess something up and break the
tenant selection logic and trick our
system no we can't have that have to
test it properly yet more time time
spent learning how to make this work and
actually building this and getting this
to actually do its job and I actually
ended up turning it into a pull request
and submitted it Upstream because it was
that much you know work getting it all
to run performant and again yet more
time burnt testing now in the end there
is a decent size test suite for this and
it does all pass and it's pretty
extensive coverage too there's a few
places in there where you know I never
could work out how to get it to do those
last one or two lines but overall
extremely high level of test coverage as
mandated by management and that's kind
of where the mistake really
lies because management wanted to gold
plate
everything and um yeah gold plating
everything is not necessarily a good
idea
um now if you're not familiar with the
idea of gold plating something it's kind
of the Monster oxygen-free Cable kind of
a a notion it it may serve a purpose but
it's probably pointless it it's a
wasteful Endeavor it serves some level
of use in the sense that a goldplated
connector will wear out slightly
differently and won't oxidize and it may
have a purpose but you don't gold plate
everything you gold plate the things
that need to be
goldplated but the the planning led to
gold plating everything and that's the
real context criticality in the middle
of all of this that just like an
abandoned subway station here this was
built for a purpose but that purpose was
never realized because the larger
context in which it was built wasn't
taken into
consideration what ends up happening is
that lots and lots of time is sunk
building this reusable core logic
testing it to the ends of the Earth and
back but the rest of the project moves
on the front end gets developed and
management are looking at a front end
that works completely but has no actual
data behind it and the rest of the
system hasn't caught up yet because this
is still being built and tested which is
very much a case of premature
optimization this was prematurely
optimized for a security posture grossly
out of proportion for what it was
needing to do and consequently it fell
to this Maxim and this is a pretty
common Trope you know there there's a
reason that this Trope actually goes all
the way back to Shakespeare because
there is so many ways that you can trick
yourself into thinking that you are on
this right path you will be looking
forward you will project yourself to be
a year from now and think I will need
this in a year it's smart to do it now
but it's not actually smart to do it now
if what you're trying to do is something
in two months what matters in a year
from now is going to be something you
should be thinking about six months from
now not now because if you're literally
building from scratch a prototype like
this was going that far down the line is
what actually led to this never actually
being used and this code being
completely wasted because the end
outcome of this was that it was all
thrown in the
bin and consequently all of the time
spent making and testing this the couple
of months solid spent building just this
one piece of functionality and for the
most part the time that was spent
building the other functionality around
it all just became pointless because at
the end of the day management were
unhappy with the performance of a
critical part of of the app and this is
why it was thrown in the bin because as
the front end was developed it got to a
point where it was fully functional they
were happy with the complete design and
with that design complete it became time
to plug the back end in and Jango rest
framework can make this work with MJS
it's a model driven system you match the
models up and the API just takes it over
for you not a lot of extra logic
required it's really nice like that so
we get the front end ready management a
happy play with it and going like when's
it going to work for real when's it
going to work for real when's it going
to work for
real but it's not going to work for real
until we're done doing all this other
stuff and even then we get to this end
point where we've managed to do this
we've delivered this it's taken longer
than they think it should have and then
we h a little bit of a snag because any
kind of quick prototype built this way
is going to run into little problems and
one of those little problems was that
there was a very important function in
the middle of it that involved a very
large list
and by default the way it did that list
update was sent one API request for
every single item in this list and some
of these lists were hundreds long which
meant that the browser would bottleneck
on the round trip because it hit the max
queries and so consequently it would
just sort of sit there and take far too
long for management to be happy with and
that became priority number one and so
we dropped everything and started fixing
this now at this point all of the work
has been done the app is functional the
only issue is a single page performance
optimization but getting it sorted out
making sure that we're doing it the
right way checking that we haven't
tripped ourselves up somewhere else and
doing it a way that doesn't break
something else because any kind of a
crash project you take a few short cuts
and you find that you actually do
something later that breaks the thing
you did weeks ago we have that point
where we kind of looks like we're doing
nothing and after a little while after
about you know 2 3 weeks of doing
nothing management are like this is
clearly fa
bin and the project eventually gets
Rewritten in a completely different
language another team that's how
business works but amusingly enough in
this particular case there was a bit of
a salvage job that could be done later
because eventually it came time to do
some other Project work and the needs
actually lined up quite well with the
entire rest of what was built beyond the
actual multi- tendency so what was able
to be done was was possible to
completely rip out the multi-tenancy and
just use the skeleton of what was left
behind from before to deliver this
functionality faster and you know in a
weird Twist of irony the fact that it
was a reusable app made it easier to rip
that out and use the rest of the
project because it wasn't embedded into
the entirety of the project so if it had
been done the correct way and actually
just built as part of the projects logic
instead of in a reusable app it would
have been harder to do that
but overall the fact that it could be
salvaged doesn't actually change the
fact that all that time was
wasted and given that this is a pretty
broad sort of Journey there's plenty of
time for questions for people who want
to dive into the specific details of How
It
unfolded Fabulous uh I'm happy to run
around and get the mic to people who
would like to ask a question please
phrase your question in the form of a
question
here we
go what language was it Rew written in
cop what
here great talk and what was unique
about that management in wanting to go
plate everything it it was a security
concern and and and not not a completely
unreasonable one just uh the the
mismatch between the technical
understanding of the potential risk
because there was a bit of an insurance
industry angle on some of this stuff
there there was a a desire to be up to
Snuff as far as an insurance company
would want to see so so there was a bit
of a bias towards needing far more
guarantees and security coverage than
they really needed in a
prototype I'm sure somewhere on the
project team someone spotted that this
was going to be a train wreck early
on what did the project team do to share
that message and why was that message
not received the the the project team
was two
people doing two different
hes so no no one saw it
coming
here I'm got to invoke that word do you
think this is a good uh argument for
Agile
development honestly yes because it
would have been easier to recognize that
the mismatch was occurring had there
been regular check-ins like that instead
of a more overarching project Milestone
based
approach is everyone
okay everyone still has their jobs and
uh you know no one's lost any work from
it that is a great question particularly
in this
track uh what would you do differently
now if you to do well I would definitely
have Incorporated it into the actual
project as opposed to a reusable app
because if this had just been part of
the project itself the integration to
make it work with all the functionality
was required would have been a much more
straightforward process evolving along
with the rest of the functionality
instead of having to be built with all
the extra hypotheticals attached to it
because as as a reusable app there was a
lot of sort of reusable app mindset
involved uh many of those helpers were
only used in the test Suite because when
it was built the thought was H I don't
know if I'm going to need this later but
I'm here building this now I should
build this now because it's reusable I'm
going to make sure it just covers all my
bases and that's leads to scope creep in
that
sense um uh I think uh we we are also
embarking the same Journey you know you
just mentioned but um uh what do you
suggest you know how uh how not to
fail ah the question of the day um yeah
no in in terms of you know trying not to
fail the lesson out of this is is to be
mindful of that context because the the
context that it was built in was where
the real mismatch occurred if if the
context had been appropriate and the
goal was to use this in a lot of other
applications in a larger organization
this approach may have been perfectly
sensible but that wasn't the context the
context was it was being built for a
single specific Django project that was
going to be used as a single Django
project in a SAS context it it didn't
need this kind of reusable thinking and
that context mismatch is what led to the
failure
we have another one over
here uh presumably there are like
multi-tenancy uh libraries already in
Django uh did you explore those why
weren't they up to stuff a lot of those
a lot of the existing ones tend to do
schema based isolation or multi-
database switching because those are the
kind of easiest ones to share um in
particular for this one uh this is
uh heavily improved from a very very
minimal prototype I found out there
which is part of the reason why the
license is all squirly but the the core
of it being that kind of extra rigor
required to do this kind of everything
lives in the one schema tends to lead to
people not building it as a reusable app
which back to the last question there is
that if you were doing a project like
that you normally make it part of your
project not built as a reusable piece so
for this specific case no actually there
wasn't
any okay I'll go here and then here and
then
here thanks it seems to me the main
takeaway of your talk in the project was
the the B you covered very briefly about
management getting impatient with fixing
a problem that took a bit too long like
what what went on there in terms of
communicating you know this is fixable
it'll take a bit more more time you know
this I don't know I mean like all the
other stuff is is clearly useful but
it's if that's where the project failed
well you're not wrong um there there I'm
I'm kind of tactically attempting to
change the names to protect the innocent
so to speak um but
the the more broad detail on that is
that the time that took from management
being happy with the Prototype of dummy
data to delivery of a real system that
they could actually touch and save data
to and begin trying to run through the
cycles with was about 2 months so it was
about two months of them staring at the
thing that they were completely happy
with but just had completely fake data
in it and actually going to use it
during which time visually for a
non-technical management trying to
design something based on their
understanding of their own business they
sort of perceived a standstill while
work was being done behind the scenes in
a sense this being done front end design
first backend plugged in afterwards
while a feature of emberjs that made
that work well kind of also contributes
to the management project delivery
mismatch that kind of culminated in that
final point of look we've waited this
long now you hit another
snag uh you can only string that along
so so far before they start to decide
you know what we're going to cut our
losses on this
one okay we
great talk um do you have any specific
suggestions on how to talk to management
to avoid a situation like this when you
see a context mismatch and they don't
especially if they're um they don't have
technical background
well I suppose that's the thing I've
kind of taken from this myself is is
trying to be more mindful of the context
and just to apply that regular sort of
talk to management skill set that you
know I have work as an independent
developer but the in in this specific
case having not recognized the context
mismatch there wasn't really an
opportunity for me to bring that up
because I hadn't seen the problem
yet but the regular skill set of if if
you do see a problem like that you
should just bring it up to the
management you know to the best of your
ability it's the soft skills component
of of learning to work in this kind of
an industry if you have that kind of
role where you have an ability to speak
to
management that's that's part of the
skill set you need to have to do
it okay I think we had one more question
which is I think we're going to have to
call it there here we go I don't know
we've got a hiveel over here but I was
thinking along the same path and I guess
sort of Management's mentioned a few
times I know there's some developers
involved as well and I guess I was kind
of curious about like where they were
you in-house developers where you're
external I'm sort of trying to work out
the motivations of both sides here so
certainly when you know like wasn't made
clear to manager that you know this is
is a front end there's a lot going the
background like those kind of project
planning aspects as well which I know um
I sort of get the feeling everyone in
this room is very bright and can develop
things but actually getting it to
getting people who don't understand what
we do on board and making sure they
understand the timelines and where the
importance is and you know were the
developers at the time even if we regret
it now was it sort of just in the
mindset of just we're going to make the
thing they asked us to or was it sort of
in the mindset of sort of trying to
solve that problem if you if you get
what I mean there was a little bit of
both um you know I I I was the senior
developer on this one um there was an
inhouse Junior uh in that particular
situation I was being brought in as more
experienced uh to sort of steer things
and do other work for other projects had
going um the the overall cycle was
actually kind of broken in those two
halves there was the development of the
front end discovering the users needs
and actually actively communicating with
them to try and find out what they
wanted and to build the best app
possible for the job they needed it done
but where the mismatch came in and and
sort of what that you know I think
you're getting at there is is that there
was that mismatch in terms of whether or
not it was fully understood by them what
was going on in the second half and and
while that was outlined as part of the
planning the amount of time that passed
in between those two steps may have led
to some sort of misconception creeping
in that wasn't identified until
at the end of the actual outcome they
decided that you know their perception
had shifted significantly from what it
was originally in that very interactive
phase for them and that that's sort of
more of a social capital kind of a
problem in terms of trust in the actual
relationship you know you deliver it
very very quickly trust builds they
start to see that things aren't being
delivered at the same rate there's going
to be a gradual erosion of the trust in
terms of that management communication
flow and and you have to be mindful of
that in terms of you know that
communication pathway and making sure
that you know they are actually on board
and do understand but once there starts
to be that sort of misconception and
that slow gradual loss of the Social
Capital they will begin to question
whether or not you're actually giving
them and feeding them Al line and you
don't really have the thing and at that
point it's all going to fall apart
anyway all righty that was quite a
story very interesting thank you we have
a special gift for