[Music]
hello
everyone um I'm very excited to be here
at pikon you today I attended this
conference uh three times in the past as
a as a listener and today is my first
time participating as a speaker and this
is also my first time speaking at a
large
conference and um as this is my first
time speaking let me start by
introducing
myself so I'm artam um I'm one of the
software engineerings at um software
engineerings directors at cover genius
however uh one of the stickers on my
laptop says developer and this is how I
prefer to identify myself at least for
the purpose of a Tech
conference um I joined cgus in 2017 when
it was a small startup of uh around 20
people and help it to grow
into uh a global leader in embedded
insurance with uh over 700 employees uh
worldwide today overall I have around 15
years of experience in software
development and most most of it uh
building backend
applications and uh during this time I
had opportunities to lead um a few few
different teams including product QA uh
devops and um infrastructure teams I'm
also one of uh C Jango uh Meetup Group
organizers and as as I have this
opportunity today I want to promote our
uh User Group um if you live in Sydney
or nearby we have our next event on 5th
of December so uh it will be at
Microsoft reactor so please come uh say
hi to me and other Jung engineers in
Sydney uh on personal front I a husband
and a dad of two kids my da is almost
nine and my son is three and a half year
old um and as you can imagine uh work in
family keeps me uh pretty busy but I
also try to find time for my hobby which
is martial arts I have been practicing
Brazilian J Jitsu for many years um and
Judah as well um and I like watching um
UFC and other martial arts competitions
at my spare
time uh my today's talk is based on my
experience working uh in a team that
developed uh excore application at cover
genius um and excore is a central
service of xcover
platform uh it is a Jango andf API that
distributes embedded uh protection
products and cover genus Partners use
this API to integrate those products in
their websites uh mobile applications
and other platforms
um we started excore development in 2018
and the fun fact is that uh only one
engineer out of three in the original
team had previous experience with Jango
uh and it wasn't
me um despite this fact we were able to
launch our first production integration
already next year and then one more in
2020 before pandemic nearly BR C
Virginia's business to
hold it was crazy roller coaster
throughout 202 20 and
2021 um but uh in 2021 we were able to
launch 20 new Integrations and a few new
business
verticals and it was the year when we
also started using multiple databases uh
in
excore migrating from one database to
multiple uh helped us to continue scale
excore and
um um and take take on board even more
Partners um in 2022 cgus doubled the
number of partner uh Partnerships and
the growth continued throughout 2023 and
2024 making next cover one of the global
leading embedded Insurance platforms uh
today with hundreds of Partners and
millions of global
customers a couple of words why we
choose
jungo um the alternative approach uh the
the alternative proposal that that I put
was to use flask and SQL alchy I was was
more familiar with this stack as I use
it in the previous application and um I
guess back then we we barely understood
the complexity of the system that we
were about to start
building uh but one of the things that
we knew for sure if the project is
successful um it will grow FAS and we
will need to scale it so before
committing into using jungo we run a few
performance benchmarks to ensure that
jungo wouldn't be slower than flask plus
scale alchemia and I um still have helps
in those results that received but uh in
our configuration Jango even slightly
outperformed um the other stack uh and
of course we considered many other
factors but one of the uh major
arguments was that um Instagram was
originally built with jungo um so yeah I
don't know how it sounds but in the
startup World um the time is very
precious so making um good decisions
fast is a key to success so looking in h
side I can confidently say that choosing
Jango was the right decision um Jango is
a great framework and one of the
features that proved to be very useful
for us is support for multiple databases
and it is also the topic of my today's
talk I will start with explaining our
motivation for using multiple databases
in
nextcore as I mentioned in 2021 we had a
a few new large Partners integrating
with xcore and several new verticals and
some of these
Partners had quite strict uh
requirements for througho and response
time and considering the complexity of
the business logic um it was not a
trivial problem to scale this
application um the computer resources
were not a problem the application was
running on kubernetes cluster and was
able to scale horizontally using
horizontal Port Auto scaler however the
bottleneck was the database uh we didn't
have another way of scaling it apart
from increasing the size but um it would
be a costly
solution um and more we still could
potentially hit certain limits um for
example dis copes um so we started
working on optimizing database workloads
we notice that um quot operation
contributed around 85% of all sccore
traffic and it gave us the narrow place
we could focus on WE analyzed the theb
work CLS generated by code operation and
noticed that from the DB perspective
there are two large steps in the
workflow in the first step the operation
would uh read various uh configurations
from database such as product
configurations product selection rules
partner configurations and so on and
this data then used to generate quote
objects which are calculated and
recorded back in the
database uh and um this quotes table uh
was the part of the same database where
the configuration data was stored so
apart from quotes we also had other
types of generated customer data um such
as transactions uh customers uh policies
Etc so the 10 tables in the database um
stored a several terabyte of data
whereas the rest 60 tables stored only
100 Megs of um configuration data in
total um so that was the the the reason
number one we saw if we could use a
separate database to store configur
option we could potentially aflo a
significant amount of read operations to
this uh new database increasing overall
capacity of the
system um the other important
consideration was the future ability to
implement Regional deployments as cover
genius uh is a global business it was
quite important for us uh to be able to
store customer data in different
locations um as required by different
insurance
regulations um so extracting
configurations into a separate database
also enable the team to implement uh
Regional deployments while maintaining
one centralized set of configurations
for uh all
regions finally splitting database would
make it much easier for us to add more
uh read replicas uh of configurations
database if necessary and increase
capacity in this
way um so yeah this is what we did
slightly later um using multiple
replicas helped us to increase capacity
and improve availability of the
system all right now I'm moving to the
main part of my talk um by the time when
we realized the need for these changes
we already had quite significant volume
of traffic in production um excore was
already serving million millions of API
requests uh per day um and uh in this
part of this talk I will speak about the
step-by-step process we took to split
the database
two so the step number one was to plan
everything uh it was important to
analyze the tables to understand better
the workloads data flows and
dependencies between models we reviewed
every table and split them in two
categories configurations and customer
data the diagram on this slide is a
really simplified example but in reality
we had approximately 70 different tables
um so it was a bit more complicated
uh we had to analyze every single one of
them and make that
decision
um so the next step was to remove uh
existing foreign key constraints in the
database and uh
Jango um can support foreign keys that
refer to a table in another
database perfectly of course there are
certain limitations with this approach
for example uh joining data from
different databases in a single quer set
won't be possible
uh but the cool thing is that most ofm
functionality would still work um the
only required change is to set this uh
DB constraint uh option to false as you
can see in this
slide
um we were not worried about data
consistency as all of the configurations
are immutable so that was a a simple and
safe change that we could deploy in
production straight
away uh now in production we had all the
tables divided into separate categories
without any relations between uh between
the two
categories um so the next step was to
set up a local development environment
um at cover genius we use Docker compost
for local development and we didn't want
to add another uh postgress container in
the local environment so we just decided
to use uh logical logical databases
inside the same postgress container so
this is an example of script that
creates uh multiple uh uh databases for
us
um and uh to start using the script uh
we only need to mount it into correct
directory and set the required uh
environment variable and well we could
use uh multiple databases in the in
local
environment um now we need to instruct
Jango on how to connect uh to the
individual databases it's pretty easy to
do uh we just need to add one uh new
connection to the databases dictionary
in our settings.py this is not a risky
change either and can be easily um
deployed in production immediately and
the reason for this is uh by default
junga assumes that only default database
connection exist um and to start uh
using the new connection in junko there
are two ways the first one is uh by
explicitly calling um a using method on
query set object and the second option
uh is to delegate that decision to a DB
r about her um in a large application
the second option is um probably always
better um as it requires less code
changes and it also probably more
maintainable
too um so the next step was to add a DB
router you haven't used DB routers in
junga before no worries it's pretty easy
um DB router is a simple class uh that
makes it possible to hook into internal
orm logic
and instruct jungo omm on how to route
the
queries let's have a little bit deeper
look at each of these
methods so firstly we need to decide how
to route read and write queries both DB
for read and DB for write methods uh
take this um uh model class as an
argument and uh return the name of
database connection to
use um and uh if we need to store um
different models in different databases
one simple way of implementing this is
through uh maintaining a mapping between
the model name and the connection name
to use um so this piece of code
demonstrates potential implementation
but our actual implementation was more
complex uh for example we can Define the
databases per uh jungo application or
per uh individual
model we also run some checks on Startup
to ensure that each model um in the
application is explicitly assigned to
one of the two
databases so next we need to um instruct
the router if it should allow relations
between the two objects uh allow
relation is simply a validation method
that is used inside um inside jungo
framework in a few places uh but as we
store individual models in separate
databases uh we need this method to
explicitly return true otherwise we
would be uh wouldn't be able to use that
was um uh using foreign keys with DB
constraint
fals um so the last problem to the
router solve is to decide what
migrations should be applied to what
databases the um allow migrate method is
um used um just um for this uh it's
called for every migration operation and
uh returns a Boolean flag indicating if
the migration should be applied to the
given database when multiple databases
are used to store different models we
need
to uh run my great command for each of
them
separately um and the potential
implementation is on this slide but uh
but please not this special handling for
use case when both connections refer
point to the same database
instance um J needs a separate
migrations table in each database it
manages to track uh both migrations are
applied and and we when we point both
connections to the same database there
is only one migration stable uh and all
of all of the changes to all models must
be stored there uh so the correct way of
handling is to apply all migration and
this is what uh the return true part
does here um when migrate command is
called on the second database connection
it will just happily SK skip all the
migrations as the do already presented
in the migrations
table
um one more caveat is for using run
Python and run SQL methods inside
migrations in this case um model
name uh argument passed in the allow
migrate method will uh contain none and
the only way for the DB router to know
what database um to use in this case is
to to instruct it explicitly by uh using
hints
um so yeah we had to review and PCH all
run Python and run SQL migration uh
operations and ensure all of them have
correct hint argument provided uh
something like on this
slide uh now once DB router is ready we
just need to enable it in the
configuration we need to specify it in
the database router setting and as you
can see um it is possible to use
multiple DB routers
um and junga can chain those calls if
required and while I could think of a
potential use case for this I think it's
probably cleaner and easier to maintain
a single DB router for all models and
this is how we implemented this INX core
to um so now introducing DB router is a
is a very significant change uh be very
careful and expect things to break if
you don't have a good code coverage I
would recommend you not not do this at
all
uh we had a very good test coverage for
excore um all critical operations um
were covered 100% and uh we had around
3,000 tests uh back then um and around
80% of the tests that started failing
after this
change luckily it was easy fix um for
many of them as um you know um it was a
simple configuration change um so by
default Jenga assumes that only default
database is used um
uh and
uh testing framework that um you know
and basically when you test the code
that requires multiple databases it
requires a special handling we need to
explicitly instruct jungo testing
framework to run um migrations on
multiple databases so um in excore we
use both junga sty uh test cases and P
test with jungo test plugin and luckily
it it was um a simple change for uh both
setups um the other reason for fail test
was the usage of Select related
optimization um so storing models in
individual databases uh we cannot use uh
select um so select real is not usable
anymore uh this op optimization operates
at SQL level and the tables um they they
they live in separate databases um and
it is not quite possible to use as
joints so instead of Select related we
switched to prefetch related where it
was possible or we did some other
refactoring to achieve the required
performance finally the last group of uh
failed tests uh was caused by um using
transactions uh the transaction methods
such as atomic set roll back and uh on
Commit uh they all implicitly use
default database um and for excore we
decided that default database will be
the one that that stores uh
configurations and in most uh of the
views we uh it was the not the desire
desired logic so uh we had to change it
um luckily that was the the last change
after pushing it into
um gitlab we could see we could we
started seeing green uh CIP plants it
was the time to test uh this in staging
environment
so deploy deploying the change was not
overly complex um we decided it wasn't
worse to automate end to endend as it's
uh just a one-off operation so we
originally deployed all the router call
changes but made uh both connections uh
pointing to the same database as I
already
mentioned and
um um yeah and after this we posted all
rights to the configuration uh tables in
the old database and created a dump of
the required tables which we loaded into
a new database um to avoid mistakes we
created a jungle management command that
generates um correct PG dump statements
with all the required uh taable
specified and after this we updated
database
configurations uh to use uh connections
to different database and that was it
after testing the deployment process in
staging uh we repeated the same uh
process in production and we had zero
down time throughout the process and no
data
loss uh later we implemented support for
read read
replicas um it was also an interesting
project but um I will leave this topic
for another day um on this slide I would
like to present the final database
architecture that we've been using until
last year um and last year we made
further changes we started using no
scale database for uh storing CS
data and uh next year we consider even
more changes as the number of AP API
consumers uh continues to grow and we
discussing such topics as isolating
sensitive data into a separate database
and um also
introducing one more for data
archives all right um what did I want to
achieve with my talk
um first I wanted to share our
experience of scaling the jungo
applications by adding multiple
databases potentially it will be useful
for someone
else um second I wanted to showcase the
junga um the junga functionality for
supporting multiple databases and I also
want to use this opportunity to say
thank you to all the amazing junga
contributors who made this framework so
awesome and also to the Instagram
Engineers who I think Pioneer using
junga for high loaded
applications my third goal was to make
um you think more proactively about the
database architecture in your own
projects uh and maybe my talk could give
you some clue or um QE or
inspiration uh but remember using
multiple databases always um almost
always imply additional complexity in
your application so this complexity
needs to be justified uh uh for simpler
cases consider using other ways of
scaling database for example using fully
managed DB cluster RDS multiz cluster
deployments uh for
example um and if you would like to ask
further question about this or related
topic please don't hesitate connecting
with me on LinkedIn uh I would love to
help if I can um in the end I want to
say thank you to my amazing team at
cover genius it's been great pleasure to
work with this team over the years and I
wouldn't be here presenting this topic
today without my
team thanks to all who listened my talk
today I'm happy to answer questions if
[Applause]
any okay uh thank you and now we have
some time for questions so uh if anyone
would like to answer question please uh
put your hands up
excellent hello and thanks for talk so
previously I heard you mention you have
like three, tests for this uh Dango
application so how do you write this
test did you use some autogeneration
tools for writing this
test sorry how do I do yeah because you
you have three, test so how did you
write this test did you write them by
hand manually or you use some generation
tools for that yeah we've been working
on this application for seven years now
so you know um so four years ago I think
these days it's around 4 and a half
thousand tests yeah we write them
manually um um yes we use uh some
parameterized test so um I'm actually
not sure if this is number it's probably
number with all the different variations
uh that that run as well
yeah um thank you uh just a question on
any thoughts on um indexing and
partitioning you know um yeah because uh
we I think we are in the same situation
as well so you know I'm I I need to
explore for um and you know get some
advice on partitioning in Jango
mhm um yes so
partitioning yeah we we explored options
with partitioning as well um basically
um you know the data that we need stor
is probably um still can fit in one uh
DB uh server so we because partitioning
would mean that you
would need to increase the complexity of
your app even further so for now we uh
decided not to do this unless it's
absolutely necessary so we have a way of
uh removing all data from the database
so there is a um you know process to
clean up um you know some of the data
that uh we don't need anymore um but um
yeah I think these days probably we have
around
um 10 to 12 terab of data and we also
considering uh switching from posr SQL
to Aurora uh for um yeah but probably
don't want to do
practitioning this is what the team
decided good day um just wondering if
you'd done any benchmarking and thinking
of the keynote particularly um looking
at what uh how your performance was
before you did the change how how it's
gone afterwards what things made you do
things like the read replic
because
um yeah we uh we have
a
um we have performance testing uh
framework we use
um oh this
days D just vanished from my head
this
um yes
um
uh yeah we have we have a suite of
performance test that we that we still
run uh we don't we don't run uh them as
a part of CI CD pipeline it's a um we
have automation we on this uh some of
the tests were on weekly
um um yeah and
um um The Benchmark I mentioned it was
even before we choose the web framework
uh it was a seven years ago I can't
remember again like what the framework
we use but um but yeah I yeah we can we
can stay after the talk and I yeah I can
speak more about this
yeah okay do we have any further
questions yes we have a couple
excellent um if you're if you're willing
to say on on on the record uh how long
do your 4,500 tests take to
run okay looks like everyone is
interested in testing topic um yeah we
we we run them in um I think four uh
parallel um CI jobs uh now um and uh I
think it's like 15 minutes something
like this for not not too
long cheers um yeah thank you it's a
really interesting idea um but I can
imagine that like losing that ability to
kind of do joins across uh different
tables in the would be perhaps a little
limiting have you found that to be a
pain um not in that application um to be
honest um joints when you have lots of
data probably using joints you know not
always uh effective and and you still
use uh joints within this uh domains of
data right so you have one domain of
generated data in which you can use
joints um and um and then for all the
analytics we never use the application
databases so all the data is pushed to
bigquery so in bigquery um it's a still
all all data is presented together so
for the analytics you can you can use
jins um as as you want but
yeah okay uh thank you very much um I'd
like you to all uh help me to thank RTM
for that uh excellent talk