welcome back to the devops track
everyone i'm justin warren your host for
today
and our next talk is called planning for
failure using chaos engineering and i am
pleased to welcome our presenter amit
sahar amit
thank you for presenting here at devils
take it away thank you
hello everyone my name is sumitsa and i
work for atlassian as a software
engineer
today i'll be sharing my learnings and
excitement about using chaos engineering
techniques to help plan for failure
scenarios in software systems
um this is the best kind of talk for me
uh because
i'm going to fumble a bit i'm going to
stutter a bit
but we'll get there then
so yeah let's get started
now
now that you're here
why should you continue listening
listening to me
in the recent past i worked with teams
to assess their production readiness
we are rolling out several new services
we had alerts we had dashboards but we
wanted to test the various
configurations help team members get
familiar with on-call hygiene
so this team members were fairly new to
the company
so they may have had prior experience
with being on call and
for the services but
the tools
the service that we had in-house were
quite different from what they were used
to before
so we wanted to get them uh you know
sort of warming up almost for their
on-call
schedules
atlasen had a very well-developed
culture around chaos engineering
practices we call it war games
internally and i adapted that as my key
strategy for this exercise
and i became a fan
of applying chaos engineering techniques
for this purpose and so i want to spread
my excitement and share my learnings
today
i will start my talk with a hypothesis
and by the end i'll have made enough
points towards proving that hypothesis
and the hypothesis is
chaos engineering techniques can help
continuously assess your production
readiness
but i'll not declare it proved or
validated by the end that's for you to
find out for yourself but i'm positive
you will find that's the case
i also like to think of production
readiness
as failure readiness since in my head um
and probably all of your heads
production
is
pretty much failure in some shape or
form whether it's point zero one percent
or whether it's point uh double zero one
percent or
um ten percent
we have uh
sort of i have divided the talk into
four key areas we'll start off with an
introduction
we'll then go over organizing chaos
engineering experiments
we'll talk a bit about people systems
then we'll talk about chaos engineering
tools
and then finally a summary to really
drill down the chaos into all of you
let's get started
so the best light plants often go
alright
and i have lived in sydney long enough
um to see enough of these that the next
time i see it which will be after a
pretty long time because of obvious
reasons
i will probably be surprised that okay
yeah we're back again here
um so this is from a new service that
was rolled out for those who are for
those of you who are not from um sydney
uh this is the sydney metro and uh which
was rolled out with a lot of fanfare
after a lot of planning uh with with
sufficient testing for that matter but
within days we had issues products so we
rolled out something into production and
we had issues
and that's why we have the train
replacement buses
always ready to go
um so getting closer to the conference
topic systems fail
uh we have hardware failures we we have
material course course that don't count
so check out this paper called silent
data corruption which is a paper by
facebook and related work by google
where they found that a task
only failed on a specific core
closer to home we perhaps don't
experience not most of us experience
mercurial cores but we do experience
exhaustion of finite resources memory
disk space cpu cycles network capacity
we are all too familiar with this
and then we have software failures
bugs
where we may not have tested a code path
properly where we may not not have
foreseen something breaking in a way
that it broke
and then we have real world failures
breaking underlying assumptions in
software
and then people during an incident the
on-call person finding out that they
don't have the necessary access to
deactivate a malicious account
and i can bet that
a lot a lot among you has actually
experienced this
and hence we put in circuit breakers we
have red limiters
we have
exponential backups in retries in our
code we put in redundancy
we know we want to run more than one
copy of our application
we put in disaster recovery plans
and we create runbooks for people to
follow
and we also have our fail safe which is
okay if everything else fails we'll
still um
have we'll say put in you know four
nines and if we don't meet those four
nines we'll
pay will sort of compensate our customer
in some way so we essentially throw
money at the problem uh by signing sls
we don't even tell them that yes we are
going to be hundred percent off for you
we we factored that in
and yet
the next incident happens
and catches us by a surprise
alerts didn't fire
auto scaling didn't happen
with a question mark
i thought we had circuit breakers oh and
we just had a slow breach and if you had
any doubts about which bike on a u track
you are in you're in the dev oops track
so what can you do um
we can borrow some ideas from the
scientific method
the scientific method
essentially is
it's a process an ongoing process where
we question we ask questions we research
an area we form a hypothesis
we test that hypothesis with experiments
we analyze the data
we report conclusions
uh
come up with action items and then we
repeat rinse and repeat
so
we are going to put our our science hat
on and we are going to learn how we can
organize chaos engineering experiments
the basic idea is we want to
intentionally break our systems under
controlled conditions
we break the carefully orchestrated mix
of sync and async workflows and verify
the guardrails
what do we break
we break our finite resources um exhaust
the cpu
exhaust the memory exhaust file
descriptors
we
we we test the network capacity at
different levels
we
we think about breaking our dependencies
our libraries our services the data
stores that we that our application
depends on
and any resource your software really
depends on
your logging your monitoring your
apm the application monitoring
management agents i believe i got that
abbreviation correct so anything that
you're running on the system
and your application is talking to
directly
should be a potential uh fault injection
point
so it's a it's really becoming curious
about your system and uh becoming the
bad
bad actor
and thinking of ways you can break it
so for example let's say you have
implemented asynchronous workflow in
your application
and
introduce timeout which is longer than
your polling duration
does your app keep pulling forever
let's say you have webhooks in your
application receive the webhook
correctly
but drop the payload what happens
does something tell you that
something like this has happened or you
are basically left scratching your heads
thinking
you know what's going on
and a final example let's say you have
enabled um you are using whiskey
applications you're using something like
unicorn or uw sgi you have enabled
worker recycling behavior and you have
been careful enough to set up a jitter
so that your workers don't restart
all that don't always start the same
time
test that send traffic to your service
at an expected request per second
how does the worker recycling behavior
manifest
um does
with that rps
are there enough workers
um there to serve requests when the
others are recycling are those
sufficient um kill a worker periodically
see what happens
a formal chaos engineering experiment
consists of four key steps
you you start off with a steady state
hypothesis
then you
have the fault injection phase
then you verify the hypothesis that you
started off it in step one and then you
identify action items and then you sort
of rinse and repeat
um as an example
let's say you have two services
service one and service two and service
one depends on service too
so we formulate and hypothesis saying if
services does not respond respond within
300 milliseconds service one is going to
return an error within
close to 300 milliseconds it won't be
exactly 300 milliseconds because
think bits and bytes take time to travel
so it will be close to 200 milliseconds
and then so that's a hypothesis
and then you inject a fault you say i'm
going to add a latency of five seconds
uh between the service to um and the
database so the application i've got the
application text here it should be
service too uh so essentially add a
latency of five seconds between service
to and database did the application
return an error after 300 milliseconds
and
in this case the application is service
one
so that's your hypothesis verification
phase or did it wait longer
um if and depending on the answer to
that question you come up with action
items if it waited longer you should
check out your timeout configurations
and repeat the experiment to ensure that
the timeout has actually solved the
problem or your and that it validates
your hypothesis
now
in in in a more production-like scenario
you'd have you would want to verify
whether your alerts fired or not as well
telling you that
yes this service one is
certainly experiencing higher latency
that's an example
so
now there are
three key steps before you run a chaos
engine experiment the first and the most
important step is to brainstorm the
experiment and formulate your hypothesis
that is step one
from the previous step
what you will really find useful and
what you will really need to design good
experiments
is a very uh good idea of the system
that you are planning to inject faults
in
system design and architecture knowledge
is essential to come up with interesting
experiment ideas because it's easy to
come up with very
generic
uh experimental ideas and those are good
places to start but eventually the
biggest uh bang for your buck almost
will come to say almost will come from
uh if you are designing this key
experiments so for example um let's say
you have an unexp i have an incident so
that instant could give you a good um
that could be your uh
seed to come up with this experiment
perhaps you could inject the same
failure that that was that that caused
the incident
and uh for for your chaos ensuring
experiment
so that this is really a key step
um i will just jump over to the next
slide so
now this is how we do planning at
atlassian
well i'm just kidding this is not how we
do it um
so so this is a a miniature war gaming
um you know
like a picture and but this is roughly
what you should so this is uh a term
there's a term we use internally called
tabletop wargaming um i learned it from
a colleague and what the term means is
you get together with the team your team
or the maybe the architect if you have
one maybe the senior engineers in the
team and you sort of brainstorm ideas to
break your system
and you do that on a whiteboard you do
it on a table on a virtual whiteboard
these days and so you lay down all the
pieces you lay down all the dependencies
for example you later on the different
interactions of your system with your
external systems and then you think
about breaking them
so once you have got the experiment once
you've got the hypothesis the next step
is how do you inject those failures so
choose your tools
uh we will go over a few tools
later in in the presentation
but sometimes these will be quite custom
tools or custom code that you will have
to write maybe perhaps modifying your
application in a way
to inject those failures so it will be a
mix of both generic tools as well as
something custom that you will have to
implement for your application
so once you're done with
the brainstorming you have chosen your
tools the third step is scheduler
meeting with all the key people so for
example people who would be on call the
team leads architects and
typically you'd block around one and one
and a half hours
for conducting the experiment as well as
then coming up with uh you know
analyzing the outcomes validating your
hypothesis coming up with the action
items um and uh just
sort of doing uh
uh like a comparison between expectation
and reality and then you repeat the step
again
so it is a fairly time consuming step it
is because to really get value out of it
you need to devote that time into doing
these experiments
the another uh key so so to
to sort of um you know like uh be a sin
have a single person orchestrating all
of these you know arranging meetings
ensuring that people turn up
uh what we follow internally is we we
dominate a person called the war game
champion who is in charge of
coordinating the various logistics
associated with the process
you know schedule the experiment and
then orchestrate the experiment along
with other team members so this is the
person who is uh driving this whole
thing um along with and with help from
everybody else who is involved with the
experiment
and of course you should ensure your
failure injection tools and scripts work
before the real experiment
the last thing you want is get somebody
in a room for one and a half hours um
and then
[Music]
figure out uh that your scripts are not
working
during um and there are two other
or perhaps one step that is really vital
is the ability to generate load or
traffic um for your experiment
locust python community favorite is is
great for http services um if you have
integration or synthetic tests they are
useful too uh the idea is that you need
to generate enough traffic so that you
can um you know there's something to
break
in your application
during the experiment get everyone in a
virtual room um
when we are back in office it can be a
single conference room like uh with
dashboards and notif notification
channels probably uh up on a giant
monitor or screen
um and then there's this other aspect is
do you do blind filler injection or do
you do predefined fault injection blind
fill injections are super useful for
like a as a training day exercise where
you want to you like maybe senior
members want to
do
use it as like a tool for training of
your junior team members
or it can be
the predefined fault injections where
the faults that are going to be
introduced are well defined and well
communicated before
another fundamental question is
staging our production
uh the key is you should be able to
easily reset your systems to the state
that they were before you started the
experiment
start gradually with the staging and
then when you're confident go for
production
and perhaps if you have automated
failure injection you may want it as an
opt-out model for staging
and an opt-in model for production
a couple of prerequisites you need to be
able to run arbitrary commands or having
admin style privileges in your cloud or
one on-prem account and you also need to
have
metrics dashboards logs
otherwise you don't really know what's
going on with your application and you
won't be able to
analyze the the result of introducing
those faults
great so people systems
and i've got a hypothetical scenario
here um
and i've got the hypothetical in quotes
because it's based on something recently
experienced so we get up on call we get
a pager um
and we see that okay we have some spam
requests coming in and
the uncle person jumps in
they realize uh okay yeah let me just
deactivate those accounts and then
jane realizes that oh i don't have the
right permissions let me get this other
person and that person also doesn't have
uh
the
right permissions and then eventually we
get this person who has the right to
access and then they start deactivating
the accounts
um so and that's why we we need to treat
our people and the the steps where
people are involved as systems as well
we need to practice um we need to inject
a bit of chaos into our people uh
systems as well um
there's this talk called chaos
engineering for people systems um by
dave branson from google who talks about
companies being distributed systems and
he and they share some ideas around how
you could uh
perform chaos engineering experiments
for for people so things like um you
know do a staycation where 50 of the
normal usual staff will not they will
just say okay no we are not present and
you know let everybody else handle uh
any failure any incident that comes up
um the the book release 8 chapter 17
talks about zombie apocalypse uh you
know simulating a zombie apocalypse
where you'd
again the same idea is make some of your
people uh not
play any role uh during an incident and
see how it goes
and that is why it's
really useful when you are performing
this kind of experiments on uh staging
environment rather than production
test your playbooks um
test your team build it and and treat it
as a team building exercises like i i
have really
found it very useful to get everybody
together and run this experience and
they're all working with each other
they're they're talking out loud
discussing you know why this could be
happening and you know what can be done
about it and so on and so forth so it's
it's really great for that too
um so couple of key points to remember
um
let people know that you're running a
chaos engine experiment the last thing
want to happen is
uh the people thinking or waking up
being woken up at 3 a.m
so that's another thing you should be
mindful of time zones of the key on-call
people you don't want to be waking
people up at 3am because you're running
a kyocera experiment
that still is from the movie war games
yeah it was great
um okay um
the
next our section we're going to talk
about chaos engineering tools
and what
there is actually a dearth of tools um
that you will find and
and that is why what i'm going to talk
about is only going to be a
representative sample
um so there will be there's no shortage
of tools in the chaos engineering
community
but you really need to understand
most of these tools probably are using
this fundamental tools that i have found
so far
iptables
it's like the duct tape of the internet
i i heard somebody say many many years
back i think um
that you will find
like
that being used
in many tools underlying tools which
give you a higher level layer tcp kill
um which allows you to break tcp
connections um and iptables allows you
to introduce dns based sinkholes
it allows you to introduce probable
failure scenarios and so on and so forth
taxiproxy and tc
which is traffic control they are super
useful to add delay to network traffic
so you can use these tools to
intercept traffic and add
arbitrary delays
the third program is stress ng
which is
really useful for simulating cpu memory
and io stress
um
so i've got a
demo um i'll not do it live i'll have a
recording so i'll just show it but we i
have this
two sample services service 1 service 2
where service 2 is talking to a mysql
database and they're all running in
docker containers that repo there has
this
you know complete demo that you can run
on your own
so what i'm going to do is
i'm going to
um
[Music]
play a video
so i've got okay so i've got
this services running and i'm going to
run the curl command uh to get like make
a http request to service one
and as you can see here i
it gets a response
there's this html sort of block that
i've got here and it gets a response in
about 0.6 seconds right okay great so
that's a normal um
default behavior
and i just do it again just to sort of
and then what i do is i inject the fault
so i run this python script
yep so my fault is injected so now i'm
going to run this
script command again
and you can see that it takes longer
six seconds
okay and now what i'm going to do is i'm
going to
um
add this query parameter so by the way
this is a this is a wrong query
parameter that i added so that's my
mistake so i'm going to add the right
query parameter and then we'll see that
yes so i've got this query parameter
it's like a
what it's going to do is it's going to
add a timeout
so now
yeah so now you're going to see that we
get the response within 0.3 seconds
remember the latency injection is still
there so the so the delay is still there
but so we get a response but this is a
500. that's because we have now enforced
the timeout on service one and so that's
how we get a 500 response
yeah so that's the demo so let's go back
to our presentation
yeah
so
and then um yes so i talked about this
fundamental tools and the inject latency
script that i have that uses
the tc
command or program to inject the latency
okay
the next
tool or software i'm going to talk about
that is really useful is called the
chaos toolkit and it allows you to
inject various faults using external
drivers so
they have inbuilt drivers for aws gcp
kubernetes toxiproxy and you can write
your own driver using python which is
great
if your service is hosted on aws aws has
a service called fault injection
simulator and you can do things like rds
failover you can trigger errors into the
api calls
so for example you could add a delay to
the auto scaling api calls and
and
you can think of the interesting
experiments you can run using that what
if your auto screen activity is delayed
by
you know 20 seconds what happens to your
application under load
so
okay great so summary
so the whole point of chaos engineering
and what it's trying to um
[Music]
uh like state is embrace the chaos it's
really about assisting your readiness
when your systems fail because it's it's
not a matter of if your systems fail
it's a matter of when
and often it'll it happens that known
failure scenarios may
come together to teach you about unknown
failure scenarios
emergent properties
because
two things you know may add up to
something completely different that you
didn't even expect
and that's what we want to get ahead of
almost by simulating this using keos new
experiments
um there's this article uh uh on coding
horror called working with chaos monkey
and there's this quote from that article
um so it begins with this sometimes you
don't get a choice the chaos monkey
chooses you and that and it ends with
this which is that's why even though it
sounds crazy the best way to avoid
failure is to fail constantly
and that is very profound
so
there are mult quite a number of
resources on chaos engineering
um and i i have got three resources here
because i find these these these are the
um
like these are the three resources which
really drill down our or at least the
first resources they really talk about
the principles that talk about the ideas
they are not about the tools uh which
sometimes the software community um
cannot get enough of um so those are the
first resources so if you if you are
getting started with chaos engineering
if you wanted to learn about chaos
engineering and the ideas and the
principles um the first resources are
the key the third book is great for
hands-on learning the author has done a
great job for for using you know basic
tools which you can run in your local
system to just play and familiarize
yourself with the ideas of chaos
engineering
great thanks everybody who who chose my
talk today um you know uh over the other
of the tracks
um thanks to all the organizers who have
pulled up this conference in this
circumstances it's just that's great
um thanks to all the next day video
folks i have had a chance to interact
with they have been super helpful super
you know friendly super nice people
um i am contactable at ecorand on
twitter you can email me i've got the
talk materials on on my github repo
and i have a website at ecoran dot me um
i i will not take questions now but i'm
happy to answer any um
in the chat or you know later on
um so great thank you everyone
awesome thanks amit that that was
amazing um lots of really really cool
resources in there um there's a few
there that i haven't seen so i'm keen to
go uh and look those up this this kind
of stuff of dealing with failure by
embracing the chaos as you put it uh is
is close to my heart so i look forward
to digging into those uh yes so as amit
said we're we're right up on time so any
the questions that are there and there
are some there already for you admit uh
if you've got any more just raise them
in the chat or chuck them into the
questions amit will be around to answer
them and with that we're now on a break
for lunch so
do get up leave the computer have some
water um after the questions of course
um grab some food um re-energize
yourselves uh we are back as a
conference we're back at about uh let me
check my notes i think it's about 1 30
yes lunch lunch break is until 1 30. um
this track doesn't have a talk until 2
15 though so you will have a chance to
go and explore some of the other tracks
and i encourage you to do so but be back
here at 2 15 p.m uh when we have a talk
by peter chu
things might go wrong in a data
intensive application
uh until then enjoy lunch enjoy the
questions enjoy the conference and we'll
see you back
you