welcome back everyone to the devops
track here at pycon
pike online 2021 it's uh it's wonderful
to have you all here and uh we welcome
our next speaker with a tremendous talk
it's molly rowe who is here to talk
about metrics for good
not evil
molly over to you
thank you
hey there everyone my name is molly i'm
here today to talk to you about metrics
for good not evil thank you for coming
to my talk
um so i'm the head of people and culture
for record point i have a pretty weird
background
particularly compared to a lot of the
other people who are talking at pycon
this weekend
um i've been a scientist a project
manager a scrum lead a hybrid cloud
consultant a recruiter a manager and now
this head of people and culture role
um
i've spent the last seven or so years
trying to solve the pain points that
plague so many tech companies
using people-centric solutions
so a combination of tooling technology
metrics and empathy
over the last seven years i've kind of
run about almost a thousand interviews
globally for engineering sre and devops
roles um and
in every single one of those i've been
asking the candidates
what kind of company do you want to work
for and then taking that data back and
trying to make that a reality
so i kind of use this constant influx of
user feedback to iterate on my idea of
what engineers want
and what businesses need
and the complexity of the problem that
separates those two things
and kind of off the back of that i've
built this role at record point
for any of us who any of those who
haven't heard of us we're a software as
a service
software engineering company
um that produces a federated information
management product
we try to reduce the cost and the risk
and the manual effort associated with
going through
data and records management audits
we're a scale-up of about 75 people
globally and 40 to 45 of those are
software engineers and sres
this means that a big chunk of my staff
are engineers
and that plays a big role in my employee
engagement
the overall culture of the business and
our retention rates so everything that i
can do to improve the processes and
remove the pain points and improve the
developer experience
uh is a benefit to
the whole business
which is why i care about slos
so
rekka point has
gone through over the past several years
a pretty common experience that i've
seen across multiple different software
engineering companies
where we've had a shifting power balance
between our engineering department and
our product department
we've had leadership transitions that
have meant that there's been kind of
rapid changes in technical direction as
well as
changes in overall delivery focus
and even though both sides of that coin
both product and engineering have had
their time in the sun had their control
um
it hasn't really benefited anyone
so until
you find a balanced approach where those
two factions are working if not in
harmony at least in concert
you're going to have struggles
so
i kind of went out to the market and
looked for
a magic wand for how to solve this
opposition nature that you see in so
many software engineering businesses
between product and engineering
uh and i want to show where i can find
one
and like most things in the life
i found that in google in this case
actually in google
so google's development of the sre
handbook and the incredible
documentation around slos slas
and slis has kind of led me down this
rabbit hole of telemetry and
visualization and metrics
that have taught me about finding a
common language of care
within
that dynamic
and the care for the user
so
a common complaint that i come across
from engineers is that product
management are changing the priorities
too often and work never gets fully
completed or the work that gets time of
day is always feature development work
and never platform or systems
improvements
and the reciprocal complaint comes out
of product you know we're not delivering
client value services being interrupted
by issues and bugs and crappy code and
unreliable platforms
and this rhetoric
justified or not from both sides is kind
of founded in the fact that those teams
are incentivized differently
product is often measured on
delivery of skus
rather than overall customer experience
and engineers measured on so many things
right features shipped deployment
frequency velocity
and this really often leaves us already
holding the bag
they're responsible for the platform
availability and the latency and the
mean time to resolve whenever something
goes wrong
but all of those metrics are heavily
influenced by not only
the priorities that come out of
engineering and and product but also the
work that's done by those two teams
and this is where the power of data
comes in and this is where the role of
sre into the future comes in
because sre are holding the keys to the
kingdom when it comes to
data and being able to create a common
language between those two factions
um
for some organizations
uh those keys are still under
construction and that's okay because all
the raw materials are still there
all of the raw materials and all of the
raw data exists within your services
today
and the future is about how you use it
i personally have
quite an obsession with metrics and data
in general and a belief that until
something's measured it can't really be
improved
at least not on purpose
um
generally
and look i'll admit this talking about
metrics will lose the attention of the
audience and cause people's eyes to
glaze over i know that
but
it's actually because most people have a
pretty crappy experience with metrics
almost all of us have sat down in a
meeting or a retro and gone through the
data and gone
yeah
but that's not right because
x
and
going we all end up going down this
justification or explanation pathway of
why that data is incorrect that even to
our own ears feels like excuses
and that's not our fault
this is because in general people are
really terrible at setting metrics and
we often default to study metrics that
address the symptom of a problem rather
than a root
a root cause
um so let me tell you a quick story
about metrics gone wrong
back in the days of colonially occupied
india the british governor was concerned
with the number of venomous cobras in
the streets of delhi
he decided to implement a scheme
offering cash rewards for cobra heads so
basically he equated dead snakes equals
less snakes in the street
seems
okay on the surface um
initially the scheme seemed to be
successful they were redeeming
or lots of people were redeeming the
cash prize
um and you know assuming that this
continues on track the number of snakes
in the street was
you know surely going to decrease
however over time that didn't happen
and they were trying to figure out why
so upon investigation
what they found was that some very
innovative individuals and then all of
their enterprising neighbors
had started farming cobras
yeah
they
basically started farming these cobras
with the full intent to kill them and
then redeem them for the cash prize
in a rage the governor canceled the
scheme and said this is not in the
spirit of what we intended i was trying
to protect you
no more no more money for snakes
and so what you then have is a populace
who have snake farms in their houses and
backyards
who no longer have an incentive to do so
to keep those snakes
and so they did the easiest and most
predictable thing
and let the snakes go into the streets
so
with a well-meaning well-intentioned
scheme to reduce the number of venomous
cobras and protect the population
the actual result was that by several
orders of magnitude they increased the
number of snakes in the streets in delhi
this law of unintended consequences has
come to be known as the cobra effect
it's so eponymous with this scenario
that i've just um described
and you know you might be tempted to say
look molly that was 200 years ago in
colonial india this is not a great
example people wouldn't do anything
quite so stupid anymore
well
the most recent entry into the cobra
effect hall of fame is uh it goes to the
university in the usa who recently had
to release a statement that said
to any students who have voluntarily
exposed themselves to covert 19 if you
continue
too many students who have done that you
run the risk of being suspended or
expelled
they had to release this statement
because the local plasma donation center
had increased the money reward for
donating plasma for anyone who had
active covert 19 antibodies so someone
who had had covert to more than a
hundred dollars per donation
the scheme was completely
well-intentioned they were trying to
treat people who were in critical care
with covid which is a a use of plasma
and instead what they created was
more people in critical care with covert
in that area
so
this is me circling back to why most
people have a terrible traditional
experience with metrics
humans are pretty bad at using critical
analysis when it comes to incentives and
are rarely just looking for an empirical
outcome like you should rarely be
looking for an empirical outcome you're
often trying to actually influence the
behaviors that drive that outcome
you're not really looking for snakeheads
i hope
you're actually looking for less snakes
you're not really looking for more test
coverage you're looking for better code
so when you're proxying the behavior
that you want with an easy to measure
symptom
this is a common downfall of general
metrics
it's pretty much just a case of be
careful what you wish for
um there's a couple of easy ways to
counteract a lot of these pitfalls that
go beyond imagine how you would game the
system
because people are infinitely
resourceful particularly when you
incentivize them to be so
pretty much everyone should be familiar
with this cost speed quality trade-off
triangle and this is actually a great
way to build metrics
because you should have metrics that
contradict each other
what that allows you to do is create a
balance where
to exceed or to gain a metric or or to
push one metric really hard you're going
to influence those other metrics to
their detriment and therefore have an
overall worse outcome than if you hadn't
like tried to gain that singular metric
a really great example of this is nicole
fourgrin's uh forsgren's metrics for
high performing teams
she outlines these in her book
accelerate which i highly recommend you
read
but basically the premise is that
lead time deployment frequency change
fail ratio and mean time to resolve are
metrics that are indicators of high
performing teams
lead time and deployment frequency
obviously correlate to speed and change
fail ratio and mean time to resolve
obviously correlate to quality or um
reliability sorry thank you um
and so
you know if you're trying to optimize
for your change fail ratio and you know
you're trying to push out perfect code
that has no bugs you're going to impact
your lead time and your deployment
frequency negatively
these are the kind of principles that
led me to the investigation of slos
right the new problem is how do we
define metrics that encompass the whole
product and incentivize good behaviors
between competing elements
it starts with kind of finding that
balance of power between two factions
so no longer talking about speed and
quality or reliability but now talking
about product and engineering as the
business
kind of equivalence of those things
the thing that's often
kind of missing when you do have two
powerful factions vying for control
is an impartial arbiter particularly
when those factions have different
objectives and they're being measured
differently
so the role of sre that's where we come
in
so sre
owning the data across the business and
being able to provide a common language
and a common
thread of discussion between those two
factions
with data to back it up becomes a new
way forward
the role of sre is defined by google
is responsible for so many pieces of
data so they're responsible for the
availability the latency the performance
the efficiency the change management
monitoring emergency response capacity
planning
of all of their services
and this places sre in this pivotal area
of control as the provider and the
purveyor of data
if we rephrase this balance again but
now in terms of sre as the middleman
providing a common language that impacts
both product and engineering we're now
talking about speed and reliability
again
but
where we have slos
or any kind of metric to come in and
bridge the gap and provide a
quantifiable answer to what is important
at any given time and why
as well as that common language to how
to productively discuss it in a way that
we're bringing
logic and data into what is often a very
emotional exchange
so
expectations from users around features
reliability availability security and
quality are all increasing exponentially
right
but most organizations are making
trade-offs and are not really set up to
deliver on all of these vectors which is
the speed and reliability compromise
that we've been talking about because it
really is fundamentally underlying the
the conflict between engineering and
product
on one axis there's this desire and need
to have rock solid stability
um and reliability but the challenge is
if that's what you're optimizing for
you're not innovating to your full
potential
and you're going to have issues within
the market if not now in the future
and on the other hand you can't spend
all of your time pushing for feature
delivery without regard to stability or
you'll rapidly accrue risk and technical
debt and potentially churn your existing
customers
so
what i haven't defined for you and i
want to touch on really briefly is what
are slas slos and slis
this all comes straight from the google
handbook sre handbook please i encourage
you to read it
it is much more thrilling than it sounds
but slos slis and slas are exclusively
used as metrics that capture parts of
your user journey such as availability
or request latency or throughput or
error rate they give you a metric for
both sides of that coin
um as like the queen of product versus
engineering if you assume that both
engineering and product care about your
user
which i hope they do
so your sla is your service level
agreement it's your external metric to
which your business has committed to
legally and with generally a monetary
obligation to meet
this is often something
reasonable but after which it's the kind
of the baseline after which your clients
are
not happy with their service
um
your slo is your service level objective
it is the internal target for the metric
that you're measuring that should
represent
where your client starts to feel pain it
it represents your optimal point where
you want to hit from a reliability
versus speed perspective or a
reliability versus risk perspective
and your sli is your service level
indicator it's the it's the measure of
service reliability it's what you're
measuring
um
slis will tell you that something is
wrong and you need to use all of your
other metrics to figure out what that is
so
this is a good depiction of how this
works right
so
your agreement is just enough to stop
your customer being unhappy or leaving
or churning that's what they've agreed
to as an acceptable level
your objective your slo has to be
tighter than that agreement and it
should represent your desired user
experience
breaching your objective has to have
consequences as well you shouldn't just
be only if we breach the sla is there a
problem breaching your objective is
where you need to be able to leverage
that data
within the business to change priorities
of what's being worked on because your
hand you're heading in the wrong
direction
and it allows you to be proactive before
there is a monetary problem
where you're impacting your sla
anything above your slo means that
you're spending
too much time on reliability and you're
wasting effort that could be used to
deliver features
so anything better than your sl slo is
wasted effort
very quickly about error budgets error
budgets is the next stage in this
scenario where an error budget is
monitoring your slo over time
if your slo
as in the previous slide is 99.95
availability then
your error budget would be 1 minus your
slo 0.05
this means that if you map that out over
the month
0.05 of the minutes in the month gives
you 22 minutes
sre practices encourage you to
strategically burn that budget to zero
on purpose to do things like
deliver new features run expected
systems changes
use planned downtime or just do a
slightly risky experiment it means that
if you're hitting or using your error
budget you're running as fast as you
possibly can without impacting your
availability and without impacting your
client
error budgets are not something you need
to do right away if you're investigating
slos for your own business start at the
start start with slos and slis and
setting those metrics and starting to
measure them and make sure they're the
right thing error budgets are things for
down the track they don't have to be
something that you start with you don't
have to go all out right away
so martin fowler very famously 2018 said
evidence refutes the bimodal it
notion that you have to choose between
speed and stability instead speed
depends on stability so good it practice
gives you both
so
what does good i.t practice actually
look like and how does data and slos and
all of the things that i've talked about
implementation of sre within a business
how does that get you there
using your metrics for good starts with
what are you trying to get out of them
so common language slos provide you with
that common language between product and
engineering of how do you talk about
what is important and what is important
should almost always be your customer
you also now have hard data to influence
that as i said previously like quite
emotional decision product and
engineering are
often rewarded and incentivized on the
metrics that they as individual
departments care about rather than that
centralized core component of the client
so having hard data that backs up that
work needs to happen to stabilize the
product is really important
it also allows you to be proactive you
know when things are trending in a bad
direction where you're going to breach
an slo or an sla you also know that you
can be proactively releasing innovative
pieces of work because you're consuming
your error budget to do so
and
the justification piece is really about
how you have that productive
discussion so having slos in place for
your production services allows you to
remove all of that emotional ambiguity
when it comes to figuring out the impact
of an unplanned change or outage
um businesses often refuse to invest in
availability or reliability until the
bottom line's impacted
so it's really important to have that
data to back up what you're trying to
say
this is where sre comes in as a whole
and the role of sre may change going
forward in the future at least
from kind of what i understand behind it
um
initiatives like slos can be really
difficult to get buy-in for and the
owner to kick off really has to be the
sre team
[Music]
but it can't be them in isolation you
know you're talking about metrics of
what impacts the clients and what the
clients or the customers care about
which means that you need to be involved
with your product owners with your
customers themselves or with your
customer success teams as well as your
engineers who have to do
the work to make sure that we're hitting
those slos because your slos shouldn't
be aspirational they need to be
something that you can achieve
there are lots of selling points behind
slos as a program
but in general
the ability to surface your technical
debt
related to reliability or lack thereof
means that you know you can advocate for
what you need in terms of allocation of
engineering resources
you're reducing your manual effort from
an sre perspective of generating this
data you know what you're going to talk
about you know what you're going to
review you know what the metrics are and
what we care about
and as i said you're also reducing your
risk around monetary risk to the
business so this is how you sell it into
the business is reducing that that
monetary risk or risk of churn by
increase improving your customer
satisfaction rates
um
these are not for everyone i'm not
trying to say that slos or slis or slas
any of those things are for every
business
uh there is a level of maturity
particularly in the businesses that have
actually already implemented this to
great success the googles the
evernotes the twitters
those guys are huge
you don't have to start there you don't
have to look at them and go this is this
huge unachievable mountain
um you know you can start with two to
three slos and iterate and work your way
up you can start to build out that team
or you can build it internally as a
proof of concept within your sre team
and then start to use
the quality and the relevance of the
data that you're producing
to get buy-in from other pieces of the
business as well
and drive that adoption
failure is going to happen like this is
the devops track you guys have been
listening to failure and learnings and
you know amazing triumphs through that
all day
as you implement these kinds of things
failures are going to occur slos are
going to be breached systems are made by
humans and we've already discussed very
in quite a bit of detail how humans are
imperfect
so what's important is learning from
these and continuing to iterate on your
slos you should be going back and
looking at them on a regular cadence to
make sure that they're reflecting the
things that you want or the things that
their clients still want
um
and like you have to be collaborative in
this iteration as well you have to
incorporate those other parts of the
business that have those touch points
with the clients as well as the ones
that pay the bills
uh to make sure that you're delivering
not just for yourself and your
department but also for the whole
business
the other piece to all of this right is
the blameless mentality and how
important it is
because you're not going to get it right
away right straight away or even
continuously
what the clients want is going to change
um but nothing's about who tripped over
the power cord but how do we stop people
from tripping over the power cord next
time
slos should never ever ever be tied to
individual performance metrics
they need to be the goal around elixir
the goal should always be defining more
slos to get greater visibility and
understanding rather than blaming teams
for not meeting slos your slos are meant
to be a way to empower the discussion of
what needs to happen next and what went
wrong and how we fix it in the future
i just want to circle quickly back to
martin fowler's idea of good i.t
practices and bridging the gap between
speed and quality
so wrecker point my my company is still
in its infancy when it comes to
implementing any of these slos sli slas
and i came to talk to you today about
the research that i'd done in the hopes
that other people had seen these
problems in their own organizations and
found this something interesting
um as you know a way to
bridge that gap and go forward
um slos are a great way to start
leveraging your existing data and
setting metrics that really mean
something both to your customer and to
your product and addressing a root cause
rather than just
symptoms um
and the more research that i dive into
of this side of sre
and its potential role as the collector
and the curator of data and acting as a
mediator between the two factions within
the business means that they get to be
an impartial driver for the great
customer experience
and the more i believe that even if this
isn't the final ansel
i think it's a really good step forward
for technology in general and the way
that our businesses need to continue to
move forward
i'm certainly not trying to say that
slos will solve any problem or even that
they're right for every business
but i am trying to say that this
framework is something that i see as a
potential bomb for the pain that i have
seen across so many businesses globally
um and every day in my own business
it's highly likely that i'll be back
here next year for the we tried this and
this is what we learnt uh metrics for
good not evil part two
and we can definitely go over the
blooper reel then
um but i hope that this has been a bit
of an inspiration to do your own
research and explore if this might be a
good solution to your problems
thank you so much for your time i know
this is not a normal talk even for this
track
but it's something that i really wanted
to share and if you've got your own
horror stories or success stories for
implementing slos in your business i'd
love to hear them please hit me up on
linkedin or twitter
and uh
if you've got nothing else out of this
talk please remember that whatever else
you do don't put bounties on snakeheads
thank you dawn that has that has been a
tremendous talk thank you um
it's
yes sorry molly what am i saying dawn i
have my next talk already queued up in
my head excellent more failure for the
for the devops track
um my apologies molly uh metric is close
to my heart though i'd love everything
that you were talking about there um
people in the chat also really really
got a lot out of that we have a couple
of questions already queued up um
so if you could jump into the chat and
uh and talk to people about those
questions either just in the chat for
the the room itself or into the hallway
track that would be superb um people
would love to talk to you some more
all right no worries i'll jump in now
thank you great
and we have another short break now um
we'll be back in 15 minutes for our
final talk of the day um which is by
dawn this time getting people's names
right is important uh which is about
accessibility uh accessibility overlays
a cautionary tale hmm i wonder what the
cautionary tale will be
and uh yes so grab a quick drink ask
some questions about molly's talk there
about metrics and we will see you back
here in about 15 minutes
you