foreign
Welcome to our last talk in this room
today
Dave has over 20 years professional
experience building breaking and
learning about things
he does a mix of software engineering
Cloud SRE devops and training with a
focus on Automation event-driven
architectures and serverless and is
going to talk to us about proactive Ops
figaroid Dave cool thanks yeah so today
I'll be talking to you about proactive
Ops or event-driven it operations before
we get into what it's all about I want
to start with a bit of a story any of
you who have been on call you may
identify with this
so it's 2 30 in the morning you're
faster sleep
and
you hear this sound
you roll over
you look at the clock
it's 2 30 in the morning
you look at your phone there's missed
calls there's messages
you get out of bed
your first thought is this
your second thought is this
you grab your laptop you open it up
you open Slack
and
there's a few messages waiting for you
there's also a zoom call going
you jump on Zoom
everyone's got their camera off no one
wants to be there you can tell it's
going to be a long night
you think that checking the logs might
be a good place to start there's some
areas
it's actually a lot of errors
I told you a lot
so then you think the lugs aren't really
going to tell me a lot let's try to
figure out how we got here so you take a
look at the get history
and then you start looking at it
properly and you realize that this isn't
going to help you find the source of the
problem
it's running on AWS so you go have a
look at the ec2 instance
it's not properly tagged there's no real
useful information set up there no one
bothered to configure monitoring
properly either
so let's just pause for a second and
I'll talk about what I won't talk about
today
so I'm not going to talk about splitting
the monolith
um have we got any managers in the room
ah okay
um you might think that devopsing harder
will fix all of your problems devopsing
harder won't fix all your problems I've
seen teams try doesn't work
um I won't fix your jira workflow but if
your jira workflow looks like this let's
talk afterwards I can't help you with
that
so let's rewind and figure out how we
got here
so the teams delivering features
um you know bit every Sprint they're
feeling all right but the technical debt
is also piling up
there's a lot of alerts there's a lot of
notifications it's generating a lot of
noise it's really hard to tell what's
real and what's not and what the team
needs to care about
uh this is the Ops Team on a normal day
they're dealing with the first issue
they're still struggling and the next
one comes along and it's just constantly
feeling like that keeper
so let's do a little retro
so if you've got to go into the office
on Friday the boss should be buying you
Donuts I don't make the rules but if
you're going into the office on Friday
you deserve Donuts
um we should implement observability
we should be stopping doing 2 am outages
if it's your birthday like the donuts I
don't make the rules you should be
getting the day off and it should be a
paid day off
even for contractors
when things go wrong publish
post-mortems everyone should know
um what went wrong and why it went wrong
if you're using safe
please stop it's not helping anyone
and we should start being proactive
so how do we be proactive
we need to build a platform and the
platform is what should support the team
being proactive now this is generated by
mid-journey you'll notice that it's a
bit dodgy in Parts your first proactive
platform is going to be dodgy as well
you just need to iterate and it gets
better you don't want to see the first
version of this image I got out of
mid-journey I had to iterate to get the
image
okay we want to make our customers happy
our customers are the people that we are
here to keep happy and so we should be
trying to make them look like this like
this and like this
so what do we need in order to make our
customers happy
we need events and so we have some
events that come around every couple of
months
we have some events that it's probably
too soon to talk about
and then there's some events that we
pretend didn't really happen
but where do we get the events from
um
so the security nodes are probably like
oh we're going to talk about Splunk no
sorry we're not going to talk about
Splunk today
the data streaming knows like oh Cafe
yeah cool
sorry I'm disappointing you as well
and the observability notes are like he
said Implement observability we're going
to talk about a hotel
no sorry we're going to be talking about
web Hooks and lots of systems have web
hooks so give us data about what what's
happening in those systems so various
tools like GitHub have web hooks if
you're an unwashed open source hippie
gitlab will give you web hooks if you're
stuck using bitbucket I'm really sorry
but you can get web hooks out of
bitbucket
also if you're using jira you can get
data event data via webhooks if you want
a really nice ticketing system use
linear
not sponsored
Asana click up
um servicenow happy to give you web
hooks zendesk less happy to give you web
hooks on webhook data for some reason
you tend to have to throw a lot of money
at them and then grovel to your account
manager speaking from experience helps
Scout loads of systems will give you
events if you're on AWS you can get them
out of cloudtrail guard Duty security
Hub and you get the cloud watch events
out of various things as well
so once we've got our events we want a
bus for transporting those events around
and why a bus well buses are really
useful and they can be fun you can catch
them to go shopping
you can catch them to go on a holiday
or you can even catch one to get real
coffee
so if
yeah pick the guy who grew up in
Melbourne so when we've got our bus on
AWS it's going to be eventbridge for the
unwashed hippies again you can use
trigger mesh or there's other services
around that give you a busted cool thing
about triggermash is they use the cloud
event standard like a lot of other
platforms unlike AWS
um but we're we're going to be using
um Eventbrite or talking about event
Bridge during my talk today
another thing that's really useful is
Step functions that allows us to
abstract away uh sorry to separate our
logic and our flow and we'll talk about
that in a bit
again you can use Apache airflow or
other such tools
um
but yeah I'll talk about step functions
and here's a little example flow so
we've got GitHub we've got a Lambda
function which is a got a function URL
on it so we can send the events to it it
goes into eventbridge matches a rule and
goes to step function and because I
really like doing animations in keynote
we get to watch our event go wee
straight across over to our step
function and we'll talk a bit more about
how all of this works
so what we want is Lambda functions
or an open source alternative is open
Faz uh there's
um Open West there's a whole bunch of
different tools out there it's up to you
to find the one that works best
um I hate managing servers patching
servers all that stuff and let's not get
started about kubernetes so my
preference is to use Lambda and when I'm
talking about Lambda functions I'm not
talking about big long Lambda functions
like this
I want to focus on writing really small
Lambda functions that can be reused so
here we've got a Lambda function
and the purpose of this Lambda function
is to just
um match zero ticket references
within any string and we can reuse this
function wherever we're interested in
finding out where jira ticket references
are I love hating on jira but all my
clients use it so I use this now the
alternative to doing this is every time
you need the regex to match a Giro
ticket you go to stack Overflow or chat
jpt or wherever and you copy and paste a
slightly different version and then you
end up with Y here does it work as I
expect it but it doesn't match the the
ticket when it runs through this bit of
code
so if we have this um code that we can
reuse over and over again it gives us
consistency
and when I'm talking about having these
little Lambda functions I'm talking
about having a lot of them we want to
have a whole collection of these
functions that we can reuse so it might
be matching particular strings
or you know it might wrap libraries or
whatever we want it to do but it makes
it really easy to grab these functions
and reuse things
now you're probably going a lot of my
stuff is integrating with third-party
cloud services or third-party
applications
and a lot of them have apis
thankfully most of them haven't gone
down the road of Twitter and expect you
to pay an arm and a leg every month just
to access an API
and they publish
um specs for their apis in the form of
open API or formally Swagger files which
makes it really easy to interact with
the apis but you still need to write
code to interact with those apis now
this is how we used to do it
traditionally we'd spend most of our day
working like this
um lots of fun
and this is how we do it in 2023
um and I guarantee you this prompt works
perfectly every time
does all my work for me
um well what I've actually gone and done
is I've generated I've gone and created
a python code generator which will
generate Lambda functions that are
clients for open API spec files and I
call it picofun because there is little
fun in writing boilerplate code
so this is a quick demo of the the
library
of the tool
um so we give it an output directory and
a spec file I've been using some
Services gives you free unauthor
unauthenticated uh requests which I use
I've also been using the GitHub API but
that has around 900 endpoints and it
takes several minutes for it to generate
the code and then over an hour for it to
deploy and when you have little bugs and
you're preparing for a talk it's not
much fun so we're going to use the
little one today
and so here we've run the first step and
it's generated all of our Lambda
functions for us we can see here there
are all the the lambdas
our way is just a requirements file and
then we've got our terraform here I'm
not going to talk through all of the
terraform because it's pycon uh not
hashicon
um and so here's the the code it's
generated again it's quite short the
amount of code that it generates and now
here we're going to deploy it with
terraform
and through the magic of a pre-recorded
demo we can actually speed up the
deployment by a minute or so
um and so here it's actually
getting ready to do the deployment I
type yes and now come on do the speed up
there we go and now it's almost done and
we should have all of our functions
very soon there we go so it has gone
updated all of my functions because I've
been deploying them over and over again
we go here we can actually see that
we've got the
um
bad edit we can see here that we've got
the the code here has been deployed and
we've got a couple of Lambda layers to
keep our code nice and short so first
we're going to call the user endpoint
and this Returns the the Json from the
the request and then if we jump to the
um test 404
we'll see what happens when we make a
call that returns a 404. it gives us
stack Trace we can do all the the error
handling within step function which
we'll talk about in a minute
um whoops so now we're going to jump
back to step functions and talk about
why I like step function so much
so here I've got a very simple
step function this one is designed to
listen to zendesk tickets now
um I don't know how many of you work in
large organizations I've worked in the
Enterprise for way too long and there's
always more than one ticketing system in
large organizations and so
um the internal tech support is handled
by zendesk but the engineers they all
want to use jira well they don't want to
they're forced to use jira
um and so what happens is you get
support tickets coming in that reference
jira tickets but there's and the Ops
Team can link them up they rarely do and
when you run get blame in six months
time and you go find the the jira ticket
and the code looks a bit dodgy and you
go back to the ticket it's a bit vague
it's actually nice to have the zendesk
ticket that says hey this blow up
production when it was deployed and this
step function will just listen to every
comment on every zendesk ticket and it
will automatically link up the the
tickets for you
um
this one here is a more complicated
example so this is checking pull
requests and doing some basic quality
checks now the first thing it's going to
do is check how many commits are on a
pull request now I'm sure there's some
of the devs here you're going oh Dave
don't you know there's the squash and
merge button in GitHub yes I know that
button exists I hate it and there's a
couple of reasons why I hate it number
one the commits it creates aren't signed
and I like having signed commits so
actually no the
um the that the right people have made
the changes it adds to the verifiability
of the code and the other thing that it
doesn't do is it just squashes
everything so you might have two or
three unrelated things it's just going
to squash it all into one commit I want
Engineers thinking about what they're
doing and organizing their work properly
into nice commits so we do that and then
we do a check to see if they commit
signed now you're probably going oh geez
this guy doesn't know the features of
GitHub GitHub will check on your pull
request that the commits are signed and
they will stop you from merging but you
can have this hooked up
um and actually send a notification to
the engineer and nag them the other
thing when you're sending notifications
um Engineers they're smart and they're
lazy so if you start spamming them with
notifications they're going to set up
filter and just send all that mail to a
folder that they never look at or set up
to Auto delete
managers
they're also lazy but they're not quite
as smart most of the time they're going
to start getting inboxes full of
notifications if you CC them on
notifications for their staff so and
then they're going to go to the person
and go why do I keep getting messages
about you not signing your commits oh
let me just fix that and the problem
goes away so tip always spam the manager
as well
um so yeah we go through do this check
and here we're actually emitting another
event so we can actually
um react to it in different ways because
if you've got a larger organization some
teams
um they may not care about some of this
stuff some teams
um may not want to have the the actual
manager they might want to have the tech
lead notified and stuff so by having
different event listeners you can have
um different logic based on the the
reaction so you can break things up and
um oh the other thing we've got in here
that I forgot to mention is we're also
checking that the
um every commit
references a jira ticket because in six
months time when it's 2 30 in the
morning and you're sitting there looking
at get history you want to run get blame
and be able to start jumping back
through tickets and understanding what's
going on or looking at the history and
jumping back through those tickets so
this also enforces that
and the idea of this stuff is that you
get streams of or not streams you get um
flows of events coming from your various
systems you identify things that are
problems that
um if they continue to happen generally
result in outages or other quality
issues so you can put these guardrails
in place and start enforcing it and when
you're building this type of platform
I'm a huge fan of green
um coding so reduce the amount of code
you're writing reuse it as much as
possible those small Lambda functions
are really good and yeah we won't do
recycling we will refactor stuff down
into those smaller bits and break it up
so we can reuse that code because if
you've got
um a whole lot of code that's doing a
bunch of different things you have to
write all the tests for it and then in
three months time you're like oh I need
that bit from this thing over here so
I'll just copy and paste it and over
time you have Drifters you copy and
paste things from all over the place
um and this is It's a real um mindset
thing you you need to be
um thinking about
um how you engage your your developers
and teach them that this is actually to
make their jobs easier at first they're
going to go oh god I've got a reference
tickets every time I create a commit
it's not that big a deal people get used
to it and if you can demonstrate the the
benefits as well it really helps the
same with writing small bits of code a
lot of developers they they like to
write a lot of code write the tests and
all of that whereas this gives you small
bits that various people can actually
reuse which is really good and we'll
talk about how to pick which bits to
um work on first so the Urgent and
important stuff these are the things
that are causing you problems all the
time so so identify what the problem is
what data can help you identify that
that problem is likely to happen and
what
um reactions you need now at first maybe
it's just spamming the dev and their
manager
um or maybe it's people being added to
your GitHub organization that don't have
a email address that is tied to the
organization when they get added you get
an event check it if it's not there kick
him out and email the person who added
them saying we need an email address if
they keep adding them back they're just
going to get kicked out time and time
again that's the great thing about
automation you don't have to worry about
it it's just going to keep doing it
um
so that's the stuff you should focus on
first then there's the stuff that is
urgent but not important for the team
building the proactive Ops platform but
build it as a platform so another team
can come along and go this thing's a
real problem for us but only affects
five developers so they will actually go
create that little automation themselves
and be able to go I can get this event
from here and I can
glue these through functions together
and I've got a reaction I need
um then that frees up the team to work
on other stuff that is important but not
urgent so you can do that later and my
favorite one on the Eisenhower Matrix is
eliminate when you've got tickets coming
in for stuff that's not important and
not urgent just close the ticket don't
give people false hope how many people
have gone through and found
three-year-old tickets that no one's
looked at and that's the occasional
comment from a user going any update
just close the ticket and then they can
stop worrying about this well so yeah be
conscious of
um you know not giving people false hope
now if you're new to working on
serverless step functions Lambda
functions all of that stuff AWS has a
great resource
serverlessland.com there's lots of
patterns there there's documentation
loads of good resources
um I'll make sure everyone's who's
grabbing the QR code has got it before
we
um jump to the the next slide
cool so we've got a couple of minutes
left so I'm happy to do questions
um if you want to get
um notified when picofund gets released
I've broken a couple of tests that I
need to fix I'll be throwing the the
first preview version over the wall
um either tonight or Monday depending on
what I get up to tonight
and I've got a newsletter which I will
be putting out stuff over the next
little while around these topics so feel
free to um grab the QR code subscribe
and um yeah hopefully you'll get some
useful information
any questions
not at all
why are they never close
uh thanks yeah it's um really
interesting
um so we sort of have a similar thing
with roll bar and slack um where we just
sort of dump things in but um yeah it
starts to get to the point where you've
got
um a mix between things that are Urgent
and non-urgent and otherwise you end up
with too many cues to monitor or you end
up with everything being you know you
don't know if it's important or not so
you end up and then you're basically
ignoring it so what's your tips to to
separate the have to know from the nice
to know or can ignore
yeah so
um first off
um having notifications going into slack
it's a great place to put stuff going
we've got monitoring in place but no one
ever really looks at it because there's
just stuff constantly
um showing up there so I I find slack is
good for how your thing happened but
it's not really important if there is
action that needs to be taken by a
person get your automation creating
tickets and have watching in there so
you can detect if it's a duplicate or
not so you're not filling up jira with
like you know 500 tickets when you have
an outage
um but that that way you've got
something that's actionable and you can
go through review it and prioritize it
accordingly whereas since like I've seen
so many times
you'll have a discussion about a
particular alert and it's like oh yeah
we should do that we should do this oh I
think it's this I'm too busy and then
three months later
you see the same type of message I think
this happened once before and
slack search isn't the best in the world
it's still better than team search but
um you know chat apps is where
notifications go to die so get it into a
ticketing system
that's why I didn't mention like you
know sending stuff to to slack
um during my talk
does that help
yep
yeah
yep
any other questions at all
right thanks everyone for coming thank
you you might enjoy a story from the
organizer's office
your VCR rewind image someone happened
to glance off at the monitor just at
that moment and had a small heart attack
but everything was very broken
thank you for your talk Dave
and please enjoy your mug thank you
thank you