foreign
welcome back everyone I'd like to
introduce Jeremy Rothstein Jeremy is a
systems engineer with a background in
industrial Control Systems dowels in
micropython and had the displeasure of
being my colleague for several years
take it away Jeremy now it was fun it
was fun
[Applause]
g'day um yeah thanks a lot Chris
um yeah whenever I go to pycon I've
always thought it'd be nice to know you
know ahead of time when a Talk's going
to get to the really juicy python
content so it's coming I promise you
we've just got to get through some
conceptual stuff first
so in one word this talks really about
the notion of reliability
saw and giving attention to an aspect of
Apple liability that's often an
afterthought or sometimes even
overlooked entirely and by then it's too
late
so when people say you know this app's
Rock Solid
yep
this app's Rock Solid it runs like a
dream
what they're really saying is that it
handles all the edge cases well it's
engineered to be fault tolerant has good
performance it does what it needs to do
well
what they're talking about is robustness
the app is robust
and it's robustness that what we tend to
think about and what we put all our
energy into achieving all those coding
standards code reviews design meetings
refactoring sessions testing all that
it's all designed to achieve a
robustness
but as you know if there's something we
all know it's that despite our best
efforts our applications can fail if
they can fail they will fail at some
point and entirely unexpected ways
so shouldn't we be prepared for that in
some way
so when it does enter that failed state
state in production
isn't reliability more of a Time measure
how long it takes to get the system
working again
so that's the repairability of your
application now that becomes the crucial
Factor
it's not just how robust your app is
that ship has sailed and I guess sank
so if you're a developer and a
maintainer so you've got Skinner that's
in the game you're developing and you're
maintaining your own product
beard can I just adjust the mic a little
bit yep
oh
[Music]
yeah cool so yeah if you're if you're a
developer and maintainer you've got skin
in the game you're developing what
you've
you're maintaining what you've developed
it comes down to doing all the right
software craftsmanship things to extend
out that time between production bugs so
that's the developer side
but on the maintenance side you also
need a knuckle down and have some tools
and techniques to keep the time taken to
fix these production bugs as short as
possible
and let's not forget what the real goal
is here it's all about us being able to
sleep well at night
and they'll get woken up
so I'm sure most of you have seen this
type of chart before it's normally used
as a lesson to try and catch bugs as
early as possible
in the process because later and later
it gets the more expensive it is to fix
not just in engineering time of you but
in costs to the End customer from the
downtime and and the defects of their
products or services
so if you improve repairability what
you're really doing is driving down the
cost at the last stage where a bug is
most costly there's other advantages for
the other stages too
but the main one where repair ability
has impact is at that last stage there
and if you if you break down what goes
into fixing a production bug if you've
first got to find the Clause so you're
in what I call a plumber situation a
plumber can't fix the leak let alone
even estimate how long it's going to
take unless they know exactly where the
leak is
so then you then you've got to apply the
fix both in terms of changing your code
and deploying it and it's that first
part
the time to find the cause that can be
the really really hard part and that's
where having Diagnostics
helps the most
so when we're talking about Diagnostics
it's its definition is is kind of
contained within the word itself it's a
tool for diagnosing it's a tool to help
you know the diagnosis it's for knowing
knowing the cause of the problem
or even finding out whether there is a
problem or not with your code
which is still really useful because so
much of troubleshooting complicated
systems with many moving parts
is about eliminating those potential
root causes you know Sherlock Holmes
stuff until you arrive at the truth
however improbable it is
and how many times does it come up
whether you know you or someone Upstream
says oh it's working for me sorry can't
replicate the problem
what they really mean is they can't find
the cause of the problem and if they
can't find the cause of the problem then
you're out of luck thinking they can
even fix the problem sucks to be you
now to me this indicates a failure
yep
I'm doing this tomorrow by the way I'm
doing his job tomorrow
yeah so to me that type of situation
indicates a failure to have enough good
diagnostic tools at hand to find the
root cause when you're inspecting the
so-called
crime scene
of a failed application
all right let's go
um
yeah so you don't have the right
diagnostic tools it's not so much to do
with having perfect replicas
of the real world installation setups
lying around on your desk to probe in
practice it's really hard to do that
there's just so many possibilities
in
on site and although obviously that
would help a lot too as well
so without much data or tools if in that
situation you're forced to resort to the
software equivalent of physicist Richard
feyman's favorite famous problem solving
algorithm at least according to his
colleagues
just think really hard and then come up
with a solution
that's that's not a situation you want
to be in when times are the essence
I'm told also in Oppenheimer is
basically relegated to a bongo drum
player so I'm not happy not too happy
with that Mr Nolan
um
so let's uh look at some of the
attributes of software-based diagnostic
tools before diving into some concrete
examples
the firstly
it's got to be on demand that is we
don't normally require the tool it's not
part of the core intended function of
the application everything works fine
without it it's only glanced at when
needed and ideally the data is only
generated and displayed when it's needed
as well making it really light on
resources
so being light on resources including
cost is really important especially on
constrained you know Internet of Things
iot devices where there's thousands in
the field
and they're charging per device
um
it's permanently attached or probably
better stated embedded so what I mean by
that is it's part of your application
the diagnostic tool is part of your
application available for use at any
time it's not some separate tool that
you need to connect or spin up when the
time comes
and it's also part of your code too like
like tests except yeah this this tool is
designed to be run in production when
you need it most
and you don't really want it to be a
separate tool because it's going to be
just another tool in another repo you've
got to maintain and potentially keep in
sync with your app
and if you have to stop the application
attach the diagnostic tool then restart
the application you're not not
preserving that crime scene so to speak
this this huge value and seeing Insider
partially failed but running application
I really hate that I've used this term
but what we're talking about here is
Diagnostics Being Human readable it
means you can't just provide a Json dump
of your App State
accessible via some API and call it a
day you can't just debug log everything
at higher resolution and spew out pages
and pages and logs and color today it's
more about being infrared information
Rich having having a high signal sort of
noise ratio if if you want to call it
that
and I guess it implies that the
Diagnostics you create are highly
customized for your application to spit
out exactly what you need to find
to Define where the issue is and not
much else
otherwise it's going to take too long
when you need it to track down the
course
and finally this one's about
being able to see inside your
application while it's running so all
the necessary Nuts and Bolts
the so-called System state the App State
it's all in service to answer that basic
question is it working and if not where
has it failed so this is that preserving
of the crime scene aspect I talked about
and this attribute really determines
whether the diagnostic tool is useful at
a more fundamental level
so let's look at some real world
examples of Diagnostics
here's your classic car dashboard car
dashboard it's uh it's pretty good I'd
say when your car acts up it tells you
where to go and look
it's information Rich lots of symbols
and icons packed into a small area
I just subtract points here for being on
demand because you do kind of need to
know the speed
you're traveling to drive legally and
you've got to know whether you've got
enough petrol so that's it's still kind
of still part of the core functionality
of the car
um here's a so-called bed of nails set
up for testing voltages and signals at
various points on a printed circuit
board you can see the Pogo pins
underneath probing the circuit
traces of that upper board
and uh yeah I I subtract points there
because it's not embedded you've got to
attach it
um and unless it comes with a really
intuitive UI all it's doing is spitting
out raw data that would be hard to
comprehend
this is github's operational status page
pretty pretty good I reckon you can
clearly see where the issue lies if
you're a customer it condenses a lot of
services behind those little green dots
which is also a little bit of a
shortcoming I reckon
and the incident history button is
really useful for tracking down
something you're waiting for resolution
on
now let's look at your tickle logging
high-res logging it's certainly on
demand and permanently attached
but it's normally quite costly in terms
of the time to trawl through the pages
to trace an issue and if you're only
logging errors so low res
logging it's often too too terse to
really know how your app's functioning
at any point in time and you might not
get that full
breadcrumbs
as Sentry calls it to track down
all the steps that led up to that that
unhandled exception
um which which leads us to our standard
python debugger again not firmly
attached and
way harder to use than logging otherwise
everyone would use it all the time but
does a really good job on telling you
the internal State of Affairs so if this
was somehow running all the time in our
app with minimal overhead and customized
just for the information we need
presented in a really easy to digest way
I reckon that would be amazing so let's
keep that in mind
now for Internet of Things industrial
control
even our connected universe
okay
that's where Diagnostics really shines
because you don't have that comfortable
environment of the cloud
so your app could be in a factory
controlling an important piece of
equipment on the line could be on a farm
in the middle of nowhere collating
sensor data and using it to control
Machinery at irrigation time
it could be in someone's house and they
certainly did not in consent to sending
you logs and rightly so given all the
privacy issues that come with that
um and yeah it also could be located in
the highly secure inaccessible private
Network self-hosted by the customer for
security reasons
so in all those examples it's quite
likely either you've got no direct comms
or you've got very very limited cons if
that
your standard high-res heavyweight and
costly cloud-based locking setup is just
not an off option in a lot of cases
so unlike in the cloud your application
in those sorts of scenarios can't just
be designed to exit on unhandled error
and rely on your horizontal scaling
behind a load balancer to pick up the
workload of the falter unit there's no
horizontal scaling
there's no kubernetes there's no
failover until there's a disaster
recovery site there's only one unit
and in this situation you're often
likely not you're relying on the
customer to tell you what's wrong
but normally they aren't as tech savvy
as you need them to be
they're you know certainly highly
motivated to help you but they're often
unreliable Witnesses and a lot of what
they observe end up just being red
herrings for you
and I'm sure you've been thinking this
whole talk like why don't you just roll
back to an earlier known good version
when this happens
but the problem is that they need this
feature right now it's a new feature
this whole they've bought this whole
product on the basis of you having of
having this feature and it's not working
so you're going to get into situations
where you just need to fix it you can't
roll back
side let's Now get stuck into how to
embed Diagnostics into your python
application there's there's many many
fancy ways but I'm going to show you
what works for me
um I use a pretty simple
UDP setup so the way it works is you
connect your production app
over UDP
um
via a client to ask it and you ask it to
send you diagnostic information
so there's no fancy web front-end no
structured data all it is is the udb
client run from the command line that
sends a number a page number if you will
to that production app and all that gets
returned is a bunch of text
that's all so there's no nested Pages no
hyperlinks no charts no ability to send
anything but a number and get nicely
formed our text back
the advantage of this is really simple
it's easy to lock down or open up if you
need to or UDP or you can turn it into a
fancy UI if you need to later on
the fundamentals are there
um I'm not going to go through this
whole thing but this is the client side
so
that's it in entirety it's basically 40
lines of code
lifted pretty much from the python docks
with a bit of exception handling
there's a section there at the top you
you retrieve a number from the user
somehow I'm just using input
you send it across
over UDP and then you wait for the
response in this case block with no
timeout again really simple and then
just print out
print out the message you get back
and on the python application side so
your Handler again all it is is a UDP
server listening on a known diagnostic
port
again straight out of the straight out
of the Python docks
it reads that page number requested by
the client
and has sub handlers to process each
specific page number request you could
use async IO you could put it in a
separate thread make it blocking and use
this built-in socket server module which
I have here
um in many cases the handling part
happens so rarely that it's just
fetching pre-computed info
that you can easily put it in the main
thread of your app without impacting
anything providing the listening
component of the UDP Port is made to be
non-blocking
um
but just as an aside I use a python
package called blessed which I want to
give a little bit of love to to make the
text output all pretty
in my Diagnostics so it just provides a
nice interface for adding in those
really complicated antsy color Escape
codes
and
you can see in the bottom picture I'm
basically printing it out
from the terminal and it works exactly
the same as if you're printing it out in
Python it's all color formatted
okay so the first thing you're going to
want to do with Diagnostics is have a
table of contents page at a known
location so in this case 255
you remember that promise with the rest
you know the apis would be
self-revealing self-documenting you
started
at initial end point and discover all
the rest all that never happened but um
table contents has been around for ages
it achieves the same thing
um and yeah you can easily
use a human to human interface where you
just shout I want diagnostic 230 to
someone on a phone in the field and they
go and get it copy it and paste it into
a chat message or an email for you
um so yeah the subtitle of my talk was
you know doing it without logging but
I've got an Asterix there so it's not
technically lying
um but it's still it's still really nice
to have high-res debug logging to see if
your app is
ticking along
but with Diagnostics
good just to have a snapshot like last
109 lines or so is all you need
every time you hit that
that
um
page 245 it just gets the last hundred
lines
and to achieve that you can create your
own circular buffer logger Handler if
you're using standard python or create
your own logging classes if you're using
something like micro python and all that
would do is output to standard output
but also at the same time storing in
memory in a list that you can unroll
when time comes to to view that
diagnostic
um but the quickest win you can have
with Diagnostics is reporting on I've
handled exceptions and crashes
yes
and you know even if you are
trapping it and you know you don't care
if it happens you still want to know
about it because you might want to make
a specific error handling for it
so normally you want to write that to a
file so it persists across uh reboots
and to carefully limit that file in size
via rotation or similar to prevent
running out of disk space
um
now
I use a decorator
um so the left is the implementation
it's it's one of those fancy ones that's
an actual class that takes arguments
um
so the way you use it is you just
whatever function you're using you just
decorate it saying you want to consume
the unhandled error and it will still
report on that
unhandled error to your Diagnostics
with getting uh crash data that's also
really really useful you've got a few
options not so elegant you can take a
look at sys dot accept hook in the
python docs you basically just register
a function to use to run when any
exception is hit including unhandled
ones I don't really use it too much
myself you can trap signals but you've
got to you've got to be really careful
with signals with
not doing too much
and
third option is the one I tend to use is
making sure every function call is
within some sort of try accept
ion uh Clause so it's catching
all the unhandled exceptions as well and
probably re-raising them
um
so in my crash file I'll go and dump the
entire contents of that circular buffer
to provide at least 100 lines of context
leading up to that crash
and then yeah using using these these
little tricks to to get it into that
file before your application exits
um so another really good use for
Diagnostics is if you've got some
complicated state or calculations you
can put it in an easy to digest format
including the results of all the
intermediate computations that went into
to generating the output
so it's kind of like a spreadsheet you
know that that shows all the steps in a
really Visual and intuitive way
so this kind of information would never
be available in a black box you know
product to the user and it really allows
you to potentially track down some some
pretty nasty edge cases while they're
happening
the crime scene
here you can see I'm showing on the left
the evolution of each variable that went
into a calculation and how it was
adjusted before it got used so that's
that's very valuable so why does it have
that value I don't understand
um
and yeah it's part of any
troubleshooting exercise particularly in
the field your technicians are going to
want to know if the sensors are working
if there's Communications back to base
Etc
um I'm showing some really simple
diagnostic page handlers this one here
you can print out the contents of a
config file that users wouldn't
ordinarily see but still still very
useful to see what's happening
um
it's useful to know what's on disk even
how much space is left even though I
didn't show that there and you can
you can run you know system CTL if
you're using system D to oh or traversal
um yeah to view app status as well and
yeah again you're condensing down that
information just to the the bare minimum
you need to provide that value during
fault finding
uh more app meter data knowing when
your application has started and stopped
and why
it may not be possible to work out you
know when it stopped you were to reboot
but I've put it there
um
and yeah on the right hand side you do
want to know all that nice stuff about
the commit ID and the build number so
you can check out the right version of
code to see if you can
find the issue by inspecting your code
and if if you want to
if you want to Branch off from that with
a hotfix
finally I'll admit it's a bit of an
abuse of the the page number system to
get it to actually do stuff but you may
want to consider providing common Quick
Fix operations
to users and to field technicians
directly through this diagnostic
interface so stuff like reboot rebooting
restarting Services clearing out files
that sort of thing and yeah when when
you do Implement a reboot
the device diagnostic you want to report
that you're going to reboot wait a
little bit
there that's the micro python example
um and uh and so they get the message
and then reboot afterwards otherwise
it's just going to hang
um yeah so that's that's about all I had
to speak about
um as I hope you can appreciate I mean
embedded Diagnostics they're really
powerful and they're you know worth
checking out so give it a try
[Applause]
thank you Jeremy we have time for a
question or two uh so signal Jack who
can see past the lights
and make him run
[Music]
um yeah so I had a question about um and
maybe this is like a detail or something
but I wanted to know more about the page
number system and if there's an
advantage to doing it with page numbers
as opposed to just like topic numbers
yeah you could use you know words or
mnemonics to describe what you want
um
I've I've taken this approach from
what's used
in Industrial Field devices they tend to
use this type of system and
if it ain't broke don't fix it but there
is also text-based
um yeah uh lookups as well yeah you
could there's there's no issue
hello I got really excited when I you
brought up the the pdb debugger and you
said I wouldn't be nice if we could have
that all the time sorry which what were
you talking about oh the the python
debugger and you said wouldn't it be
nice if we could have the python
debugger all the time yeah and I thought
oh yeah that I would love that too and
it reminds me of attaching a remote
Rebel enclosure and I saw your python
Ripple there as well yeah
do you know of a way we can just do that
can we get a remote Rebel in Python if
you've seen something like that that we
could do and then we could find anything
whether we'd instrumented it or not
um
apart from just running GDP and
production I don't really know the way
um
yeah
it's it's a tough one no thank you I
just I feel like performance here as
well obviously I should because I should
have I should have instrumented the
commit hash but I forgot if only I'd
remember to instrument the camera
there's something what you can do is run
GDP over a network connection
so I've seen that being done okay as
well which is quite interesting so cool
take a look at that it's a bit more
lightweight
I I know a way this isn't a question
okay
um
the Twisted framework has a module
called Twisted manhole which uh is I
think it lets you tone it into your
process or SSH or actually like SSH into
your process and from there get a repo
and do whatever you want
um I remember a presentation given by
someone at the very first kiwi pycon in
2009 on the subject
so I'm not too sure how up to date that
is
um but yes there's there's a simple
library that you can add to your process
which will just let you SSH into your
process and then do whatever you want
yep
yeah not sure I recommend it
yeah I don't know if you want to be
modifying variables of a production
application that's controlling a you
know really expensive machine
while it's running yeah
well thank you everyone uh
sorry oh wait track
thank you Jeremy please enjoy your mug
thank you very much
thank you
[Applause]