foreign
's career has spanned two centuries if
not millennia
he has worked a variety of roles within
the I.T industry and one day he may even
pick one and stick with it
today however is not that day
Dave is going to tell us about how one
person's ansible is another person's pie
infra take it away Dave
Daryl sorry
[Applause]
um I'd like to pay my respects to the
traditional owners of the land on which
we meet today the Ghana people of the
Adelaide region and pay my respects to
Elders past present and emerging I'd
also like to thank our interstate and
overseas visitors for visiting my home
state it's lovely to have you here and
you should definitely come back when
it's a lot warmer
preferably when we're not on fire though
as an ex CFS member I'm qualified to say
that
I would also like to acknowledge my
employers if they're watching
um
technically I'm self-employed but the
people pay me to do money people pay me
money so to do things to their computers
and My Views probably don't represent
theirs
um and I'd also uh would like to ask for
questions at the end of the talk
um I'm a little bit nervous up here this
first time for me so
quite easily derailed in my thoughts
I'd also like to add to that the GitHub
repo isn't ready yet
um the talk will be uploaded probably in
the next couple of days so uh don't hit
it yet
so I'm going to make some assumptions
um that you probably all got a passing
understanding of python and you probably
all got a passing understanding of
interval which was why you're here if
you're thinking this is an ansible
beginner's talk not quite
um and I'm not really seeking to convert
you I'm just providing an alternate
point of view
so previous talk titles that I've
wrestled with
um
is
uh yes what you see here these are the
the more family friendly ones
um
so and to be clear these are talk titles
that were rejected by myself not by the
program chair the program chair here has
been lovely
um I've been struggling with the theme
of this talk and it's something I
wrestle with constantly
um because I'm very uh OCD about diving
into problems and problems that I
perceive
so this is kind of what that talks about
but I still haven't got basically the
essence or the Zen of this talk out yet
so
um
still a work in progress
so it's also sort of documenting my
struggles with ansible over the years
and so
which will lead me I guess to confession
time which is what was taught currently
kind of is
um I struggle with ansible like I've
been using it for probably five years
now and not what it does or
what it is but more how it does what it
does
I find ansible
tends to be a tool there are some tools
that sort of fit into this model where
you're trying to solve a problem and you
have to think about that problem in
terms of the tool that you're using
against it and and I find that quite
often that was my case with ansible is
like I'd have to
structure it in terms of the way ansible
thinks too to do things
so and it sort of it it caused a fair
bit of cognitive workload on myself
doing that just having to deal with then
I won't call them shortcomings they're
just it's the way ansible's been
developed and grown over the years
um
and it sort of it takes away the
workload from actually thinking about
the problem itself
some of the
uh the cognitive workloads that I
mentioned
um
keeping track of global state that sort
of makes me feel like my old GW basic
days and to be fair I came to GW basic
after I'd worked in other Basics that
had things like local variables and name
subroutines so it was going back and
trying to have to think of global
variables again
uh and winter and winter not Ginger
Ginger coding still kicks my butt on a
regular basis uh
the obtuse error messages and this is a
true saver message is more a case of um
if you've forgotten to download a
collection that you're using you'll come
up with a rather bizarre error
it's sort of it's a the equivalent of a
code smell but in the design space
because it's a parsing issue when that
happens and it sort of it it again when
it occurs I have to think about it and
think why are you doing that is that
something I've coded wrong and it's like
oh no I've just forgotten to incorporate
the correct collection
um data structures I tend to a lot of my
playbooks tend to be well complicated
playbooks tend to be build data
structures around how I want things to
be deployed
um if when they start getting moderately
complex and coupled with uh templating
it gets
pretty messy pretty quickly
include task versus import tasks I
struggle with that all the time I can
never remember when to do which I've
always got to look up the documentation
for that
nested looping as probably also my
favorite is there is many ways to do it
in uh ansible
not necessarily
every way is equal
there are just some dead ends that
you'll go to well they're not that ends
until you've actually gone down them and
then you'll realize no I should have
handled this differently so
so
I I tend to think long and hard about
these things I spend a lot of time lying
awake at night staring at my ceiling
um
I shouldn't do that I should have
hobbies but I'd only waste them uh so
while I was sitting there and again I
tend to like research these things and
trying to dig into the nature of what it
is
um somewhere between giving the previous
version of this talk and this one which
was probably about a year ago now in the
python local Adelaide users group I came
across this quote
um
which sort of
sealed the deal for me it was like yeah
it was it was quite an opener
and it sort of it did actually appeal to
me
um it was it was pretty much a light
bulb moment when I came across this
ironically the
coincidentally the uh question talking
about this was talking about ansible uh
and I wasn't actually researching
ansible stuff in the time I was just
having to be looking for something else
but when I came across this it was just
like oh okay he's
studying he's suffering from similar
issues with ansible and I was finding
there's actually a small group of us
like that that are struggling with some
of these issues
this is sounding like a self-help group
sorry about that
um
and coincidentally uh he while I was
sort of looking for a new product to
pick up I'd already picked it up by the
stage and he was also talking about the
same product
hello
certainly so you have your my permission
to touch me
okay
it's me it's my uh well now I can hear
myself that's worse
please don't do that
I'm sorry that's okay
it could be my metal implant
maybe I'll just try not to move so much
was that better
I would just
this is going to be awkward
um so
I was reading um lox blog post
um he was talking about problems with
ansible but he was also saying oh I've
also found the solution which I'd
already started dabbling in and by
dabbling I mean actually wholeheartedly
committing
and the thing in question was Pi infra
when I came across Pine for I sort of
went oh because it's rare that I do that
for a project these days
um but it was hitting all
it was solving a lot of no solving's
probably not the correct word but it was
covering a lot of the pain points that I
was having with ansible
um
so uh basically it's very similar it
runs in the sort of the same problem
space as ansible it's uh
an agentless
orchestration tool a impotent for the
most part
um
you can still sort of work around that
but that's
something you shouldn't do
and it has inventories and playbooks
well the equivalent of playbooks
um
but they're all written in Python
it's
so generally it's it's it's a similar
model to python it runs basically on a
control node sorry python ansible
however it sort of differs from ansible
ansible tends to upload well what tends
to ansible compiles a module uploads it
to the remote end and runs it expects
you to have python on there
Pi in for a dozen buying for just
expects
a working shell of some sort at SSH is
across and not necessarily just SSH it
has a whole bunch of connectors but SSH
is one of them and is the default one
executes a bunch of commands on there
copies the results back or retrieves the
results back and then processes that and
then continues on from there it has
facts for doing a lot of that
so it will use the fax to determine the
current state if you
structure you deploy correctly it'll use
fax to determine the current state of
the system
then apply a set of operations to bring
it to the desired state that you want so
this is where the eye dependency comes
in high dependency
sorry practice that word and I still
have trouble speaking it out loud so
this is where the idepatency comes in
where you would sit there if you
re-execute a Playbook and there's no
changes to be made it should not make
any changes to that
to the desired state of the system uh I
was saying before it has connectors what
it what it calls connectors so SSH is
usually the bog standard one you use
unless you're talking to your local
machine and it's got a specific one
called local for that basically a
connector is any means that it can use
to talk to the remote endpoint other
connectors that it includes a connectors
for Docker terraform
and there's a few others as well which
completely escaped my mind the SSH
sessions remain open for the duration of
the run
if you run ansible and your playbook has
multiple plays in it it tends to do an
SSH connection for each one if you've
got uh like some environments I've
worked in previously
um sorry waving to a colleague there X
Colleen
um
uh for SSH connections if you're
spawning off a bunch of them a
connection create quite a sort of uh
quite a workload behind
um sorry that's me making noises again
sorry
I don't think I can keep still
um if you spawn a bunch a large bunch of
SSH connections you can actually create
uh quite the workload behind on your
authentication back-end system so uh
with this it spawns the SSH connection
but it remains for the duration of the
entire run it doesn't uh sit there and
reconnect
it also opens ssh in a slightly
different mode to what ansible does
ansible is quite noisy if you do uh for
example it logs in using a regular user
login account so if you do a who on the
system afterwards you can see you've
just done a manageable run because your
ansible user is logged in quite often
this uses ssh in a
uh in more of the remote connect mode
like when you're typing SSH hostname
command so it doesn't actually appear in
your login but it still appears in your
ortholog so if you need to go back and
do any sort of research for that
um so it's saying it uses the remote
Shell at the far end it's not
particularly wedded to the remote shell
that you use it's got a variety of um
different sort of plugins where you can
sit there and and depend on your back
end shell that you connect to you could
be connecting to some proprietary device
like a switch for example
and uh and it's particular set of
commands to use for that
the deploy process runs as two passes so
the the initial run
um sits there
executes all the code
gathers all the facts that it requires
to do that and it builds up a set of
operations and then the second part just
blats those operations on top of the the
system to bring it into the state that's
required
so I present you a concrete example
um this is a
lab maintenance script I call it a home
lab maintenance script that I used so I
run a small set of vps's and whatnot
um I quite often
SSH into them as we all probably do
um but
what I don't want to see when I SSH into
them is that
or usually I don't want to see the
initial connect message that the initial
connect message is usually you've never
logged into this host before do you want
to and it's like I think it's the host
I'm connecting to I always have that
moment of Doubt I've
dagnabbit now I can hear it sorry guys
um I always have that moment of Doubt
logging into it and I know of a previous
Linux conference where someone did do
that and ignored the warning and were
subsequently slightly pwned
so
uh for SSH key hygiene so
I tend to now be quite security
conscious I tend to make sure that I
populate my global
known hosts SSH keys with all the keys
of the machines I'm connecting to so
that when I do connect to them if I ever
get a scary message like before I know
something is really wrong and I never
get the you're connecting to this
machine for the first time
sorry just
what's up so host names this is uh good
Lord my slide is incredibly small sorry
I've got to appear around at this one um
so this is what we're manipulating
uh the top one is the name of is the
the public
uh known hosts also the public host key
that the server will be known by and
what we've got to do is get the contents
of that into the SSH known host file the
global one
um
slight wrinkle these days is that
um we hash the host names because
security I'm 100 sure why but it's
probably
if they do it it must be a good reason
so we don't tend to keep the host names
in the clear these days anymore
uh the other
key point there is that there's also the
uh so the in the SSH known host file
there's the uh the hashed hostname
there's the key type and the key value
and usually there's a bunch of key types
per host
so this is the ansible Playbook that I
originally had for it
um
and to fit it I've stripped out no I've
stripped out a lot of the comments and
now when I go back to look at it I quite
often find it quite difficult to read
without the comments which is the part
of the whole beef of this talk
um so
this Playbook basically it logs into the
room well it uses uh
it fetches from the remote host
uh the list of files that we need to get
actually prior to that sorry
um
we know the remote host by one of three
ways we know by its fully qualified
domain name it's fully qualified domain
name with a DOT on the end of it and
there's good DNS reasons for that which
I won't go into here
and then it's also known by its IP
address which is the long convoluted
host vars underneath it
we then connect to the server and we get
a list of the public key files that we
want to bring back across
and
with a new slip to fetch them slip
brings them back as a base64 encoded
file because it's not uh it's only ASCII
clean the connection so
um the
we retrieve the the key back actually
we've achieve all parts of the public
file The Next Step then splits it up
got
it's boring me even
it's not boring it's a lot of detail and
what I find and this is
partly what I'm trying to convey is what
I find is that with the comment strip I
get confused as to what it's doing
easily and I have to stare at it for
quite a while and again it's the problem
I'm trying to solve not the problem I'm
trying to solve in the tool that I'm
trying to solve it with
um there's all sorts of chicanery in
here there's the uh the use of default
for the SSH host key because SSH host
key when you do the original pass
through it is blank or it's actually
doesn't exist so if you don't have the
default ansible then throws an error and
saying this key has never been used
before and it's like well I'm just
looping through it typically if I was
doing this in Python I'd just set the
variable beforehand so I know it's
default State and then proceed through
it
uh
this goes on
this then gets to the the
the uh
the uh the end of the wedge of the
matter I guess
um this then actually updates the known
host file
we do this on the local machine we have
delegate 2 on localhost which says for
all the data you've just collected from
all the remote machines we're now just
applying it to in this case it's usually
the laptop that I'm running it from
um we're doing a lot of looping we're
doing looping with products and again if
I don't go back and review the ansible
documentation for I think what is it I'm
doing here
it just slips out we also have in Loop
control where we can sit there and as
it's as edible is looping through it it
sits there and reports only on the
subset of the information otherwise it
just tends to blurt out
great deal of noise onto the screen and
if something breaks you're not actually
sure what's broken
so I come to
there's
there's some ways you can work around
this like you can
deploy you can modularize stuff to tasks
but again once you do so you've got to
sit there and you've got to be aware of
what variables are going into and out of
tasks you've got to track that yourself
you know we have methods in Python where
you can pass in local variables
these are things I don't have to worry
about these are just things that occur
naturally to me when I'm coding it's
like but when I'm coding an ansible I've
got to like no I can't do it that way I
have to track all of this state myself
that's the computer's job I don't want
to be doing the computer's job
I've got other things to do stare at my
ceiling
so
again as I'm obsessing over this
um
I love this quote
um
it's from an old web comment which no
longer exists unfortunately but I've
still talked to Phil Barnes occasionally
Owens the program manager and Desmond's
the programmer
and this is one of the things I
struggled with is like this is part of
my struggle with this whole concept is
I'm still writing code you know and part
of ansible's reason was to be able to be
usable by the every man sorry every
person
sorry that's gender inappropriate
um usable by everyone
it should be relatively simple to put
together and for simple cases it is
ansible was quite simple but when you
get to start doing complex things with
it it just for me and it might not be
everyone here but for me personally it
goes off the routes fairly quickly when
I have to burn brain CPU Cycles on
details that I'd rather be burning
elsewhere
so I was concerned
when moving to python when moving to Pi
infra but not just paying for it I was
looking at other programs in the same
space I was concerned am I just swapping
one program from another one problem for
another
but I don't feel that's quite the case
um again it's more the power of
abstraction that you get from Python and
being able to express problems in Python
so I've converted that script I'm sorry
that Playbook
um
Pi infra
and as I said point for basically the
inventory and the deploy scripts what
its versions of playbooks are
um
for all intents and purposes python so
first up is the inventory
um for so for my example
um we just this is the inventory.pi that
we pass into paying for it's just
basically a list of servers normally
it's just a straight list of servers I
actually
like again I like my running dialogue as
things are running to be reasonably
cleaned so you don't have to pick
through and try and pick through the
error messages so I like my server name
short
and my SSH known assh hostname's long so
basically when playing for runs it knows
that the host that it has to connect to
known as server one is actually
server1.example.com and so on you can
pass other
uh variables in sorry you can assign
other variables into this these are just
the ones that I needed for the purpose
the
upshot this though is that it's
executable python code so you can use
something like requests and
grab your inventory from you know a
convenient Json server that's located
nearby but has all your stuff there if
it doesn't come back in the format you
know that's that's like this you can
Mudge it within the script you don't
have to write
a uh a separate inventory source as you
would do with ansible you can actually
do it it's it's python you can do it as
is
uh so paying for when running this it's
just concerned about prod servers it can
it basically any uh variable that's
sorry any yeah variable that's set in
the module space it uses
uh as the inventory so you can have prod
servers you could have Dev servers and
they'd be selectable from the command
line as well or it'll just execute
against the lot
uh
uh I'll come back to it local shortly uh
but at local as I said before it's uh
basically saying I want to connect to
the local server not virus sh
but I'm sorry I won't come back to it
just I'll come back to it right now uh
the reason we do that is that Pi infra
doesn't have the concept of a delegate
to so when previous Playbook had a
delegate to to say I need you to run
this on the Local Host
time for it doesn't have that but it
just lets you run your playbook against
localhost anyway
uh and we have group files as well
same as ansible so almost feature
complete for that and this is basically
saying this is the SSH user which isn't
my user in case any of you thinking
about trying to hack a new my framework
um and the key
which one of you have either but um be
sure to commit that to uh GitHub later
uh
interesting bug and it's probably worth
keeping and this will be uploaded to the
repository um interesting but paramico
and modern versions of Debian and Ubuntu
SSH servers
can't negotiate
can't play nicely there's a couple of
algorithms there that it seems to it
tries to use and then dies badly but
it's a simple matter to disable those
and say just exclude these algorithms in
your initial setup and negotiation
and then go nuts
um the last line uh
that's also a paramico thing uh paramico
just uses your local your personal home
directory SSH known hosts by default
holy duly I am so sorry
have you wiped that five minute thing in
me yet you did sorry about that um speed
run
I apologize for this because this is
what I swear I wouldn't do
ah the deploy script proper
ly to the deploy script
uh the deploy script only needs to
execute on the local host and this is as
you can see pure python
um I grabbed some bits from paying for
it that I need I also grabbed some bits
from help script that I need
and then I sit there and I go right I
need to slurp up all the keys from all
the hosts that aren't me
hence the I name does not equal that
local and the ones I've successfully
connected to so if there's any servers
offline it skips over them as well it
grabs the connection IP the SSH hostname
and then executes that fact SSH known
host keys
it then sits there and creates all the
host names based on that so that's the
fully qualified domain name with them
without the dot and the connection IP
and then bunks that all into a list of
lists
and then it calls this is H known host
subset which sits there and says passes
that all off to the SSH host module
um
which is the operation that Pi infra
runs
so it does that tells you the name of
the path the path to the message you
want to update name is the descriptive
field that comes up as it's running so
you can
so you know what it's doing in the play
and of course we need pseudo access to
do that so just say give me sudo or give
me death
I have cheated slightly but to be fair
answer we'll start at it
ansible has a lot of building facts and
modules that we can leverage on
um
that we do not have so I've actually had
to write
two facts and an operation
I'm going to really speak on through
this
this is a fact
these are very simple
these are so simple I've spent no time
thinking about writing these as it
should be again this is the problem that
I was trying to solve I do not want to
be thinking about how to solve these
problems I want to be thinking about the
problem that I'm trying to solve in
ansible sorry the automation problem not
in ansible no
um so a fact is basically we derive it
from what's called fact base which is
just the you know the the superclass
there's two methods that we need to
implement command which in this case
we're actually there is a helper script
that applying for supplies that just
lets us go hey we just want to cap the
output of all those files
and then there's a process which it runs
which basically grabs the output of all
those files which is executed across SSH
brought brought back and then we just go
hey
do with it whatever and what I'm doing
with it here is basically just returning
a dictionary that says this is the key
type this is the key value imagine says
one minute
um
but this is the thing is that the fact
is structured in a way that helps me
solve the problem that I'm trying to
solve
and skip over the next fact
should note right
and then this is the operation the
operation is basically it grabs all the
data that you've collected and then sits
there and
where what I'm doing here is irrelevant
skip that
um
but we need to basically say in the SSH
known case files we need to delete Keys
we need to append Keys we sit there and
we work over
we do basically technical Mumbo jumper
don't need to care about it but what we
do at the end is we go for the lines
that I want to lead out of the key file
this is the said command I want to
execute for all the keys I need to
append these are the keys I need to
append and if I've made any of those
changes
it's almost up if I if I have made any
of those changes then I need to use SSH
Keygen to sit there and reformat the
output to rehash the output
and I think I'm done and I apologize
um
30 seconds
right
there will be no questions
um
if you do two months fee operations to
Pine for when you run it you get to see
the commands that it spits out over the
top commands are the facts that it's the
shell commands that it executes for the
facts the bottom commands are the ones
that the output of the operation that I
ran and that shows you yep exactly what
it is it's doing if you use three minus
fees it actually sits there and shows
you the command the response from those
commands coming back this is great for
debugging
any Lessons Learned yes time up talks
more properly uh no questions
this time we're really done I apologize
sorry
honestly pretty good speed run