[Music]
greetings greetings and welcome back to
the devops track sorry Dev oops track
oops
oops uh thank you for joining us here
today at Pon it is great to see everyone
here in the audience uh kicking off in
our first session now we have Evan uh to
talk to us about who tests the testers
making testing
pipelines Evan is a senior software
engineer whose passions lie in improving
the developer experience by enhancing
code Health optimizing workflows and
working towards no human errors. comom
when he's not thinking about security
informatics or giving talks about his
many projects you'll find him noming on
subway cookies or chasing bunny rabbits
take it away Evan thank you very
much I I love this screen saver I think
it's very fitting for the talk um but
yes hello everyone uh hope you're having
a great day um so I'm I'm Evan as you
might have heard and I want to talk
about pipelines because while working on
the platform team I've developed on
pipelines as part of a gitlab repo to
run tests for our
codebase but I noticed there were quite
a few oopsies that were made that could
have been prevented if there were
tests and it got me thinking if we as
the platform team help developers run
tests for their code who helps us the
platform team write tests for running
the
tests so so today we'll unpack that
topic in three easy
steps I'll briefly introduce how to make
a pipeline and what it
is show how they can
break and finally Deep dive into the
methods to stop them from
breaking all right so for this section
could I please ask everyone to raise
their
hand okay wonderful we're going to do a
little survey so keep keep your hand
rais also I'm going to ask to take a
photo because I didn't think that would
work and I think this is really
funny wonderful you look great Okay cool
so keep your hands raised keep your
hands raised okay wonderful um I I'll
let you know when you can you can lower
them okay so keep your hand raised if
you know what a pipeline
is okay so we got a few hands down don't
put your hands down just yet but keep
your hands raised if you've worked with
a pipeline with r being code or work or
whatever okay still quite a few all
right now keep it up if you've written a
pipeline okay so there's still quite a
few of you that have your hands raised
and that's great all right one last one
keep it raised if you've had a pipeline
that you've written fail on
you okay none of you lowered your
hands I was not expecting this okay
we're in for a great talk all
right so so that we're all in the same
level uh for those of you who hasn't
seen a p plane before they can look
something like this and to put it simply
you can think of them as a factory line
where each step performs different
operations to get to the final product
but in our case our operations can look
something like
setup build test with our final
operation deploying this built and
tested
product so why are p P lines good and
why should you care about
them well they allow us in one case to
create a streamlined process that
automates what would otherwise be very
manual and
tedious and in other cases they act as a
quality control similar to a factory
line if any stage of the factory line
Fails Like building or quality
assurance that item or in our case some
code won't progress to the next stage
and thus won't be allowed to
merge for this presentation I'll mainly
be talking about gitlab
pipelines however if you are a GitHub
user or you use any other kinds of
pipeline tooling most of the content
should be General enough to be applied
and additional resources will be given
at the
end okay so to make a gitlab pipeline we
first need to create a gitlab c. yaml
file that gitlab will use as
configuration and that's kind of a
mouthful so I'm just going to refer to
it as a CI config file from from here
on and if you're new to yamal think of
it as a way to store dictionaries and
lists similar to Jason but more human
friendly we'll make a simple one now
that one can use on their python repo to
ensure that tests pass before merging
here we Define that we wanted the job to
run the latest version of python within
a Docker
image and then we'll Define a job with a
name test
job and Below we'll Define script as the
command to run for that job which for us
will be a python file called
test.py this is quite simple um probably
one of the simplest pipelines we can
create that being said it probably
shouldn't be use as a reference as it's
UTI utilizing a lot of what gitlab um is
offering by default and we can get away
with not having to install any packages
as none are being used so if you are
looking for a reference pipeline I'll
have some links in the resources at the
end that being said given how simple
this pipeline example is it probably
doesn't best mimic the issues that a
real pipeline might so let's extend it
to do so
to do this we'll add some rules which
decide on what conditions the job
runs we then specify that we want the
job to run whenever there's changes that
have been
made such as changes to any python
files this is so that we can speed up
our pipelines by not unnecessary
unnecessarily running a costly job on
unrelated changes to other files within
our repo such as
documentation another thing that we
might want to extend on is having to run
different types of testing jobs you
might not want to just have a single
test python
job for example a database test
job which we could set up in a similar
way except it might uh depend on some
test databases to have been
created so for that we require a new
setup job to be run
which needs to be run under the same set
of
changes and additionally we need some
way of specifying that the database test
job can only run once the database setup
job is
finished to do that we use the needs
keyword which specifies that the
database test job needs the database
setup job to finish successfully before
tests can be started and completed
correctly with the idea being is that we
want as much running in parallel as
possible such that our previous tests
don't have to wait for setting up
databases if they don't depend on
them and so now what we've landed on is
great it's efficient as an example and
it works when it's merged
in however as new PRS are made and time
goes on one PR causes the pipeline to
fail not because the tests themselves
pass or fail but because the pipeline
itself is
incorrect gitlab will tell us that the
database test job couldn't run because
the database setup job didn't run now if
you go back to our pipeline
definition can anyone tell me why it
didn't
run
PPU yes that's right there was a typo in
the changes and so the database test job
ran because there were changes to files
ending in py but the database setup job
didn't run because there weren't changes
to files ending
inpu and the reason I give this example
is because this is something that's
happened to me right it's just common
error and so while it's great that we've
all figured out why this happened and
how the fix could just be a one
character
change in our line of work it's more
important to understand how this was
allowed to happen in the first place so
it doesn't ever occur again and we don't
have to spend time going and fixing all
these one character
changes so can anyone think of some
examples for why this could have
happened feel free to shout
someone yes you could have had some
careless developers you know they're
just typing really quickly and was just
easily missed anything else
that's a good one I haven't thought of
that one so the pipeline was updated to
a new version that doesn't support the
previous things okay cool yes set
relies the setup job relies on
configuration files that just happen to
bepu you want to rerun set yeah yeah we
don't want to keep having to rerun
things y uh anything
else yes lack of testing safety nets
yes right yeah yeah so someone like
right so I repeat it for the for the mic
um
someone someone ran their stuff locally
and they had it but they hadn't
committed it and so they wouldn't have
noticed a problem in in the production
yeah um some other ones could have been
like the config is too complex maybe the
reviewers weren't sharp enough um or we
could have Ru in a better way to um use
um don't repeat yourself and so forth
but those are all good good
points and to me I think that all of
those are all human errors um and as you
might have heard earlier I don't like
human errors that could be avoided by a
systematic change which would allow us
to blame a system instead of a
people so so I would say that at a
higher level this occurred because while
this configuration looks like code it
isn't treated like code because we're
used to testing code with tests but who
tests the testing infrastructure with
tests so now you might be thinking Evan
this is great Insight uh but Solutions
speak louder than problems how do I stop
this from happening to me it's a great
question
to answer it there's a couple of methods
that we'll dive
into the first is validating our CI
configuration using linking
tools the second is running an endtoend
pipeline before committing
changes we'll then look at how we can
run those pipeline jobs before sorry
locally and how we can run static
analysis testing on the CI configuration
finally we'll look at dynamically
generating pipelines using
tools now I'm not going to claim that
any single method is the best but I will
cover the pros and cons to help you
decide which one or combination works
best for
you so the first method we'll dive into
is linting the CI config
file linting is great because it's a
fast method of finding problems that can
be solved before going further and gab
offers quite a bit here they have a vs
code
extension which will tell you within
your editor whether your yaml is set up
correctly they have a pipeline syntax
Checker which will do the same thing as
the extension but through a web UI and
an API
endpoint they also have a visualizer
which gives a quick way of helping you
see the dependencies between the
jobs they have a validator which
simulates a git push event on the
default Branch with the rules and needs
evaluated and finally they have a tool
to expand your yaml into its full
configuration which helps you see what
the pipeline looks like with all of the
special syntax
expanded however all of these methods
have their downsides too
the VSS code extension as you might have
guessed is only for vs
code and even if you instead use the
pipeline syntax Checker whether it be
for the web UI or the endpoint they only
tell you if there's something wrong with
your schema not with how it
runs the visualizer while handy doesn't
help you understand the issues that we
had around the pendencies with the needs
and rules and thus won't uncover those
issues up
front the validator while it says it
simulates and evaluates rules and needs
and I could be misunderstanding here
feel free to correct me at the end
wasn't able to pick up on our problem
and also doesn't have an API endpoint to
make testing
easier and the full configuration
tool the issue with that is that the
expansion is limited and won't expand
out defaults since technically Ally each
job has an image but we've just taken it
here as a
default and so while lending tools have
their use they are only a single piece
of the puzzle just like how we can't
expect linters to catch logic
errors so if lters are lacking due to
not testing end to end why don't we
Instead try running an endtoend pipeline
before merging
changes well the good news is that
gitlab lets us run a pipeline from a
particular Branch such as the one that
is introducing the CI config file
changes and it's also lets us uh specify
variables for example overwriting those
that would be predefined by
gitlab this is great because it lets us
run a p a pipeline
identically excuse me let us run a
pipeline identically to how one would be
run by gitlab such as a post merge
pipeline however there is one limitation
being that the this pipeline depends on
changes for that particular branch and
therefore we won't be able to catch our
problem as our Branch doesn't introduce
the changes that are needed to reveal
that
bug however there is another way and
that is by creating a branch off of our
CI config file changes with the changes
that are needed such as changing a file
that we expect to run in the test
directory this is great as it would
reveal our
problem however as you may have noticed
this is quite an intensive and costly
process as it's slow and manual and may
not scale with all of the combinations
of changes that may be introduced with
lots of different
rules and so if our third method we have
something in between something that's
not as static as linting and not as end
to end as running entire
pipelines for that we can use tools like
gitlab c local and gitlab c
local Yes you heard me right the
difference is in a
dash
gitlab-ci loc is a typescript tool made
by firal in January of
2020 and gitlab C- local is a python
tool made by Adrian DC in January of 201
20 honestly it's just bizarre how
similar these tools are both in what
they do and when they were made and
their
name but anyways at the end of the day
these are both tools that let us run
individual jobs of our gitlab pipelines
locally using
containers and the core benefits of
these tools are that they let us remove
the dependency on
gitlab we might not want to spin up an
entire pipeline line just to run a
single
job and this is helpful as it quickens
our feedback loop by allowing us to
locally replicate job environments that
might otherwise be too different if
simply run as a
script that's not to say that these
tools also don't have
downsides for one being simulators
they're not always going to match the
real
thing and if we're solely testing our
pipeline changes they don't give us any
itic information in advance and require
us to wait for the jobs to finish
running and so in our examples above
both of these tools also won't help us
uncover our problems even though they
might have other
benefits and so with the fourth method
let's explore a different kind of
testing in this front I wasn't able to
find any kind of static analysis testing
tools for CI config files and thus to
solve this problem I begrudgingly made
my own tool CI test CI test acts as a
missing piece of this linting endtoend
integration testing puzzle by
functioning as a framework for static
analysis testing think of it as a
toolbox to add to your shed of
equipment by analyzing the CI config
file and how it's changed we can gain a
better understanding of what the
pipeline is meant to to look
like as an example it can it can produce
snapshots of the pipeline by collating
relevant information such as pairing
together lists of jobs and
changes and so using our example when
the change was made we would get this
diff this immediately lets us know that
an additional rule set was created that
is different from the expected rule set
from all other
jobs and so really we should have seen
this instead where all the jobs
undefined under the same set of
changes of course like everything else
it has its drawbacks and can't solely be
dependent on since it doesn't run in
integration like all the other
methods however it will allow us to
quickly iterate on rules with confidence
which was something that we couldn't do
before and that being said don't take
that as me on the record to say that
this will be the holy grail for all your
pipeline tests and and so forth there is
plenty of room for improvement and it's
been written in a way to be highly
extensible so if you want to help
contribute to this or any other
developer tooling I'm more than happy to
help Reach Out make a pull request on
GitHub or come to the
Sprints all right so the last method
that I'll cover is using tools to
Dynamic ically generate pipelines using
programming languages like
python
gitlab-ci python library is the only
upto-date uh Library I could find to do
this although if you are interested
there are some typescript libraries that
I had dismissed due to needing some more
attention
anyways gitlab C python library or Gip
for
short is a library from Thomas Stein
buff and Daniel F Essen
that allows you to replace your CI
config file with
python now you might be wondering why
would you want
this well if we quote the library we can
utilize all the features of python such
as code
reuse variables control flows and the
like programming
paradigms
libraries
testing Packaging
and anything else you can imagine the
world is at your hands with
python so the way that these generated
pipelines work is that in our CI config
file where we've been defining our
testing
jobs we instead Define a generate
pipeline
job which runs our python CI config
file to create a generated config CI
file that contains all of the previous
jobs that we want to
run the original CI config file had also
defined a run pipeline
job that has a needs on the generate
pipeline
job as it uses the generated CI config
file to then create a child pipeline
that runs all the previous jobs that we
wanted to
run this is great because it means we
can replace writing all of our yaml with
python
okay so what does that look like if we
want to replicate what we had before our
config file would look like
this we import the library create a
pipeline specify the default
image then we create a
job we give it a name add a script and
its
rules doing the same for all other
jobs adding our need
before finally adding all the jobs as
children of the
pipeline and then finally writing that
pipeline out to
yaml and sure enough we can have this
pipeline execute in our
repo and have all the expected down
strep jobs run
successfully however has this caused our
initial problem to go
away well it doesn't give us any
protections from the pipeline failing to
run because again there were no reasons
for it to run correctly since there was
nopu
changes and so fortunately it seems like
generated pipelines won't directly solve
our problem that being said they can
help as the authors mentioned earlier
this programmatic approach does let you
import your pipeline and perform tests
on it allowing us to do something
similar like we've done with CI
test however there is one other thing to
consider since our pipeline is now in
Python they do let us easily create a
change set using a
list that we can apply everywhere and
thus help alleviate any rule mismatches
as all the rules will now be the
same this may have those of you
knowledgeable and yamal thinking doesn't
yamal also let you create a list that
can be refer
and you'd be
correct we can create a key with an
anchor and reference them with aliases
to do the same thing and solve our
problem but if we wanted to do something
like extend this chain set it wouldn't
be as easy as yaml doesn't allow merging
of
lists well that's not entirely correct
gitlab does allow you to merge
lists however this doesn't work for all
cases cuz it only lets you do that
sometimes and it won't result in valid
yaml so are generated pipelines a good
idea I think the authors have a good
view on this which I'll do my best to
summarize the cons are that it could
allow for writing complex pipelines that
nobody else understands when they could
be basic things that build and test your
code and shouldn't be in code
too but if you're not fond of yaml your
CI files are growing to over thousands
of lines long or they're LED with
complex rule sets maybe writing them in
code is
cleaner importantly it's a choice that's
up to you your team and the
situation so to close out if we look
back at our methods and ask what is the
best
approach well it really depends on what
your CI looks like if your CI config
file has become thousands of lines long
gener gener ating it dynamically or
writing tests may help relieve that
burden but at the end of the day all
these methods are only tools to assist
us in our Pro in our problems and if the
underlying problem is the thousands of
lines of CI
configuration it might help to start
there
instead thus my view from all this is
that Simplicity is key choosing one tool
over another will still be adding
another component to something that is
likely already complex
that you'll later have to deal with
question everything that is there and
utilize the least you can to get the job
done as always I open to feedback as I'm
continuously striving to be better you
can find my contact all related repos
tools references and sources at CIT
test. nohum error.com
and if you're still wondering who test
the test is I'll be around after for
questions
thank you to my friends my family and uh
you for supporting me as well as the
open source contributors and finally to
you for
listening I'll leave this up now so that
you can actually take pictures and take
look thank you very much