foreign
[Music]
next up we have advanced testing and
rust uh and we have turban Harding uh
come to speak to us today about as you
would expect from the title of the
speech uh some Advanced Techniques for
testing your rust code please give
turbine a warm welcome
how's everyone doing
good
cool I'm Tobin I'm going to talk about
program um testing and rust Advanced is
a bit of a subjective term so by
Advanced I mean I feel like I've
advanced in the last six months and I
thought something was pretty cool so
hopefully you'll think so too can I get
a show of hands how many people write
rust for a living
written a few projects in Rust
Court never heard of it
so I always wanted to do always wanted
to see a talk on testing
cool no worries um if you want to look
um play along there's a repo there I'm
going to be showing a bunch of code I'm
just going to be in the editor in the
terminal most the time so yep
um also it's not a formal talk it's just
like a show and tell so if you want to
know something yell out I can take
questions anytime if I'm wrong feel free
to stop me I'm not a QA expert I'm not a
rust expert I'm just got six or seven
years programming professionally and I'm
just a guy likes rust
cool
so I'll run quickly over unit tests and
integration tests and directory
structure then I'll talk about rustock
examples a couple of different ways to
do mutation testing on what that is and
why I think it's so exciting
we'll touch on property-based testing
and why I haven't done it much and then
we'll do fuzzing and code verification
I'm a bit light on knowledge on the last
two topics but uh you know when I wrote
the submission I was hoping to learn a
bit more about it before I got here but
yeah
[Music]
cool basic stuff unit testing
integration test is language agnostic
yeah it's not all important about the
layout and whatnot there's rust specific
ways to do it but integration tests the
main difference is they test your API
surface unit tests have access to your
internals
so the question that often faces me is
you come to some new code looks alright
it's got tests they look all right bit
messy about their tests right
um I read through the code
Linus doesn't test so we're right to go
right
um but no
how can we check how can we assure
ourselves that we're not full of bugs
and that things work
one metric is test coverage people like
to say a line of code was executed by
their tests
another way is mutation testing
so mutation testing is as the name
suggests mutating the code so you have a
test Suite you have software that
changes a line of code and then you run
the tests if the tests pass then
obviously you didn't test it because you
just broke it and the test will pass
so it's kind of a cool concept I never
thought of it before until a few months
ago and I came across it and I was like
sweet so we have this one is Cargo
mutants what cargo mutants does this
default default is it just drops the
body of any functional method and
Returns the default type for the return
value so if you're returning an INT
it'll just return zero
it's cool
all right
now I'm going to jump around I might
talk a bit fast I get a bit excited so
if I'm going too faster you want to know
something really stop me
this is just a wrapper type something
you do in Rust often you're wrapping
into something that you want to add
abstraction to so I hacked this up it's
a bit contrived but it's the aim is to
show different tests right so we have a
struck num represent a signed integer we
have a Constructor from a sign type we
have a Constructor from an unsigned type
we have a getter two signed and two
unsigned and then we have a more
interesting method ABS to get the
absolute value
[Music]
then we have some standard rust things
triform is conversion functions and some
error logic because I work on a library
all day I do all the error handling
manually and
we have two tests for abs
a positive test case and a negative test
case
sorry testing a positive value and a
negative test value
sorry
looks good right
all the non-trivial methods I tested the
one non-trivial method is tested
it's got two tests so is it all right
let's see what cargo mutants thinks
cool so if you remember we replaced the
return value with the default return
value it's as it shows there we're going
to ignore display
for now and we'll just look at so we can
see num2 signed a num to unsigned are
both not tested
and if we go to the tests
we can see that's clear right
cool
so we've added
some stuff
let's talk about that
rust docks
rust docs are specific to rust but I
believe they're based on the python
system they're similar to what the
kernel uses now as well they pull the
documentation out of the source code and
one neat thing I don't know too much
about the others but one neat thing is
you can have code that's actually run in
the examples so you can have
oh and they only test the public API
they're good because they demonstrate
usage and
their document better yep
so
see now
from sign now has a bunch of stuff the
standard examples header and then with
the three back texts we have code there
that runs
now this is nice because
um it renders I didn't open that before
I'll just leave that if you want to see
the rendered version and I'm going too
slow we can look at that
yeah so you've got when people read it
in code they can see the example it gets
rendered to HTML and all looks lovely
syntax highlighting and whatnot and the
cool thing is we can now
spell nightly correctly
so you see the top two are our unit
tests and the next two other tests we
just added the examples
and let's run cargo mutants
awesome
so we killed two of the mutants that one
like I said we're just going to ignore
for now
um but we've only done one test
and it's telling us the whole function
is tested so that's probably not true
right
we had a bug just the other day where it
was a wrapper type and we had a sub
function and an ad function and all
these things and there was cut and paste
error where the sub function called add
so like it's pretty embarrassing thing
to merge but this would have caught it
so it is useful
cool so now we have awesome docs we've
got a good test coverage of the API
because if anyone changes the API the
test will fail straight away so if you
see a patch come in with changes to the
examples
then you know that change it to the
example code then it's a good hint that
you've changed your API right
cool but is it tested sufficiently
let's give mutagen a go
so mutagen is a mutation testing as well
but mutagen is a little bit more
aggressive than cargo moons cargo
mutants you didn't have to change the
source tree did you notice we could just
run it we can run it on any tree and it
does every method mutagen needs some
Source tree changes but it can change a
lot more things so let's have a look at
that
we've got a dependency on mutagen coming
straight from git
we've got this line
10 got numbers yep line 10 um 10 and 11
we add a use statement and I use an
attribute don't worry about that it's
not super important it just means that
only include that line under certain
conditions
and then we're mutating as first start
this one method
so again the attribute is a little bit
complicated it's not all that important
it just says to the compiler too I'm to
mutagen to mutate this
so mutagen is going to mutate from
unsigned let's give it a run
cool so it's created three mutants and
they've all survived
um mutagens written by Geeks so it's all
about survivors and mutants and zombies
and stuff
um
we need to kill them
so what to do you can see here all these
numbers the first number 51 is the line
so we can go over to 51.
[Music]
and as this line effects is less than or
equal Max
you can see it says what the mutant is
doing replace less than or equal with
less than
so it becomes there's only a few it does
a bunch of things I don't know all the
things it does but a few of them um
often survive and they're usually
boundary cases it changes most num and
if Clauses it changes most numbers but
and it's usually at the boundary that
that's interesting right so now we see
already that this is pretty useful
testing boundary cases is useful
so we have three on one line we can see
it's a boundary case so of course let's
check if the value is Max so we go down
let's
I added another we added another test
and here we go so we set X to Max we
create a number and then we send it to
sum we use it's a round trip so we
create it with the front with the
Constructor and then use the getter and
assert that they're the same
so yep that should look good that looks
good right
profit both killed except one
all right
everyone good so far
yep you can read the fonts and
everything's fine
awesome
all right so this
um the next one is a little bit more
curly we've got the mutant here is less
than a replace less than or equals with
greater than or equals so that seems
like it's obvious right but when I saw
this I was like well why is that even
working of course like it seems strange
it's not a boundary case it seems quite
wrong
and then
I looked a little harder and I worked
out that there was an invariant that
wasn't documented
and that is that from unsigned
takes an unsigned 32-bit integer but
ours is assigned 32-bit integer so
obviously there's some bits that are out
of range right
so
the test that I wrote
is in the documentation so it's an
examples so we added this error section
that's like in Java where you it will
put a heading errors and everyone knows
that's describing the error tone so I
have um described the range
and then added a test that round trips
to and from the Constructor in the
gutter
this is nice because if someone comes
and changes one and doesn't think to
change the other then you know and it's
like really obvious then that they get
us and um set us for people new to the
language
the library I work on Rust Bitcoin we
try to do things that help people that
are new to us because there might be a
hardcore Engineers come in that haven't
done rust so we try to explain things if
they're a bit funny and also we try to
explain Bitcoin stuff so that people who
are new to bitcoin can learn from the
code base too so it's kind of a nice
principle
all right let's see if we killed them
all
win all dead
so
now we have pretty good confidence in
our tests right and here I'll segue to
why I put this talk in in the first
place testing something that I always
want to have done good then it's often
messy you feel bad doing code chain
fixing people's things but and it's also
sometimes boring but
there's the idea that you can read the
code and know what's going on but it's
never the same as if you wrote it right
whereas when you when I was doing using
mutagen I was breaking the code and then
really having to look why did it break
and I so I really had to understand the
algorithm so having to write those tests
was really good and I really thought
that after writing those tests I had
covered the whole thing
there's definitely could be ways you
could break it that are not like that
mutagen doesn't cover but I felt much
more common that I'd really looked at
each line so it's yeah it's pretty cool
and I was pretty confident after doing
mutagen after doing mutation testing
that the API service was covered that
the internals were like pretty good and
that all the boundaries were tested
but
is it always correct
what's always
all right do you have any questions on
fuzzing
oh sorry not on fuzzing on mutation
testing
um I don't mutagen needs nightly I don't
know exactly what it's um what's using I
didn't look in there's yeah one of the
dependencies requires nightly yeah
awesome so um the question was why did I
use nightly so that plus nightly that
there's a nightly tool chain of stable
tool chain and a bunch of old ones and
so nightly has some features so most
likely one of the dependencies uses a
feature that's not in stable yet so
that's why I was using nightly
that was the question
oh sorry
yeah more
cool um does anyone want to know about
the attributes did they look funny
if that's what you wanted to know about
okay also I've got plenty of time so
I'll show you guys that
so
um CFG
and then the attribute is everything
inside the brackets
and it's an all is just um like a
logical and so CFG says if um test and
mutator both enabled they exist and
they're enabled and then include the
next line
test is enabled by um cargo test these
are called conditional compiler
configuration options cargo test enables
one called test they can be any string
and I created one called mutate and
you'll see when I run it I pass um
Pastor rather compiler CFG mutate so
that mutate can be anything and that
just sets the variable to true
so then that'll be included
and then CFG attribute what it does is
if the first thing is true it changes
that line to be CFG the next thing so if
test and mutate are enabled this will
change to B this line will be changed by
the compiler before code gets compiled
to be CFG mutate which tells our mutagen
to mutate the method
okay
so the if you look up online how to do
this they don't do the rust Flags
business that's something I added
because I wanted I didn't want mutagen
to run anytime anyone ran the tests I
only want to just run when I read NCO
and I really wanted it to run so that's
why I did that we do the same with
benchmarks um let's yeah it's kind of
neat
yeah so if you look up how to run
benchmarks is another nightly thing
where often benchmarks use this crate
test and it needs nightly and
if you do want the suggested online is
put a line in the code that says if
you're running tests enable this test
grade and that breaks compiling with all
features so you now cannot compile with
cargo check or build dash dash all
features and I like to be able to type
that because it's heaps easier when
you're I'm developing so I came up with
this trick yeah to add a CFG flag and
use that yeah
cool sorry fuzzing um I don't know a lot
about fuzzing but what I do know is that
it's taking arbitrary data and feeding
it into your functions and checking that
they don't explode
I don't have anything particularly good
to fuzz in this code so I added
oh sorry I do want to see the last few
mutations
thought that went quick
so I added mutations to all add a
mutagen to all the rest of the
non-trivial methods and so now we've got
13 mutants and
they're all killed by two
so you can see now from signs done
two unsigns done and abs is done so
we've got two unkilled mutants at 86
all right
all on this
self-zero just gets the inner field so
the inner field which is the assigned
integer if you remember the sign integer
is less than zero
the mutations are replacing the zero
with a one and the less than with less
than or equal so standard boundary cases
can anyone see what's the problem
this if Clause determines which branch
to take right
changing less than or equal to from to
eat less than to less than an equal and
or changing the zero to one functionally
just makes the zero value take a
different branch
but the zero for the value is zero it
doesn't matter which branch you take so
zero minus zero is the same as zero
so I talked to the guy who wrote mutagen
and he called this I don't care Claus I
don't care statement so the value is
zero I don't care you can Branch either
way but mutagen can't actually handle
that because it it can't tell what's the
difference like why you can branch on
either
so an open question that um you guys
could help me with is I don't know how
to put this information into the code
because I don't want I don't want to go
and spend two weeks writing mutation
testing do all this stuff and then a
month later someone else comes back and
they try and run it and there's 1500
false positives
and then they've got to go through
everyone and check them that seems like
a waste of time so the best I came up
with is putting a code comment putting
those lines and saying like lying such
and such is false positive but that's
like pretty hard to maintain it's going
to go style pretty quickly but I thought
at least if someone's just trying to
skim over it they can go oh there's two
thing false positives there that's
probably all right
but yeah I'm open to tips on that one
so tests
ABS handle zero
that's an obvious um is that a boundary
case when it's right in the middle
um it's kind of obvious that those that
should have been tested
and
so the suggestion is a conditional
compilation and explicitly test for zero
I think I did try that at some stage
because
so when you mutate your test for zero
and then I'm I believe I did try that
it's a good idea because I had it I
believe I did try it and you get another
false positive
yeah there's I I think that's what
happened at the end if we have this time
I can type it up we can see but thanks
thanks
any questions about those tests not
nothing crazy there
all right that's fuzz
all right to fuzz I added
a from string implementation so that's a
trait from straw that converts from a
string to whatever it is you're doing
you're whatever type
and that's a candidate for fuzzing it's
a bit contrived in this example but like
I said I couldn't come up with any
better so that's the best we've got so
we take a string
and I'd like to know that you can pass
any string in the whole world and my um
code's not going to blow up
so
to do that we create a crate called fuzz
the um when I was going over this talk
yesterday I realized does everyone know
about Edition 2018 2021 2015.
so every so many years Russ puts out a
new a new addition and that's when they
bundle up a whole like a whole heap of
change or like right now here's a new
edition and then people can jump from
one to the next and this is Cargo cop
program from code that used Edition
2015. so some of it's going to seem
strange like the fields down the bottom
but if you don't know about that I
wouldn't worry about it if it's strange
just don't worry
so I just write a little script that was
just going to run it for 10 seconds
fuzzling needs to run for a lot more
than 10 seconds obviously
so it's just going to keep feeding data
into this hung fuzz provides this macro
fuzz
and that takes a closure D is the data
it's going to pass in
and it's going to call the function that
I defined do test now do test is a
little bit contrived
here the standard library is pretty much
taking care that I don't blow up so it's
not really this is not really useful but
you could imagine a time when this would
be more useful so I convert the data to
a string using from utf-8
and then if it converts so if that
random data happened to be valid utf-8
then we can drop in and we can try and
create a string from it this is because
the from string extra accepts it takes
us through as argument so
I create a string and then imagine that
we've written a library and we've got
another Library somewhere else and we
want to test our inflammation
implementations against them in this
case I'm using the stood lib which is a
bit pointless but you get what I'm doing
so I create a string from using from
string Radix if that error is I assert
that our library are it as well and if
that doesn't error I assert that we got
the same thing
now I said I was going to mention prop
testing prop testing property-based
testing and the idea is that you pass
random values in and then you write a
test that tests the prop just the
property not the exact value
property testing is a version of fuzzing
and this is like prop testing would
probably be better in this case if like
for all the other Constructors the
weather line is between fuzzing and prop
testing I don't know exactly
and I'll explain to you why I don't use
prop testing in a minute
cool so it's kind of fast for 10 seconds
and miraculously is going to find no
bugs
but that really hasn't given us that
much more confidence right
um yeah we have fuzzing and we run it
during the night for a set amount of
time and it gives us some confidence in
some cases but we'd like something more
we want to know if it's right all the
time
code verification
so
prop testing passes some number of
random variables to your method or
function but imagine if you have a
function that takes two u64s the space
that you're going to test is pretty
small compared to the whole thing right
so it'll be nice if we could have some
way of knowing if it's for everything
and what we can use is something called
canny Kenny's a library that does code
verification it uses math magic to
verify that all of any value
mathematically every line can't be
reached or and can't crash it can't
handle all logic it chokes on Loops a
bit as you can imagine the mask quite
complex and if there's like nested Loops
it just takes forever to verify so how I
did it is I would add cany tests run it
and if they ended within like a
reasonable amount of time a minute or
something then I was like sweet if not
there's some things you can do otherwise
you have to use a different type of
testing so it's good for a subset of
things that you can test
foreign
's pretty easy to use you don't even
have to add a dependency because you're
going to we run it with a different
command you just add this with a
compiler attribute again conditional
configuration attribute and then
where you have the attribute proof which
says this function's a proof
now similar to if you've done prop
testing so County any will give just
gives me a random
sorry it um signals that this can be any
value
and then I write the test you can also
limit that so I could say any i32 and
then I there's a method I can function I
can call that says assume that it's less
than 10 if I want to test everything
less than 10. and then I just write a
test very similar to what a prop based
test would be
sorry very similar to what I testify
pasta value because I know
it's a property isn't it because we
don't know the value so we converted it
and then we just checked that it was the
same so unsynabs is coming from the
standard Library
ABS is the function we wrote and we're
just comparing they're the same so again
a little bit contrived but we were doing
this with like a hashing library and
then we use another hashing library and
we check if we get the same thing so it
can be useful
cool so I've got one failure
that's nice we've got three successes I
won't dig too much into it because I I'm
like I said I don't know I don't
understand it all that much it's got
good documentation but basically I just
looked for red and then tried to work
out why it was red
so if we look here line 89 attempt to
subtract with overflow
all right so it's the same line we're
looking at before now it took me a fair
while to work this out and actually I
had to work it out again yesterday when
I was going over it because I wrote this
a little few weeks ago
um Can anyone say what's wrong
attempt to subtract with overflow
the clue is our inner type is assigned
integer and we're returning an unsigned
integer
how's your two's complement knowledge
so the minimum value the minimum
negative value of a type of an unsigned
to other sign type can't fit in the
unsigned type
so if it was is it two bytes would be
minus four three bytes minus four
doesn't fit into three bytes three is
the maximum value
does that make sense
yes
not correct
well
the gate
and let me
let me see
[Music]
so three bytes there if it was a three
bit it's a 32-bit number if it was a
three bit number the minimum value would
be -4 and for an unsigned
you can't have positive four
so that you think that's not the reason
that that's that that arm test is
triggering
yeah so that's why the Overflow
the
fixes
[Music]
outside the the what was said was a
suggestion is to change
to convert to signed
if you cast a sign then you get another
warning for that you're losing value I
believe
[Music]
it's already signed
so yeah we can't cast one sign because
we don't have the we don't have the
value right so I didn't add the fix the
fixes to to return an error right and
then check the value and if it's out of
range you have to return an error
would that be right
[Music]
you can simply check for the
value and return
value
in this case for the other approach
involves magic involving ignoring errors
on arithmetic
in Gold that has a bit stream
so if I got you right you're saying um
ignore errors
if you negate
so who's talking so I can look at you oh
yeah yeah
wise representation
of your answer is the same regardless of
whether it's signed or unsigned
so in effect if you can convince
rust to take the bits
and just manage it you get the right
answer yeah yeah awesome awesome yeah
good idea
thank you
so we saw that um verification is useful
we saw that it found a bug that I
wouldn't have seen like that that's
pretty that's reasonably I'm reasonably
confident to say that I wouldn't have
seen that I wrote it and I didn't see it
I think that could have gone through
review and not been seen so is that um
Kenny for the win
yeah pretty sweet right it is um can be
slow to run and mutant ninja testing has
a problem that I wrote like a bunch of
tests and then I looked back and I'm
like that's all super trivial that's two
hours of time that I don't think that
really adds any value
so I don't know exactly where the line
is like how much time to spend on this
I'm still learning that
and I don't know how much time to wait
for the verification was one minute too
long too long or too short I'm pretty
impatient
and yeah if you've got 20 000 lines of
code and you start putting you know you
put in 100 200 mutations
so yeah a few open questions
so summing up we looked at rust dock we
looked at examples we saw that the tests
run those we looked at mutation testing
basic with mutants cargo mutants and you
didn't have to change the source tree we
looked at mutagen which needed Source
tree fixes and possibly maintenance
burden it's definitely a pain in the ass
to run because you get a lot of false
positives and it takes ages once you
start putting a lot of these it takes
ages
um I was trying to mess around with
attribute so you could run specific bits
but I couldn't find a really nice way in
the end I was just like using said to
change everything change all the lines
the attributes and then making a change
in one place and then resetting the um
the index
we've got Hong fuzz for fuzzing uncanny
for verification
[Music]
my talk
so listen
[Music]
you guys want to say anything or ask any
questions or give me any ideas
thanks very much for the talk thank you
coming to us so like my background's
Java kotlin Dart and they're like the
most common like
rough measure of unitesque is coverage
I'm just wondering like you didn't
mention that or would you be able to
contrast like using like like code
versus for instance mutation testing and
like do you think one's more useful in
the other in terms of rust or like one
doesn't like coverage doesn't coverage
doesn't make so much sense in Rust sure
I can only give an opinion it sounds
like Tully's my personal opinion
um I've never been a cure like employed
as a QA guy I always was suspicious of
coverage because I just have to execute
that line of code in a test
but like
there's different kinds of Courage c0c10
ah cool okay there are different models
of coverage um C zero coverages did this
line execute yeah one coverages did you
test all the branches
etc etc so you get a different depth of
coverage cool levels of things and a lot
of people just stop and did I execute
this line but that is not the only model
of coverage that exists excellent okay
thank you that's awesome all right
that's something I've learned
so yeah I can't answer is better and I'm
definitely going to go home and check in
coverage I always just thought the badge
and was like what does that tell me
thank you that's excellent
I fully answer your question
cheers
any other questions or anything else
you'd like to see further I'm about all
day so you can um come up to me if you
want to chat yeah I literally got really
excited when I was doing that mutagen
stuff look in this algorithm when I
found a code path uh false positive and
I worked out that the the algorithm was
correct but it was so complex it took me
like 10 minutes to work out that was
correct and I was like that's awesome I
wouldn't have looked at it that hard I'd
just gone yep sweet looks right so yeah
it was really cool
off the back
I've been asked before yeah
I suggested before about conditional
compilation of an explicit zero test and
you said there was an issue with another
false positive do you want to try it yep
so what was the idea we wanted to check
if the value is zero and then do
something different on zero to one right
so is your idea to when mutation is on
when mutation testing is on to do that
I don't actually use this form factor of
keyboard at home so there's a lot of
stabbing
so that didn't work at 90
. cool I'm happy to say this is what I
did last time and that's cool that it
did do so now we've got the zero one
problem
oh yeah so here we can't add another
line the zero one problem is a false
positive you can't fix so we've replaced
one false positive with another at least
there
and then we've still got a problem in
94.
the same problem we could we could
assume that we could change the next
line to remove that and we'd get another
so we've got another false positive
we'll replace one false positive with
another
do you think you could get rid of those
are you convinced that I didn't face it
any further than that
the quest
it's not very clear why we've got the
new false positive
oh that's because it's the this boundary
case where you've got a zero so you
could change
oh no if that's a one that should be
fine that's
oh yeah because one is two zero minus
one is one and one is one
no
yeah so that's the same right I know
that's not because it is trans zero that
is not clear why that's false positive
ah because we haven't written a test
so we have to add it so we'd have to add
a test you want to see me do that have a
good time we'd have to add the test
for one
it's a bit lazy do it like this
foreign
system works
so now we're left with the ones at 94 so
we could probably get rid of those so I
guess then the question is how much
um
the function's a lot more messy now
it's got that is that is that a net win
to remove a false positive
I don't know I tend to think that's not
a net win
yeah I kind of get I kind of feel like I
don't want to come into the code base
and make a bunch of changes and then the
next guy comes along is like wow what's
this 15 lines of [ __ ] that I have to
look past now just so that you could fix
one small thing that probably wasn't a
problem in the first place
so I'm always a bit cognizant of that
what are you out of blocks I reckon that
more experience blocks is that true or
is it worth having all this
probably not worth having it yeah
yeah
do you have an opinion on the idea of
putting the false positives in the code
comment do you think that's worth the
noise
probably not was the comment before
[Music]
yeah
yeah so I used I put it above the
like just here
and I just wrote
um
muted in false positives and put a
couple lines then I use the same form
everywhere so it was easy once you've
seen one you didn't need to keep reading
them
suppress it yeah yes yeah so in in the I
the the comment was there needs to be a
way of suppressing that in the test and
I went into the refund talked to the guy
and it shouldn't be that difficult but I
only put two hours work into it I
couldn't get it done in two hours and he
doesn't want to do it yet so yeah it
shouldn't be that hard
you have to um it's like reasonably
Advanced for us you have to mess around
with sin and proc macros and things
but yeah once you're um passed rust 101
then you can mess around with Brock
macros and lose a lot of days
but yeah it would be really cool if we
had that
awesome thanks for your time thanks for
coming
[Applause]