foreign
Au education
session and uh Stephen here has the
enviable position of being the last
session before the afternoon break
we're going to give him all of our Focus
Stephen is a senior lecturer at the
University of Sydney in the field of
Statistics data science and machine
learning and teaches python in all of
those and what I found exciting is that
he also has a bit of a pawn shot for
microcontrollers and has a fully
automated Greenhouse in his backyard so
we're going to be talking later okay
thank you uh thank you for coming to
probably the most boring titled
presentation of pycon assurance of
learning through testing I'm presenting
this on behalf of work from both myself
and Allison who you saw earlier
and this is work that we have sort of
developed over a number of years it's
not something that we have put together
just for pycon this is something that
has been an ongoing project for us and
in this talk I'm going to go over the
problems we're facing and some
suggestions about how we've solved them
and hopefully they're helpful for you in
the future
so first of all I want to talk about
um cover off what is Assurance of
learning
um the key thing that we're trying to
get across with Assurance of learning is
that you need to make sure that students
are actually learning what we want them
to learn
um or in other cases we want them to
achieve the learning outcomes that we
set so for our units and our in our
context the University of Sydney we set
learning outcomes for our units so these
are the things we want our students to
achieve by the end and we want to make
sure they do that and you could take
that further and say that there are
learning outcomes that each week
subtopics that you want them to learn as
well
so we want to demonstrate this through
assessment and it's not just for the
sort of promise that we give to the
students through those learning outcomes
and what we want to give as Educators
but we actually at the University have
some requirements that we assure that we
do this
and what I'm going to talk about today
is mostly in the context of using Ed
stem but tools like reflect and GitHub
through Renee stalk you would have seen
earlier actually have the facility to do
this sort of testing of your students
code so keep in mind that even though
I'm going to give examples to show
screenshots of ads this is quite General
so don't think that you have to use add
to uh to do this
and here's an example of one of the
exercises that we might actually give
our students to do the description of
one of them anyway so we might get them
to write a simple code a Python program
to
read in a number from the user so
collect some user input and then tell
the user whether that was a big number
or not so this is a very simple exercise
we might give our students in week one
or week two of an intro programming
class
and the students will get this exercise
description and then they'll write some
code the code might look something like
this I've intentionally put a bug in
there hopefully you can spot it for that
exercise
and then they'll go ahead and submit it
so after they've submitted it they'll
get some feedback from a number of test
cases that we've written ourselves so in
this example there are four test cases
shown we have the they're just based on
the number that we input into the
program to test where the student passed
or not so in this case the student
didn't pass the 107 test case because
they actually put out or printed out on
the console that's a big number when
they shouldn't have
and in this example through this talk
we're going to go through the problems
of this sort of environment of uh making
sure that students are doing what we
actually want them to be doing and
learning what we want them to learn
through this automatic automatic testing
environment
and just as a little bit of context for
behind the scenes this is what one of
those diff based tests would look like
on Ed's back end we tell Ed this is the
name of the test case on the left hand
side this is the sort of the parameters
or configuration and then we tell it
what's the expected input for the
program what's the expected output for
the program and then above I have some
examples of what those input and output
files look like creatively named in and
out
and from here Ed will do all the heavy
lifting for us normally of actually
producing all that nice pretty feedback
for the student
and this works okay for the great
majority of cases but there are cases
where it's it falls down
and the cases are really where we want
to do more complicated testing like
longer programs that are more
interactive and dynamic or where we have
students that aren't trying to hack the
test cases so only when it only works
and we have compliance students
and so we by creating this sort of test
case setup and evaluating our students
in this way we've created a sort of
student versus teacher Paradigm where
we're kind of chasing each other's Tails
they're trying to beat us all the time
and we're trying to beat them
um so here's an example of a devious
student's solution for that example they
might just uh hard code in the answers
for the test cases so for the other test
cases like two 107 you actually don't
need to print any output to pass this
test case so this is not what we want
the student to do we want them to use an
if statement and produce a more general
program we don't want them to be hard
coding against our test cases
we want them to we want them to
demonstrate their General understanding
and have the ability to generalize to
new problems
here's another example of a more
complicated case where we might want
them to do some sort of simulation and
then that simulation results in a random
output in this case this is a Monte
Carlo simulation so they're sort of
evaluating an integral the area under
the curve by actually just generating
around a bunch of numbers or coordinates
on the on the grid and seeing which
proportion of those fall underneath the
Curve
so each time you run it as shown on the
right hand side you might get a slightly
different result and our diff based
testing that Ed has provided in the past
is not actually going to be suitable for
that case because we have a fixed input
and output
because we can't tolerate different
outputs from our student even though
they've got a perfectly valid answer
um and here's another example of a
longer form program that a student we
might ask a student to build
um so they have this console program
that is just uh collecting key value
Pairs and sort of a data entry program
and in this case the student has to has
to implement a whole bunch of uh
commands like the add get and print
commands and these will also have
Dynamic outputs based on the previous
state on what has been entered into the
data storage system beforehand so being
able to test the the state of the
program is correct and the students
doing what we want using that diff style
test is not really uh very user friendly
to us as teachers because it means we
have to write thousands of tests to make
sure that everything actually works
and we want something a little bit more
General and flexible that can cut down
on the work for us when testing long
complicated programs like this
there's also other issues that are a
little bit more python technical like if
we want students to demonstrate their
understanding of some object-oriented
programming Concepts we might want them
to use like pythons inbuilt inheritance
features like on the top we might expect
them for their checking class of a bank
account to use the super
method rather than just sort of
re-implementing that same code in the
child class and doing a diff based test
on this is kind of weird like I don't
think it's possible
uh and again there are some cases where
we want students to do things by hand
rather than using an existing library
and doing a diff on that is not going to
reveal that to us so these are some uh a
devious student might go oh this
exercise asks us to calculate the
average or the mean so I know numpy
implements that for me so I will just
use numpy's implementation rather than
write it from scratch squares we want
them to write it from scratch to
demonstrate their understanding of
variables data types loops and so on
so in summary we want students to solve
Dynamic interactive and interesting
programs and we want them to be able to
demonstrate achievement of learning
outcomes not to take shortcuts and
things like that
so the start of our solution here is to
use a python Library called P expect
based on the expect console program if
you've used it before
the important thing about p-expect is it
allows you to
programmatically control another script
and you can feed it input and read the
output as if you were a human typing
those commands yourself
so p-expect is very powerful and you
don't actually to be useful you don't
actually have to use the full gamut of
what it gives us but we only use a very
small number of he expects features to
get our good results
here's an example of how you might use
pay expect
so you just need to import it and then
the next sort of block of code there is
to spawn your students codes like that
big number python file in a sub process
a child process set up the logging so
that you can collect the output of that
program as it's running and then send
line you can send input to that program
and then you can expect so child.expects
the peer expect eof end of file just
means that you're going to wait until
that program ends so in this case it's
very simple that big number exercise
only requires us to send one line of
text in and then just wait for the
program to end
afterwards you can collect the output
it's in byte format and then you can
decode it into something sensible
so here's a little cheat sheet of things
you might want to use or do with peer
expects so spawning a child process
number one thing sending lines of text
as well
you can expect things that are quite
interesting so you can use regular
Expressions to wait for output so you
can wait until the program from your
student actually matches a regular
expression if you want or if you know
that it's not going to be changing you
can expect an exact string as shown at
the bottom so if you wanted to the
student to Output hello world you could
expect exactly that string
so this gets us onto the pathway of
those of defeating those earlier
problems so if we want to input a random
number into our students program like we
didn't win with those earlier test cases
we only put in four numbers for them to
test and they could then counter that
we can instead feed in a random number
from Python's random library or maybe if
you want to use numpy or something like
that you can do it too so the first
example at the top is showing how to
send in a random number
with the the heart to defeat hardcoding
you probably also want to repeat these
tests a few times not just send in one
random number they might get lucky so
you might want to repeat your tests as
well
here's an example at the bottom of how
you could send in some random text so on
most sort of uh Linux or Unix
environments you there's a file called
words under user shared dict and then
you can just randomly sample from those
words and send them in as well if you if
you wanted to send it some random text
to your students program
I'm going to duplicate side there it
also allows you to tolerate Randomness
so in the example of that Monte Carlo
simulation doing the integration
this is a regular expression that you
might use this is going to give you
allow a student to report a result by
printing it out that's a a digit a
number up to four decimal places this is
just a simple example you might have
something that's tailored more to your
needs and you can help your students a
little bit here or restrict the
complexity of your regular Expressions
by saying that okay you need to print
out the final answer to a specific
number of decimal places and so on
so how do you actually do more
complicated interaction this requires a
little bit more code
the strategy that we've adopted is to
have a file maybe it's called inputs.txt
that lists on each line all the inputs
that you're going to send into your
students code so we can just read that
in and split it into the separate lines
and then use just a quick little Loop to
send in those new lines so the first
thing we're going to do is just wait
expect a new line maybe you the way we
tend to structure our um or the ways I
have been structuring the programs is to
get the student to print something out
the first time and then wait for the uh
Slash r n which is the new line or End
of Line character set and then
afterwards at the end buried at the
bottom is to send in the next line of
text so this way you can just feed in a
whole bunch of text and just listen to
see what your student has produced at
the end
afterwards you could then using the the
um the example that I gave before
collect all the output and compare it to
um to see whether it was what you
expected if you wanted to this example
sorry this example will just spew in a
bunch of text into the students program
and it doesn't care what the student
puts back every time if you cared what
the student gave you back every time
then you might have a corresponding
outputs file and then sort of go Tit for
Tat with the inputs outputs file in your
Loop
and maybe if they failed at a certain
point you could stop them early if you
want to stop their program early
okay so that's the the relatively
straightforward stuff we do encounter
more difficult problems which I want to
talk about now and this is the stuff
that is a little bit more Niche and
maybe more applicable to higher levels
of Education in in high school like is
11 and 12 or university onwards
so cases where you have going back to
the checking bank account example doing
a diff based test is not going to reveal
whether a student has actually
implemented the child class Constructor
properly but you can use Python's unit
test library to help with that so you
can patch the parent class Constructor
in this case bank account init
and listen to see whether the the uh
that was called by the child class
so here's an example of doing that so
um to do the monkey patch you just use
this nice decorator from the unit test
mock Library
you supply the name of the function that
you want to actually patch
and then associate that with a function
in this case it's called test super and
inside that function I'm just creating a
new instance of the child class and then
after it checking whether the child
class was actually called so mock super
assert called and that refers back to
the the name of the function in Grain
and then to run this you just run it as
a normal function like test super
so that way you can get an error if the
chart if the student hasn't actually
called the function that you expected
them to call
this doesn't just isn't just limited to
object-oriented programming you could
apply this to almost anything you can
think of that is in the form of a
function
the next issue is literal problems
um
sometimes you might want to prevent
students from using things like you
might want to prevent a student from
using a or the built-in python
dictionary for example so going back to
the earlier data storage program that
long interactive program part of that
exercise was to actually get students to
implement a subset of the Python
dictionary class itself but I didn't
want students to just wrap the
dictionary class and hide it in their
own little implementation I wanted them
to try and apply Concepts that they'd
been learned they've taught and been
learned sorry they'd learned in class to
actually Implement some of a dictionary
themselves but what I discovered was
that students were actually just
creating an empty dictionary as a
literal and then using that as their
sort of backhand and hiding their sort
of deviousness away from me
so once I discovered that I did some
research into this problem and the
solution that I've come up for this is
to use the abstract syntax tree and this
is a little bit more scary so just a
quick mention to explain what the
abstract syntax tree is the python
interpreter will convert all your python
code into the abstracts into an abstract
syntax tree format before it gets turned
into bytecode so you can leverage that
with Python's built-in AST module
to actually inspect what the student's
code is doing and detect things like the
creation of a literal dictionary
so this is the solution that I've come
up with because monkey patching is not
going to work in this case because it's
treated somewhat differently to a
function in Python
so here's an example of how you might
iterate over a student's class
definition to
deep drilled drill in and dig down and
see if there is a creation of a
dictionary in there somewhere and you
could put this in as a supplemental test
case to say uh I detected that you are
actually cheating you need to go back
and fix that or change your code
and I think this brings us to the last
technique I recommend less last
technique before I get to the
recommendations
is checking for illegal Imports so if
you want to stop your student from using
numpy or pandas or something like that
to take a shortcut then you can use the
built-in module finder Library so this
is
relatively straightforward to use but I
think it's a little bit unknown
I think a lot of python programmers out
there don't know that this exists
so it's quite straightforward to use you
just need to create a module finder
object evaluate that so run the script
whatever that script is you just refer
to it by its name and then you can go
through the list of modules that were
imported and if that one of the matches
one of the ones that you want to ban you
can then fail the student on that test
case
okay
um onto recommendations before before I
get there I want to point out that these
are examples of starting points for
these sort of detection strategies I
don't recommend that they are always
used and this should only be seen as
sort of a starting point for your
journey in the sort of automatic testing
space
don't feel like you shouldn't do it
because you don't see any value in doing
this hard stuff even some of the easy
stuff is quite important so I don't want
to I want you want you to remember that
earlier easy examples that I showed you
so the first recommendation is think
first so
this automatic testing procedure and
detecting
um sort of irregularities in students
code or devious students is time
consuming so it may not be necessarily
worth your time doing you might think oh
I saw Steven's talk I'm going to go try
it out but you might spend far too long
doing it and not get any results and it
might not actually be applicable to your
students so you need to think about
whether it's actually worthwhile
there are also going to be imperfections
in your test cases and these test cases
tend to be a little bit of a hurdle for
students students will come to you and
complain and say hey I couldn't pass
this test case and
often the case with his automatic
marking system is that students will see
the feedback from the market and say
that they see that they failed and they
won't think to look at their own code to
sort out the problem they'll come to the
teaching staff first to try and fix
their issues so you may have to deal
with students coming back to you a
little bit too soon
and you might have to send them away and
say hey you need to go and try again
um you've been given some information
about how to fix that
so just be aware that there's a
additional time complexity of students
complaining at you all the time you
might want to start with simple tests as
well I don't think there's any real need
to start with really complicated tests
this is again ties into the time issue
you don't have to write crazy tests you
should think about what is going to
bring you the most value in terms of
your time and also the most benefit to
your students
um and you can also limit yourselves you
don't have to deploy this across all of
your assessment items or all of your
exercises or all your homework at once
maybe one week out of a term or a
semester you might try this out and then
slowly over time you can come back to
your test cases and improve them or
extend them to broader parts of your
assessment strategy
the other thing is that I wouldn't
recommend this for
everyone everywhere or all students and
it's not a complete replacement for sort
of a human connection and feedback from
a person
so don't think that if you're well
particularly for us it's attractive
because we have large number of students
but even still we think that it's
actually quite helpful to get feedback
directly from your the instructor the
teacher person to person rather than
just having a computer yell at them that
they got something wrong
the other thing is that most students
will do the right thing so the devious
students are usually a small proportion
of your students and actually throwing
these test cases at all of the students
can be a negative thing so you need to
again evaluate whether it's going to be
worthwhile to do
uh and lastly you may accidentally
narrow the path
so often we want students to be creative
and to be thinking for themselves but
sometimes these test cases make students
sort of funnel down a path towards a
standardized solution which you might
not want
um and it's not necessarily true that
students have to follow a narrower path
it's just more that they feel that way
or perhaps at prior examples that
they've learned from lead them down that
narrow path with your test cases
so just be aware that sometimes your
test cases can be a little bit too
strict and you need to make a trade-off
between those strict test cases and your
students needs
okay that's all for today thank you
[Applause]
thank you very much Stephen do we have
questions in the room gosh we have many
questions in the room let's go
okay a great talk my question was around
um when these are used for assessments
do you give the students all of the test
cases beforehand like do you let them
run all of the tests and kind of
essentially know exactly what Mark
they're getting before they submit do
you give them a subset or once you go
there uh it varies
um it's normally well the test cases are
actually for us in our tutorials as well
so we have them in tutorials
um almost all the exercises are covered
by test cases
um and in the assessments for my
assessments personally I kind of run
them as a bit of a test fail or you
succeed or you um don't there's no sort
of marking in between them and then but
the trade-off with that is that I give
the students quite a lot of help to get
over the hurdle I can say this is going
to be difficult for you I know it's
going to be difficult I'm going to give
you lots of help and you you there you
can see the test cases straight away as
soon as I unlock the
um the homework or the assessment they
can see it they can try hacking against
the test cases immediately
still a question over here yes
I've got like three hopefully they're
quick
um first of all how is this deployed uh
is it like all on replay do you have to
create an API how does this it's on Ed
um I could give you a demo afterwards
but um yeah if they have a system where
you can uh input information about uh an
exercise so you give the description
information which I showed uh was an
example of a description here's an
example of a description on the left
that the student sees and then on the
back end as well we can put in uh
information about our marketing
procedure and with ads they allow us to
just run an arbitrary bit of code to do
it to do the marking and we just have to
return the results to add in a specific
format they give you uh it's in Json and
they say you need to return your marking
to the ad system from your marking
script in that format and they do all
the rest Json Ed there's some that is an
online education platform okay
edstem.org Ed stem and the feedback that
students get it's in entirely after the
problem is submitted or is there
anything ongoing while they're building
it uh no so for the students interface
basically they have two options run and
then they can sort of interact with
their own program or they can Mark and
then the Mark will then marking button
will run our marking code and then they
will be presented with the example of a
feedback like these test cases here so
you've passed all the test cases right
so they get the feedback after they run
off yeah after they click the mark
button yep okay so you can't really run
the your regex thing in a loop and tell
them hey you're ticking off you've done
this and then this and then this well we
can do uh in our marketing code we can
do anything we want we can run the test
cases multiple times if we wanted to
like we can reevaluate the student's
code each of those test cases you should
think of as like uh so the test case two
107 199 that would be us running our own
marking script for that test case uh and
we within that we can do whatever we
like basically cool and the last
question is one word answer there was a
special library for check King o o
program structure I forgot the name of
program structure like object oriented
uh well you could you can monkey patch
things
um to check that uh certain functions
have been called as you expect as the
instructor but then there's also the uh
ASD yeah thank you
thanks very much for the presentation I
was curious to if you could provide more
details on that narrowing of the path
like coding is a very creative thing
that really brings a lot of creativity
from students
um and when you try to put all these
blockages like could have like some
negative trade-off like sometimes brute
force or hard coding things is the easy
way but sometimes hard coding or using
brute force it may be more complex and
maybe may show that the person he or she
may definitely know the stuff and he's
just trying he or she may be trying to
trick the system but at the end that's
great for me if the person understands
what they are doing well that's that's
fantastic so you mentioned that you have
to manage that trade-off but what can
you do to avoid the risk of penalizing
someone that really knows the stuff and
it's just just try to trick the system
yeah so for me my strategy is actually
outside of this automatic marking it's
to offer other assessment opportunities
that are more free form so this I I
haven't run a unit where this is the
only way we assess I have always had
other assessment items that give the
students way more opportunity to express
themselves and do what they like instead
and meet a more I guess
less well-defined goals and if as long
as they meet those
um according to our marketing rubric or
whatever that's fine so that that's the
opportunity for them to to take that
creative path yeah
okay thank you very much here is your
mug and your thank you card can we give
Steven another round of applause