[Music]
I do love that welcome back to devoops
next up we have sanin who is a former
Computing teacher and an avid fan of
python since 2014 he has a back in
developing programming courses and small
educational applications for use in
schools as one of the creators of the
Victorian coding challenge sun has been
building and maintaining a coding
competition platform that that has
hosted quiz style events Capture the
Flag challenges and in 2014 agent-based
challenges where students sorry 2024
reading agent based challenges where
students program agents to compete in
challenges based on Game Theory please
welcome sanun
hello
um is this mic working yes I think so
works really well so let me tell you a
little bit about the Victorian coding
challenge so this is a competition for
essentially students that find Computing
in school a little bit too easy um it is
funded by the Victorian Department of
Education through the mathematical
Association of Victoria and over the
last five years we've had over 7,000
students participate it takes place over
two stages stage one is an open-ended
problem set stage two is the competition
which is the interesting part which I'll
be talking about so just a quick show of
hands um put your hand up if you found
Computing in school some combination of
too easy or
boring all right you guys would have
been the audience you would have been
the students participating um so here's
the team that makes it all happen danela
who works for the mathematics
Association of Victoria essentially
managed to get the funding deals with
the Department of Education deals with
the schools a million emails Nathan and
I essentially make the problems in the
platform over time I've become more of
the platform person Nathan's become more
of a problems person and uh along with
us there's been probably about another
seven eight different volunteers some of
whom I'm going to mention in the talk so
a little bit about the background of the
the challenge so the first part is like
an open-ended problem set we're
currently up to like 30 plus problems
and the goal here is just to do
something that's a little bit more fun
and more challenging than school is
normally so two examples of problems one
is um it was a it was called I always
win the game of nim that is a Cod game
if you do a binary representation of the
game state for that game you could
basically win 100% at a time so this was
a fun thing that the students built
basically like a card Hustler that they
showed to all their friends a computer
that would win 100% of the time in a
seemingly fair game another challenge
again this is something that kids would
never come across was like this time cap
capsule problem where they get given a
time capsule that'll like reveal on a
certain date and they can either hack it
in various ways you could break the
cryptography you could study the API
code and find a way to like encode the
data in a tricky way where you
manipulate the time stamp so that it
opens early so these are the type of
problems that we put in front of kids um
however when the competition started it
was covid time and this was all pitched
as a live event and we couldn't do a
live event so basically we had to do um
like a remote thing so in 2020 we did a
combination of recommendation engines
and um we gave students the Wikipedia
API and from the get-go we when we got
the the the actual submissions by
students there were about 40 for each
year band we basically Blown Away by the
quality students we had a team build
their own machine learning model that
was making quizzes I mean we didn't even
know how to mark this stuff was it on
the basis of like the most interesting
and the most creative or the most
advanced code so yeah that was uh it was
a really really interesting time going
through all those
2021 2021 was much the same except worse
from our perspective we had a theme of
games the 78s made text-based games
which went really well n910 this is my
total mess up um we chose to use a 3D
game engine in Python that had too high
system requirements given that I was the
one testing it on my computer laptop I
yeah somehow even the school gave me a
very nice laptop and my students a very
nice laptop it never really really
occurred to me that the requirements
were too high we ended up only having
five submissions for 910 so this is the
third year of the project so we had a
three-year Grant and the department
essentially is telling us
hey this year you get to do the live
event we're going to pull the students
and the teachers make sure you give us
feedback and based on that feedback
we're going to decide whether we're
going to renew your um your Grant
so really we were in need of a spectacle
here and here I have another question
for the audience uh show hands if you
know what a CTF
is okay leave your hand up maybe if
you've participated in
one okay maybe I'll do a little bit of
explaining so in this particular class
um so I was a teacher that year I was
teaching at the King David school we
were um learning networking and that
year the theme of the class was uh CTF
so we we
essentially the challenges from
something called
picoctf and I basically took part in
creating problems and little puzzles for
students that they would solve by sshing
into my computer on our local network if
you see there's like a switch in the
middle everybody would come in with a
school issued old teacher computer that
had Ubuntu on it or um a Raspberry Pi
and the students themselves came up with
this scoreboard this live interactive
scoreboard where where they took those
problems and they scored themselves and
they had like live rankings and this was
done by by the way like halfway through
the semester so we spent like I don't
know 10 15 lessons every year students
every class students rushing in and then
like doing problems and having these
rankings on a scoreboard so for me I've
never seen more enthusiasm in a
classroom in my entire life and this was
the idea that I wanted to port to the
Victorian coding challenge the first
thing that I did was um rewrote the
scoreboard program um the initial thing
that you know when you have very
enthusiastic year NS that have a lot of
talent but not a lot of experience it
was like one function with 20 parameters
and God knows how many globals I turned
it into something really simple one
worrying thing because I wanted to keep
it keep the coding basic so all my
students could understand it was that
Json file like there was no database
there was just a whole game state was
one Json file and I was like thinking
that's a ticking time bomb that never
somehow went off um so yeah we used this
for at least like at least 10 lessons
and it worked perfectly so if it ain't
broke don't fix it this is more or less
the program I was going to take to the
Victorian coding challenge in terms of
the challenges really there were only
three types um one was um they would
download a file and they need to solve
something maybe it was like a a um like
a password protected Word file that they
needed to find a tool to crack maybe it
was something coded or encrypted data it
could have been like a zip file with
like a f Bolder jungle where something
was hidden that they had to find uh
another type
was that they would SSH into a python
script so you create a user on Linux and
then like you create a rule in sshd
config and then they SSH into like a
puzzle you created they don't see the
source code they're just like inside
your your sort of text Loop and the last
one was a little bit more elaborate um
have you show hands if you know what a
truth jail
is so like you create essentially like a
not really a virtual environment but
like a folder with all of the necessary
tools and libraries to have a very
restricted like Linux system and in
there they perform a task so those were
the three types of uh challenges in
terms of how this was deployed now this
is terrible I mean people talk about
like being embarrassed by past code like
wait for this so three python files that
do the admin easy so each challenge was
actually the users didn't have accounts
each challenge was a user
so they would SSH into that user you
could have multiple SSH sessions for the
same user from different IPS Linux is
flexible like that they would go in and
try to get a flag then there would be
another user called a scoreboard they
would SSH into that to submit their flag
along with their password and they would
get their scores updated so that's how
it worked and with nine students in my
class it never broke now I did take a
look at other projects that did similar
stuff and they were about at least
magnitude more complicated and I was
wondering like this is so easy I don't
know what why why do people make it
complicated so um in terms of the user
interface was very basic Google site you
can host files there you get the
instructions to the
SSH and then like the challenge went for
three hours every hour we released new
problems and new challenges so this is
the competition day um and let's see how
we're
going so in the morning there's
something called relays like unplugged
problems
the kids smashed that everybody had a
good time at 11:00 I'm standing in front
of a crowd about three times the size
going we're going to do a a systems
check everybody has a flag it's there
it's on paper you're just going to type
in this into your terminal submit and
we're all going to be good turns
out works for about two-thirds of the
teams and then all of a sudden a bunch
of people can't submit it and I have no
idea why luckily this is deployed in in
a in a Leno server that takes less than
90 seconds to restart and so restarting
it fixes the problem and that was my day
like this is a student there looking at
me I'm probably restarting the Leno
server right there and everything
somehow just rolls over the line um
until about 10 minutes to go where I
mentioned sheru jails they are
escapable um the score of the top team
which was winning by a country mile I
mean like they probably risked this
qualification but I was really impressed
just went up by 3 Zer and then it went
down back to normal so yeah I I really
when this all ended I was almost shaking
I really didn't even know how we went
but was the funding lost no we were good
so despite all these technical
issues every team submitted everything
they found in terms of the flags the
feedback from students was amazing and
the teachers did mention some technical
glitches but they were happy to and like
not only did we get our funding renewed
like we actually got way more funding so
this whole thing ended up being thank
you ended up being a big
success um the tech postmortem now this
is the if it ain't broke don't fix it n
student
thing if you open up the sshd config of
any Linux system you're gonna you're
going to find commented out the default
settings and I only would have needed to
change that Max sessions from 10 to
something like 30
that wouldn't have fully fixed it
because there was another problem where
students used like these terminals that
were in the browser and if you close the
browser you don't close the session
so that was a major issue another thing
that could have been a potential issue
was there was a challenge that was
around cracking a hash but they had to
enter a password into an SSH and and and
a team try to brute force that so I
don't know if this Mac startups thing um
that sort of limits the amount of
waiting connections to 10 and then drops
30% after 10 kicked in or not but yeah
the part that keeps blowing my mind
again and again and again is that not
having a database wasn't the issue that
single Json file that was edited just
somehow
worked uh yeah the mega score kid I had
a really good conversation with him he
he actually learned that on the
spot like nobody finished except that
one team and he had enough time to read
a Blog about escaping sh jails and apply
all that so that was
impressive um Lessons
Learned um if it ain't broke try your
best to break it like that's just I I
had that attitude of
like this works in all the circumstances
I tried it but you can actually quite
easily simulate more terrible
circumstances um for me personally the
mix of technical failure and the student
enthusiasm like I've never been more
motivated than at the end of this to try
to work and try to make this a much
better platform so that was the perfect
thing and yeah in a space of two weeks I
had two babies a human one my daughter
was born about a week before the
challenge and then after all this
happened I the project became something
I started to take you know very very
personally and seriously so where to
next um more founding allowed me to step
away from Mostly making problems and
worrying about the
platform now the first thing funnily
enough the first thing I did after this
broke is I probably I spent all of my
free time in the next two months
rebuilding the CTF platform which never
got reused because we ended up deciding
that we're going to step away from the
CTF thing and we're going to try to
build a platform that was more like
Google quizzes so where they we could
ask different types of questions except
that it would incorporate a live
scoreboard because the live scoreboard
was the pretty big deal there
so yeah that was my job to build it and
this is the tool set that I chose based
on the situation based on like a mixture
of what looked interesting to me and
what looked not intimidating to me so I
did teach you know normal CSS uh
JavaScript HTML so it was just vanilla
js on the front end initially SQ light
given that it worked seamlessly with
python was going to be the choice for
the DB fast API now I had some
familiarity with both Jango and flask
before fast API but that fast API seemed
like it just seemed easy and obvious
from the moment I first looked at the
documentation so that was going to be my
back end and uh yeah I was going to do
the serus developer thing with GitHub
you know automated deployment automated
testing for every commit and so on and
going to use uh AWS for the first time
now I only use more or less
these apps but I get built for other
apps I don't know if other people have
similar issues with AWS but like it's
really
annoying
um
so I don't have that much time for
technical detail but this is more or
less how I ended up slowly building a
more robust platform so by April uh in
2023 I had a prototype that just worked
it it all the features that we needed
students could ow short answer questions
multiple
choice
um like I used all of the libraries I
wanted and it was deploying
automatically so that was pretty good um
by September I had added an admin page
which made it easy for us to populate
questions there's also a Junior Day for
years 56 that I was very little involved
in but I made it easy for those guys to
just create questions and put them on
the site and by October with about a
couple weeks to go I had gone to town
with testing I used something called um
show hands if you've heard of locust I
have no idea if this is popular it does
stress testing um basically there was
local testing there was stress testing
and I found every excuse I could to get
every circle of friends every
professional group to try to use the
platform in some other version I
addressed it up as AI trivia look AI is
making these Json files which are trivia
which was just an excuse at that's my
platform
so basically after about three four
successes like I decided to let it go
there were a couple of weeks left this
is Locus by the way that was a
screenshot in the WhatsApp group I was
running about 142 requests per second I
expect the competition only needed about
five so I was like safe in that regard
this is what the competition looked like
these are groups of questions this is
actually a primary question and this is
going to be either like multiple choice
or a short answer they have three
attempts to the short answer there's
going to be scoreboards you're going to
see them on the
pictures um and so yeah a couple of
weeks to go and I have an idea that I've
been thinking about a long long time and
it is the idea for like an agent
competition and the idea here is like
imagine some simple game and the
students would program an agent to play
that game and the Agents would complete
against other agents and then they would
get some feedback and then they would
try to like adjust strategy
so you know this is this is bit of a
technical challenge but I thought that
in the two weeks I could do like a
10-minute preview and show it to the
students like this is what we're going
to do next
year so
yeah like I said I only wanted a
10-minute
demo I mean again student code running
on my server this is a bit of a
challenge but here's what happened in
one day messing around with Chad GPT I
had a proof of
concept and all of the sort of big
challenges I thought I had which was
like making sure that the student code
is properly formatted and you know I
don't have random silly indentation
errors and and and syntax errors was
sorted by something called uh inspect
lib and it was converted to like a
playable game object uh using something
called import lip. util and So within
one day I started thinking well maybe I
could do a little bit more than a
10-minute preview I mean I know it's
really risky but we're going to cage it
in you
know this is experimental might not work
so we basically decided I somehow
convinced everybody else that we should
do a 45 minute mini game with the
students and uh it'll be worth 5% not a
lot of consequence and the student
experience was really really easy so
they played a game called greedy Pig and
greedy pig is a bit like Blackjack in
that there's a dice rolled you keep
adding to your total unless the dice
rolls one so it's about kind of like
knowing when to stop and there is a
statistically correct point to stop but
if you have a game state where you know
where the other players are doing you
can do a much more sophisticated
strategy so this is like they got a a um
they got a reple on a website called
replit they had a working player with a
working strategy and all they had to do
was tweak that in 45 minutes they had
you know we had acted out a game in
front of them with 10 minutes they they
were told what the game was a week in
advance and interestingly enough a lot
of uh Primary students in Australia play
this math gaming class
so yeah like from the student point of
view we knew we were good we knew that
like they could understand it and play
the
game okay um so yeah like essentially
from the student point of view a couple
of if statements if they understand the
game they could Implement a very good
strategy so the live event in
2023 going back to that live platform I
built with fast API SQ light that all
worked perfectly it was tested to the
Max and the only bug we had was in the
competition teams would log in with two
lap stops to the same account and the
scores were added up on the front end
not the back end and so they all had
wrong scores because they were logged in
with two computers other than that
having to tell them what the scores were
we didn't really have any bugs so with
an hour ago it was all smiles so now the
78 day and the 910 day they're about a
week AP pot so with the 78 day I set the
expectations guys this agent games thing
we're about to do is an experiment it'll
probably not
work but let's give it a try this is
what we're going to do next year it's a
really cool concept
and it worked other than like the server
was overloaded with submissions and it
crashed a couple times so I fixed that
over the week for the
n910s I got this um slow API rate
limiter that only allowed one IP to do
three requests a minute and I also added
all these security features for bidding
certain libraries and certain functions
and so for 910 I was confident and the
speech wasn't this is just experimental
probably not going to work guys you're
going to love it the 78s loved it and
here's what
happened that remote
address in a venue which was the library
of the docks somehow every single IP in
the building sorry every single computer
in the building had the same IP so
instead of limiting one team to three
submissions a minute we limited all the
teams to three submissions a
minute and it was like a car crash in
slow motion because it's not like
totally failing ah one team got in one
team got in but we're running out of
time and so we ran out of time and it
was really really really
frustrating but uh yeah still the
feedback of that was oh I really want to
do agent games I really wish that worked
I really wish that worked so like I was
fully committed to uh to doing agent
games right so this is very much exactly
the same thing that happened with the
year before
except this time I'm going to make it an
open source project I'm going to try to
get other people involved the quiz was
staying that was all fine I have another
kid and the history repeats itself um so
making agent games bulletproof having
student
submissions run on my own server the key
here was a mix of
validation which used all of those
things I realized I'm running out of
time really quickly but most importantly
every simulation ever ran in a Docker
container and if your code was overly
analytical overly slow it was rejected
if it managed the simulation against a
bunch of dummy teams in under 3 seconds
your file was saved on our server and we
were going to simulate you and that was
all going to run
so this is what the back end looked like
when we run a
simulation um students and us we get to
see this uh feedback in PDF that they
get to see how they did against other
teams and adjust their strategy based on
that
feedback um in terms of the platform the
coolest thing that I did was I made the
game itself really kind of extensible it
only takes about two to four hours to
make a new game by the way I'll be here
for the developer Sprints and if
anybody's got cool game ideas I would
love to hear them um and yeah
um I'm going to have to skip a whole
bunch of slides but I'll definitely show
you this one the agent games the event
went really well but first we had
prelims and in the prelims we had about
700 submissions and several successful
hacks and teams just beating everybody
with a 100% score the most interesting
of which was the one on the screen
there a kid figured out that we're using
the random
Library he then found a way to Brute
Force random so that every single time
for every single thing he did the dice
roll a six and so even after a thousand
simulations like he stops at a 100 he
stops when he wins he like doesn't
because he only get sixes it was like
the most genius thing I've ever
seen so um yeah I mean I'm GNA fast
forward the competition went well we had
one funny
bug we had a Myst team appear now I
checked all the server logs nobody
logged into the server I was really
paranoid what had happened is because I
had outsourced my vanilla JS front end
to a freelancer who did it in react he
was kind enough to put a sign up button
there um and that was the only security
breach we had thanks to you know get up
automated deployments I uh I got rid of
that in about a minute but I was really
sweating for about an hour a couple of
uh nice things to say about people at
help uh this was the freelancer Muhammad
he wrote the the front end Matthew there
who
is a master student helped me with the
docka um Han who is the head shooter of
the first year subject in Union
Melbourne did a whole lot of testing and
made the agents more secure and Nathan
made the markdown
feedback and this is the
repo and in the future I plan to try to
share this with uni institutions School
other
competitions yeah and uh thank you very
much I had a lot of fun um for the uh
platform if you can help out with any of
these um Docker performance
improvements uh deployment on Windows I
hate windows I wish this thing could
just work on
Windows so if somebody could do that
that would be nice the feedback tends to
be too long uh we need to make that
markdown searchable so yeah those are
some of the issues uh that I'm about to
document nicely on the repo and yeah
that's everything thank you very much
[Applause]