[Music]
Welcome back everyone. Um our next
speaker is Michael Gribbon. Um Michael
Gribbon is going to be talking us
talking to us about uh a talk called
Beyond the Hype using AI coding agents
today. Uh over to you Michael.
Thank you. Free round of applause
please.
Uh hello, I'm Michael and I have about
four years of experience as a software
engineer and for
slightly less than one of those years,
coding agents have been a thing and
that's what I want to talk about today.
Um I want to make you slightly more
productive at your job using coding
agents. Now, as I've learned uh
yesterday watching the Dal and I track,
every talk mentioning AI has to mention
the elephant in the room. There was a
lot of hype about AI and about coding
agents. And by a lot of hype, I mean
like a lot of hype. Like some people
think there will be uh no more software
engineers by 2026,
which is a bold bet to make given 2026
is like five, no four months away. Um so
I don't know about you though, but I've
seen this type of hype before. Does
anyone remember GitHub copilots?
uh quite a few people. Yep. So I do and
I want to tell you a story. So back in
2021, I was in my last year of
university and I was uh in on a meeting
with a bunch of academics and we watched
uh GitHub co-pilot which had been
released that October ace a first year
programming assignment in one shot. Um
there was a lot of like fear,
uncertainty and doubts and a lot of hype
back then. um like what could GitHub
pilot GitHub copilot not do and it turns
out like actually quite a lot. Um I have
no doubt that it helped like a few
students cheat on assignments that was a
problem but it didn't radically change
coding like didn't radically change how
I coded and I haven't heard of like mass
job losses thanks to GitHub copilots.
Um,
but it did change coding. Like 20
million developers use GitHub copilot.
That's a lot of developers. Um, I don't
think there's like any going back. The
generation of developers nowadays coming
out of university just expect to tab to
the end of every like line of code. They
just they love that tab button. Um, and
what lesson can we learn from all of
this? Well, we're going through a hype
cycle. And I think the lesson is coding
agents will change coding just like
somewhere in the middle of all those
predictions like from it's a fad that
will go nowhere to uh by early 2026
there will be no more software engineers
somewhere in between because no one
really knows what they're talking about.
Um, Claude Code started this explosion
of command line based coding agents I'll
be speaking about today. And it was
released in February, which is 7 months
ago. So if someone says they know what
they're talking about, then they have at
most 7 months more experience than you.
And I don't think seven someone with
like seven months of experience is an
expert at anything. Um, but that also
means I have to give you a disclaimer.
Um, I don't know what I'm talking about.
I have less than seven months of
experience on this topic. Um, anything
and everything I say in this talk might
be out of date within a few months. You
have been warned. Uh, don't take me too
seriously, but this is where the fun
begins. Um, most of you like might write
code at your job. Um, and the most
obvious way to boost your productivity
if you write code is to get coding
agents to write code. Um,
if you just heard the other talk, you
know what I'm talking about. It's vibe
coding. Um, and I found from experience
that like talking really loud about vibe
coding in front of developers generates
a mix of reactions. Um, I have a theory
on why. I think part of it is like the
people who talk about vibe coding talk
like really big about vibe coding. Um,
but my experience is like it's like a
New Year's resolution. Um, everyone
likes to talk about their New Year's
resolutions, but by February, like, you
know, you made a big promise, you just
don't talk about it that much anymore.
Um, don't ask me. Um, I tried out vibe
coding. It was it was pretty fun. It was
even a little useful, but I don't vibe
code that much at my job. I mostly just
push artisal handwritten code um,
like a craftsman. And it turns out most
software engineers are like me. This is
from the Stack Overflow. It's still a
thing apparently. Um the Stack Overflow
developer survey from this year. Um most
developers don't vibe code at their job.
But there's something going on here
because vibe coding is really really
popular. Like it's a these vibe coding
startups were apparently the fastest
growing businesses in human history. Um,
and
it seems to be really popular amongst
like the general population. Um, and to
be clear, that's a great thing. I think,
uh, bit of a side note, but like the
vast majority of the population can't
code. Um, I think this is like roughly
50 million developers, so like assuming
the rest of the code can't code, the
rest of the world can't code. Um, I wish
more people could code. That's my
opinion. I like I think it's a really
empowering skill. Um, but we've been
trying to teach people to code for years
and some people just the reality is some
people aren't going to learn to code.
And a few years ago, well, not a few
years, one year ago, it was kind of
impossible to write a few hundred line
like Python script to automate part of
your job. Um, or to write like a
thousand line JS game to play with your
kid. But now that's possible for
basically everyone. I think that's a
really good thing. But we're left in the
weird situation where the people who
can't really code just love vibe coding
and the people who code for their job
like don't love vibe coding. Um
something's going on here and now I have
to explain large language models the
type of um AI model that powers uh
today's coding agents. Um, I don't want
to talk about Transformers and attention
and just like say all these other jargon
words in front of you. So, I'm going to
do what everyone else does and just like
add sparkles um and also dumb things
down a bit. So, here here's how I view
it. Um, you give a large language model
some text, you get back more text. Um,
okay, fine. Let's take away the
sparkles.
We actually get tokens. like you give a
large language model some tokens, you
get back more tokens, just one token at
a time. Um, and like what are these
tokens? Um, this is FSBuzz going through
OpenAI's token tokenizer. Um, tokens
look like this. Um, and large language
models generally allow imports of up to
200,000 tokens. Um, that's called the
context window. The input is called the
context window. Now, as you can see
here, a token is about like one word
roughly. Um, as you can see from the
symbols, they kind of mess things up a
bit, but one line of code is about 10
tokens, give or take. And in practice
though, like larger language bottles
have trouble as you fill up more of the
context window. Uh, this is something
called context rot. So, 200,000 tokens
is a bit generous. However, you know the
every single day apparently there's a
new model like the tech gets better all
the time and a state-of-the-art model
these days can have a context window of
a million tokens but that's about
100,000 lines of code um using the
heristic I said earlier and that's just
like not enough um like most software
engineers work in large code bases with
more than 100,000 lines of code Um, I
know Wikipedia says these are the
notably large code bases, but hey,
Google, that's not good.
Can I Ah, we're back. Um, Google also
employs a lot of software engineers. So,
what ends up happening is, uh, companies
with large code bases employ most
software engineers probably. And my
experience is writing a few thousand
lines of code from scratch is a very
different problem from writing code in a
very large code base with millions of
lines of code. Um where shipping too
many bugs too frequently will lose
someone a lot of money.
So I'm going to make a bold claim. Uh
the biggest productivity boost you'll
get is probably not from writing code.
Um, few software engineers
uh actually use AI to write code. And
this is just the software engineers who
use AI a lot. Okay, I didn't actually
give you a productivity tip, but here's
where the fun actually begins. Um,
here's how I personally use coding
agents to be like slightly more
productive.
Uh, number one, searching the codebase.
So, I know I just claim that um these
large language models can't handle more
than 100,000 lines of code, but
neither can I. Um I work in a large
codebase. Um I can't name every file off
by off by heart. What I do is I use the
search bar.
Something's really going off with the
internet. Let me let me go back. Can I
continue?
Oh, no.
the right hand side.
Right hand side.
There we go.
Continue. There we are. Um,
yeah, I use the search bar and that's
also pretty much what they do as well.
Um, Claude was one of the first large
language models get really good at
coding and they got really good by
calling tools. tools like uh search a
directory or read a file or edit a file.
Now I'm going to give you an example. I
know very little about Django, the
Python web framework. I've used Flask
more, but I wanted to I just really
wanted to know how select for update is
implemented in the Django. Um no reason
other than I needed an example to
motivate this. Um this is Gemini code. I
recommend Gemini Code to beginners just
because it has like a super generous
free tier. Um, and if you're cheap like
I was when I was preparing this, it
might help as a good like cost-free way
to introduce yourself to this
technology. So I asked Django, how is
select for update implemented? And just
look at the go. Now I'm not expecting
you to read all this slop, but make a
look make a take a note of the tool
cause. So it's searching text and it's
reading a file. searching text again and
it's reading another file and so on and
so on. If you enroll that it kind of
looks like this. Um I give it a short
prompt. Gemini starts running. It
decides to call the tool to search for
some text. Um it pauses some someone
else something else goes and searches
for the text. Um it goes back to Gemini.
Gemini is like I found a file. I want to
read that file. Um, it calls another
tool and then Gemini starts running it
again and then eventually decides to
just return an answer. Um, yes, I
skipped a few like steps like there's a
lot of tool calls sometimes, but that's
the gist of it. And it eventually gave
me an answer. I did have to tell it to
like be more concise so I could fit on
the slide, but it found the answer in
the end.
Now, Gemini Code and Claude Code are AI
agents, you might call them. Um, you
might have heard of that term before,
like AI agents. Um, you might be
wondering, what is an AI agent? Um, it's
probably one of the most illdefined like
terms around nowadays. Like, anything
that has like a sprinkle of AI is called
an AI agent. Um, I've heard dozens of
definitions, but there's a definition I
really like, which is an AI agent is
just a large language model calling
tools in a loop. A tool like search code
or write code to a file or search the
web or you know whatever, but they just
calling tools in a loop.
Um, cool. Now, the second thing I like
to do is debugging stack traces.
Maybe someone here will surprise me, but
I haven't met that many people who just
love debugging stack traces. Um,
thankfully, you don't really need to
that much anymore because, well, here's
a slight digression, but as a general
huristic with large language models, one
that I use a lot is that going from many
words to a few words generally pretty
good. Going from a few words to many
words is a bit more sketchy. Um,
this is like a pretty common joke about
AI. Like, I'm going to write a single
bullet point. I'm going to turn that
into a long email and then you're going
to take my long email and turn it back
into that single bullet point. Um, I
think that's a bit unfair because
turning a single bullet point into a
long email is begging the model just to
make stuff up. Like, you're telling it
to write slop. Um, but when you go from
a long email to like a single bullet
points, um, it might miss a crucial
detail, but it's compressing a lot of
text, like it's it's not it doesn't need
to make stuff up because it has a long
email.
I think that debugging a stack trace is
fairly similar. So, um, a stack trace
has a lot of lines. Generally, like one
or two lines actually matter, and you
can tell fairly quickly if you found the
right line. Um, that just saves you the
time of reading the entire stack trace.
Um, side note, this is not important.
Um, I didn't know what to put on these
slides because who would actually read
the stack trace? Um, this is just some
software go I found. Um, here's some
more software go this time with a
password being leaked in a stack trace.
Um, just for fun. But yeah, the the cool
thing I found with debugging stack
traces using coding agents is you might
genuinely be 10x more productive. Like
10x programming is a thing. Um that
doesn't really work out because like
most of your time right now is probably
not debugging stack traces. It's like
you know 2% of your time. Now it's 0.2%
of your time. Um but one survey found
that just debugging stack traces was the
single um biggest time saving from using
AI tools um just because it's so
radically um efficient.
But there is a caveat um it helps to
choose boring technology. Some people
might have heard of this motto before
basically you know prefer popular
frameworks with lots of support rather
than um a cool new library I published
yesterday that no one knows about. Um,
coding agents kind of turbocharge that a
bit. Like more code in the wild means uh
more training for the large language
models. That means they understand your
tech bit a bit better. Um, you'll have
an easier time with a popular framework
like Django than a library published
like this year. It does make me sad
sometimes knowing that if I ever wanted
to, you know, make a cool new Python web
framework, I'll have a hard time
competing against Django. But I think
the ship has sailed.
Um and of course you know luckily for
everyone here this is a Python
conference. Uh Python is a boring
technology these days. Um last year it
took overtook JavaScript as the most
used language in public GitHub repos. Um
Python is generally the most
wellsupported language by um coding
agents.
Okay. Number three. You want to fix some
bugs. Um, this seems to be like the
bread and butter for casual coding agent
users, like just um fix my bugs for me.
But I suspect most casual users aren't
doing it super well. Um, like picture
this. It's the first day in your job. As
part of your on boarding, you just get
the few assigned a few bug fixes and
your boss comes over and just says two
words, fix this. That's all they say.
they just say fix this. Um, personally
that would suck. Um, but I've seen a lot
of developers and just like try
debugging by just saying fix this and
you would not say that to a new hire on
the first day of their job with a few
new like bug fix tickets because they
have no context. Um, don't really expect
coding agents to be any different. Like
a newbie on their first day needs
context just like a coding agent or as I
like to say context is king. The
huristic I use is just give as much
context as you would to a newbie on
their first day. Um I have seen coding
agents fix very complex bugs from
detailed bug reports even with images.
They can read images these days but I
have not seen many um coding agents fix
many bugs from like a one or two word
bug report. Uh, two tips I recommend to
ensure you have enough context is just
like prompted to ask clarifying
questions. They sometimes won't by
default, but literally just add like if
anything's unclear, ask me a few
questions before going ahead. Um, it's
the same way that, you know, I'veed
newbies at work a few times and the
newbies that ask more questions
generally on board a bit faster. The
other thing is, this might sound a bit
crazy, but literally just talk to it.
Um, people usually speak faster than
they type. So, use your microphone and
use a transcription tool and literally
talk to your computer. Um, large
language models understand language
really well. They'll get past like, you
know, grammar mistakes or transcription
errors. They won't get past a lack of
context. Uh just a warning though that
if you do it in an office you might get
some weird looks but
um I have I've done it for you. Uh and
last thing is you know avoid context
rot. So as I said earlier like these
large language models struggle a bit as
you fill up more that context window and
if you fill up entirely it's kind of
game over. Uh yes cing agents can search
really well. they can call tools and so
on, but if you're working in a
sufficiently large codebase, um my
experience is like 100,000 lines of
code, you know, from with no context, um
is surprisingly small. So, you do want
to protect your context window if you
can. The most obvious way to do that is
these agents are configured to read from
like a markdown file. In Gemini, it's uh
gemini.md. In Claude, it's claw.md. um
it's just the place the go-to place to
dump tip tips and tricks about your
codebase and often um the coding agent
will like offer to do it for you if it's
the first time but you know might be hit
and miss. Um this is like Gemini code is
open source and so is that like
gemini.md they use to work on Gemini.
um it's a bit meta what you'll mostly
see it's about uh running command line
tools and style conventions. Okay,
number four writing code. I know I said
earlier that the biggest productivity
boost won't be from writing code but I
confess I have used coding agents to
write code. I have committed that code.
Um, but you might have heard people
complaining very loudly that Claude Code
just loves deleting unit tests. Um,
I'm one of these people, by the way.
I've seen it firsthand or that they
might just delete your database um once
in a while just for fun. Uh, and that's
why I have trust issues. Um, I have
trust issues with these coding agents.
Like maybe I'll trust them in the future
when they get better, but right now the
future isn't here. Um, I've learned that
the hard way too many times. And I'm not
alone. Like, very few developers seem to
highly trust AI. Like 3% seems like a
small number to me. Um, but why do
coding agents just love deleting my unit
tests? Um, okay. So, now I have to talk
about how large language models are
trained. This is going to be a massive
oversimplification because there's so
much you can fill into 30 minutes. But
one way of thinking about it is large
language models are trained to maximize
a reward function. A reward function
like write code that passes the unit
tests. Um you might see like the flaw in
this but we'll get there. Um the success
of chatbt for example is credited with
just like um you know getting trained at
getting really good at writing text that
humans like as measured by that little
thumbs up or thumbs down button.
Um, but in practice writing like
ungameable reward functions is really
hard and sometimes large language models
find an easier way to achieve the reward
um without doing the hard work. That's
called reward hacking. Um, and sometimes
coding agents just figure out like why
do I have to fix the tests when I can
just delete the tests? Um, it's obvious,
right? Just that zero tests pass
successfully. Um, and that's why my
experience is, you know, you won't save
that much time generating a thousand
lines of untested slop to a critical
system that you can't go down because
you're on call and you like to sleep.
Um, I have tried. It was not fun
reviewing every single line and fixing
all that code. Um my general rule is the
time saved with coding agents is sort of
inversely proportional to the time spent
verifying like um that's why I really
like using coding agents to debug code
because bugs are usually one or two
lines and usually obvious in hindsight.
It's just finding those two lines is
really hard. Um it's just easy to verify
so you save a lot of time.
Having said that, you can um save time
using coding agents to write code. Uh
for example, you'll save more time
refactoring code if there already exists
a comprehensive set of unit tests. Just
make sure they don't get deleted. Um if
you don't have a comprehensive set of
unit tests, just be prepared to review
every line of code very carefully.
You can try to ask your coding agents to
write some tests beforehands. that can
work, but in my experience, it's still a
bit of a slog than, you know, a set of
coding agents that you really, really
trust, set of tests you really, really
trust. Um, and coding agents are also
really good at writing shell scripts.
Like remember five coding from earlier?
Um, running a few hundred lines of code
from scratch is bread and butter for
these things. And most shell scripts
don't do anything too spicy. Um, most of
my shell scripts just transform a CSV
file to a different CSV file. Uh, maybe
like add a common or something. Um,
nothing too spicy. And the neat thing
about those types of shell scripts is
you can just look at the output. Does
the output look same? If so, you don't
even need to look at the code. Um,
yes. Number five, generating synthetic
data. So large language models are
really good at making stuff up. stuff
that is realistic but not real. And a
couple years ago I was working at Canva
on autocomplete.
And we were gearing up to announce a
design generation tool called uh magic
design. Given the prompts, it will
create a presentation or social media
post. But there was a problem we had in
autocomplete which is the vast majority
of search queries are fewer than five
words like usually one or two words. Um,
magic design prompts are more than five
words. So, how do we get autocomplete to
work on magic design? Uh, we literally
just made stuff up. We got one human,
uh, not an engineer, by the way, to
write down like several hundred prompts
that were realistic and real and then
used um, chatbt to generate like
thousands of more prompts from that like
seed.
I'm skipping out over a lot of hard work
here, but it kind of worked. Um, and we
had solved the cold start problem in in
like these recommendation systems where
autocomplete is almost magical if you
have a lot of data, but if you don't
have any data, you're in a hard spot.
Large language models can help bootstrap
across that like bridge by, you know,
generating realms and realms of of fake
but realistic data. Yes, real data is
better than fake data, but fake data was
better than no data.
Okay,
those are my five big tips and I want to
ask, you know, what did we learn? Um, my
takeaways are, you know, coding agents
are really good at some things, but
they're also really bad at other things.
Um, but you only find out that coding
agent is good at something by wasting
time. um productivity doesn't only go
up. Um so you know you will waste time
figuring that figuring out that putting
coding agent up to a task was just a
really bad idea. But then you'll save a
lot of time like figuring out it's
really good at something. Um and that's
it. If there's still time I'm happy to
take any questions.
Thank you very much Michael. Um before
we have any questions, if Tom Ellen is
in this room, could they please head to
the next to next door for their talk?
Thank you. Um we have a couple of
minutes for questions. Uh do we have any
questions?
We have a question over here about
halfway down.
Thank you very much. Um hia, I'm Renee.
I work at Microsoft the wonderful world
of co-pilot. Um, and I've been recently
using uh, as you showed with Gemini, the
um, Gemini don't mark, we've got
co-pilot instructions.mmd
going on both at a global like level for
all of your repos and at a repo level. I
like including things like uh, you know,
make all my variable names in Australian
spelling um, and things like that or
like this project is using fast API. I
was wondering if you have any overall
tips that you would include in those
kind of instruction data sets, markdown
files.
Uh yeah, sure. I think the best tip is
generally like get it to develop a plan
and then get you to approve that plan.
Um yeah, that's my biggest tip. Um I've
sometimes you like I I have a lot of
command lines which are really really
slow and sometimes it takes a long time
and I realize oh it's cuz I said run all
the tests and then runs all the tests
and I have to wait 5 minutes. So you
know it differs a lot depending on your
project. Um but main one get to make a
plan.
Uh, one more question.
No, just up here at the front.
Great talk, Michael. I don't think you
mentioned anything about using AI for
code reviews. Do you have a take on
that?
Um, my experience is they are really
good at nitpicks. Really, really good at
nitpicks. Um but you again the biggest
mistake is like people just expect it to
be good but you have to kind of refine
it with a lot of explanation of like
what do you want it to review exactly um
to get good results otherwise it'll just
be like you know in Python we expect
variables to be like this and you did
not do that you should do that um that's
my experience at least
cool uh that's all the time we have
thank you very much Michael here is your
mug
are you around for the rest of the
weekend
yes I Um, thank you, Michael.
Cool. We've got