foreign
today we have uh David Lawrence from
South Australia giving us a talk
analyzing and sharing genetic data with
python uh could we give David a warm
welcome
[Applause]
to you David
um good morning
um hi yeah so I have a pretty weird
career um path so I was interested in
computer games at the start and then I
realized I needed more money and not
working insane amount of hours and then
worked in the corporate world doing Java
programming for a while and then went
into bioinformatics and that's actually
been looking at I looked at it today I'm
like wow I've actually been a
bioinformatician longer than a
programmer so that was
pretty strange
um yeah but there's a saying in science
where the best you know as statisticians
they say the best thing about it is you
get to play in other people's backyards
um and I feel the same about programmers
so you know um if you want to if you a
programmer you can
um you know you can end up working on a
national
um electricity Grid or genetics you know
so there's a lot of cool things we can
do
um so the place I get to play is in
biology
um which I I'm biased but I think it's
one of the most interesting things uh
there is
um so everyone you know because of the
vaccines most people know about mRNA now
so DNA makes RNA which makes protein and
they've actually you know it was only in
my grandparents era so maybe their sixth
or my parents actually in the 60s they
discovered how the DNA gets turned into
protein so there's actually a code it's
called codons
and the the bases get changed um the U
is the RNA version of a t but yeah
basically this is how it's translated
and basically the DNA gets translated
three at a time into proteins and then
proteins they fold up that's a
computational problem that people are
working on and it folds not just a
single protein it folds into multiple
ones
then into complexes and then it gets
ridiculously complicated and hard so
there's a lot of computational stuff to
do in biology
um so yeah and bioinformatics when you
think about it it the business
requirement is basically to discover all
about this stuff about life and we know
a lot but there's
an astonishing amount that we don't know
there's definitely lifetimes worth of
work to to find it all
um yeah so there's a big demand for
programmers and the more python
programmers in bioinformatics the better
I think the better it needs are some
more a lot of software skills there's a
lot of people are sort of self-taught
biologists and um yeah if you can come
in with um the software skills you'd be
very welcome so yeah check out the jobs
um
yeah um and if you're interested my
recommendation for getting into
bioinformatics is this um online web app
it's basically leak code for
bioinformatics project Rosalind
um and yeah basically it starts off
really simple counting nucleotides so
you basically just have to they randomly
generate an output you run your Python
program or whatever program against it
paste the answer in if you got it right
you move on to the next one starts off
easy gets really hard so lots of fun so
I'm going to talk about my particular
job so this is I work in pathology so
here's an example of something that that
happens to me so basically in well me
and a massive team so in the Women's and
Children's Hospital the other side of
the river
um generally people you know a baby
might be born with heart problems and
the question is is it due to infection
is it due to a genetic predisposition is
it due to birth complications all kinds
of troubles and knowing the reason
um helps the clinician decide what to do
so how pathology works is basically the
clinician sends an order to the
Pathology company essay pathology is the
public service provider for pathology in
South Australia we take blood we take
information and we send back a report
that informs the clinician's decisions
on what to do
so there's that you know that's how it
worked for things like um you know blood
concentrations of some you know calcium
or something it's a bit more complicated
with DNA sequencing so we still take
blood we extract the blood
um put it in a DNA sequencer process it
on HPC machines work out the difference
between like a reference human genome
and and this patient's Human Genome
analyze it and send back the report so
this is at foam Road the other side of
um North Terrace this is my colleague
and our expensive new sequencer the way
it works is basically the DNA gets put
on this uh sort of glass tile and they
sort of wash the whole thing in bases
and when the next base at a time gets um
Incorporated and it flashes a certain
color and there's a camera in the
machine and it just like
um basically sees that you know there's
a million flashes of this color and that
color and it works out what it is and
eventually it turns it into DNA
sequences so this is the raw output of a
sequencer the new one we have is 16 to
20 billion reads in 44 hours and these
are usually like 250 bases long the
other things are like quality score and
how confident the base calling is
so what do we do how do we you know 200
bases where you know what does that mean
um see the 200 bases we work out where
in the three billion human bases that
came from so there's like a string
matching with a bit of fuzziness in
there where we map to the human
reference genome so
um and here's an example of a mapping
um the uh these all there's no cut no
letters in the map reads that means it's
a perfect match but the G in the middle
that's a mismatch so the patient differs
from the reference genome the reference
genome has an a the patient has a g half
of the time so you have two copies of
chromosomes one from mum one from Dad
and the on one of them we don't know
which either the mum's copy or the dad's
copy um there's a g instead of an A so
this is called a heterozygous mutation
so where do these human genomes come
from uh it's basically was it extremely
expensive project in the early 2000s and
yeah you can basically download this off
the internet
um through you know it cost insane
amount of money to produce you can
download a text file I use it we use it
in genomics for everything uh it's
extremely useful thanks for paying for
science back you know 23 years ago I
will use it every day
um and Yeah the secret you know it was
insanely expensive to do sequencing in
the past but the it's actually much uh
falling faster than Moore's Law so if
you think computers are getting uh
faster over time well uh genomics is uh
going even crazier and when things get
cheaper people want more of it this is
our demand for the stuff running through
our software
um that starts around about 2013 and
it's getting nuts right so
good times so how's python used in
bioinformatics well generally uh the
tools for doing the uh the hardcore
computational stuff are written in C for
Speed or sometimes rust now but pythons
very commonly used to sort of uh join
the multiple tools together so you might
Define a workflow and coordinate the
running of all these different jobs in
Python so here's an example I'm just
going to skip over this but there's a
program called snake make which is
basically like make but it allows python
so it's pretty cool using bioinformatics
maybe it's cool useful for you guys as
well for dependency stuff
um so for research typically
researchers work in small teams and the
output is basically a paper so someone
has a biological question and you work
with the biologist for six months to
three years and a lot of the time it's
stuff like we have this data we have a
biological question what can we do and
you often try a whole bunch of stuff you
write one-off programs you investigate
it and python is amazing for this
because you know I can write python 10
times faster than C or something I don't
know what it is but it's like it's so
much easier and and the thing is we have
HPC we can run we've got hundreds of
cores we can run things in parallel it
doesn't matter how fast it is often we
only run it once we're just
experimenting and if we can experiment
twice in a day rather than once every
few days we converge on the answer
faster and that's great
um so yeah to go back to varying tools
when so the original sequence is at the
top and the bottom one is at the bottom
that's called in point mutation
we have a file format for this it's
called the VCF format and basically
there's a chromosome location a
chromosome name a position and then the
original sequence and then the new
sequence and a bunch of other stuff so
um I've written some software that some
a user essay path to analyze this kind
of data and it's called variant grid
um so what we do is
um we start with the sequencing which
has a huge amount of data we run then we
do the get the diffs and we do all this
on an HPC and then from this
um the VCF for the variant calls then it
gets ingested into our program and I
take over from that point there
um so it's basically a Django app
um it's running on g-unicorn with
postgres being the central database
and whenever we have anything you know
two more than a couple of seconds I farm
it off to a um to a queue and salary
workers do the work most of the time the
the longest running jobs might be say
half an hour which is for annotations
they run things like computational
prediction so for instance you might
have a sequence and it'll do things like
um look at the charges of the the new
modified protein and work out how much
of a sort of how wonky the protein will
go and classify it as pathogenic or
something and so there's all these
different tools and public databases
that we use to work out what a variant
does
so yeah here's the VCF again
um and uh so the way I represent it is
in Django
um yeah because this isn't in the Django
stream not sure if everyone knows Django
but
um basically you can declare classes
that represent
um database tables
um so here's my Chrome contig is just
another biology name for a chromosome
position reference and there might be
multiple variants for a location like
you know
people might not have a t you might have
a c someone else might have a g
and so you can have multiple ones there
and you can write your python classes
and it sort of handles the SQL for you
so instead like on the bottom right here
is the database and on the top left is
how you write the Django code and it
means that instead of writing all the
you know technical computery stuff
you're just sort of working with
variants and samples and you you end up
sort of working in your domain which is
really useful instead of computer you
know code
but
um yeah so there's a huge amount of
um
biological data and we get basically
everything that we can find we'll we'll
run it and that you know there's
hundreds we have hundreds of columns of
different tools and public databases
that tell us what a variant does
um so yeah there ends up being uh uh you
know hundreds of thousands to millions
of variants um and almost all you know
they're things like be a millimeter
taller or shorter or have um a slightly
different color hair or something so you
don't care about that when you're
searching for the um genetic cause of
disease so most of the work is throwing
out all of that stuff and to fight to
search through this it's not easy you
have to know um it's like a mini
research project you have to know about
the disease how common it is the family
history you end up going and doing uh
looking through the literature and so
these met these team of people called
medical scientists who basically do this
investigation and the goal is basically
of this program is to allow those people
to run their own filters on huge amounts
of data sets because they've got
excellent medical knowledge but not so
you know they're not going to be able to
run Jupiter notebooks or something
so a naive way to do filtering would be
something like this where you basically
build up filters but the trouble is
we're offering off and running something
like 50 to 100 filters and this gets
unwieldy very fast
so basically I have a mate who works um
in video compositing and he showed me
one day a directed a cyclic graph often
these are sort of tilted sideways and
basically the output of one node goes
into the other one some kind of
filtering or thing happens and that's
how it works and that allows people with
domain knowledge to apply computational
work which is exactly what we want to do
so ripped it off and do it for genomics
so the way it works under the hood is
each node returns a Django Q object
which is basically Django's way of
filtering a query set or an SQL query so
you can take these Q objects do logical
operations on them and make a SQL query
so here's an example so I have
um here um there's populate like I've
got a whole bunch of queue objects one
of them is population frequency greater
than or equal to 0.01 one of them is
Gene symbol in this Gene list and the
other one is damage greater than 0.0.5
and then you can run
um you know you could write this by
saying you know q and Q 2 and Q3 but you
can also just reduce it down applying
and like that and you can if you want to
have basically you know apply this
filter or this filter or this filter and
this filter you can build all kind of
logical operations in this way
and this is what the GUI looks like
basically on the left is the nodes the
filters and you allow and you click on
one so we've clicked on the red one in
the middle and that loads the population
frequency and you can choose what
databases you want to use in your
population frequency and below that is
the grid of the variance like the 31
that appear in that node and it allows
the sort of the medical scientist to
sort of jump around and like apply more
and more or less stringent filters if
something comes through or not they can
adjust the population frequency if it's
not quite known how common is it disease
is things like that so it allows them to
do their own filtering
uh and yeah we try and allow the GUI to
work nicely for
um the particular filters they want to
run
and if you click on a variant behind
this loads a page per variant and we
have all that annotation details um that
you can view
um yeah because the data size we have
something like 30 000 samples each with
a million rows we store everything in
per patient part uh postgres partitions
and we also filter those we might have
multiple partitions per sample so 95 of
a patient variants are common but a lot
of the time if it's a rare disease you
know it happens one in ten thousand
people you just want to throw away those
really common variants so by putting
them in separate partitions we can
choose to jump to just the partition
we're interested in which makes it
really fast so here's an example where
at the start We have basically
collections is the name of the
partitions that we're after so we always
apply the rare one but then if someone
asks for common variance which means
they haven't applied the you know
population filter
then we also include the common
collection which is like uh 20 times as
big and then we do a filtered relation
which is Django's way of joining to a
table with a constraint and because of
that constraint like we say just this
parent table just this one and then the
postgres knows okay well because of that
constraint I only have to look in this
partition and that makes the query
really fast because yeah if we if we
didn't jump to the partitions
we're talking about billions and
billions of rows and that's not going to
perform in real time
so once we've found these variants we
have classification and this is the
classification uh part of the project so
basically we Auto populated but the
medical scientists can also tweak it go
grab information from various other
sources some things you can't automate
like interpreting literature uh well not
yet
um and yeah so ultimately what they do
is they follow a framework called the
acmg guidelines and they finally
classify a variant in one of five ways
it's benign likely benign
um uncertain and finally likely and then
pathogenic that means definitely disease
causing and then we send that back to
the clinicians and they can look up
drugs and stuff to use
so one of the troubles with this is that
um the state uh the health in Australia
is basically run by States and so
everyone sort of works by themselves and
don't talk to each other so if someone
in Victoria came in and there was a
variant that was pathogenic and someone
in South Australia and they had the same
one but they said it was unknown
um then they would never we would never
know and someone one of them got a bad
diagnosis
um so ideally we'd like to share but
it's actually really difficult because
um it's patient data and there's all
kinds of ethical and security issues to
do to deal with that
um so there's a group called Australian
genomics and they sort of sit as a
national group um that sort of works
above the states to try and help the
individual states work better especially
better together so they came up with the
the plan of coming of sharians which is
basically they have a central system
where you take the state's individual a
class classifications and send them up
to a central server with all of the
anonymity and all kinds of stuff that
you have to do with and if there's a
discrepancy so two different states have
different results then it basically
sends an email and says look sort this
out
and then there's a little like workflow
that we send them down
um so yeah this um Project's been
running since 2019 and so far we've
Linked UP basically every public
pathology provider in Australia and
um also just recently New Zealand we're
now starting to go after
um the uh private Labs pathology labs
and we're also starting to go into
collecting the cancer ones which are a
little bit different and require a bit
so the ones we've traditionally captured
are the inherited ones and we're
starting to capture the cancer ones
which are have a bit of different
information
so the way it works is basically so
essay pathology you know they use
variant grid and so we can do whatever
we like there but in other states they
use different software and so we sort of
have to deal like right integration
software with them and talk to their
system and then translate the data from
their format into the common format and
then send it up um via API
um and then if two different Labs get
their different results then this is
what we show them a diff
um and then they have them basically
they usually have a meeting bring in the
clinicians and the big guns of their
labs to decide what happens
um and then they sort it out go through
a workflow one of them change well
ideally one of them changes to agree
with the other one and then uh it
eventually Cascades through their system
through the API again and then we mark
it as resolved
so yeah one of the troubles
um is how to handle data from many Labs
every state does their own thing and so
we had to write custom converters to put
into Json so sometimes we run it on
their systems sometimes they just dump
data on an S3 bucket and then we run the
code on our cloud system and send it
from our Cloud systems back to another
through the API and yeah so we we allow
basically everything we allow people to
to give every bit of information they
have but after but after we've
investigated and asked for them what it
is then we can sort of add a bit of type
information such as float or int and we
try and get them to use a we use a
controlled vocabulary like an ontology
so if a lab calls it population
frequency another lab calls it pop freak
or something we will convert that and
send it all up on that you know with the
consistency on the API so we try and
hide the um the differences in those
connector programs
so here's what the Json looks like and
the advantage of this is we have I don't
know maybe 100 150 different fields that
we collect from all the different Labs
um and uh what what we do is we've like
um so here there's Nomad allele number
which is how many
um counts this has been seen in in a
hundred thousand person population
survey we know that's going to be an
integer so we have a basically a key
with an integer on it and we then when
we get the data we can run a validation
and say this is supposed to be an INT
um you know uh it's a it's a string and
we can like raise a little warning and
stuff like that which just helps keep
the data pretty clean
so um yeah here's the project timeline
um so basically I started this so I work
as a I work as a researcher on genomics
and I people kept coming to me asking
for me to do filtering and I said why
don't I write some software so that
people don't bother me anymore and then
it got used and then the researchers
started using it then the diagnostic
people started using it we ended up
putting basically we won a um a project
to sort of people have this very rare
disease and a um they basically have
this online site where all researchers
from around the world can upload their
data and share it and we got we're
getting external funding from that from
the states and that's still ongoing that
was um 2018 the first funding and then
um we hired then we won the share uh so
sharing was a project that we had that
Australian genomics wanted to do and
variant grid 1 the tender for that and
we used thanks um yeah the tender for
that
um and use variant grid technology as
the base for sharing it so we've sort of
have the ability to skin it and turn on
and off things through settings to make
the site work however the client wants
it and you know swap the CSS out and
stuff
um yeah so in 2019 we've got a second
developer which was um amazing
um and 2020 we went open source and you
can go to the GitHub page and check it
out I've um in science
um you're supposed to write a paper to
advertise your project and I've been um
I don't know I'm a programmer not a
writer and I've my boss keeps on my back
to write a paper and one day I'll get to
it
um but yeah and then so it got
diagnostic use and it's been used by
sapath
um and yeah so that's how it's going so
thanks a lot
um cheers
[Applause]
[Music]
[Applause]
all right thank you very much David what
a great uh sequence of slides can we
have a round of applause for David
[Music]
[Applause]
now we do have a few minutes for
questions so if anybody in the room
wants to put their hand up or if anybody
wants to type their questions into
Discord and we can read them out here
please do so
I'll give you a moment to type things
into Discord because I know it can be
hard to raise your hands sometimes and
I'll check and answer them later if um
anyone else say anything else so but
yeah feel free to ask any questions
all right we have a question from down
here near the front
um yeah I guess I was just wondering how
kind of easy and usable you find the
kind of ecosystem of
data apis like that ncbi provides for
looking up your sequences and stuff
because I've made my way slowly into
this world and yeah I found it a
struggle yeah um okay that's a great
question
um there are tools there's like um
entrees um there's a python library
biopython and some that sort of manages
that we use that for things like um
querying say how many
um Publications there are for a gene or
you know when was the last time or or
other things but basically the
um yeah the tools aren't
um awesome
um we write yeah basically we um we
always wrap our
um
API calls in uh with timeouts and
exceptions because the services are up
and down uh I don't know it's um but
yeah it's a bit of a weird
um
yeah basically a lot of work was done a
long time ago um by
um people that yeah there's a lot of
some somewhat questionable decisions
that you wouldn't necessarily do it this
way uh now and um but yeah it's um it's
a big yeah that's why we need more
programmers in the field
um get get our sort our apis out and
yeah there's not there's a lot of um uh
custom file formats and wacky old um you
know really inefficient old file formats
that are there instead of you know in my
in my ideal world every service would
have a rest API and it would be trivial
to get everything but that's not a world
we live in yet so but yeah more
programmers would would make that world
a reality so
consider bioinformatics
yeah
all right we have one question from the
Discord uh do the findings push to or
pull from open targets genetics
open targets genetics
um I'm not sure so we go to clinvar as
well which is a very common oh that's
the most popular as far as I know
um variant classification sharing uh
platform and we collect a lot more data
than they do and we also
collect stuff that hasn't sort of been
approved for public use so it's behind a
like a private server and allows the um
uh sort of labs across the country to
sort out any problems out of
um you know internally and then after a
while they've sort everything out and
then they share it and then we do once
we sort of set things to the public
share level then we go through and
collect everything and send it to the
external databases for everyone else to
be able to look at so we do talk to
other databases and push it out and we
also pull those in so we can do
comparisons if if ours are different
from theirs so we both push and pull to
external databases
all right that's all the question all
the time we have for questions uh I'm
have the traditional speakers gift for
David
the wonderful
pycon a youth speakers mug thank you one
more round of applause for David