[Music]
good morning everyone my name is Jay and
I'm going to be the session chair for
this morning and we've got a great talk
coming up we have ten lenberg who's an
experienced data science scientist and a
developer of the scores open source
project and he's here to talk to us
about Lessons Learned From developing
that open source project please welcome
Tennessee to the
stage yeah thank you very much uh people
all morning have been kindly asking me
if I'm ready to give my talk or not and
I have to say the answer is no not
really and the reason for that is like I
think I've got a good deck and a good
content and lots of good information but
I've been struggling to find like what's
the story what's the narrative spine you
know where's the kind of hero hero's
journey what's the three act structure
or the the five room encounter that
we're going to build out of this and the
reality is that making an open source
package isn't really like that it's just
a whole lot of stuff and and a lot of
this talk is just a whole lot of stuff
and like the reason I made this talk is
that I've got 20 years of development
experience working with python and and
like it was hard you know with all that
experience it was you know pretty
difficult to you know find out all the
information and make all the choices and
figure out how to put it together and
the story of of what was involved was
just dealing with a whole lot of stuff
so uh you know the talk this sometimes I
give funny talks this isn't a
particularly funny talk at least you
know not deliberately um and we're just
going to go through the Journey of all
of the things that you might need to to
you know tackle and deal with and the
slides are pretty dense that's cuz I
want there to be like a you know like a
good record and a good set of content
and things can people take photos of or
pause the video on I'll try to talk
separately from what's on the slide I'll
I'll attend to the general theme I'll
call out a few stories and a few few
points we went through um but if there's
sort of you know a lot up there or I'm
not addressing all of the points you
know fire them up if we've got time for
questions or you know drop them in the
chat and the other reason I'm not ready
is that I don't think that I'm really
terribly much more qualified than huge
numbers of people in this room
particularly and it sort of slowly
dawned on me that I'll be presenting my
own sort of experience and the
experience of my team and building and
constructing this package to a room full
of people who may well have known better
than we did uh how to make those steps
along the way a lot of these lessons
will learned the hard way as I went and
no doubt there are few surprises
remaining waiting for me
too so a few things that aren't C
covered number one naming your package
if anyone's figured out how please let
me know uh legal advice I'm not
qualified you'll need to find your own
own course through uh through legal
advice like choosing a license Dido
approval processes wherever you find
yourself you may find approval processes
are part of your life they're up to you
and I'm definitely going to T not touch
the choice of a text
editor I think it's always helpful to
start with one thing to take away if
you're going to take away one thing
these are good places to
start I I had found them before I
started I did not find them before I
started I did find package manager
thingo I don't know what these are
called one of them is called hatch and
it describes itself as an extensible
python project
manager that that sounds helpful and the
other one claims to make it all easy
which made me skeptical but it is a
commonly adopted framework and both of
these things do a lot of things see also
many blog posts discussing which one is
better which one is not better why you
should adopt one when it's new when it's
deprecated and yeah there's a lot of
complexity uh we chose hatch in the end
but we're really using only a tiny part
of hatch um largely because all of the
various sort of decisions it's made we
wanted to be more kind of like mindful
of so if I'm talking about me I'm
talking about me if I'm talking about we
I mean the develop velers of scores and
the people who've made contributions um
I may fall find myself falling into
either sort of patent at various points
along the time so both of these have you
know more or less to do with a bunch of
the rigging and infrastructure and
processes that are involved in you know
the activity of software development uh
explaining it to the internet testing it
assuring it going through release
processes all those sorts of things um
but there are also Choose Your Own
Adventure versions of every single part
of these as well and sometimes those
Choose Your Own Adventure versions might
be better documented or even a better
choice for
you okay so how to lay things out on
disk this is one of those things that
that's changed uh and become more mature
but there are some undocumented
weirdnesses to how to lay things out on
disk that I don't see talked about as as
much at least I didn't come across them
so tldr definitely put your source code
in a source folder and don't rely on the
class path for finding your modules um
do use an editable install and a virtual
environment for doing your software
development okay so here's how I lay
things out on disk and you know this may
seem like a dry topic like I said
building an open source package is just
a whole lot of stuff um you'll notice
there are some files sitting at the top
level that kind of look like
documentation and there's a
documentation folder uh that's largely
because various other parts of the
ecosystem just expect things to be in
particular places and quite a lot of
what we found out the hard way was like
what things just have to be in some
particular place or another for it to be
found by for example GitHub or Google or
the cicd package or you know various
types of Integrations so the the
integration ecosystem around open source
packages has got a bit harder so I'm
pretty happy you know so each time we
kind of go through one of these things
I'll comment on whether like we're happy
with our choices in terms of whether
this is going to apply to every package
you know like probably not um but I
think this is a reasonably sound
starting
point okay configuration files uh tldr
do use P project. toml so more and more
projects uh are using like P project or
toml I think this is great please do
more of it this would was one of those
we went one way then the other way then
the other way then the other way type
scenarios for us like a lot of packages
use dot files I come from a generation
that used setup dop by default so I kind
of started there before moving over to
Pi project. toml so there were some
Journeys sort of backwards and forwards
like I didn't realize you don't need
setup dopy anymore that was that was a
discovery you don't need setup.py
anymore and it it it seems to be better
without it so uh I will comment that
like with the dot file type
configuration like your pilot RC and
this RC and the other RC and cicd
subfolders full of stuff that tells your
cicd what to do I feel like that's just
hiding stuff and it's it's like a
contradiction like you it's in your
package but you want to hide it so it's
not cluttering things and being
distracting but your developers need to
know it's there because it's how it all
actually works so I feel like you know
that's something I feel like isn't maybe
sort of sold perfectly
yet versioning oh boy did I get this one
wrong so I thought versioning was just
you know it was just kind of chill you
know particularly in the early days of
starting a package I thought it was just
all cool so I kind of you know I made
the numbers go like generally speaking
up but I oscillated on whether to put a
v in front of the version numbers for a
little while and I started with a V and
then I pulled off the V and that
confused the heck out of the internet
that that was really not a good call and
I also didn't think it like mattered
whether you had the like you know I just
went with like 0.1 2.3.4 I didn't
realize you needed the patch part of the
version number to to kind of comply but
you really need that third one and I
think it played into a bunch of things
in our early experience like search
engines didn't find us or our updates
for a really long time and it was
incredibly confusing to understand why
so even at the outset
it was pretty important to use a really
recognized convention for this and I
didn't
realize uh and the the ecosystem plays
really well with semantic versioning it
plays much less well with Calver and uh
it plays mediumish with a new one called
f and it doesn't play very well at all
with just whatever someone felt like
that morning you have to be fairly
specific about it um so yeah that was a
surprisingly important thing uh we went
to version one we sort of to you know
tossed up that change like there's a lot
of articles about what going to version
one means it seems to mean a lot of
different things to a lot of different
people python packages seem to like
cruise on you know like low version
numbers to quite a high level of
maturity for a while it's not amazingly
clear to me why um we just went for it
so we we did go for it when it was ready
so when I say we went for it we didn't
just go for it we carefully approached
it and then mindfully decided to do it
but we didn't fear it and I think that
was a good call uh really at the Crux of
the sver system is information about
what what's a fix and what's a bug and
what breaks things backwards and that's
kind of what the integration needs to
know uh and it is better to provide that
information if you can and it hasn't
been a
problem right branching there are a lot
of different branching strategies out
there um yeah this is something where
like there's a tradeoff okay so like
what we do is we have tags for each
release it's pretty standard we put our
development on develop not on Main which
is like semi-standard we merge domain
just before creating each release kind
of
semi-standard uh but it means like
there's this kind of contention between
like when you go to GitHub you see the
default branch which is our develop
Branch but when you go to read the docs
you see the latest release and so they
could be different and I'm not really
happy that they're different and there
doesn't seem to be a way to resolve that
so this was also something we changed
partway through our branching strategy
like which one was developed and which
one was main the change didn't make many
big problems um but it's probably better
to be stable up front because I think it
did result again in compounding that
like searchability of the documentation
site issue
that I referred to before so it it
confused it confused the Bots until I
had until the documentation was
configured really nicely and it really
it it got happy when we did the tagged
releases it got happier when we went
past version one it was like that like
the search engines were less happy pre
version one and then once we started
using sort of sver properly then it
started to understand how the versions
of our documentation site like hung
together and direct people to the right
one and things like that so there was
sort of these unexpected links between
things like branching strategy and
whether when we told someone about our
package they could find it so there were
these sort of like connections at a
distance that that you know as far as I
knew weren't written down
anyway automated testing it's 100%
totally worth it uh it was worth it for
us I'm pretty sure it's going to be
worth it for you uh I really think this
is important there are still a lot of
packages that don't have 100% test
coverage there are like really major
python packages where you go to their
GitHub and it's like 84% coverage tests
failing and you know these things are
like running the world and you just
think really and if you're a new package
and you're trying to like you know
establish some credibility and trust
those sort of those badges and those
status indicators you know they matter
they make a big difference to whether
people take a look at what you've got
and go yeah I'll happily use it or
whether they go I might just wait so I
think it's important for that trust in
the package it's also really important
for agility like if um if a dependency
has a security issue because it's you
know become outdated you know you can
just like push push go on that merge
request very confidently and you don't
feel like things are going to break at
all and it's also been very helpful in
making sure we support good Library
compatibility compliance like people
might be aware one of the major
mathematical libraries recently went
from from like the I think it was the H
the exact versions don't matter we'll
call it 1.6 to
2.1 and just like tons of stuff on the
internet broke and it took like 3 months
for all of the packages to go oh now we
need to fix it and that was because
people didn't have well partly because
of the sver thing where people were just
using the latest package regardless of
major version numbers but it's also
partly because the test cases weren't
hitting it in all cases so being able to
just kind of confidently move forward
know that you're going to discover new
issues during development rather than
finding them on the release it's just
incredibly helpful if you're a package
manager to have this in
place uh some experiences with automated
testing troubleshooting and a couple
from Beyond the scope of this you know
our open source Journey so like a common
one performance tests are too slow to
automate people will often have like
large realistic test data files they use
to drive through their package to make
sure everything okay that's fine that
doesn't need to prevent you from doing
good 100% coverage so that's that's a
really important thing and we've taken a
number of the scores contributors
through like how to do that you know how
how to go away from this kind of like
large file on dis mentality to a
separation of what the automated testing
is covering and what else you also need
to cover sort of separately um guey code
is hard to test I just couldn't help
throw that in there that's more from my
past life um just don't put your
application logic inside the click
Handler make it separate make it
callable make the click Handler call a
callable function that's testable if at
all possible make the the memory and
state model of the application shouldn't
be the state of the goey uh similar for
command lines except life's generally a
little easier uh applications processing
a lot of files on dis is pretty common
in the in the sort of science space
where you've got like you know carefully
craft Ed you know mathematical array
based inputs that match precisely
various outputs a lot of the sort of
domain people so you know in my case
scientists but in you know any case the
domain people might be quite comfortable
creating those test rigs as like files
and file formats they know how to make
maybe in applications they're familiar
with um it gets a little and as a result
a lot of the functional part of your
application can end up just going to
direct to disk and taking files names
and passing file names around and then
when it comes time to test it like it's
really hard it's either slow because
even with SSD disc is slower than memory
or it's really flaky CU you've got to
like pass around strings and manage
program State you know manage the state
of the variables in your automated tests
and you can't just get in there and
interpose terribly easily it starts
getting a little bit bit narly moving to
something which passes around fil like
objects allows you to create data on the
Fly into straight into memory without
hitting the disc it's much quicker uh
and it allows you to have a more modular
design that's going to let you interpose
on
things so this is one of the thing like
like a point like that is why I'm not
ready like this is this is like my story
I don't know if I can give that advice
to to kind of everyone you know what's
exciting about that well it mattered to
me and it mattered to creating this
package rather than so it's an example
some random tips on testing uh hard to
cover all of them but work out how many
decimal places to test to and do it
consistently um computers are both
brilliant at maths and like kind of bad
at maths um if you do the wrong sort of
order of operations or something changes
you are going to get different numbers
down in the fine scale of those decimal
places and you will need to figure out
if that matters to you um so yeah we
figured out a number of decimal places
and more or less standardized on it it's
a good idea um with the test data I was
referring to previously essentially we
stored it inside piy files that makes it
pretty easily discoverable by the test
Rigs and the test fixtures means you can
import your test data e easily and you
don't have to sort of you know do a
little bit of used to be called like
reflection apis to like discover what
directory you happen to be sitting in at
the moment and where your source code
happens to be coming from and where you
think your test files probably are in
order to load them into memory if you
can just put it straight in a pi file
it's a bit bit more straight
forward um and if your code is test hard
to test can indicate a problem with the
code okay typ hinting was a journey uh
I'm not bitter at all um so this is a
situation where I've got a pretty
different opinion to the norm so I want
to both give you my opinion but also
explain to you that it is not the
commonly held wisdom so that you know
like both of those things are clear okay
so a lot of people really like typ pins
like and if you're like well I'm
handling forms all day long and I really
really need to know the difference
between an integer and a string like
that's fair that's crucial to your
application logic uh I wanted to write
really General code and a lot of the
functions like I could support like an
INT or a list of ins or a python array
of ins or a numpy of ins or any kind of
number really or xarray object or pandas
or any one of about three or four
different pandas Alternatives and the
major xarray alternative and it would be
kind of nice to support like the back
ends for like pie torch and tensor flow
at the same time and that's like that's
all reasonable for maths so trying to
typ hint that is actually kind of pretty
complicated and like one of those
libraries and I'm not going to name them
you cannot inherit from like that they
have done funky funky metaprogramming
and you just like you can't do it and if
you want to have an extension to their
classes you have to use their funky
registration process to add your add
your functionality into their classes so
TI penting was very difficult and not
very helpful for these complex compound
types where you're trying to write
polymorphic code that can just handle
whatever you throw at it which I was
happy to do you know we were happy to do
we could easily handle it but on the
other hand people wanted that sense of
safety comes from type hinting and I've
also seen this gradual move from type
hints which sounds kind of you know free
flowing and relaxed to type annotations
which sounds a little bit more formal
and a little bit more expected so I
think you know the world will thank you
for providing type hinting and type anot
annotations that will thank you for
providing that information on your apis
which I think is fairly appropriate in
terms of like return types and things
like that but at the same time it's not
necessarily inform and it can be
actively misleading when the code itself
will comfortably support multiple
different types um which passed um but
it's easiest to just
conform linting and static analysis this
there's there's a lot that's been said
on lenting and static analysis uh this
is kind of my take like black is useful
and fast just use it isort is useful and
fast just use it rough is a recently
developed tool which is kind of really
interesting so it's very fast and a lot
of the time like Fast tools are very
comfortable to adopt because they fit
very neatly direct into the editor
experience um but it doesn't have enough
uh kind of functional coverage for me at
the moment compared to like you know
pilent or or something that's going to
check a few more
things um and yeah security static
analysis you'll have to do some reading
on on the Security Management as
well
documentation uh we we lent into
documentation and we we lent in hard and
it was just well worth it so here are
the kinds of documentation that you
might want to consider writing uh we did
all of these uh and all of them have
helped in their various ways so dock
strings and code comments the major
dispute that I hear on dock strings and
code comments among people from
different perspectives is that well the
code does what the code does it doesn't
write doesn't do what the comments or
the docs string say um I would generally
argue that even even an incorrect dock
string or comment can be remarkably
illustrative to the intent or hopes of
the person that wrote the code and that
there is not really a source of truth
about what should be happening in the
code regardless of what is happening in
the code uh and certainly people
appreciate having a lot of good
information uh particularly around the
API
docs contributor guides have been mainly
helpful for me as the package maintainer
like in theory there for everyone else
but actually like to a large degree
they're a reminder to me of what I asked
people to do so that I can be reasonably
consistent uh because left to my own
devices I'll be wildly inconsistent and
confuse everybody so it's it's been a
helpful Aid to memory for me to be able
to work through things and it means that
if someone else needs to you know
oversee a merge request process or pull
request process for a period it's fine
uh websites like you know like
technically you don't need one you know
you can just put your code out there on
on GitHub you don't need to go to Pipi
either you can just make your code
available it won't be like you won't get
in trouble for not doing it but it's
obviously essential to
discoverability uh consider examples
tutorials and notebooks I think there's
a rant later in this talk about making
your tutorials work but just in case I
forgot to include that slide I'm going
to do it twice it's really important at
least to me to make the tutorials
actually work um I lost I would I would
estimate like straight up weeks like you
know 8 hours per per day times 5 days a
week times some number of weeks on
tutorials that just didn't work just
like debugging someone's tutorial out
there that is simply like nonfunctional
and maybe my success rate at like making
the tutorial function is maybe like 25%
like 75% of the time it's like well the
source data is moved and is gone and
requires a sign up and all the versions
of all of the things in this in this
tutorial of changed and I just you know
like didn't get there so yeah again not
bitter at all but our tutorials work so
you know
like perfect perfect testing like could
they ever break for a short period of
time until we fix it yes it's possible
but we actively maintain them so you
know like feel free to go find a broken
tutorial and prove me wrong but we will
fix it right back at
you okay so how to do strings give every
function a dock string the exceptions to
this rule aren't worth arguing over
certainly not in a
presentation uh comment liberally uh
relating a few nebulously connected
comments like dock strings math Jacks
web pages tutorials and Doc strings like
they have helped me find bugs doing code
reviews in other people's code often so
it it is very important because
programmers all of us we're all just you
know flawed humans and our expression in
code is not necessarily perfect and so
expressing our ideas twice in two
different forms fixes bugs uh which is I
think you know important I don't think
that's like the most important like the
most important thing is that other
developers can go back later and read it
or like you in 3 months or if you have a
short memory like me like a couple of
days later it tells you what you were
trying to do in the first place that's
very important um we've used Napoleon
style do strings I have no idea why
they're called Napoleon style dock
strings I assume Napoleon didn't invent
them but I haven't gone further um but
it has been a reasonable Choice uh and
we use it to drive the API docs
obviously so the documentation text
stack uh I only found this one like this
YouTube video like this just fixed it
for me this was this was the answer um
so thank you to whomever uh and
apologies I don't know the pronunciation
of the native language here Yan Louie
Koo Rodriguez thank you um this was what
allowed us to string together the
technologies that we wanted to work with
Sphinx Can Go the Distance on the
functionality you want in constructing
the web page um markdown at least among
this particular development team was
universally uh more pleasing to write
documentation in and this particular
Tech stack allows us to stitch all the
tools together it was a great choice
right the source code for documentation
okay this is kind of like the problem
you don't want to have but it's going to
be there uh the documentation is like
this little world of its own and it
looks so innocent how markdown files can
have so much complexity is remarkable
they will render differently everywhere
and it's a challenge and it's a thing
and there are rules that I've tried to
document that I can't even express in
English I just like remember that
magically that part of the of that style
of markdown in that kind of API string
in that kind of way I have to do this
slightly differently there and I've just
never been able to explain all the
details so we've tried to achieve
consistency between and I'll just go
through the list pii GitHub front page
GitHub viewing the documentation folder
GitHub looking at the tutorial folder uh
cuz the read me goes onto pii uh local
builds of the documentation people
actually running actual Jupiter lab for
our tutorials as opposed to the
documentation rendered version of the
tutorials read the docs versions of all
of the things that I just mentioned and
binder and NB viewer so that's like 10
different like rendering contexts plus
actually reading the markdown files uh
we had to make some compromises some
things were just not possible to support
the greatest thing we ever did was make
this 228 Branch where we can just push
stuff and every one remembers what the
URL is and if someone says hey my pull
request is good to go can you show it to
me we just force push it to 228 it
renders and read the docs on a in a
hidden link and we go check it out and
that's been incredibly helpful because
not everyone's it's not actually hard to
build the documentation like that that
bit is really quite straightforward um
but a lot of people feel more
comfortable when they see it for real
zase on the
internet the good news is that everybody
really really appreciates good document
a um if if your part of your motivation
is to get adopted like do good
documentation it's going to make
probably a bigger difference than
anything else people will happily have
half as much functionality and twice as
much documentation and something
reliable um I feel like narratively I've
covered a lot of this already um but I
think it's very much helped with
onboarding people and if you have a
particular domain or field that you're
working in it can also help people from
the domain perspective on board into
into their side of the process as well
and I really like looking at math
Jacks okay ecosystem the rules are not
obvious and this was the this was the
biggest thing where like no amount of
like internet search could answer these
questions for me it it just experience
only taught me these things possibly I'm
wrong possibly I'm not I was naive at
the outset um but these were the things
I discovered so sver you basically have
to do it release notes that's totally
free form you can do anything you want
with your release notes within reason uh
the process like how to publish to Pipi
like discovering the one recommended
path was very difficult because there
are very very many paths that are
referred to online and a lot of those
guides are outdated and I had a really
hard time figuring out our process is
straightforward I have one minute to go
we tag the releases use compatible
versions they're much better than
greater than or equal to how and what to
automate we don't didn't automate
anything we retained manual control of
the final version publishing to places
that was a good call basically
everything else is automated lean into
automation Community considerations and
I've just been given the one minute flag
so uh this should be like there are and
should be many talks on community
considerations um the main thing that we
kind of did upfront was make sure to
have a good code of conduct and and make
sure the contributor and the contributor
Covenant is now widely adopted and and
well respected so in the past it was
much more much less clear how to how to
do this in a way that's going to work
for everyone now it's pretty
good okay we published a paper in the
Journal of Open Source software in some
disciplines uh academic citations are
really how people are going to be able
to get recognition for their work and
it's very important so if you do publish
your own package consider an academic
publication such as in the Journal of
Open Source software uh it's a great way
to support your colleagues if you're
working in an environment where you want
to get the scientists on board with your
work and we are one slide away uh but
I'd like to thank all of the
contributors to our open source packages
for a moment just to appreciate all of
their
work in conclusion it was a worthwhile
experience and I hope perhaps that this
may have help may be able to help
someone else avoid some of the pain that
we experienced along the way thank you
thank you very much Tennessee for that
awesome talk we've got a small gift of
our appreciation this great Pon mug um
we're not going to have any time for
questions right now but I'm sure
Tennessee is having a chat uh throughout
the rest of the conference um we're
going to have a small pause uh just for
people to move around but the next talk
in this room is going to be Simon Aubrey
talking to us about duck DB so thank you
let's W let's thank Tennessee once more