[Music]
after brief a hiccup we are back with
Tom who is talking about whyu uu IDs are
secretly incredibly fascinating please
make Tom welcome
hello I'm weirdly nervous this morning I
um most of the talks that I give have a
kind of point this one was purely sort
of special interest information dumping
so um I ended up just talking about this
randomly to random people forever and so
now you guys are forced to sit here and
listen to it
um in terms of real world information if
you've already read RFC 9562 then you
probably already know everything that
I'm going to say in this talk and you
just kind of here for opinions and and
Vibes I guess
um
I'm oh yeah because I Yammer on about
this way too much I've got the last
slide of my talk first just in case I go
over
time
um
so most of this is just weird interest
stuff but like if there's useful things
that I want you to take out of it if
there's useful things that I want you to
take from this talk they are that it's
really worth knowing what the
differences between these U IDs are
because they can actually be quite
useful for certain use cases um and uid
V7 which is a new standard that was
finalized in April of this year but has
been being worked on for the last couple
years might be really useful for certain
use cases of yours and we'll go into
more details on that later um if you
need to invent your own kind of
universal ID system there's a now a spec
for you to conform to while you do that
which could be very useful for you so
make sure you know about that
but I'm talking about a thing that may
or may not be useful to you depending on
your use case so probably the most
important lesson that I realized is
probably applicable to almost everyone
in this room if you work
in what we do uh whether or not your
uids are a good primary key for you or
not or whether or not they're useful for
your use case I've now worked in a
couple places where we've had real
problems having with the default primary
key value of databases particularly
postgress so maybe you don't the the the
one lesson that I think is probably
applicable to everyone is maybe you
don't need 128bit universally unique
identifiers uh but I think that nowadays
you will regret using a 32-bit
identifier at some point and by the time
you regret it it's a really painful job
to fix so I cool you can all go
now okay I don't know why I was wasn't
able to squeeze this all into a
lightning talk um so why am I talking
about this because I thought this was
really interesting in Django land first
of all if you how many people here work
in Django do web development do database
design have to choose your primary key
yeah okay so you guys are vaguely in the
right place right um you get really used
to Django's defaults of integer primary
keys if you don't decide on one it'll
make one for you and by and large it
works perfectly fine Forever Until your
table gets too big I've now worked at
three different companies where each one
has had huge projects that have each one
taken like over a year of developer time
because of something going fun with with
your primary key Choice um I worked for
a company several years ago who who were
merging multiple production environments
into one production environment which
was multi-tenanted and naturally they
had to remap a whole bunch of primary
Keys through a huge amount of data sets
and it was gigantic and talking about
that would be a whole talk on its own um
I've worked for a company that chose to
use uid fors and text columns for every
primary key in every table of their
database which had some real I see some
of you shaking your heads and you're
right but also it had some real real
advantages that I will also go
into and then most recently uh at Kraken
Technologies we've had an amazing
project recently where we've had to with
some urgency quite quickly change a
bunch of primary keys from 32- bit
integers to 64-bit integers because we
were running out of space and by the
time you're running out of space your
tables are already really big and if
your tables are really big it's probably
because you've been reasonably
successful so you don't want downtime
and so trying to do this sort of thing
without downtime would also be a whole
big talk of its own
uh Tim Bell will be here later in the
conference and I'm sure he'll love to
revisit many months of his life so be
sure to ask him about
it for a long time all I really knew
about uids was that they looked like
this you know you've got you've got
those hyphens in slightly different
places you've got a whole string of
hexadecimal numbers and there's a bunch
of different versions of them and in
practice you only ever really see uuid
V4 which is the version which is
completely random who who here like I
don't go into much about where they came
from I just go into sort of the weird
stuff in this talk but all of you have
seen this sort of thing before yeah um
you I knew about uid v1s I knew that
they were constructed from several
ingredients and I knew that people like
just don't use them for anything and
then if you go into the python standard
Library you'll see U uid versions two
three and five didn't know what they
were and nobody uses them and so that
was what really got me
interested we've had this standard uh
uids have been around and not a standard
since like 1995 and they've been a
standard since 2002 or 2004 and nobody
uses them and everybody invents their
own Twitter invented lexu ID and
Snowflake and Google invented push no uh
yeah push ID for Firebase and uh doc ID
which is something they use in Google
Drive and mongodb invented their own
ones and Instagram uh which is a big
python Django project invented their own
one called stting ID um everyone has to
invent their own because for some reason
the standard that we had was never fit
for
purpose
so I am going to give you a crash course
in what all these things were why
they're interesting why they weren't fit
for purpose and how that's being fixed
nowadays
so this is the executive summary of all
the versions that are currently in
Python the V1 was built out of the
timestamp the MAC address and the
sequence and so each of those chunks of
the uid become those things um the V4 is
the completely random one that doesn't
use any other kind of logic it's just
121 bits of completely random
information and then the little version
specifier which is like whatever the
remaining bits were seven bits right um
V3 and V5 are the ones that you probably
heard the least about and they have a
very interesting Niche use case because
you can turn any other identifier you've
got into a uu ID I'll show you an
example of that and V2 I'm not going to
talk about because it's almost the same
as V1 but it was specific for a certain
application back in the 90s for a
certain operating system so none of it
really exists anymore so it's something
that you can actually just kind of
ignore so we'll talk about V1 they
decided that a universal ID something
that well hang on the point of a
universal ID is that you can have a lot
of things generating these IDs and
they'll never Collide
okay one easy way of doing that is
making sure that you incorporate a few
ingredients that the other machines
generating U IDs won't have so the time
that it was generated a node ID like a
machine ID and so each machine
generating U IDs has its own node ID and
then a few other bits and pieces like
sometimes a bit of random data or a
clock
sequence when they were designing uid
ones they made a couple interesting
decisions that kind came back to haunt
them uh the time stamp is this readable
for
people cool okay um the time stamp is a
60- bit value for you id1 this is
represented by the coordinated use time
blah blah blah blah blah blah blah
blah 100 nond intervals since the
Gregorian reform to the Christian
calendar in
1582 okay sure why not um and then the
node address was the MAC address of your
computer computer so just find any
network card on your computer grab its
Mac address and throw it into the ID
so this example this U ID right here is
the one that's like in the spec and if
you can decode it and get that
information back out the spec suggests
that you don't do this that you don't
rely on any information in the uuid but
it's interesting that you can decode it
and so you can find out that the time
that this U ID was generated was the 3rd
of February
1997 um the clock sequence which is
actually just kind of a random number is
1085 and the node ID is that Network
address and of course because it's a
network address you can also tell that
that's from Intel it's from an Intel
Corporation uh machine and that block of
Mac addresses was allocated on the 1 of
January 1980 like around the first
allocations of ethernet addresses um the
reason you can know that by the way is
because maybe up until recently or maybe
it is still the case that all Mac
addresses the first
half of it are allocated to a network
card manufacturer and they get to decide
on the sequence number from
there
um
so the other important thing to note is
the time in ID 1es is backwards so it's
not like year first then month then day
or it's not it's not most significant
bit first it's least significant bit
first so um they don't
enter in order so uid 1's could their
their sort of overall numerical value
could be all over the show as the least
significant digits
increment that will be
important um so that's id1 I've skipped
over id4 because it's just pure random
data I told you that id5 is quite an
interesting one sometimes you have a use
case for this and I did have a use case
for this and it was great um you take
any other identifier you've got and you
turn it into a uuid and so you do that
by the the way the algorithm works is it
takes a namespace which is just another
uuid and it takes the canonical
representation of your value of of your
name and it does something like an hmac
you know it does like it concatenates
the two and then it does a sha one hash
or an md5 hash and it takes those bits
and turns that into a uid uh the
difference between V3 and
V5 is that V5 used md5 and V5 used sha
one and so they don't recommend you use
uh V3 anymore at all but it's still
around in the standard and
so and in the standard there's a couple
predefined name spaces so if you for
some reason wanted to
turn a DNS
name uh into
a uuid there's a DNS namespace and
there's a URL namespace and there's a
x509 namespace and a few other things um
and so this can actually be genuinely
useful and I did this trick a couple
years ago because I had I had a primary
key that I needed to turn into a uu ID
for a certain table and the in the
integer ID corresponded to a sequence of
financial transactions
and I you you invent any uu ID and it
becomes your name space for whatever the
identifier is that you have and from
then on you can basically just map
anything onto any name in there to a uu
ID without it colliding so they're
deterministic this way so unlike the
other ones where you expect to get a
different unique uu ID every time these
are designed to get you back to the same
one every time so for transaction one as
you can see I've got the same value you
can see my mouse yeah okay transaction
one you can see the same value came out
when you feed at the same input
transaction two is a completely
different ID so you can't so basically a
one-way hash right basically like a
password
hash hang let me
just and id4 pure random
data and so nobody uses these except id4
is still fairly commonly used when you
just want purely random
stuff a lot of the questions a lot of
the time people wonder if uh collisions
were why you wouldn't want to use these
if they were too likely to collide if
you go to the Wikipedia page you'll see
various absurd examples of um what is it
like with uuid4 which is completely
random if you generated a billion of
them a minute or no a billion a second
for 90 years you'd have a coin flip
chance of maybe having generated two of
the same one um so the the numbers that
we're talking about for random
collisions in a way that would affect
your app are negligible like humans
really don't have a great way of
understanding in their head just how
gigantic a 128bit number is um but in
the history of people using uids there
have been collisions but they've always
been sort of um implementation bugs so a
reasonably prominent example was I
think it might not have been VMware but
it was like a it was a VM company
product that was incorrectly seeding the
random number entropy pool on booting
VMS and so the VMS had a higher chance
of generating the same U IDs uh so there
was there there's been that sort of
thing in the past but that's not why
these became unused
it was because by and
large everybody needed one feature of
primary of identifiers that uids didn't
provide which was they needed numbers to
go
up um especially so a perfect example of
this is is a relational database like
postgress I I use postgress for my
examples because I just use postgress
for everything
um most relational databases use uh B
trees for their indexes and berries have
been heavily optimized for insertions
kind of happening at the end of the
stack and so in real world terms you
want locality of data that was sort of
generated around the same time and so
you would like your IDs for similar data
to be close to each other on dis or
logically uh uid the uuid standard all
of those numbers come out pretty much
random including the uid V1 which had
time but like I told you the time was
backwards so the low significant B bits
are wrapping around all the time inserts
will happen all through the table
inserts will happen all through the
index and that's fairly
inefficient um some other complaints
cuid V1 um the Gregorian Epoch thing
just was annoying to implement uh
if you did want to sort them by order
you kind of had to parse the uid which
is not a great idea and using the MAC
address was considered a bad idea people
either wanted kind of arbitrary node IDs
that they assigned or they really didn't
want the potential security or identity
problems of using um your L your literal
Network address in your primary
keys and so yeah if we went back to that
list
uh all those Twitter ones and I think
the mongodb ones all of them are like
64-bit integers that basically have some
variation of like the time and then
random data but the time is in correct
order so that basically IDs will go up
over time
um snowflake has a little bit of extra
data in the middle it's used by Mastadon
as well for those who use that uh it's
like the
time a type identifier and then random
data um but the thing that ended up
being really popular was time so that so
that number basically goes up as you
generate them and random data so that
brings us finally
to the new versions the new versions
will probably be in Python by
three4 um there's PLL requests for them
and I see discussions on like the python
discussion lists um but there's plenty
of libraries that use them anyway um oh
yeah I was going to demonstrate some of
these I even have like a terminal right
here look there you go uid ones um
actually yeah let's do this now so I
just generated 20 uid
ones and what you can see is that
they're actually not that unique right
like you can see that the time sequence
is going up on the left and as soon as
that actually yeah perfect timing let's
just hang on I always go over time and
now I'm suggesting we wait I always go
over time when I'm talking about this
and now if I just sit here for a little
bit
longer oh it's going to take too long
but you can see what's going to happen
the D the D is going to wrap around to e
and then it's going to wrap around to F
and then it's going to wrap back around
to zero and so that means that the
insertions of these IDs as they were
being generated all over the time all
over the place um they're not going to
be at the end of the index now if we use
du id4 they're completely
random right so they're also always
going to be inserted in random places in
your
index um except for as you can see the
little four down here which is the
version specifier
and the other non-random bit is this
column here you'll notice is never lower
than
eight in heximal value because the first
two bits of
that number are also um part of the
version
specifier so they're never quite as
random as you totally thought anyway
let's go on to the new ones uu ID V6 is
exactly the same as uu ID V1
except put the time in the right order
so it's still the Gregorian 100 Nan
thingy myob um but you put the time in
the correct order and the spec the the
uh spec says don't use the network ID
anymore just pick a random number and
stick to it so can I generate some of
these for you yeah if we
go what do they look
like they are going up over time oh and
this implementation as you can see it's
not sticking to a static node ID this
last chunk here is the node ID uh and
it's not it's so they're just generating
a random one each time um but the point
of the uid V6 they don't recommend you
use it they just invented it because
it's literally just uuid V1 with the
bits rearranged so that you can
implement it really easily if all you've
got is uid v1s you can just take that
move this bit to here and and then
you're done uuid V7 is the one that they
recommend you use for every nowadays um
and it's really dumb simple it is just
the Unix timestamp you know the number
of milliseconds since the January 1st
1970 which pretty much every modern
computer just uses you all are familiar
with it Unix time stamp in milliseconds
um for the first 48 bits and then a big
chunk of random data like it's still a
big chunk because you got 48 bits then
you got like what 67 bits of random data
so that's enough to make
it really ESS um and then there's a
couple optional rules in there for
submillisecond precision so like this
little this little chunk right here
instead of just being a part of the
random data can be if your system clock
goes down to nanc Precision you can kind
of use that for some more time Precision
um I'll generate a couple of those and
you'll see more numbers on screen won't
that be exciting
um and they are very random except for
that front part and they go up over time
and they're guaranteed to go up over
time that's the other rule of uv7 is if
there's it's written into the rules that
implementations need to account for like
clock Jitter and stuff and need to
either increment a counter but you can
you can place design guarantees that it
will go up over
time and finally they created a uid V8
and uid V8 is screw it build your own um
you can decide what any of these bits
are you can decide what chunks they are
they just have to have the version bit
in the right spot and the variant bit in
the right spot um and this sounds like
it might be a bit of a cop out but it's
actually incredibly useful
because 128 bits is probably you can
construct a really useful identifier out
of that you might need to put some kind
of implementation specific information
in it um but having a uuid spec for it
means that you can use things like
postgress is native binary uid column
and indexes for it and so this is
actually this is this is might actually
be a really powerful useful new tool for
people who do have to invent their own
identifiers for something because you
don't have to stash them in big strings
in your postgress
database so it's kind of the general
purpose uh do whatever you want and they
still recommend that you don't depend on
what's what information is in there like
you shouldn't really be decoding these
things anyway as as a best practice but
by having that you use Python's uid type
you can um yeah database native formats
that support it
uh yeah those are the new versions so
for the last bit of my talk I'm going to
talk about like why you may or may not
want to use these in your database
primary key you don't you don't need
uuids until your project or your data
set or your system is so big or
distributed that you can't afford the
time or network cost of asking
permission to use a an identifier as a
primary key right um postgress by design
your primary key is always going into uh
index with with like a uniqueness
constraint on it right so no matter what
even if you're using uids in your
postest database you're still kind of
asking permission if you can use that
identifier because you're checking to
see if it can go into your your your
index and if it can't then there's a
collision so uids aren't designed for
this use case but they're good for it
any anyway um there's some real
advantages
uh migrating your primary Keys when you
realize they're not fit for purpose for
is a really expensive project the thing
that I was just telling you before about
moving from 32-bit to 6 64-bit integers
by the time you've got billions of rows
in your database and you don't want your
database to have downtime you're in real
trouble there um yeah by the time you
realize you have to do it your database
is always really really big um this one
fascinated me this was the big lesson
that I got for working for a company
that had just off the bat because it
wasn't ajango project probably they just
were like no we're going to use uids as
our primary keys for every
table it was a company that also dropped
to Raw SQL all the time um quite
usefully like it was an SQL Alchemy
project and so that meant that you're
it's quite easy for you to go use the OM
when it's convenient but also when you
have a query that of some complexity
that you want to construct you feel the
convention was that you feel quite
casual about dropping down to Raw SQL to
write it but when you're writing raw SQL
with a modern database schema there
there's a lot of risk of like doing the
wrong join on the wrong identifier you
might have several models that are like
closely related closely stacked and you
might join the wrong one on the wrong
one and you know if every table has a
row in it of ID 100 you won't notice
that you've made this
mistake a real advantage of your primary
Keys being globally unique at least in
terms of your entire application and not
just your individual tables
is that entire class of bugs is just
eliminated because a developer will
write a join and it won't get any rows
and they'll go oh because I'm joining on
the wrong foreign
key and so I've got no idea how much
time that saved us because it's one of
those things like the bug can't happen
so it didn't happen so I don't know how
much time we saved but I've had that bug
in the past and so you know working for
a company for a couple years where that
sort of thing just didn't happen um you
don't really know how much it bought us
but it was not
nothing um if you are using uids in your
database don't store them as strings uh
if you store them as strings instead of
being like what is it eight no 16 bytes
of binary data it's about 56 bytes
including the hyphen um a uuid with all
capital letters is a different uuid than
a uid with all lowercase letters and
that's not great um and you're looking
up collation rules every time you do any
kind of comparison or database index or
B tree insertion or anything which means
effectively that when you're inserting a
uid into your table it has to check if
it's like a Mexican uid or a Polish uuid
or a UT utf8 uid because you order the
letters in a different way when actually
you just want to be treating them as
binary data so other piece of solid
advice if you're using postgress and you
want to use uids use the uid column way
faster way more
efficient um
and yeah I've now built a couple
databases just for side projects with
using id7 uh as a primary key yeah it
works and you don't have to think about
it it's just like sure okay but of
course this is something with like a
couple million rows so it's not like you
know my little side project that's
collecting the temperature sensor in in
my in my bathroom is um going to hit two
billion rows anytime
soon and then V4 is still useful for
completely unguessable stuff V5
keep an eye on V5 sometimes you might go
oh actually like I have a use for
this um where are
we yeah and if you need to invent your
own seriously consider using uid V8
because it lets you use postgress as
column it lets you use all these this
tooling that you've already got and then
finally yeah like the the only bit of
absolute solid advice that I think I
have for you
is if you work for a company you might
end up with a couple billion rows sooner
than you think and so uu IDs might not
be the right answer but 32-bit integers
like you're not you're saving Yourself
four bytes a row right or eight bytes a
row if it's indexed as
well for small tables that's not very
much and for big tables You'll wish you
had changed it so the one piece of
advice I have for you is move away from
move away from the small numbers for
your primary keys and that's my entire
slide deck I am done um we have like a
minute for questions
[Applause]
thank you Tom we are unfortunately out
of time for questions cool we have a
gift for you as token of
appreciation mug that is
gorgeous thank you very much that is the
only mugging
allowed I'll be I'll be hanging out at
the Kraken desk some of the time this
weekend and otherwise just come and grab
me at any moment if you want to chat
thank you so much for your time thank
you