foreign
okay welcome everyone back to All Things
data we're going to jump straight into
the next talk and um you know we often
talk about people process technology and
uh our tools and as a python conference
you know we have a lot of tools coming
up uh and um uh but today we're going to
be talking about people in process and
in particular around data contracts and
how they can help us uh you know get
more data quality and you know all that
good stuff that we're actually trying to
do with data so I'm
uh really excited for uh the talk next
from Ryan Collingwood and I'll just give
a quick introduction to Ryan
um he uh is a boundary spanner between
business and I.T concerns uh he's lived
product management consultancy iteration
management business analysis data
analysis software developer quality
assurance to where it started as it
support every day he practices a bit of
all this to deliver pragmatic and
maintainable solutions he loves the
journey both as a contributor and a
coordinator from the wrangling modeling
sampling digging for insights and all
the important communication of insights
and further actions and he gets a kick
out of teaching others to finish the
many water bodies of data whether they
be ponds lakes or swamps so um let's
hear put your hands together for Ryan
Collingwood talking about data contracts
consensus as code
thank you thank you thank you um I'm
really excited to be here at pycon in
Adelaide it's just fantastic energy and
thank you for being him
so who am I and what is my current
context because I feel like context is
really important to share whenever
you're dispensing advice uh I'm
currently working on the team of data
and analytics at origin if you're
unfamiliar with origin origin is
Australia's at least by my Reckoning
oldest fashion brand Australian fashion
brand founded in 1938 and we are a
centralized team very small centralized
team looking after a much bigger rest of
the organization and our Tech landscape
is dominated by one or two monoliths
namely our enterprise resource planning
application and our pause system and
it's orbited by a number of little
sassers you know so you know these two
big planets and all these smaller ones
orbiting around it and our data is
mostly moved around in a batch fashion
why I think you might care about this
conversation so over here we have a
reference diagram of a data stack
if you are in the pinkish purplish
bucket of data engineering you probably
know all of the pains that I'm about to
talk about if you are in the blue bucket
typically the consumers of the magic
that happens in the pink butt bucket you
probably have felt the impacts of this
pain and if you're on the green bucket
well fantastic to have you here because
maybe you don't know about all of this
and we need more allies from that bucket
before I get too far along with this I
want to give a shout out to Andrew Jones
Andrew Jones is to my knowledge the
person that crystallized the term data
contract within this current context uh
published a book at the end of June uh
was really helpful for me putting
together my thoughts I just wanted to
say thank you
and before I get too far I just love
this quote it may be from a song that
you recognize narrated by baz luhrman
it's actually from an article on the
Chicago Tribune and the gist of it is
take this advice from when it comes that
comes from my context and uh if I could
tell you anything about the future where
sunscreen so what are data contracts
over here I have two quotes one of them
is from Andrew Jones himself the other
is from atlan now there's a lot of words
on screen let's not try and read all of
them at once but what I want to draw
your attention to is the mention of
people that you have a generator
consumer parties if you think about a
contract in the real world it involves
people and that's a core fundamental
thing that we need to keep in our minds
here the other is about an agreement
so that's the blue text there and then
there's a mention there about having a
structure you know or or an interface by
which you access this and then finally
the the Big Kahuna at the end there at
least for my situation right now there's
a bit there about data quality or having
certainty about the what you're going to
get out of this
so let's take a scenario we have team a
team B Team C and let's just say they're
all working in the same organization
data is Flowing everyone's happy but
what I haven't told you is that team C
have set up a non-consensual API
what I mean by that well team C have
found a read replica of one of the
databases controlled by team A and are
pulling out data and using it for their
own products Downstream
but then as always change happens
team a their database goes through some
changes
teammate had an understanding with Team
B so Team B knew this was coming and
they made the necessary steps to
mitigate this change and so their
product is
carrying on still working whereas team C
because they didn't have an agreement
didn't have an understanding this change
blindsided them and so their products
are now impacted and potentially not
working
so if you've been working in that pink
stack in the middle as I was saying in
the beginning you know about this you've
experienced this and maybe you've even
caused this
so what makes up a data contract let's
let's get to it well a data contract
could be as simple as this
this is a yaml document that's
describing an event
it has some information about the
contract itself you know the version who
authored the contract it also has uh
service level objectives so we're making
a promise an expectation about
how complete this data set will be how
fresh it will be and then below you'll
have a there's a collection of things
that you might recognize as being schema
but what I want to really draw your
attention to is that a data contract is
not just schema
as I mentioned it's people if there was
anything that I had to prioritize and
don't take the ordering as a
prioritization however I will say this
if you don't recognize and understand
the people as in the people who are
creating the data and the people who are
looking to consume the data and even the
people who are responsible for
maintaining the underlying
infrastructure if that is not captured
and part of your data contract
you probably don't have a data contract
other things that you might want to have
in your data contract are a schema we
love schema schema is great uh contract
governance so this could be things about
say the version of the contract or the
publishing state of the contract then
there's semantics semantics is different
from schema and we'll talk about that in
a second but the tldr is schema for
machines semantics for people Fair bit
of overlap then there is a bit there
about service level objectives because
again if you are providing a contract
another thing that is inherent contract
is an expectation of being able to get a
benefit
then there's the governance of the data
set so this might speak to how
privileged is this information is it
classified is it internal is it public
and then finally maybe something about
the mechanisms of transmission so this
might cover how the data was captured
um whether it was from a human entering
data into an interface or was it a
sensor and then your expectations about
how this data can and should be
transmitted
let's take a minute to take a slight
detour about schema and semantics
because this thing is near and dear to
my heart as I said schema exists in my
mind for the benefit of systems
so you have a data type in your code
it's an integer
it's an integer because your system
needs to make certain assumptions about
it so that if you try and do an
arithmetic operation on that variable it
knows treat this like a number
whereas semantics is for humans and
human expectation
let's take an email address from a
schema perspective what is an email
address it is a series of characters one
after the other you might call it a
string or if you're speaking in database
parlance maybe it's a varchar right
but for a person
we don't see it that way we see it as
this is a piece of information that has
a very specific format that I can use to
communicate to somebody and I have
expectations that I can use this
information to have a conversation right
and while we're on the topic of email
addresses there's a fantastic GitHub
repo called awesome falsehoods if you
want to have a crisis of confidence
about what you think you understand
about email addresses about names about
any sort of what you might think is a
standard data type go check out that
repo and have your confidence utterly
destroyed but that repo is really
speaking to sort of the semantics you
know there's as I said schema is about
being able to store and retrieve
information and do operations on the
information by systems with confidence
whereas semantics is ensuring that we as
people make the right interpretations
and assumptions about the data that
we're looking at
so what's the minimum amount of stuff
that you need to be you know thinking
about if you're looking to implement
data contracts well first of all don't
set out to implement data contracts set
out to solve problems but let's say
you're looking to solve problems with
data contracts well again you're going
to need someone
generating the data and someone who
wants to consume it that's again people
first and foremost
then
you're going to need to define the data
contract now this is a this is from
Andrew's book and it my mind is fairly
optimistic as we'll see as we get
towards the end of this talk where I
bring it to my context there is this
optimistic hope that your data
generators will Define the contract and
then you know from that contract you can
do things like provisional interface
which represents a service that has its
own database that the consumers can then
interact with via that interface
I've put an extra little database
diagram up there because the guidance
that I've seen in the literature is you
don't want to write a data contract
against say your operational database
you want to have sort of a layer of
abstraction like an interface over the
top so maybe it's that you publish
events of interest to a topic queue or
you provide a database right which is
it gets its definition from the contract
itself or maybe it's just a view that
you offer up to your consumers but you
don't want to write it directly against
your concrete implementation because if
you're needing to then change the
internals you then have to reroute the
contract right so you want that level of
abstraction so that is a key Point not
to be overlooked
so you've got this data contract what
can you do with it a couple of things
from the data contract you could
generate some Json that you can then
feed into say cloud formation or
terraform or whatever your
infrastructure is tooling uh of choices
and then provision infrastructure so you
could take a data contract and provision
a big query table that has a schema as
defined in the data contract right
you could take a data contract and
generate just on schema so if you're not
familiar just on schema is an
opinionated way for describing entities
and their attributes and from just on
schema you could then generate apis so
you're given your consumers not only
access to your data but also the
mechanism they don't have to write new
code here's a library have one for
python have one for JavaScript off you
go
another thing you can do with data
contracts and this is you know what
brought me to the store is being able to
Define data quality checks all right
there's three places broadly speaking
where you can Implement quality checks
the first is that publishing time so if
you have influence over the system that
is capturing the information before that
data is even written persisted you can
do data quality checks to make sure that
it's conforming right or at the
infrastructure itself so say you pass it
on to a message queue you can then
identify does this event conform with my
expectations no put it on a dead data
queue or manage it in some way and then
finally there's after publishing now if
you're living in a batch world this is
probably going to be realistically where
you're going to be doing a lot of your
work
the one downside to doing it off the
publishing is that if you have errant
data it's out there all right so then
you're going to have to think about okay
what is our case management Incident
Management around data that's
potentially leaked and you know now
there's Gremlins all over the kitchen
so I'm going to bring it back to my
context as I said in the beginning I
have one or two big monoliths in my
world and one of them is the enterprise
resource planning system so for those of
you who are not familiar with what an
Erp represents just think of it as
perhaps all the microservices that you
might have but except in one box and one
interface right
so people from various functions you
know from procurement from Finance from
design they work in the Erp in various
places and that information gets written
out to the erp's database and that
eventually Finds Its way into my world
and the data Lake pretty standard bronze
silver gold situation going on there and
from that we as the data team pick it up
clean it up conform it to a model that
we then present in a variety of ways be
it reporting be it you know sort of
analysis and Discovery and
that's the value chain of data in my
organization but when I look at this
again through the context of
producers
what I see are producer boundaries so
there's a producer boundary all the way
on the left where people are entering
information into the Erp zy
there's another producer boundary
between the Erp database and where it
gets picked up and adjusted into the
data Lake and finally there's another
producer boundary between where where's
the data team have now generated these
opinionated gold standard models and
surfacing up to the rest of the
organization
and the people on the right may also be
some of the same people on the left so
again
in the beginning we had that nice linear
diagram of producers and consumers
they may actually be the same people in
some situations
so at these producer boundaries when I
think about it these are inputs to a
data contract so between the people
working with and interacting with the
Erp that's where we can get an
understanding of the semantics
one of the
depending on your point of view perks of
a lot of these Erp systems is that you
can configure them in any way that you
choose to often you're advised not to
but no one listens to that they go ahead
and configure it anyways
or conversely
they don't configure it and they just
enforce Rules by convention
so there is a understanding within the
people operating the systems that okay
this field over here don't populate it
if these other things are true you know
so if it's if it's a purchase order say
for apparel don't fill in that field
over there but if it's a purchase order
say for materials fill in that field
over there but that isn't enforced by
the system right so that is where again
the semantics of the data in the world
that we're living in that's where I can
get that information from having that
conversation there
similarly between the Erp database you
know it materializing its information
and its schema and its point of view I
can then start to pick up on the schema
and perhaps the service level objectives
that I can expect profiling the data um
I see it looks like there's a purchase
order raised every day so maybe that's a
test that I could write you know expect
that we will see at least one record
every day appearing in the table that
represents purchase orders and then
finally
um I will take those two things in my
context and then generate the checks and
tests which I'm applying at that last
boundary over there
the reality that I have defined since
starting this journey it's been about a
month and a half now is
the people at the very end also have
their interpretation of the semantics so
again that initial diagram that had this
ideal linear you know left to right flow
in reality it may be a little different
so consider the people at the beginning
and at the end because they may have
slightly different but still significant
interpretations of what the data means
so how are we going to make this all
happen well we're going to use awesome
people that understand modeling that
understand abstractions and understand
constraints and I think I'm looking at
them we could even do it in code
and you should definitely Version
Control it
but you might be thinking hold up a
minute a couple of slides ago you showed
me a yaml document and said this is a
data contract why are we talking about
code now a couple of reasons
um predominantly
documents
you know be it a word docket Excel
document even to a lesser extent a
structured document like joson or yaml
they suffer in varying degrees from what
I describe as the entanglement of
meaning and representation so the
simplest example I can give with us is
say you've got a Word document you know
a rich document that people can apply
formatting to
you can start typing a sentence a
statement of fat
later you can come back to that document
and apply formatting that strikes
through that sentence
when reading that document you as a
person interpret that oh that sentence
is no longer valid it no longer applies
however if you parse that document you
may not pick up on that
so again it's the entanglement of
meaning and representation when we write
code we say exactly what we mean because
we have to
another reason why you may want to
consider doing this as code rather than
a whole bunch of documents is finding
references so when you have your code
and you want to find where a particular
variable has been used it's pretty
straightforward these days we've got
wonderful tools just right click find
references there's all the places but if
you're doing this with plain text you've
got to do text matches
and that just opens up a whole can of
worms that you now have to figure
through
finally there's some other good stuff
like we can test code and ultimately if
you need to represent that knowledge as
say a Json or yaml document you can do
that from code right
and again speaking about refactoring
text you might think Ryan but I am a
wizard of regular Expressions I got this
I believe the same until I have to do it
and then I find myself doing a whole
bunch of Googling or playing around with
a regex tester right not knocking
regular Expressions they're fantastic I
just think that my little brain can't
keep you know track of them for long
enough to you know remember what I need
to do the next time I need them
so coming back to my context what was
considered as I said the first thing is
I didn't set out to implement data
contracts I had a realization that there
was this misalignment around the
expectations and the understanding of
the information that was being entered
in at the beginning of my the journey
that I this last that I showed you there
and at the end there was a lack of
knowledge there was gaps in our
understanding so first identify a
problem that you believe can be
addressed by establishing and
documenting your consensus
next step is of that big problem find a
small scope let's let's do something
small first rather than trying to boil
the ocean and make sure you've got
allies people that are willing to work
with you on this
secondly identify what your constraints
are every organization is different
every situation is different you may
have more Engineers available to you you
may have few you may have great
engagement through business you may not
the next thing I would say is keep a
people and process Centric right if you
are looking to address a
part of your world that doesn't involve
people
I would say that a data contract isn't
necessarily the way to go about it if
you're looking at interchange between a
system a system B no people involved you
could probably get done what you need to
get done with an API yeah just keep it
simple
next because I was looking I had this
hairbled idea of doing this all as code
I had to think well I'm going to need
some sort of way of modeling all of this
right so I describe this as my meta
schema
and then finally and this is where I'm
going to contradict myself in a couple
of slides I was looking for ways to
maximize uh opportunities for people to
contribute
let's talk about those guiding
principles the primary objective for me
was establishing consensus and the first
outcome that I was looking for was Data
tests as I mentioned you can use data
contracts to create infrastructure
create other tooling but for me it was
really about I want to have confidence
and be able to confidently say to other
people that you can expect the data to
have these characteristics and these
guarantees
then around that meta model when I was
thinking about it what I wanted to
capture was the meaning and
understanding of our data from the UI
all the way down to the database right
it was really about and just having that
end-to-end lineage going on
and understanding and recognizing that
there is schema and their semantics
they're similar but not the same
and truthfully I'm still figuring it out
I was sitting in last night you know one
not being able to fall asleep because I
was worried about catching my flight and
then two thinking I don't like the way
I've named certain things and we'll get
to that in a second here we go here it
is so here's a you know a representation
of my meta schema for my data
contractors code so the black stickies
are what I describe as contract
governance so the version number the
publishing status who are the people
involved and any service level
objectives that I'm trying to achieve
for this data contract and then within
the data contract I've gone for keeping
it again centered around events so an
event occurs say Supply created
and all for that event you may have one
or more entities I don't like that word
I'm still thinking of a better word but
bear with me and an entity has one or
more properties and a property and this
is where I was keeping the the reference
the representation of the semantic and
the schema separate a property is a
composite it has an attribute which is
to speak of the semantics
and that it has a source which speaks of
the schema so I'm living in a relational
world so my source is comprised of a
table or column in a database but if
you're working say with a document
database that might be say a key a
record a collection or if it's hiding
away in an exile document it could be a
workbook a sheet a cell right you know
allowing myself dynamism for the future
potentially
the other thing I want to call out there
is explicitly modeling the semantic type
now the reason that I believe that this
is worthwhile and necessary to do is the
implementation of a semantic type let's
say email address May differ wildly from
one system to the other so if I have
knowledge that system a we have a simple
you know attributes of the semantic type
of email and I need to exchange to
system B I can then write some code that
does that translation for me because I
have that hook that they're both
semantically an email address
for example but bear with me
so how are we going about doing this
I'll take a moment and give you a chance
to look at this
because you're probably seeing that
Excel icon going Ryan what the hell
you're just saying you're doing this as
code but there's an Excel spreadsheet
over there are you crazy probably no
again coming back to my constraints and
principles you may recall I said I was
looking for opportunities uh for other
people to contribute so going on this
journey I have a business analyst you
know that has been linked to the team
they're fantastic Randy couldn't sing
high enough Praises about them and when
I was thinking about yeah let's do this
as code I thought they couldn't Learn
Python you know
they're smart they'll get it but again
the constraint was we needed to move
quickly we don't have a lot of time to
do what we need to get done or rather we
need to prove value very quickly and
thus I came to the realization of well
if they're going to be doing a lot of
the legwork talking to people and you
know having these conversations about
what does this mean and where does it go
spreadsheets acceptable right so
we have the spreadsheet that where
they're capturing all the knowledge
about the events and what's involved in
them
that's but she gets committed to a git
repository where then I've got a crcd
pipeline set up where it reads in the
spreadsheet generates my code models
those models are tested and now we have
new contracts expressed as code models
and then soon in the near future from
those code models I can then generate
all sorts of things right now I'm just
generating very simple SQL based tests
so if an attribute is marked as it must
always be present or it must be unique I
can generate tests that automatically
test that for me
and you could do it you know using Great
Expectations or soda or any other the
testing libraries right now I've just
implemented as basic SQL because I have
yet to make that decision about where to
go next in terms of testing
so who's going to do the work coming
back to this optimistic diagram so
hopefully these people the data
generators
but in my experience being part of a
centralized team
because I'm I'm in the category of data
consumers in reality but I'm having to
derive that process so again your
mileage may vary so if you go out and
read the literature and you see the sort
of idealized state and you think well
hold on that's not going to really be
how it plays out for me don't worry
I'm having this too
so we're the path and conference let's
talk briefly about what we're using to
do all this stuff
so some helpful python libraries pandas
for reading in the spreadsheet what you
may find is that if you do go this route
and hand a spreadsheet to somebody and
say please fill in the spreadsheet you
know you give them a very opinionated
structure you may get back what I
describe as a sparsely populated Excel
spreadsheet so the First Column my
spreadsheet is the event and I had the
expectation that they would write the
event name on every row not the case
they write it on the first row and then
I have five blank rows and then the next
event name appears so long story short
pandas read in the spreadsheet and then
forward full and also pandas is great
just for any other manipulations you
need to do pedantic for defining my
models what I love about python amongst
other things is python is a what I would
describe as a gradually typed language
working on python if you're not sure
what a variable is going to be you don't
need to tell it hey this is going to be
an INT and if it isn't you your
application immediately goes Kaboom
right but however with the introduction
of Thai Pence and live use like pedantic
as you get more certainty about what
you're going to be bringing in
you can model it use it and off you go
I'm using rope because the models that
are initially generated are quite
verbose and rope is a fantastic library
for manipulating your python code and
effectively changing it so you use path
and code to manipulate python code and
then Pi test my pi and black just for
hygiene and you know having confidence
so talking about the generated model
here is an example of one right as you
can see it is a very verbose there's a
lot of inline variable declaration going
on here
and as you may recall one of the reasons
why I want to do this as code is so that
I could get the benefit of being able to
do things like refactoring or finding
references
what I would prefer is something that
looks more like that
because over on the down below there
I've highlighted a variable attribute
company and because that attribute
conceptually appears in multiple on
multiple entities in my code because
I've refactored it and extracted as a
variable I can now in my IDE right click
say find references and on the very far
right hand side you can see all the
other places has been referred to
I've got a little collab notebook here
which demonstrates the use of rope to
extract and refactor extract variables
and refactor your code don't have the
time to go through it all right now
um I'm sharing this because rope wallet
is a fantastic Library it's
documentation as to how to apply it in
terms of recipes I haven't really found
a whole lot to help me out so have a
look have a play but to expand this code
very briefly I'm first finding the in my
abstract syntax tree the nodes because
that's how your code is represented that
I want to manipulate
identifying their character position in
the code and then I am telling um I'm
defining a new variable name for it
and then for each of these variables
that I found I'm using rope to extract
the variable telling rope to identify
the changes it needs to make and then it
makes the changes and thus I go from
this very reverse code to this nicely
condensed code
so some further options as I said
the code that is generated is quite
verbose and there's some more things
that I would like to implement to make
it a little bit easier to work with
um I won't go into all of this but if
you have any thoughts about how I might
achieve this I'd love to hear from you
so some key takeaways you may be a data
producer and you may not know it okay so
make it worthwhile for people to
register
you know with you as a data consumer
and a data contract is a way to go about
doing that and then finally code is
easier to refactor
finder references and generally maintain
than the alternatives
and with that thank you for your time
and I look forward to catching up with
you
thank you Ryan that was really good uh
yeah it turns out when you add people
into the mix it gets a little bit
squishier and Messier and especially
many of them it's a really good sort of
actionable uh or you know solid
strategies for digging into it we
probably have time for maybe a quick one
or two questions do we have any
gonna end this
can I
there you go
hi I was wondering with your contractors
code is it your intention that these the
code will test the inputs to a system in
real time or just in the test
environment or where are you going to
deploy it potentially
um so as I said my world is very batch
driven right now
we do have aspirations for moving some
of our flows to be more event driven now
so things that are critical say the
activation of a gift card you know we
want customers to be able to buy a gift
card and use it immediately rather than
waiting for the current you know
uncomfortably long period of time that
it takes to activate so in that scenario
I do see a very real extension of what I
have to be able to you know receive
events test them validate them and you
know pass them along should they they
fit yeah
cool alrighty thank you um we might call
it there it's time for our break so
firstly before we move on to the next
things a little gift for you
all right thank you very much and
um one more round of applause for Ryan
[Applause]