hello welcome back
opening
opening data science and analytics track
this morning
laura summers who is a
multi-disciplinary designer researching
technology ethics and technology ethics
and building tools to promote fair
machine learning laura is passionate
about feminism digital rights and
designing for privacy which feels like
something that should be kind of
baseline for anyone practicing machine
learning and artificial intelligence but
unfortunately isn't
laura is the founder of dubai sai and
also the human beside behind open source
and community projects like the ethics
limits test fair archive and the
melbourne fair machine learning reading
group
laura can be found on twitter her handle
is at summerscope and she can also be
found on the venuless platform where she
will be answering your questions during
her talk which has been pre-recorded she
will also be available for questions at
the end so please
ask your questions and some of them will
be selected for a live answer
so laura starts with important questions
about ethics but also about practicality
because we all want to be ethical or at
least we all say we don't want to be
ethical but how
so drawing on her experience as the
ethics ops consultant embedded with a
small fraud detection team in a big
telco
laura gives us tips on how to move past
the
ai ethics hype
and how we can start trying testing and
implementing practical approaches
so please give a virtual hand of
applause for our virtual keynote speaker
laura summers and her talk
game the ethics of telco fraud
okay let's get started
i've put together a google docs um
document online you can go to it here at
this tiny url
you can see the links to the slides
extra resources and also any q a you
want to pop in there go for it
so just to quickly point to what i'm
going to discuss today
the lineup is we start with deconstruct
your proxy
we then go to explicit data design
assuming good faith
and failure first design and hopefully
all those will make sense by the time we
get to the end of the talk
so let me set the scene
starting in early 2020 i was embedded in
a small fraud detection team as an
ethics ops consultant
and as part of a larger telco
it was really a pretty small team there
was one data scientist one project
manager one ops person who was like
communicating with customers and
researching stuff with us
we sort of had half a data engineer and
a few other devs kind of floating in and
out a bit but mostly that was the team
we were using a combination of internal
tools and detection mechanisms and a
third-party tool that was doing payment
gateway fraud detection um and that was
using ml but none of our internal stuff
was like ml it was all kind of pretty
straightforward deterministic and
business logic type stuff
um i can't share specifics on the
features or the thresholds we were
working with but um you'll get an idea
through the talk um and we were looking
primarily for a kind of fraud called
international revenue sharing which is
essentially jacking up the price of the
call of a call for an international call
um by sending it to a mobile or sending
it to a premium number where you can
charge a higher rate and then sort of
taking a margin taking that higher rate
um slicing it off the top and um
pocketing it as
as revenue essentially
so instead of you actually getting a
service or talking to someone they're
just making those calls and letting them
sit there in a crew minutes and then
charging them back to the telco
but we also had other kinds of um
issues we were dealing with like um
service overuse and promotions abuse or
people were sort of pushing the
friendship or trying to use like
promotions more than they were
technically allowed to um so you know
there's quite a quite a mixed scene um
yeah and if you don't know what ethics
ops is or if you haven't heard that term
before it's um something i've heard
around the space it's it's i think of it
sort of as like devops or sec ops but
for ethics with the vu to try and
improve process
and give ourselves tools and deployment
sort of monitoring
approaches that will help us get better
at improving the ethics of our system
another metaphor i like to use is with
sre is like you thinking about this
question of how good is your uptime and
you know that you never get to 100 but
you're sort of aiming to get better all
the time similarly with um ethics ops
you don't assume that it's going to be
done and you're never going to have 100
perfect ethics but rather you're trying
to observe the state of the world now
and get to better
so yeah fraud is an interesting space in
which to discuss ethics not only are you
the company sort of capable of
inflicting harm on your customers you're
also under attack by a subset of
customers who mean you harm so you sort
of have to balance the real and often
urgent costs of identifying bad behavior
and sort of mitigating those costs that
they're they're charging to you while
also acknowledging the possibility of
false positives and considering how you
might identify and support people who
get caught in the crossfire
and i think an important way to do that
is to
develop this gut feeling this sense
track that we mostly i think already
have to be honest which is keeping that
healthy cynicism about our data sources
um and certainly in the team i was
embedded in like the thing i observed
was it was only when we felt really
comfortable and confident in what we
thought was going on that something
would
be really surprising and like kind of
turn us on our heads
so yeah this idea that like we have to
always
be sensitive to the distance between
what it is we can observe and how we
draw causality to events in the world or
behavior in the world or why people are
doing things even is something i think
is important to keep in the back of our
heads at all times um and a good example
of this was we were observing a
particular customer who was sending a
whole bunch of sms's overseas um and it
was it was a lot it was a very high
volume and we were all kind of pretty
confident that you couldn't be sending
this many sms's just like manually so we
thought maybe there was some kind of
script or bot or automation platform
that they were using to broadcast sms's
um and sort of the
sort of extension of that was we thought
probably they're doing some kind of
marketing push so maybe it's you know
spam or maybe they're trying to
get visibility somewhere but you know we
thought it was basically like them
trying to use our our personal services
essentially a marketing platform
um
and that really
that really seemed obvious but then when
we called this person up and had a chat
with them we found out that actually
they were sitting there with their
thumbs messaging just heaps of people
they had a ton of group messages set up
and it was essentially um you know a
communication
uh communication and talking to people
was still like the sort of underlying
reason behind their behavior um so that
really like flipped everything we
thought we knew on its head and it made
us worried that we were um gonna have
difficulty working out
what was the malicious behavior from
what was just like overuse
um and i think one way to think about
this is to like take some time and
deconstruct your proxy and that's sort
of assuming that you acknowledge that
pretty much every data that we might
collect as a proxy it's it's an
approximation for the thing we really
want to know but it's never the actual
thing we really want to know
so i'd like to offer a concept framework
for this and we can use the sms person
as an example
so we start with the signal so in this
case we were observing how many sms's
the service was sending and at what time
and looking at you know like the sort of
um frequency of their sends and that
kind of thing
um from there we were
then inferring an activity that we
thought would be causing the the data
that we could observe um so on the top
you can see a mobile and that sort of
someone sitting on their phone typing
out sms's and on the bottom was what we
thought it was which was some kind of
bot or script
and from there we can then make another
logic leap which is inferring some kind
of persona or motive behind the activity
that we think
is causing the signal that we're
observing
so you can see here how i'm constructing
this like it's kind of forcing us to
acknowledge that we're taking one or two
or more
leaps of logic
in getting from what it is we can see to
what it is we think is happening in the
world and that helps us keep a really
healthy um humility and nuance and
understanding what it is we can observe
and not getting too confident and saying
like this equals this and there's no
other interpretation
um so when we look at persona like again
we went down the bad route and we
thought oh you know they're sending spam
or sms or something but in fact it was
just a really large number of people
talking on group chats and like
communicating and sharing information um
and you can think of other ways that
people might generate that kind of data
as well for instance you could be a
developer testing out a twilio api and
that's also not a malicious use of your
phone but it might be unexpected
so the next thing i want to talk about
is this idea that
once we really get into our heads that
we don't know
everything perfectly and that we have to
like allow for uncertainty we also have
to try and give ourselves the biggest
leg up we can to interpreting and
handling information we do think we know
especially under stress so i want to
make a distinction here um between what
you might think of explanatory data
science where you're trying to describe
the world you're trying to understand
what's happening you may not be sure
what the right answer is and you're sort
of
you know going at it from different
lenses or cutting the data in different
ways and that's that's a really
important and useful thing to do but
that's not really what i'm talking to
right now so what i mean when i talk
about load-bearing design or
you know data science under pressure is
when you have an alert set up and
you know like that alert triggers some
activity or it means that you have to go
check out a service or you have to go
like have a look at a customer and see
if something unexpected is happening and
it usually means that you have to
respond
in the moment whether it's you know
eight in the morning or two in the
morning
um so yeah i'm really talking
specifically to load bearing in high
pressure situations for the rest of this
section
and with that in mind i think it's
really
important to emphasize that data and
misinterpretation is not just easy but
likely under pressure
we're putting like or putting our data
science outputs through a ringer or like
into a crucible and like really seeing
how well they stand up um so if we want
them to survive that environment we have
to design for a cognitive load that
means not putting too much information
in the view and that means being really
cautious to make everything as
interpretable and explicit as possible
and i i think it's important to remember
that simple
design doesn't imply that you are stupid
the person receiving that design or
maybe even your friend or your colleague
or the junior or your project manager
it just means that you're recognizing
how important it is for this to go right
um and another anecdote from my time in
fraud detection was that we had these
alerts coming through to slack and the
alerts were
coming from data that would be ingested
then a batch would run and a feature
would get calculated and then depending
on the outcome of that
there was another delay and then they
would come into slack so there's sort of
a time frame between when the data would
be ingested and when we would get this
alert and whenever the alert was bad and
we were worried about things we would go
check our logs and have this like
momentary panic because we would forget
about the delay between when the data
was ingested and when this alert was
coming out
and we would be like oh [ __ ] oh the log
is showing something different to the
alert oh no something's so we'd think
not only that something was going wrong
as in we were under a fraud attack but
also that our systems had fallen down in
some way and kind of doubly panic um so
i really wish i could go back in time
and tell laura from the past to like
take the time to make that as explicit
as possible so what are other pointers
for making your data as interpretable as
possible
this is probably an obvious one but it's
bears repeating avoid dsl avoid
domain-specific language avoid tlas the
three-letter acronyms um and
particularly don't put more than one of
them in a line or in a sentence right
like the more that you have the more
opportunity for people to get confused
um so an example here of this like going
wrong on the left is where we see usn's
breach tier one what's the usn
um on the right you can see that it's
actually unique service numbers that
makes that a bit more clear
another again probably obvious one but
one that people easily miss is add units
we can see on the left i've got a tier
one is a breach of over 30 something in
24 hours but what does that mean it
could be sms's it could be hours on call
it could be number of calls made in
total
on the right we can see it says 30 calls
in 24 hours so
now i know what it is we're talking
about
this is a bit of a bug bear of mine if
you are talking about things like delta
or percentiles and you're looking at
change up or down say over a rolling 24
hour period or over a week
that's a perfectly valid way of looking
for
spikes in behavior or changes but it's
really important to ground that with
your absolute or total count of the
thing
so for example here on the left we see
12 and 24 increases in tier one and tier
two that seems like tier two is really a
problem but then if you ground that in
well it's only three unique surface
numbers for for the top tier and then
two unique service numbers for the
middle tier like they're about the same
they don't actually have that much
difference it just it's just because the
volume of the initial
breaches wasn't was quite different so
yes that kind of getting sucked into
percentiles without knowing how many in
total we're thinking about really
matters for deciding what needs action
for instance
um and then again to the anecdote i was
sharing at the beginning of this section
if your data currency is like
different to the time you see the alert
or see the dashboard or whatever it is
you're looking at make that as clear as
possible and you know don't assume that
people will remember because you're just
adding more stress into their heads at a
period when they're already under stress
i think it's important to remember when
we work with fraud but also when we
think about um you know fairness and
ethics
it's easy to kind of get into this
mindset of
being a bit spy versus bi and seeing
seeing like kind of bad actors and
malfeasance everywhere
um and i think
not not that we don't want to be able to
imagine what can go wrong
but i think it's important that we start
from the assumption of good faith
particularly for our comms and then work
our way down from there so we don't want
to start it like you know a 10 for
suspicion we want to start at a 1 and
like kind of add points of evidence and
then you know if someone's really acting
in bad faith then we can
up the ante on the tone
and especially remembering that there's
no need to suspect bad intentions when
laziness or incompetence explains the
same thing just as well
so i think it's important to
avoid falling into that trap of being
ultra suspicious and seeing bad stuff
everywhere and i think a good way to
avoid that trap is to be explicitly
deliberately
kind and respectful in all of our
communications and when i say
communications i mean not only how we
talk to customers whether it's actually
a communication like a phone call or an
email
or whether it's a email template or a
notification template that we're setting
up or if it's how we talk about them
inside our teams about a customer or an
end user like the more we can set this
intention and stick to it i think the
safer we are and you might be saying to
me well laura what if it's a bad actor
and
to be totally blunt my answer is so what
um i don't think it matters if you speak
respectfully to a bad actor but i do
think you can cause a lot of harm if you
come at you know a confused actor or
actor who's just like doesn't understand
what's what they're doing that's wrong
with like a lot of aggression or a lot
of suspicion or sort of interrogative
tone so if you can
bring it all back and start from the
assumption that they may just not know
that they're not allowed to do the thing
or that it's not appropriate
and try and give them as much sort of
principled support as you can and then
work from there you're much less likely
to cause brand damage you're much less
likely to lose customers because you
know you've alienated them
um and i think if you if you simply take
it down to the bottom line you're much
less likely to lose money and on top of
that if you do talk to a bad actor in
good faith you sometimes catch them out
right like they kind of think oh they
don't suspect me and as a result they'll
tell you things that confirm your
suspicions about them so there's this
like additional value in coming at them
with real like you know clean neutral
respectful language
and
to add to all of that
we're in the middle of a pandemic people
are having a bad time you don't know
their circumstances you don't know how
much money they're earning or not
earning and you don't know certainly
what's driving them to do what they're
doing so i think it's important to
remember also that we don't want to be
moralizing about this or judgmental so
it's better to try and keep that aspect
of it out of it and just focus on what
it is you need to do so if it's
preventing behaviors if it's um you know
nudging people along the right path like
keep those as your as your guiding star
and get away from this issue of i was
right and they were wrong or i was
better than them
another tip for trying to get this
better is to to focus on describing
behaviors and not people or activity
that you can observe and not people so
for example in the fraud team we were
calling people fraudsters and abusers a
lot and it was something that kind of
didn't sit right in my gut and i was
sort of interrogating why do i not like
this this um use of language it was like
oh you know we're
doing the thing of making the
assumptions saying that's definitely
what's happening but also it's got this
kind of judgmental tone
and we're adding a bunch of like
probably unnecessary
uh like um judgmental and moralizing
attitude to to something that can be
quite simple so we just started talking
instead about you know
i can observe like this person's sending
a lot of sms's to country x or oh
they've had a spike in phone
phone call behavior and when you start
doing that it it again like
it focuses you on what you actually know
and doesn't take you past that into like
the assumptions that you make from that
or the inferences that you make from
that
don't mistake
this this whole rant for saying that i
don't think you should intervene on
people who are behaving inappropriately
in your systems i absolutely do think
that that's important and in fact often
i think we can solve these problems
better upstream and product by like
designing in
clearer like boundaries for what's
acceptable or making the product offer
more clear or you know otherwise adding
in those guardrails
but i'm just saying you can do that work
and still stay respectful in your
communication style
so the last finding i want to talk to is
this thing i like to call failure first
design and that stated simply is is
saying that we start from the assumption
that the system will sometimes fail and
then we plan for it
this quote i love from stuart russell
and he says assuming perfect knowledge
of the objective
decouples the machine from the human
what the human does no longer matters
because the machine knows the goal and
pursues it
and i think that's a really important
way of like helping us think about this
like we don't even know our own
objectives perfectly let alone people's
objectives we often do a really poor job
of capturing preferences for instance
and we live in a world where there's no
bug-free software and when you add in
stochastic noise and non-deterministic
systems you have all of these
abstraction layers and um
aspects of our tech that can go wrong so
just accepting that that's the world
we're living in and trying to say well
we'll inject some humility into our
planning and into our design and we'll
try and continue to do these little
micro adjustments based on what people
tell us about their preferences um you
know their experiences etc the more
likely we are to
not get down the path of having you know
done something horrifically bad to a lot
of people and not even known about it
until something terrible happens
so i think about this work as like
designing an escape hatch or adding
lifeboats to a big ship right like
it's not necessarily something that we
like to think about it's not necessarily
fun work and in fact you may not you may
think that success looks like never
seeing it get used
but you still have to make sure it's
there and it's functioning and if people
do need to use it you want to make sure
that it's actually usable
so feedback loops um in order to
discover things about people's
experience in the moment need to be
intuitive contextual and timely and what
i mean by that is we really can't just
throw some support channel at the bottom
of our page and assume that people will
talk to us right we have to actually ask
them questions about the the specific
contextual thing they're doing in our
tool at the moment
so some little examples of that
we might for instance on the left we're
looking at botify which is a music
streaming service and it's showing me
like the different genres that i listen
to the most now i might know that i
listen to a lot of lo-fi hip-hop but
actually prefer to listen to a bit more
classical so i've got a link which lets
me adjust my preferences
on the right a little bit more of a dire
example i've applied for a loan of 85
000 and been rejected and what i've done
is added a couple of buttons that give
me ways to kind of
further explore that explanation i've
seen and maybe like let me talk to
someone or let me kind of break down
how that algorithmic decision worked so
that i can actually understand if i
agree with it and see if i need to take
more action or not
another big one with image
classification is just letting people
tell you when you're wrong especially if
it's their images
in this example we can see it's
incorrectly classified as a cute puppy
as a cute kitten
another one that i like to talk about a
lot is adding in the level of confidence
of your system um just kind of letting
people know that you don't think you're
perfect is much more encouraging for
gathering feedback as well
this is a pattern you've probably seen a
lot it's getting a lot more popular now
where we see
spikes of activity or things changing on
your account unexpected logins and new
machines um and we'll sort of send
someone a message immediately in the
moment and just check and see it was
them and most of the time it is but when
it's not it lets us sort of
smell out suspicious activity very
quickly
and the last one which i think is
pretty
easy to do and important but easy to
forget is if you're working in a domain
where you're expecting someone to
experience harm or you're worried that
that's a likely outcome for them you can
just ask you can ask them you know
in the moment that they're doing a thing
like filling out the form or using your
tool or you can ask them like as a
screening question before they join um
but asking these questions of if they've
experienced harm before or if they're
experiencing harm currently in your tool
can be a surprisingly fruitful way of
eliciting what it is that's happening
so it's really important that if we do
do these things we don't just want them
to fall in a heap right so we do have to
plan and do a bit more work
and that can look like customer support
or writing a support script or thinking
through
policies for the use of your tool
it also means you have to be really
meticulous about capturing your
learnings you don't want them to just
sit in a bunch of support tickets you
need to have some kind of data schema
you know it can just be a spreadsheet
somewhere but you still want to make
sure you know what it is someone did and
what you classified them as and if
something changed how that
classification changed
and um importantly if you do find out
that your model isn't great or if
something's going wrong in your logic
that you can leave time for product or
model improvements
and also
preparing for pushback um this is a
little bit of a sad story part of the of
the talk but
people don't like it when you make bad
assumptions about them and that's okay
you just have to prepare yourself
mentally for that experience
um
if you think to yourself oh well i'm i'm
making something visible that previously
wasn't visible and that means that some
people are going to say oh yeah that
makes sense to me and i agree with it
and some people are going to be like oh
okay i don't like that anymore or i
think this company doesn't know what
they're doing
you know you're inviting the opportunity
for people to disagree with you or to
think that you are doing something a
little bit different than they
originally expected um but that said
you're still going to find out much
sooner if something is going wrong
than you do if you just have no feedback
loops baked in in the first place
um and certainly in my experience in
fraud
we had a few instances with these kinds
of feedback loops where we had some very
angry customers and some very like clear
mismatches of their mental model of what
they thought they were buying and what
we were telling them they had bought
and it was uncomfortable but it was also
really important for us to know
so just to review we've talked to
deconstructing your proxy
explicit data design assuming good faith
and failure first design and hopefully
those are useful little like flag posts
for you to remember and think about how
to do this work a little bit better
and um just to close up with a metaphor
i think of
making tech as us working together
sitting on a a big ship a cruise liner
and when you go below decks it's quiet
and safe it's maybe even boring like you
don't even have a sense that you're
going anywhere
but when you come back out on deck you
have a sense of the speed and the mass
of water displaced
and i think as technologists and as
people working you know with data
systems with real humans in the world
we don't want to insulate ourselves from
that sensation we want to feel the wind
rushing through our hair
we want to have a little frissona fear
and not be too comfortable
because i do think the real game is
trying to catch fraud yet not defraud
ourselves
thank you so much for your attention i
hope you enjoyed that um i just want to
add one little note which is i am
working on a tool which is coming soon
which is called sweet summer child score
um it's going to be a
test which scans for harm to people and
communities by data driven systems
that's going to be free and open source
and i'm looking for testers to help me
kick the tires so please do reach out to
me at laura depis dot ai if you think
you could help
thanks again
welcome back thank you laura that has
been an amazing talk and you know as
someone who already knew the outline and
had learned something from it i've
learned so much from every one of your
slides there are questions the first one
is um what have you found helpful for
getting people who might be you know i
have some resistance some friction to
changing the ways they operate they have
operated traditionally like traditional
fraud teams at you know this kind of
institutions
yeah look it's it's a great question
it's one i ran into a lot in this
experience um
the the short answer is it's hard and
you can't fix it straight away um the
the longer answer is trying to come up
with champions people who like genuinely
care about the work and being really
visible about
you know awarding them applauding them
um you know like recognizing their work
when they do
um they they do kind of model these
behaviors of um holding themselves back
from being too judgmental or sort of
focusing on behaviors and outcomes and
not focusing on you know moralizing or
judging people um it can be a long slog
because you're really like changing
mental models and you're changing like
these very ingrained behaviors
so also being patient with yourself that
it's not going to happen overnight i
think