[Music]
hi everyone welcome back for the last
talk before we have lunch uh would you
please join me in welcoming to this the
podium uh Clinton Roy who'll be talking
about adding file system contexts to
path
lib it's always helpful to get a
reminder of um what is between you and
lunch so no pressure uh hello everyone
my name is Clinton I'm a software
engineer uh most of my career is has
been around helping researchers uh write
code so that they can do more research
and more science and make life better uh
at the moment I'm at the Australian
synchrotron um a few of my colleagues
have given uh a couple of talks earlier
in the week on the synchron stuff so I
don't have to go over that
fortunately um I'm talking today about
something that is not uh synchrotron
specific uh but it certainly has come up
from uh some of my work at the
synchrotron and it is a very small tweak
to python that I'm suggesting um and you
know as if as with any change the change
is the hardest
part um so during this talk I'm just
going to outline what I'm proposing um
why um how it could be implemented and
what are the next
steps um there is a public part of the
Python Community public website uh uh
called discus or discuss um there is a
section of that called python ideas I'm
trying to choose my words very carefully
here python ideas is a place where ideas
go to get schooled um a lot of people
have ideas they think they're really
good um uh ideas many people have ideas
and they're much like getting a song
stuck in your head they're stuck in your
head they're not stuck in everyone
else's head uh they might sound really
good inside your head but when you sing
them they sound really bad ideas can be
really similar to that they sound really
good in your head but they're not so
crash hot when everyone else looks at
them um there's there's roughly um um I
forget the exact number I think there's
roughly about uh 50 or 60 threads uh of
ideas posted to the python ideas um um
Forum a year and most of those ideas the
response to them is like the last person
who brought it up here's are all the
responses to it that said it's a really
bad idea go and have a look at this
thread or even uh more concretely that
idea um was proposed in the standard
python way uh uh a pep which we'll get
on to later on um and these are all the
reasons that pep didn't
proceed so whenever I have an idea uh
because I subscribe to the python ideas
mailing list um I know how to set my
expectations for any ideas that I come
up with and they are
l so I had an idea and much like a song
I couldn't get it out of my head um and
it was rattling around and matter how
many side projects I said yes to I
couldn't get rid of the idea from my
head so I decided to post it up on
python ideas and let other people shoot
it down so that I could quietly put it
to to
bet um so you know I freely admitted
that I was using a a public resource I
was using other people's time for my own
mental
health um unfortunately someone came
back and said that's kind of an
interesting idea and you should do it
and then he uh uh listed out uh a
technique for doing it
um if you don't know Barney Barney is
the this is the code owners file from uh
cpython GitHub um Barney is the owner of
the path lib Library so unfortunately I
posted an idea and the person who owns
the library said it's an interesting
idea and you should go and have a look
at it so that was that was the first
fail so um I'm not I'm not really
suggesting it is a very small idea as
you you will see uh I'm not suggesting
huge code changes um so most of the
coding examples I'll be showing will be
relatively small um for space reasons
you can assume that all of the examples
have uh this this import here just so
that we can save on Space going forward
so it's a very standard uh import of
path from path
lip um and the change that I'm
suggesting here is there are flags that
are passed to operations in path lib and
the most common ones are to do with case
sensitivity and to following sin links
and I'm suggesting that we move those
flags from the operation so from the
verb and uh we move those flags uh to
the noun uh so to the object so um in
this before case here uh pathlib has a
glob operator so we're asking path lib
to give us all the st. Tiff files and um
if I say case sensitive false um it will
give me all of the star. Tiff lowercase
files and all the star. Tiff uppercase
files um and if one of the elements of
Home Clinton R Pon Au is a Sim link um
I'm telling it to not follow those Sim
links so if there's a Sim Link in that
path don't follow it so all I'm
suggesting is we um
move those flags from the operation to
the object um there is a little bit of a
name difference between the recurse Sim
links and the follow Sim links the
concept is identical there so don't get
hung up on the name of the attribute
changing too much so that that's the
talk so if you need to go off and have
lunch you've got the the guts of the
idea um and I kind of have to describe
why we want to do this now so if your
stomach's nice and comfortable um for
lunch now I'd leave now because this is
where it starts to get a little bit
quasy I am a developer I'm therefore
right
[Music]
so this is this is the essential core
problem when people post their ideas up
to python ideas this is the the core
motivating factor that they've got um
this is the one that you've got to get
over when you post your ideas in
public um I I think long term with path
lib we're just going to start seeing
more and more flags and and I think
those flags are good uh so so having
having those flags uh to be able to
change the way that operations happen I
think is a good thing but I think it
might be a better thing if we have one
place where we put all of those flags
rather than adding Flags to lots of
different
operations and that's that's a little
bit of a taste thing uh but I also think
it means that when we add a flag we are
not uh just adding a flag to one
operation we're thinking about how that
new flag can uh alter the behavior of
the entire path lip uh
Library um and it means that if we add a
new flag we're adding it in one place so
the documentation for the path lib init
uh function would grow all of the uh
Flags but none of the other operations
would grow uh in their complexity
um the the main thing I think here is
that it's very possible to have paths as
um parts of an API where you're passing
a directory along to an API and that API
you don't want to have to give it all of
these extra flags as part of that API
because the path is really just a input
or output operator and you don't want
your non-path API to have to deal with
those flags
so this is where the work side of things
come in let's say I have a science API
that I give a directory and that
directory might have oh I don't know
hundreds of Tiff files from a camera
that I
control um at the moment my science API
would if if there were sings in that
path because it was a nested NFS Mount
from three servers away and there were
lots of Zim links in that chain at the
moment I would have to tell my science
API that doesn't really care about Sim
links that it needs to follow the Sim
links so if I get my change in uh that
op option would get passed along to the
path object the science API wouldn't
have to think about Sim links wouldn't
have to know about Sim links uh but it
would get modified uh by those flags and
in my mind that's appropriate because my
science API doesn't know what SIM links
are shouldn't have to know what SIM
links are
so it's that separation of concerns in
the
API um so as long as well as uh Barney
saying it's it's an interesting idea he
also gave me a complete recipe for had
to go about implementing a prototype so
the rest of the talk is going through
that
prototype um so I am making a class
called context path because all of these
flags are context in a certain way um in
the fullness of time if this idea was
accepted there wouldn't be a context
path it would go into path itself but
for this test run we're making a new
class called context path the really
important thing here is that in Python
312 and 313 Barnie's done a lot of work
to rejig path lib so that you can subass
from path and do so in a really useful
manner so if you want to do experiments
around path and path lib it is really
easy to do do um so much so that he's
kind of got a rot response about how to
try these things
out
so um the second implementation step is
passing along my my context Flags to the
init uh operation so my init operation
here uh uh very standard python where
I've got uh my star ARS which is all of
the parts of the path
um I've got my case sensitivity flag my
follow Sim links flag I call uh super of
in it so that's the the path in it so
that initializes path how it always has
been and then I store my Flags in the
object so very um very normal standard
basic python
there uh step three uh is Implement with
segments with segments is the Workhorse
of the path uh path lid path object uh
anytime you are making A New Path based
on uh a previous path like you're using
the slash operator to add a new
directory or file name into it it
eventually gets called down to with
segments so the um the path
implementation of withd segments is
really quite clever it grabs self asks
me what the type of self is and in path
that answer is path and then it passes
along those path segments to its own
class so it's calling in it again
essentially so my context uh path
implementation very very similar does
exactly the same thing but just passes
those flags along so I'm doing exactly
the same thing as the original path
class but I'm just passing my context
Flags
along I can't hear anyone's stomach
rumbling or anything like that so that's
good
um so that's essentially the
implementation uh like a test
implementation of my idea um in terms of
uh creating a context path passing the
flags along now it's time to use the
flags I'm not um like like the really
important thing here is I'm not talking
about changing the behavior of path lib
I'm just talking about changing the way
you can figure uh path lib so um one of
the operations that that path lib does
is a stat uh a stat uh asks for
information about a file or directory
that you're you're pointing
to so um the implementation um of stat I
can call that in my context uh
implementation of stat by calling super.
stat and when I pass so stat already has
a uh a flag on the verb to follow Sim
links or not so all I do in my
implementation is grab my uh uh
internally stored flag self. follow Sim
links so I'm not changing the operations
of anything it's just um how and where
the flags are stored and passed
along uh and similarly with glob where
we're looking at globbing start. Tiff
files before very similar um I call the
current implementation of glob but I
just pass along the case sensitive and
follow Sim link flags for my own
internal uh flag uh registry
memory so to think about um how this
would operate for testing we have a
testing directory here um we've got two
Tiff files here one's lowcase one's
uppercase so when we switch the case
sensitive flag and we do a glob
operation we'll expect to either get one
or two Tiff files there and we' got uh a
destination D at the top and we've also
got a Sim link um uh pointing to that D
at the bottom
um don't worry the context um important
parts of this slide will be presented
further on but that's I've only got four
files that I'm testing with just as a
proof of concept
test so now we look at some basic tests
um for context path so again these are
common Imports so stat is a standalone
uh python module uh it's a built-in
python module when you do a stat uh on a
file or directory you get back a whole
bunch of numbers stat essentially has a
whole bunch of constants that let you
ask um is this file thing a file is it a
Sim link is a
directory and for my tests I import
context path as
well uh so these are just Imports that
are repeated for all the tests and I've
just put them up here to save space
further
on so this is a case sensitive test so
this is the subset of the test area that
um we're interested in here so we got
the lowercase Tiff file the uppercase
Tiff
file um we create our context path we
pass along case sensitive true so we
care about the case when we do um a glob
on star. Tiff and that is a lower
lowercase do Tiff we expect that we'll
only get the lowercase image. Tiff file
back so that's our case sensitive uh
test there and very
similarly same um area with the two Tiff
files uppercase and lower case this time
we do we set the case sensitive option
to false so we don't care about case so
this time when we do a glob on star.
Tiff it will glob on both the lower case
and the uppercase you can see the assert
there we expect to get those two files
back so that's the the test so we've
tested uh the on and the off parts of
the
flag um Sim links are a little bit um
little bit harder to reason about to
think about um I'm a Unix person and I
know through many years of experience
that I get them wrong all the time um so
this is the um this is the the test for
uh following Sim links when we're doing
a stat operation the subset of the test
area that we're interested in here is
the destination dur so it's a directory
and the Sim link that points to that
directory we make a context path to our
Sim link and we say uh follow the Sim
links um and then we do the St
on that so because we're following the
Sim links when we do the stat we won't
be doing the stat on the Sim link we'll
be doing it on the destination directory
so when we ask the stat results is it a
Sim link it'll be false because we're
following the Sim links so we'll
actually hit the directory and when we
ask the the state results is it a
directory because we followed the Sim
links and we've hit the directory it
will
be and similarly where we're in a mode
where we don't want to follow the Sim
links where we want to Halt on the Sim
links um it's kind of
Switched so we pass along follow Sim
links to be false um which means that
when we're doing the stat um when we're
uh realizing that path it'll stop at the
Sim link so when we check the state
result we can see that it is a Sim link
and that it is not a d because it hasn't
followed that Sim link through um
and I don't think I'll tell you how many
times I had to do this test until I got
it
right um so I've got the idea I've got a
very basic implementation I've got a
conference talk out of it as well um so
what's the next thing to do now that
I've got a really basic implementation
of the idea I need to get back to Barney
um show him the code see what he thinks
um and and that's kind of like the
code's the really easy part like the the
point of this talk is to to show that um
you know software a lot of the time is
really easy and it's dealing with the
people and the change management that's
really hard so the next uh so assuming
that that Barney likes the code and
wants to go ahead now we have to think
about how we're going to incorporate
this change into the py python
ecosystem um the structured way of doing
this uh in the python Community is uh
with a python enhancement uh proposal or
pep um it's a structured document um
which talks about all of the technical
challenges of the idea uh it explicitly
brings out um how we're going to teach
this change because uh at the roots of
python uh it was designed to be a
language that's easy to teach so if
you're making a change to the language
you have to um outline how you're going
to teach this change uh which I think is
uh very useful thing um but there's also
uh thinking about backwards
compatibility because it's a change so
it's it's we're going to um you know if
this gets up it's going to change how
people use path lib and we have to think
about all of the code that's out there
that uses these flags on the verb and
how are we're going to migrate to having
the flags on the noun do we have the
flags in both places do we deprecate
them do we think about eventually
removing them do we use the Tri-State
nature of these flags where by default
they're none um you can actively set
them to true or false if one of the
flags is none and one of the other flags
is uh true or false does that override
things or does the verb always override
the noun or does the noun always
override the verb there's lots of
options and fortunately um I won't be in
charge of making that
decision um so if we go down this track
um it's it's it's one of these changes
where it sort of uh opens the door just
a crack and it can sort of see through
all the things that you would do with it
so um the buzzing in the brain this is
like the flapping in the brain versus
the buzzing in the brain so Windows has
things that behave like Sim links but
regular users don't have the Privileges
to use them they do have things called
Junctions which are kind of like Sim
links but they're only um if I'm getting
if I'm remembering
correctly uh Junctions are Sim links
that you can only point to directories
and you can only Point directories on
the same file system there's already
code inside of path lib to check if
something's a junction and part of part
of my um impetus for doing this is
instead of adding a follow Junction flag
to all of the calls that it makes sense
we just add that follow Junction flag
just in the init function and we call it
done um there's lots of fun things with
lots of different Windows operating
systems where the path can be encoded uh
with different um Unicode mappings and
different you know pre- Unicode mappings
so that you can have uh nicely spelled
uh human names in file names and
directories um at the moment python sort
of takes a punt on that and says we
always do this which is totally fine
totally legit way of doing things but if
we had options on path we could set
those options and we could get nicely
printed names out of paths on Windows
that our users like to read without
having to do any
work um one of the things with Sim links
is that it's very easy to do a really
long chain of sim links ping-ponging all
around your file
system uh you can get into Loops quite
easily if you uh if you forget Which Way
Around The Operators to creating Sim
link uh go which you know I never did
that anymore um so it's very possible to
request following Sim links attempt to
open up a file on a Sim link and just go
round and round in a loop now if we let
that happen um that would be a nice way
of doing a denial of service attack on
on some code so the security people have
have added limits to uh the underlying
file system code that says um if you're
following Sim links and you hit 100 Sim
links that's it we fail and you can't
open that file wouldn't it be nice if at
the python layer we could do something
similar and say you can only follow 10
Sim Links at the at the at the python
layer so that would be an interesting
way of doing security stuff at the path
liway
um there's lots of things to think about
from the backwards compatibility point
of view and and I've already mentioned
this like what would take prec what
would take precedence would the the noun
or the verb Flags take precedence um uh
do we use the Tri-State nature of these
flags to uh make things nice um would we
actually deprecate the arguments uh on
the verbs or would we just leave them
there as an optional extra for those
times when you do actually want to
overwrite
um if you're writing a program or an
application and you've got lots of paths
coming in and out maybe for that
application or Library you want to
always have the same flags for all of
the paths that you create so maybe we
provide a path Factory that sets all of
those flags maybe that's something to
think
about um so that's that's ESS the talk
um I like using old technology um so uh
this slid uh deck um was made with emac
and org mode and uh latch and TI Zed um
the new thing for me is I've actually
paid for a type face which is a sentence
that I never thought I would say but the
uh the monotype face there is uh
courtesy of berley mono and with that I
have time for questions
[Applause]
and who's up for the first question
yeah
no come on you knew i' have something um
so you you already answered my first
question which was the default flag
model um but the other thing that was
giving me Vibes of was decimal contexts
and whether you'd considered the decimal
context
type
approach yeah so
um so similarly so programming is you
know not all the time but a lot of the
time it's
about uh making assumptions versus
knowing the assumptions that you've made
so we've done that in a lot of
contexts and one of these context is
with decimal numbers uh so there's a
whole bunch of flags that you can set
around dealing with python decimal
numbers and this is not a python
specific thing this is the the standard
uh for decimal numbers that the python
mation bases itself on but in order to
make life easy we hid a lot of those
details inside the module um now I might
be getting this wrong so so shout me
down I am um it's it's difficult because
those contexts are built into the module
it's a little bit ugly to change those
contexts for two different decimal
numbers and get them to talk together
nicely so I like I I don't know like in
my mind the the decimal context way of
doing things is not great and and maybe
shouldn't be a recipe for how we do this
in the future but you know that's uh in
an attempt to make decimal numbers as
easy and as nice to deal with uh uh
integer literals we hid some of the
context it's still available and you can
still get it but it's not as nice as it
could be it I I I don't know if you
agree with that or not
but yes yes and and fortunately I don't
have to make that decision think we got
one right at the back oh
yep I'm glad that the video operator
didn't invent a cute way of
communicating to me with the video
recorder I don't have a mic they didn't
give me one they don't trust me with it
uh I wrote my my question down because I
forgot it immediately uh I know you told
us to disregard the uh rename of rer sim
links to follow links yes is that rename
partially so that it is easier to um
introduce that number limit or is it
also just to make it clearer that it is
stopping any any Sim links rather than
like yeah yeah yeah so it's it's very
possible that we would need to think
well we would need to think about the uh
naming of these flags if we change the
location of the flags we need to think
very carefully about the naming of them
uh so it is entirely possible that
without me thinking of a explicit case
of it that we might have two sink flags
that mean slightly different things for
different operators that's entirely
possible um and that option is in the
back of my brain I can't think of the
case but it's entirely it's entirely
reasonable and possible that that will
happen hi you mentioned a few potential
future options on paths like Junctions
and windows en codings is that something
you start off the conversation with with
the code owner of hey I've got a bunch
of ideas and this is a prerequisite or
do you just take it like one step at a
time and see how far you get um I I
would I would I would give B probably
like this talk video basically so I'd
give him the code and and the talk and
and I'd sort of suggest you know these
are all things that it might improve
things um and you know uh bnie will have
a much better idea of what the right way
to go is for these things um you know I
like none of this stuff is
really
wow all of these things are possible
with python now it's just just that you
got to do explicit code handling in your
paths and around your path handling um
and I think python could be nicer if
some of that stuff was automated using
the
contexts thank you very much Clinton um
I present to you the
mug thank