foreign
hi everyone thank you for being here
this is the last talk before we start uh
well before we all enter Chris and
oigabauer's sardom of lightning talks
there there is afternoon tea first so
stock up
um
in the meantime Amanda Hogan is not only
a teacher she's also a teacher teacher
so she teaches teachers how to teach her
in Computing
yes that's what yes Esther that is what
yep good great
um if you've been on the Discord she's
also the person that's putting up all of
those really cool visual diagrams of
talks they're awesome go and find them
yes
yes I'm just gonna let it do the thing
yeah thanks very much Maya okay hello
everybody
um I know that you thought this would be
the quiet talk when nobody picks on you
and you can just hide at the back of the
theater but that's not what this clock
is going to be because I'm really
nervous and I need to use audience
participation so there's going to be
some audience participation in here
somewhere are we singing cilia am I the
right person to find out we have no idea
let's find out okay so why am I doing
this talk
thank you
I am the luckiest woman in the world uh
every morning I am woken up with a
coffee delivered to the side of my bed
I'm not kidding this is a photo of my
side of my bed and I I love my husband
and he's never allowed to go anywhere
because I need this
um so every morning we sit down and we
drink our coffee and we play hurdle
hurdle is a game that came about when
when Josh Wardle did the whole world
thing and there were a million clones
made and hurdle was a thing where you've
got 17 seconds of audio to identify a
song and then once herder was was made
then there were some spin-offs that were
also made and they were decade based
hurdle was bought by Spotify and then
immediately killed and so we stick with
hurdle 80s 90s and 2000s
and I thought
um there's an interesting thing it's an
like uh it's interesting that my husband
and I uh we identify songs differently I
work with lyrics he works with music and
bands and so it's in my mind okay
foreign
we also are a family of Music listeners
we listen to music while we make dinner
we all have headphones we listen to
music and our commutes
um we all have overlapping interest but
also diverse interests my parents were
from the age that you know the big pop
decades the 60s and 70s I was Peak
listening in the 80s and 90s and my
children are Peak listening in the
post-2000s world so there's a friendly
competitiveness between the generations
which generation is the best generation
of music
interesting question I have rattling
around in my head
I used to do data stuff for a living a
long time ago I don't want to
misrepresent myself here I was not a
data Legend but I was a dashboard
builder at Microsoft a long time ago it
was a ton of a time ago now but and I
thought
that I could do with updating my data
skills
I have heard about libraries like pandas
and I wanted to have a play but when do
you find time to play you never find
time to play so I
like the two things started to mesh
together could I use this as an example
of how to learn something new while
trying to find out something interesting
you know something that I'm interested
in can I finally tell my children that
their generation of pop music is worse
than my generation of pop music okay
so confident confident that you can't
get deeper than the lyrics of Nirvana
and Mariah Carey I set out so I had to
have a whole lot of procedure involved I
had to make some decisions and there are
some decisions that some of you will
disagree with but I went with those
decisions anyway and you're welcome to
go ahead and do your own procedure but
we'll we'll work through my procedure at
the time being so I needed to find some
I needed to make some decisions the
first decision was what music what music
do I sing in the car what's music do my
kids dance around the house to what
music I went with pop music because it's
the thing we sing most often and then I
wanted to find Trends I wanted to find
some music to analyze and so I went to
billboard.com billboard is a magazine
that's been around since the 1800s they
are the ones that publish the number one
songs
um of of a year they have the hot one
they have hot 100s
um it is an American Magazine so there
is a little bit of
um consternation there because it's how
Americans sing pop music and ours is
slightly different but but Aria charts
aren't comprehensive enough and so I
needed to pick something that had a good
list of data
okay so I've gone with I've gone with um
pop and I'm okay with American Music so
I went to billboard thinking that I
could find all of the pop songs that
they'd ever hit uh that had ever hit
number one and see if I could grab it
from their website and of course I
couldn't navigate it at all it's just
very difficult to find anything so
handily and I use this term with love
there are some nerds who put some stuff
on the internet
so Wikipedia has a full list of all of
the billboard number one singles of the
70s 58 to 69 the 80s and 90s 2000s 2010s
I did not go into the 2020s
because it's just not a full decade so
I think it's great that there's also
consistency for the most part these tape
these pages are laid out very well and I
didn't want to go and copy and paste all
of that stuff manually so I went and
discovered that pandas has a fantastic
functionality that allows me to put a
URL in and have it read the HTML of the
page for me and then I can say go and
have a look at table two in that page
and grab all of that data as a data
frame which is like saving my life here
okay so the downside of Wikipedia is
that those people don't call each other
very often and so
we had the 20 the 2000s be inconsistent
where instead of hash I have no to main
number instead of artists I have artists
with an a
and a link instead of single I have
single with an A and A Link and instead
of weeks at number one I have weeks at
no dot one like it's just annoying but
uh then there's also uh instead of reach
number one date they had issue date and
there's nothing I could do with that I
just had to assume that they were the
same date for the time being
um I did briefly think that I could go
to the Wikipedia
list of Billboard Hot 100 number one
singles of the 2000s and update it so it
matched the list of Billboard Hot 100
number one singles of all the other
decades but I have actually been in open
source flame Wars before and they're not
very fun and I decided that I was not
courageous enough
okay so but thank you Wikipedia for
giving me that data and then how do you
get the lyrics of all the number one
singles since 1958 well you find an API
and there is a library called lyrics
genius thank you very much lyrics genius
for existing there's a website called
genius.com and lyrics genius will allow
you to use Python to poll
genius.com and get about 1100 songs
worth of lyrics
um it's a bit of a mess to be honest uh
I made it made my life easier because I
didn't have to go and download 1100
songs worth of lyrics by myself but it
was such a mess that I remembered why I
had exited data analysis in the first
place in the cleaning data is a bit of a
pain
okay so this is what the python looks
like you you have to get yourself a
secret key but once you've got your
secret key then you can search for a
song using the single and the artist
that which is the title and the artist
and it will either return or not return
there were some problems with this and I
will talk about them next okay so my
problems
the first problem is that data
codification is really hard and we don't
all call each other or hang out on the
internet and agree as to how we're going
to codify things so
this is from 1958 the Chipmunk song also
known as Christmas don't be late in
Brackets by let's just check this page
this is the literal album cover David
Seville and the Chipmunks
however
in wikipedia.com it is listed as the
Chipmunk song in Brackets Christmas
don't be late by Alvin and the Chipmunks
feature with sorry with David Seville
and ingenious.com it's listed as a
chipmunk song Christmas don't be late by
Alvin and the Chipmunks so
that doesn't match up this happened to
me a lot this is the kind of thing
that I saw a lot in my Jupiter notebook
while I was trying to run my genius.com
um lyric search so there are a couple of
problems here the first one was matching
of data so slight differences in how
artists are named slight differences in
how songs are titled a lot of problems
with feet or featuring and so matching
up was a problem there was also some
connectivity issues apparently lyrics
genius isn't prepared for people to just
pull it with 1100 songs worth of trying
to get the lyrics and so I would have to
run this and I'd keep track of
all of the ones that had failed and run
them again and again until I got down to
a a core list that were actual problems
that I had to go and do manually
um but the rest of them were just it had
dropped out at the wrong time it was it
was a pain
okay so then my other problem was even
with songs that I'd returned the lyrics
for I could get something like this
this is what you get when you search for
Gangster's Paradise by Coolio you get
a review of Gangster's Paradise by
Coolio by dude called the rap critic and
I'm I'm a teacher in my normal life and
so I had to notice I can't stop myself
from noticing that we're going to count
the full stops in this review they they
go to there and then oh my goodness
uh it's a bit of a stream of
Consciousness thing but also like just
punctuate please uh okay so I had some
issues and then I had some decisions I
had to make and some of you will
disagree with these decisions but they
were the decisions that I made and
you're encouraged to go and do your own
projects and fix mine sure so the first
problem is I love Unicode it exists for
a reason it's a very good thing and also
it's very
painful and so I needed words to match
with other words and things like
smart inverted commas
were causing me pain also accents were
causing me pain and so were Spanish
upside down exclamation marks and I know
these things all have a place and also I
stripped them out with abandon
so everything has gone back to ASCII it
was where the world started so the other
thing were that there were some songs
that really didn't fit into my data set
so here's where some of the audience
participation kicks in the first one I'm
going to see if anybody in the audience
knows this song by the image
no okay
okay
so we had Obama it and non-english
language songs I had to strip out
because I was going to be analyzing the
language from an English perspective and
it just doesn't make any sense for me to
apply those algorithms to non-english
songs so there are a couple of these
let's have a look at some others
anyone
[Music]
okay so Macarena and then this one
should be much more likely to to be
identified
there we go
there we go despacito
um and there were a couple of songs that
were number ones but were instrumental
and so they're not used to me at all so
I had to strip those out uh so can we
see if we can identify our instrumental
number ones it's actually really unusual
for a number uh for a song to get to
number one as an instrumental except
especially recently it was much more
common in the in the earlier uh decades
um so this is one anybody know what this
might represent
Rocky yeah
this song
so not Eye of the Tiger but the actual
theme
has a name I forgot to write it down uh
and this one uh it was a meme early in
the 2000s
uh my children said Mom that has lyrics
but four lyrics don't count as lyrics
[Music]
okay we have the Harlem Shake and this
one playing to my audience
yes here we go
Cantina Band from Star Wars so those
were all they're about
um uh 11 songs that are non-english
language and there are about eight that
are instrumentals
um and so they were gone killed Dead uh
okay so I needed a process in order to
analyze what was left I have a lot of
data and I need to work out how to how
to determine what silly is what good is
how do I tell my children that my
decades were the better and so I
invented the Jabberwocky procedure
named because I was going to take a
two-prong attack one was reading ease
reading ease is a thing that teachers
use to classify books
um and it is an algorithm that shows you
the kind of reading age of the text and
this is a book for children and and
nonsense because Jabberwocky is a poem
that is predominantly nonsense uh so
those are the two things I'm going to
look at
the first one was nonsense so over the
decades these are the number of song a
number of nonsense words in the Whole
Decade in all of the lyrics of number
ones
um I had to chop out 1950s because the
number was so low but the number's so
low because 1958 and 1959 don't qualify
as a decade uh so from the 60s then it
goes up a bit for the 70s back down for
the 80s and 90s and then smashes through
the roof in the 2000s and then back down
a little bit for 2010.
and uh if you want to see what that
looks like in the 2000s we have this so
this is across all the lyrics we have
some swear words which I have censored
and we also have a real dislike for the
letter G
every word that ended in ing got turned
into a word that ended in i n and that's
kind of how we got to nonsense so this
is running
all of the lyrics against words.text and
pulling out proper nouns so uh names
names were pulled out because they're
not nonsense but these are all the words
that don't fit either into words dot
text as a dictionary or uh is not a
proper noun or identifiable as a proper
noun and so that's what we had left and
there are tons in 2000s
so now oh I just have to make a side
note that that my husband claims that
word clouds are not data analysis and he
works in data analysis so he's allowed
to mock me for that but also across a
Whole Decade I claim that the lyrics
like grouping together into sizes does
give you some information
so now I have I'm down to
[Music]
um
1069 songs
um and that's a lot of lyrics and I have
tidied them all up and now might be a
bit on the thin side
1069 I like I went with number ones but
maybe at this stage I'm realizing maybe
I should have gone with top tens or top
100s but actually it's taken me a really
long time to get to this point so I'm
throwing it open as an open question
please extend my research I'm not even
sure it qualifies as research but it was
fun and so I'm going to take those 1069
songs and I'm going to apply the reading
ease calculation to them
all right so what is reading ease
uh well there was a dude called flesh
and you might have heard of the flesh
Kincaid formula uh the flesh Kincaid
formula is just taking the flesh formula
and then turning it into like grade
levels for books so if you are reading a
grade level four book then it will be
graded by the flesh Kincaid formula and
so this is the flesh formula and he's
basically read a whole lot of material
and then tried to to pitch it between
100 as being the readable and zero as
being unreadable like very hard and so
that's where those literals come from
but
you can see that the relationship with
my is my thing going to reach yes the
relationship is total words over total
sentences so words per sentence and over
here total no my thing's not
it's there somewhere there we go total
syllables per total words so syllables
per word so these are the two things
that impact the readability of any piece
of text the number of words per sentence
and the number of syllables per word and
you can see from this formula that the
number of syllables per word has a much
bigger impact than the number of words
per sentence but they both have an
impact okay
and now I'm going to show you what that
looks like across
across the different
reading ease from what flesh said was
100 is fifth graders
and 0 to 10 is professional or
postgraduate level okay so I have babies
up here and post grad down here you have
to remember that because it does kind of
feel a bit counter-intuitive that 100 is
the the score on reading ease but it's
actually for the lowest level reader and
zero is for the highest level reader
okay just remember that and as we as we
look at this graph syllables per word
are going up and words per sentence are
going up
okay so I've decided to look at
sentences first
but what is a sentence
Collins Dictionary says it's a group of
words usually containing a verb that
expresses a thought in the form of a
statement question instruction or
exclamation and starts with a capital
letter when written okay
but that's a PhD
and I don't have a PhD and I'm not
getting a PhD between July and this
presentation so I tried some natural
language processing libraries to see if
I could get them to determine what a
sentence was in my lyrics and they ran
really really slowly and still didn't do
a good job
so then I thought I will use the primary
school definition of a sentence ends
with a full stop or exclamation mark or
a question mark and turns out that
lyrics are not well punctuated so that's
really difficult so I came up with my
estimation for a sentence
every two lines in a song is
approximately a sentence and I figured
that since I'm using it and applying it
consistently that that'll do and also
remember that the scent the words per
sentence impact is much lower than the
syllables per word so let's focus on the
syllables
all right so
in school when we teach students
syllables I'm a high school teacher I
don't teach syllables but they the
primary school teachers tend to teach it
with a clapping method so we have
syllable what we're going to do is we're
going to clap with the emphasis so sill
La book can we all clap together
syllable right and that's what that
should look like okay
so it's really hard to teach computers
to clap
and I found it really I I found that I
could not do that so I went and found
some algorithms and this dude Michael
Holtz sure has written an excellent blog
post and the summary is you follow this
algorithm you Loop through the letters
of the word if you reach a vowel a e i o
u or Y
and it's not next to another vowel add a
syllable to the count if the word ends
with e then subtract a syllable from the
count and if the word ends with l e then
put the syllable back so this looks like
this we get to the Y we add one we get
to the a we add one we get to the E we
add one we take one off because it ends
in an e we add one back on because it
ends in an l e all right so it's got to
the three that we identified with the
Clapping method so the algorithm seems
okay and I did run it through a bunch of
different words
okay so then we get to my code
I am getting sentences I am getting
words I'm getting syllables and then I'm
I'm applying that flesh formula which
has the Wikipedia link in there to the
flash formula page
um and then I end up with a number
once I've applied this formula across
all of my pandas data frame I'm learning
a lot about lambdas more than I ever
thought I needed to know and I get
this graph
and you can see here that there are two
things that jump out as kind of weird
something's going on in 1958
and something's going on in 1961. so
let's have a look at 1958 so this is per
year
summarized average per year and so I'm
going to look at 1958 these are all the
songs that were listed in 1958 as number
one and there's only a few of them
because it actually didn't start till
August 1958 but you can see they're all
way up there in the hundreds in the 100
area so they are actually all have a
very like a very low reading age a high
reading ease so that kind of what that
checks out but what's going on in 1961
this is going on in 1961. we have an
outlier
so I was interested what song in 1961
could possibly
be postgraduate level
postgraduate levels reading in 1961
and then I found out
[Music]
[Applause]
[Music]
all right so The Lion Sleeps Tonight by
the tokens may not seem to us that it's
a very complex song but turns out it has
an awful lot of syllables per word
because
um we it is very long and a whim away
has four syllables so it becomes our our
most complex song in number one history
okay so what's my outcome
well if we look at the graph that I
snuck past a little earlier you'll see
that I have a very clear trend line from
1960 to 2020 that shows that the reading
ease is getting lower which means the
reading difficulty is getting harder but
at the same time I have the nonsense
following the Jabberwocky procedure
spikes at 2000 and 2010
so clearly if you're looking at the the
overlap of those two graphs then the
1990s which happens to be my decade is
the correct and only best decade for
singing pop music yeah
so
there are questions still unanswered
about my process but I do want to point
out it's a little bit of a joke but
there's actually much more keeping us
together than keeping us apart if you
look at all of those word clouds for all
of the lyrics in all of the number one
songs in the 60s 70s 80s 90s 2000s and
2010s you see that love no baby these
are words that we're singing a lot so I
think that there's more that keeps us
together than keeps us apart
so this is my
GitHub the whole point of this talk was
that I had a silly idea and I went out
and I did a whole bunch of investigation
and I went through some pain and then I
came up with a talk and if I a high
school teacher from Sydney can do that
then so can all of you
that's it
[Applause]
[Music]
can you have a mug thank you
um any questions for Amanda we've got
time for one or two
thank you
that was great
you mean the 90s being great yeah
obviously
um as our resident expert on what songs
people should sing What song should
people sing at the karaoke birds of a
feather in terms of in Discord
I think we should start with something
from the 90s clearly so maybe we're
doing uh Blaze of Glory by Bon Jovi
I think that's that's my expert opinion
I don't feel stitched up at all
I think we'll end questions there
thank you everyone
I think