foreign
[Music]
welcome everybody now for this afternoon
I'm on the wrong page
whoops
I'm thoroughly on the wrong page I
apologize
I do apologize but I won't waste any
more of your time
so I would like everybody
to make Christopher feel welcome thank
you
first you have New South Wales today I'm
going to be talking about was that was
that not on before
okay today I'm going to be talking about
ceph and drbd two distributed storage
systems I'm going to be comparing them
talking about what I liked and didn't
like what their strengths and weaknesses
are
so
at trustworthy systems we provide
networked home directories to our
members you can log into any machine on
the network and you'll see the same home
directory with all the changes you've
made to it
the way we accomplish that is using a
combination of something called drbd and
NFS
so the file system that those home
directories are in has to be stored
somewhere
drbd handles that task and it's better
if that file system can be stored in
multiple places at once
yeah
we have drbd does the work of storing
the file system on multiple servers at a
time and keeping it consistent across
those different servers and then we use
NFS the network file system protocol to
export those home directories over the
network so they can be mounted on
whatever machine someone's using
so let's do some more detail on that
the drbd stands for distributed
replicated block device
uh block device is what drbd is designed
to work on it deals in Block devices
replicated it means that the contents of
that block device are going to be copied
identically to several places
distributed means that that copying is
happening to several machines on a
network so multiple machines each have a
block device those block devices on
separate machines have the same content
and drbd handles making sure that that
stays the same then we Mount that file
system in one of the machines and use
NFS to export it
so this setup works pretty well but it
does have a couple of weaknesses
one of them is that NFS and drbd don't
know about each other
drbd has no idea that the block device
it's replicating is actually being
served over the network onto other
machines higher up NFS doesn't know that
this file system it's exporting is
actually replicated across multiple
physical servers
that means if you ever want to change
the configuration here there's more work
you've got to do there are more places
you have to make the changes
also because they don't know about each
other
um
a dlbd resource can only be in primary
mode on one or two hosts at a time
primary mode is the mode where you're
allowed to read and write the data on
that block device
so the easiest way to deal with that is
usually put the drbd resource in primary
mode on only one of the servers and
leave it in secondary and the others so
it can't be read from or written to on
the other servers and then run your NFS
demon that's doing the exports on the
same server where the resource is
primary
if that server goes down however that
means that the RBD is not going to
automatically put the resource up into
primary mode on another server you're
not going to get an NFS theme and
automatically spinning up on another
server and even if they did do that
there's no guarantee that their efforts
to recover would work together because
again they don't know about each other
there are tools you can use to try to
automate this thing and get them to work
together but that's also adding an extra
layer of complexity on top
another issue with this setup is the RBD
doesn't have the greatest conflict
resolution support
if two servers end up concluding that
the contents of the block device should
be different if they disagree
often you end up having to just manually
issue the command to re-synchronize the
data on the device and that can take
hours to happen
so we've been thinking about
alternatives to how we store our home
directories and an option that we landed
on is called ceph
ceph is a distributed storage system
which is designed to be able to scale to
really large deployments
it supports three different ways of
mounting file systems over the network
and it dynamically redistributes where
the data restored on different hosts
it's really intended to be an all-in-one
solution and I like that I think it's
it's nice to have a system where the
components are designed to know about
each other and work together
but it's also a bit more complex than
drbd is so to determine whether or not
this would actually be a good choice to
replace our existing nfsdrbd setup I
created test deployments of each of
these two structures so that I could
compare them
first point of comparison we come to how
easy are these things to set up from
scratch how much work does it take to
get the RBD and NFS going how much work
does it take to get stuff going
the RBD has a big Advantage here drbd is
a fairly simple technology there's not
that many moving parts to it
drbd consists of two main components
there is some user space software that
you use to configure the resource and
give it instructions and there is a
kernel module which does most of the
work of replicating the data and
communicating with other servers
the steps for getting the NFS drbd
deployment going will look something
like this
first you need to download the user
space configuration software and you
need to load the kernel module
you then have to write a configuration
file to describe what your resource is
going to look like and you started
running
you need to decide where you're going to
have the resource in primary mode
Mount and then you can create a file
system on the resource and mount the
thing
and once it's mounted on one of your
servers you can start up an NFS demon
and tell NFS here's the file system
export this thing over the network
when I was setting this up the NFS part
was pretty straightforward I didn't come
into any problems there
setting up a drbd resource had a couple
more steps to it it was mostly not much
of a problem drbd's parent company
limbit provides a user manual which
tells you just about everything you need
to know to get the RBD going there was
one substantial problem I ran into with
drbd setup and that was at the kernel
module and the user space software can
be on different versions
I wanted to be using the RBD Version 9
because that allows replication to three
or more hosts at a time
in the RBD version 8 and before can only
replicate Between Two Hosts
now the latest Damian packages for the
user space software were on the RBD
Version 9 but the version 9 kernel
module hasn't been integrated into the
Upstream Linux kernel yet
so for a while I was trying to send
commands to a kernel module and they
didn't understand and the format of the
configuration file changed a bit with
the versions as well so it had no idea
what I was telling it to do
fortunately that wasn't too hard to
solve Lin bit also provides source code
for the kernel module so I could just
download compile the version 9 kernel
module and load that and from that point
drbd pretty much worked
so on to Seth
Seth has a lot more moving Parts than
drbd does a ceph cluster is made up of
many different demons all working
together and doing different tasks these
are the four most prominent kinds of
Seth demons
there are the object storage demons or
osds each OSD is responsible for
managing one unit of storage such as one
hard disk one drive there are monitors
their job is to make sure that all the
other demons are performing their tasks
correctly and to track where the data is
manager demons watch the cluster's
health and performance and metadata
servers store metadata for the file
systems that your ceph cluster is
managing to make them easier and faster
to access
but these four are just the four most
prominent ones there are a bunch of
other demons as well in a ceph cluster
Seth has so many moving parts to it that
it comes with its own Configuration
utility called SEF ADM to help you set
up the cluster
if you're using SEF ADM then making
yourself cluster look something like
this
one you tell safe ADM to bootstrap the
cluster which means it's going to create
a single monitor a single manager and
some configuration and authentication
information
that bootstrapping happens on one
machine you then have to connect other
hosts up to it so that the cluster can
start spreading
take all the devices you want to
actually use for storing things and
create an OSD on top of each of them
then you tell these F cluster to create
a file system and it'll use those osds
to get the storage it needs to put that
file system on
finally a client needs to be
authenticated to the ceph cluster before
it's allowed to access the cluster and
mount the file systems that are in it
now Seth has documentation and quite a
lot of it there is
a documentation website
you can also look at Man pages on the
ceph commands and you can often run a
safe command and stick dot h on the end
and it'll show you what are the further
ways you can extend this command
one pretty substantial challenge I've
run into with understanding Seth is
it doesn't feel like any of these three
sources are really comprehensive
I found things in each of them that
don't seem to appear in the others
you kind of have to check all three to
be sure that you've seen what your
options are and even then you can come
across a situation where you've got say
two different commands that seem like
they're supposed to do the same thing
but in fact they might not
self documentation is good at giving a
high level overview of how a cluster
works the four key types of ceftemons I
listed earlier that's an example of
information that is very easy to get out
of seps documentation
there are also a lot of instances in the
documentation where you are told in
order to perform this action run this
command
low level instructions
but I find the documentation kind of
struggles with giving mid-level
explanations of what are my options why
would I want to do this versus the other
what does this really mean
for instance
step three of setting up a ceph cluster
was create osds these are two different
commands both of these commands appear
on sef's documentation website on
different pages both of these commands
will create an OSD for you in your
cluster on top of a particular logical
volume
something that the documentation does
not make clear is that these two
commands are different in a significant
way the First Command will create that
OSD under the management of ceph ADM the
tool which is explicitly designed to
help you with deploying all the demons
in SF cluster
the second command will not create the
OSD under the management of ceph ADM
and we'll be able to detect that an OSD
is there but won't be able to do
anything to it until you explicitly
adopt it
this is the sort of thing that you can
run into when you're trying to
understand what's going on inside Seth's
documentation
on the Simplicity of setup drbd is
definitely further ahead than Seth
that's to be expected though drbd is a
simpler tool it does not do as much it
does not support as many options it's
not surprising that it's less work to
set the thing up
so what about features what do these
tools support how easy are they to use
what options can they give us
well I just mentioned drbd is a pretty
simple technology
all it's really doing is replicating the
data on this block device
the RBD doesn't have any higher level
options on top of that it can't for
instance put a file system on the Block
device it can't do any kind of cluster
awareness for you
you have to create your own file system
on top of the block device
once you have drbd replicating the thing
you can from the command line view the
status of your drbd resource you can see
where is it primary where is it
secondary where is it up to date this
was taken while a drbd resource was
synchronizing to the two other servers
but that's about all the information you
can get there's just not that much high
level stuff going on in the operations
drbd is performing
in contrast sef's feature set is
probably my favorite part of it
I mentioned earlier that safe
dynamically redistributes data it's not
like drvd which is essentially copying
the exact contents of this block device
onto another block device on another
server several actually decide this OSD
is the size this OSD is this size I'm
going to give this one more data because
it can hold more it will move where the
data is replicated around based on what
the storage options are so it's really
quite easy to add and remove osds to
your cluster as you get more storage or
as you lose storage options
if you have bootstrapped yourself
cluster using ceph ADM then it will come
with a graphical dashboard that you can
reach over the web
I mentioned that ceph managers watch the
cluster's health and performance you can
see a lot of that information here
you can also perform a lot of the setup
tasks for a Steph cluster from this
dashboard you can add hosts to your
cluster from here you can deploy
services from here you are supposed to
be able to create osds from here too I
haven't managed it because of an issue
to do with how ceph detects whether a
device is empty or not
one of my favorite things about Seth
from a system administrator perspective
is how automatic it is
I've had times when I have taken down
one of the hosts in a ceph cluster
and then I've brought the host back up
and rejoined it and initially the
cluster status up in the top left will
say health warning bad data
but in a few minutes without me having
to issue any commands except we'll have
cleaned up the problem and the cluster
will be back to full health again
ceph is intentionally designed to be
self-healing
so that it will find and try to fix
problems for you
when that same kind of thing happens in
drbd if I take a host down and bring it
back up and try to reconnect it into the
cluster I will often need to tell it to
re-synchronize the data on the drive and
that can take hours to happen
so
so far I've been talking about
qualitative things
how do these tools feel what sort of
options do I have on them let's get some
numbers involved how do they perform
I mentioned there are three different
ways you can mount a ceph file system
over the network there is a kernel
driver you can use for mounting set file
systems you can mount them in user space
with fuse
and you can export a set file system
over NFS
similar to what we were doing with drbd
although it's all handled inside SEF
so when we include the RBD that's four
different ways we can mount a file
system to run tests on
for testing performance I add three main
strategies the first was a program
called postmark the way postmark works
is it creates a couple of hundred files
and then it performs creates and deletes
and reads on those files it completes a
set total workload and then finishes the
faster you can get that workload done
the better your file system is
performing
NFS bench Forks itself to create several
processes each process creates one file
and that process is the only one that's
going to work on that file
the process each process will then open
its file write data into the file close
the file repeat that many times over
again there is a set total workload the
sooner you can get through that workload
the better your file system is
performing
and the last thing I did I took the NFS
batch program and I modified it so that
rather than completing a set workload it
would just run continuously for about 10
minutes
and then I mounted a file system on
several different clients at a time and
ran that long-running NFS bench program
and all of them at once and I watched
CPU usage on the server and client
machines while that was happening
actually now I'm going to stay on that
slide for now
so
if you don't really understand a system
well there are a lot of ways you can end
up measuring the wrong thing
especially with something like this
distributed file systems
there are a lot of components that go
into the end results you get when you
try to test their performance
you've got to think about what are the
capabilities of the clients what kind of
network connection do the clients have
to the service what are the capabilities
of the servers how are the servers
talking to each other what sort of
algorithms are they using to decide
where the data ends up
if you don't really know what you're
doing it's easy to end up measuring the
performance of some bizarre
configuration that you shouldn't be
using in production anyway
that happened to me several times over
first I originally created myself osds
so that they were storing all of their
data on hard disks
well the self documentation recommends
that if you can you should put the right
ahead log for the OSD on something
faster so you don't have to wait for the
hard for the slower hard disk as often
when I made that change I then had a
look at NFS bench I noticed NFS bench
was using the sync system call
I tried running it using the fsync
system call instead both of those system
calls are basically designed to commit
the changes to the underlying file
system or storage
and I saw a huge difference in
performance of
that are optimized for everything
nope turns out the difference is fsync
commits one file at a time sync commits
the entire file system every time you
call it it was the difference between
each process committing its own file and
each process committing every file
it Steph was just doing quadratically
less work because I changed which system
call I was using
and then when I was running the CPU
status the CPU usage
experiments
there were some cases where I found that
the clients were spending almost 100 of
their time waiting for Io to happen but
the servers were virtually completely
idle
somehow the clients were hitting huge
bottleneck and the servers had nothing
to do
it turns out that was because the switch
that was connecting the clients to the
servers just couldn't keep up with all
the traffic the problem was nothing to
do with safe or drbd the problem was to
do with the network wasn't fast enough
so
having been through those experiences
here are the results I ended up with on
my last run of these tests
so this is results for postmark on the
y-axis we've got how long the experiment
took to run remember a faster completion
is better
um on the bottom on the horizontal we've
got how big the experiments were
the yellow bar is a ceph file system
mounted using NFS green is sapphire
system mounted using the kernel driver
blue is set mounted using fuse and red
is drbd mounted over NFS I'm going to be
maintaining that color scheme for the
next couple slides
so on the postmark we're seeing the ceph
kernel is doing quite well for itself uh
drbd and ceph over NFS are actually
doing extremely similarly
I was hoping coming into this that Seth
would just visibly outperform the RBD
but to be doing quite similarly is still
pretty good when you consider that ceph
is rather more complex and does a lot
more for you
set fuse however is having some issues
I actually had to cut off the top of
this graph so that we could see
what was going on with the other Mount
types
cephus is not happy the Seth kernel
driver and Seth over NFS are the ones
we're particularly interested in the
safe kernel driver seems to be
performing very well and ceph over NFS
is useful because the tools that allow
you to do the SEF specific file system
mounts are not available in every
platform
we want to be exporting over NFS where
we can because that's something we know
virtually every machine is going to be
able to accept
so it's good to see that Steph is not
doing worse than drbd over NFS
now under NFS pitch nfsbench has a lot
more parameters than postmark does for
varying the seismic experiment again on
the y-axis we've got how long the
experiment took shorter is better
this is specifically each process was
opening and writing into its file 400
times and on the bottom we've got how
many blocks of data were written and the
minimum size of the file that was
getting written into
kind of interesting thing here that I
noticed the number of blocks written in
the minimum file size aren't really
changing anything we're getting pretty
similar and levels of performance as we
vary those
the only parameter that I found visibly
made a difference with NFS bench was the
number of opens when I doubled the
number of opens the time it took to
complete the experiment was a lot longer
that's not remotely surprising though
the number of opens is roughly the total
workload that you've got to do
so if you double it you can expect a
roughly double duration on how long it
takes to get the experiment done
okay so for the CPU usage experiment
remember the way this works is I've got
10 client machines the file system is
mounted on all 10 at once and then I'm
running NFS bench on each of those
machines all at the same time and
watching the CPU usage on the clients
and on the server
NFS bench incidentally I was running it
so that it would Fork itself to make
four worker processes for each instance
of NFS bench because the machines I was
running on had four CPUs so all the tpus
have things to do but they're not
overloaded with loads of tasks
so here we have when mounted using set
fuse what percentage of total CPU time
was spent waiting for Io
so the y-axis is percentages notice it's
not going up to 100 I had to shrink it
so that we could see what was going on
here not much I O is happening these are
pretty small numbers really
there are a lot of lines on this graph
because I have 10 clients plus three
servers the black and the gray lines are
the servers all the colored lines are
clients I'm again going to be keeping
that convention for the rest of these
graphs along the bottom is just
as the experiment progressed what time
it was the experiment is designed to run
for about 10 minutes so we're seeing
these experiments between 600 700
seconds
um
so yeah not very much I O is happening
here
we can see one of the client one of the
servers the light gray is getting a
little more IO than the others but the
numbers are so small that's that's not
really a big conclusion
for idle time what percentage of time
did the CPU spend idle we can see the
black and the Grays are below the colors
the server is spending a little less
time idle that's not really weird the
servers are having to service everything
from all the clients each client is only
doing its own work
now under the Seth kernel driver we have
a much more interesting picture
the clients are spending a whole lot of
time on io8 under the Seth kernel driver
something about the kernel driver is
really inefficient in its usage of the
CPU
or at least in the the time it takes for
Io to happen
the servers however are not getting very
are not having to do much I O it's a
little hard to see because there's so
many colors going everywhere but if you
look at the bottom you can see the gray
lines and the black line are still down
the bottom
uh the servers are not having a hard
time it's the clients that are having to
wait a lot idle time looks
like it's just about a mirror of the i o
wait time again the servers are at the
top they are mostly idle the clients are
the ones who are spending big chunks of
time not idle but presumably that's
mostly i o weight
the drbd mounted over NFS looks a lot
more consistent than the kernel driver
does
so drbd it's holding at around 50-ish
percent of CPU time is waiting for Io
it's a lot but it's not going everywhere
it's pretty predictably the same thing
you can see one of the servers here is
spending a lot more time on iO than the
others if you remember the way drbd
works we can only have it in primary
mode on one server at a time so only one
of the servers is actually exporting
over NFS that's the one that's getting
the the longer I O wait times than the
other two servers that's why one of the
servers is getting a higher i o weight
than the others
and again idle time looks like it's
pretty much a mirror image I I did also
record things like the percentage of CPU
time spent on user space spent in kernel
space as opposed to idle and IO and so
forth but those numbers were so
consistently small across everything I
didn't bother showing them it's the i o
and the idle time where we get
interesting looking graphs
and for Seth mounted over NFS
interestingly like drbd mounted over NFS
it's holding around the sort of 50
region it's just more Jagged Seth seems
like it's a bit less consistent in the
amount in the
time spent on i o weight than drbd is
but it is holding around the same area
and I think that commonality is related
to the fact that both of these are
exporting over NFS
uh the idle time again basically a
mirror of the io8 time
so what do I think of these things in
the end
well the RBD is definitely simpler to
form your mental model of if you need to
spend time understanding how something
works before you can start using it drbd
is going to take you less time to
understand what the pieces are and how
they fit together there aren't as many
moving Parts it's gonna probably be
easier to set up from scratch
you may need to compile the kernel
module if you want the RBD Version 9 but
that's the most major snag you're likely
to run into
of course that Simplicity comes at the
expense of drbd does not do nearly as
much it will replicate the contents of
this block device but it has a hard time
recovering from conflicts and it doesn't
really support any additional features
on top of that
Seth much more automatic it does a lot
of things for you without you even
having to issue commands it is very
capable of fixing up issues with the
consistency of data for instance
but it has a much more complex structure
to making SF cluster work there's a lot
of demons that all need to exist and
communicate with each other for the most
part
need to actually set up all these demons
yourself
you might have noticed when I was
talking about setting up a ceft cluster
from scratch I didn't say anything about
creating extra monitors or managers or
metadata servers you don't have to do
that safe ADM will spin those up as
necessary when your cluster gets bigger
but you will need to for instance
probably create your own osds
ceph also probably needs your
configuration
in order to get the best out of it
you'll need to spend a bit of time
changing the setup from what it is out
of the box like I had to realize
actually just putting an OSD on top of a
hard disk on its own is not getting the
best out of an OSD I should also give it
a faster drive for its writer head log
that sort of thing so I think drbd is
going to be easier to start up with if
you've got a sort of a small sort of
deployment you're working with drbd
might be a good idea it's going to have
less initial investment to get it going
but as the size of your deployment gets
bigger as the complexity of what you
want to do gets higher sep is going to
become more and more and more valuable
because it is so capable of dealing with
different situations different machines
with different storage options and
rebalancing and redistributing things
as the capabilities that it's asked to
deal with change
that's all I've got to say
[Applause]
[Music]
[Applause]
thank you Christopher have you got any
questions for them
just a couple of implements
Implement questions around NFS
um
first
to the NFS that it's exports from CES is
that
just the normal Linux kernel NFS engine
or does it have it on its own NFS engine
and another question is you mentioned
exporting Rover FS because of the
clients would need it as opposed to
Native connectivity decepts
what clients are supported natively
process
Okay so
step over NFS let's do that first
um the ceph NFS export you set up inside
the Sev cluster so there's a place you
go you can do this from the dashboard
you can say I want an NFS export and
then cephal create some demons that do
that
so ceph does the NFS from inside the
cluster
as for
what
platforms the sap is available on I'm
not sure what the full list is I've been
doing all this on Debian so I know for a
fact it works on Debian
um
I know I'm pretty sure one of the I
don't believe for instance set for work
on Max that is I don't believe you can
uh you you can mount things as a ceph
client on a Mac and some of the people
in our organization use max so we need
to be able to set things up so that that
will still work
um
full list of what safe supports I'm not
sure I've basically learned what I
needed to get this going
hi thanks for the talk
um I wanted to ask about the original
use case I think you kind of actually
answered before that it's within your
organization
um uh but you are on unsw campus
and like what are the clients that are
accessing these uh home directories
running
um yeah the use case and the clients
that are actually using it
so the people who are accessing the home
directories are members of trustworthy
systems we have so we are attached to
unsw but we have our own servers and we
manage our own little infrastructure for
our research group
so yeah our home directories are not
part of the general unsw computer
science and engineering department
[Music]
what's the actual reason for that other
than if someone's using the same
potentially yes in some cases there
might be someone's wanting to do some
kind of tests on a particular machine
but that's not like their work their
work machine or their laptop that they
bring in with them there can be
situations like that and I found it is
useful to have these
home directories that mount on
everything to move stuff around
sometimes
[Music]
so in regards to give you a simple
scenario a directory that has say a
million files in it that might hit one
OSD or you have a very very large file
which lots of processes are hitting
which again could have an impact on
metadata performance have you done any
event tracks like that
no the the tests I've done are the ones
I've talked about here
uh I I I nowadays spend a lot more time
on Seth though more on the Block side
but in my past web hosting life I did a
lot of drd NFS in the late 2000s and uh
we were using was then heartbeat now
Peacemaker to do like the failover stuff
I was curious whether you actually
leveraging that left it out or whether
you weren't leveraging that to handle
the you know moving stuff around and
yeah our current our current setup does
not use pacemaker I've been working on
redesigning our setup and I'm using
pacemaker for that uh
but for our home directory is storage
I'm using SEF because I like
I like the self-healing that's one of my
favorite things about it it fixes
problems for you
and when it does that it does it seem it
I found it seems to do it a lot faster
than drbd does as well
thanks so one thing that I've seen quite
often with NFS mounts is a lack of
Transport encryption so Transit being
plain text
um have you like how does safe compare
in that regards out of the box
capabilities I don't think Seth does any
encryption on the data it sends either
thank you I noticed you talked about the
networking side of it and how you
switched was a bottleneck at one point
have you then decided to leverage a
separate switch for the set clustering
back plane so the switch that I was
running those tests over is not a switch
that's responsible for most of our
running the the main networks that
everyone's working on this switch was
connecting to a set of machines that we
generally use specifically for
benchmarking things they're useful
because they're all basically the same
and since we're using them for
benchmarking there's not other things
going on on them at the same time so
that the switch that was causing the
bottleneck there is not a switch that's
really going to be coming into play in
the day-to-day use of Seth look although
of course we have replaced that with a
faster switch for the benchmarks I
showed here
so that we could get rid of the
bottleneck
anything else right now
you are going to
handle as well and I'll take it you'll
be around for the rest of the day yep
lovely in the meantime thank you so much
Christopher there's a little gift from
us for you thank you for your time thank
you