Good afternoon, I'm Jeff Layton,
I'm from Dell in the United States.
I always like to start off all my talks in Europe
by apologizing that my parents speak
between them nine languages, I only speak one
and only a little bit better than George Bush,
so I apologize for that.
But with that, what I wanna talk about today
is NFS storage in HPC.
Lately, within about the last year or so,
we've been seeing NFS as kind of,
it's suffering a renaissance, I guess, suffering,
if you like, it's making a big comeback in HPC.
We see a lot of customers using HPC
as the primary storage system.
We're not talking about petascale,
we're talking about kind of the mid-level,
lower-level market, but that's actually
a pretty large market overall in the HPC world.
So we really think NFS, the comments up here on the screen
are just showing NFS as the enabling technology
that kind of started clusters many, many years ago.
But NFS is still really remaining very, very popular,
so the questions we always ask our customers
is why do you like NFS?
It's easy to understand.
You have high school students
who can even administer NFS systems.
It comes with every Linux distribution
you can possibly think of that are gonna be used in HPC,
and most of the time it's good enough.
You don't have to have that last few percent of performance.
You can actually use NFS and do very, very well with it.
But over time, when we used NFS originally for HPC,
over time the systems got bigger,
the IO needs were a lot larger,
so NFS kind of fell out of favor.
You know, I don't know if everybody remembers
that years ago you had stale file handles,
stuck NFS mounts, things like that.
So it kind of fell out of favor,
but in the last few years we've seen kind of a renaissance.
Even the larger customers, for example, University of Texas,
they still use NFS for things like home directories,
storing applications, in addition to having
a scratch file system for the high IO needs.
So for these smaller systems, though,
NFS serves just the primary working space,
so home directories plus the scratch space,
the working space.
So the question we always like to think about
is why is this coming back?
Why is NFS having this big renaissance?
And I think we can look more kind of in the commercial market
and the ISV market, where a lot of these applications
still do have one rank, usually the rank zero,
that does all the IO for the application.
So we'd be running over several hundred cores,
but usually the rank zero does all the IO
for the application.
So in that case, MPIIO is not a big deal.
You don't necessarily need a scalable parallel file system.
You can get away with NFS and it works quite well.
In addition to that, customers, you know,
as I mentioned before, you have high school students
that can do NFS.
So you can build your own simple NFS system.
It's not difficult, everybody, a lot of people in the audience
have probably done this at home.
But what customers are expressing to us
is they've been there, they've done that.
They now want a complete solution.
They want someone to come in and say,
here's the hardware, we've tuned it,
we've optimized it, we know how to architect NFS
for the best possible performance.
That's what we want.
We're tired of doing it ourselves.
So a simple example that I want to talk about
to show you how NFS has become so important
is just a simple case.
Let's look at LS Dyna.
People know this application fairly well.
It's a crash, it's actually a time-dependent
finite element application.
We're looking at a simple example.
We ran on eight processes, two processes,
or two nodes with four processes each.
So if you look at the table real quickly,
you can see in the bottom right,
only 38 seconds out of about 3,300 seconds we're using IO.
Less than 10% of the time, well,
much less than 10% of the time, 1%, was used for the IO.
So at that point, do you need something really fast
like a 100 gigabytes a second?
You're not gonna impact the overall performance
very much at all.
So in this case, NFS actually can work quite well.
We can also see, you can look at the application,
and if we skip through a couple of these slides,
you look at one process dominates overall.
It writes about at the very bottom,
it writes about 1.7 gigabytes out of the entire application.
So you can see LS Dyna very quickly.
Only one process does the majority of the IO.
So putting it on a fast scratch file system
is kind of a waste.
You're not gonna impact the overall performance.
So what we have at Dell in the United States,
and we're bringing this globally,
is something we call NSS.
It stands for Net NFS Storage Solution.
So what we've done is we've taken the Dell common parts,
PowerVault storage, PowerEdge servers,
and open source software, and we've gone in and tuned
and figured out the best options
for the best possible performance.
But in addition to that,
we've actually taken two approaches.
So Dell, we believe in open source,
so we've taken all of these components,
and we can deliver a complete solution,
fully integrated, three years of support,
all the way to L3 if you want,
or we give you the full document
that tells you exactly what we did.
You can buy the hardware, download the document,
and have at it and do it yourself.
So we give customers the option either way they want.
So we actually call this,
in this grand scheme of things, primary storage.
So for larger clusters,
it would be home directories and applications.
So for the sake of argument,
these are just two PDFs that we show you.
The top one shows you how to take Dell hardware
and actually tune them for the best performance.
Same thing on the bottom.
But we give you the full white paper, our NSS solution.
So all the tuning options that we've done,
we expose to you.
So, and actually customers have been very receptive.
They can see exactly what they did,
but then we'll deliver the solution for them.
Because as I said before,
they've said, we're done with that,
we want a full solution.
So the NSS solution has taken the guesswork,
we like to say, out of NFS configurations.
Right now we have a range of capacities up to 96 terabytes,
and I can explain why the 96 terabytes
is our limit right now.
The simple reason is our partner Red Hat
only has 100 terabytes of storage
so that they can test on to meet their SLA.
But we're working with Red Hat
to break that 100 terabyte limit,
and that should be coming soon.
We also have HA configurations,
but overall the bullet that people look at the most is,
do we have good performance?
And using IB, we've gotten up to one point,
gigabytes a second, on writes,
and up to 2.4 for reads, for just a simple NFS system.
It's very cost effective,
and what we do with the solution is we bake in
all three years of support with Dell and Red Hat
all the way to L3, as I said,
give you all the redundant parts as well.
But we also give you the spare drives kit,
we call them cold spares,
you can just set them on the shelf,
hard drive fails, you just swap it out,
call Dell and we'll send you a new one.
And it's very easy to install,
we can do it, or you can read the white paper,
whichever way you wanna go.
But the secret sauce that what we've done with Dell
is we actually worked with Red Hat,
did an initial install,
and that's the gray line that you see in performance,
this is throughput versus number of clients.
Then we sat down and looked at different I-O schedulers,
different parameters within the kernel that we could change,
and again, we then got the red line performance,
we got about a 30% increase in performance,
just by doing some tuning options.
So this is all published in our white paper,
but this is the advantage of going with the NSS solution,
we show you all the details of what we've done.
In addition, we actually,
this shows you a simple example,
we don't test a single node against the NFS system,
we actually test up to 64 clients,
and in the case of HA, we do 64 clients under heavy load,
and that's when we start pulling cables
to make sure that everything recovers correctly,
no data corruption.
So we begin with a simple server, this NFS gateway,
and then we have two options,
lower left is a simple NFS solution,
just a simple RAID card,
on the right is our HA configuration.
So this is what we call, we do it in small, medium, large,
kind of as initial starting points to make things easy.
So we have a simple RAID card with one JBOD,
then we go to two JBODs,
then we can go to two RAID cards and four JBODs,
we actually found this to give us better performance
without much increase in cost.
And you can see in the green box, lower right,
you can see the InfiniBand performance
is about 1.4 gigabytes a second on rights.
In the HA configuration, we do cross-connect to an RBOD,
so we actually have a full failover
and then we have JBODs behind it,
but we go beyond that and we actually add in
all the fencing and all of the,
we cut the power so that make sure the other node
isn't gonna come back up to avoid data corruption.
So we include this all in the solution, fully baked,
and we can show you how to do the fencing,
it's all in the documentation
and we can deliver this for you.
On the case of the HA configurations,
right now we have the medium configuration,
this goes to 48 terabytes,
and then we have this large, just a few more JBODs,
this goes to about 96 terabytes, but in the lower right,
you can see we get about 2.4 gigabytes a second on reads
and about 1.3 gigabyte a second on writes.
Very good for the small to medium market.
So real quick, actually I'm gonna end early
since I'm in the last talk,
virtually all clusters from what we've seen from our clusters
regardless of the size all the way up to several thousand
nodes, you need a shared file system for your applications
and for your directory, but for the small to medium clusters
we've actually seen this Renaissance and NFS,
this is what they like, this is what they understand,
and this is the solution they want.
So what we've done is we've actually taken
all the guesswork out and we put this NSS together,
so you can either look at the cookbook if you like,
or we can deliver the fully baked recipe,
whatever you want, all the way right now
we can go to 100 terabytes,
we hope to be breaking that limit this year by quite a bit.
It's actually just a hardware problem,
just four meeting SLAs.
Very cost effective, easy to configure,
and again we can deploy it if you want,
or again we give you the cookbook
and you can enjoy it yourself.
Thank you very much.
Thank you, Jeff.
Frank Beran, please.
Yeah, thanks for the presentation.
So besides the interesting sales pitch,
I would say you have a very valid point here.
Valid point and so far that even in the HPC arena
we will have a stronger diversion between two types of,
at least two types of customers.
Customer type number one is heading
for the peak performance at every cost
and at every moment of the system,
every aspect of the system,
but I would say, and Horst Siwan
had a point about this this morning,
but we will face 90 to 95% of the applications
where good old solid technology,
both in hardware and software,
at least in my opinion will be sufficient
to deal with a lot of the problems the customers are facing.
And most of the customers
maybe not be too willing to try all the interesting aspects.
How do the customers respond from your point of view?
Do they ask for the more reliable,
for the older stuff they know better?
Or do they ask more for the highly developed
new computer science guru stuff
which makes people feel better because it's new?
It really depends.
We've actually had an interesting effect
that customers come to us and we say
we have this NSS solution pre-baked and they say, okay.
And then we give them the white paper
and they'd say, okay, I wanna do it myself.
Here's the white paper.
Soon as they read the white paper, they say, this is great.
I wanna buy it from you pre-built.
Because as I said before,
they're kind of tired of doing it themselves.
But they like the fact that everything is open and exposed.
They get all the details.
So God forbid something happens to Dell.
You have all the details of what we did.
So we found a lot of customers really like that.
Even the ones that like to go play or try something new.
Let's try a different knob or something.
Even then, we found actually the customers
really like these kinds of solutions.
And I'm just using this, by the way,
it is a sales pitch to some degree,
but I'm using it as an example of what you could do.
So yes, thank you.
And most?
Check.
I guess this is the last question finally of the day.
Make it a good one.
So Thomas' talk.
So let's see.
I guess I have a concern here
about your taking high performance computing seriously
with only 96 terabytes of storage available.
That's not very large as you point out.
I understand you're prepared to fix that in the future.
Yes.
But it just makes me concerned
that you don't take our community all that serious.
But you'll correct that, as you said.
But maybe you could say something
about the power and cooling of your system.
We didn't hear anything about that.
We actually, to be honest,
it's our system, we're not using the densest storage unit.
We actually rely on more generic, I think.
So we've got a 2U with 12 drives.
We're not going to a 4U with 60 drives.
So with these customers,
we try to keep the power density in the rack,
keep it well within the air cooling range.
So well below 20 kilowatts.
So a full rack of NSS is well under 10 kilowatts.
It's more in the six to eight range.
We actually have measurements on the web.
I apologize.
I don't know them off the top of my head.
Okay, thanks.
Thank you again.
Thank you.
Thanks.