Ladies and gentlemen, welcome to the hot seat session number
one.
Welcome all in the hall three here in the lecture hall.
But also welcome to the 1 billion attendees
in the internet.
This session is broadcasted to the internet.
The format of this session we have already successfully
experienced in the last seven years.
Each performer representing an HPC manufacturer in this session
has been invited to give a 10 minute presentation
of his area of speciality or choice.
After his presentation, the speaker
will quasi in the hot seat be asked two probing questions
on the topic of his presentation by carefully chosen
inquisitors, thereby allowing you, the audience,
the opportunity to judge the manufacturer's flexibility
to problem solving and the like.
The questions of our inquisitors promise
to be a challenging and tricky five minutes for the speakers.
Ladies and gentlemen, you be the judge.
We will hear nine presentations in this hot seat session
in total this afternoon as already last year.
And tomorrow we will have hot seat session number two
with also nine presentations.
Since our schedule is very tight this afternoon,
all speakers have been informed in advance
about the rules of the hot seat sessions,
especially the time requirements.
At this point, I would like to introduce, first of all,
my co-chairman as every year, Professor Ann Bode
from Technical University, Munich.
To you also our inquisitors sitting to my left,
I start with Donna Crawford, Associate Director
of Computation, Lawrence Livermore, National Lab.
Dr. Happy C. Toole, Director, Center
of High-Performed Computing in Cape Town, South Africa.
Professor Thomas Sterling, Professor of Computer Science,
Louisiana State University, USA, and Professor Gerhard
Wellein, Head of High-Performance Computing Group,
Regionales, Reichencentrum, Erlangen, Germany.
And here you see our countdown timer.
And we have already introduced this
in the last couple of years.
And this timer is operated by Herbert Schneider,
as every year he's sitting behind of me.
He is, if you wish, our friendly executioner.
Our first speaker this afternoon will
be Dr. Tao Lang, General Manager, Market and HPC Solution
Super Micro USA, USA.
OK, Tao, may I kindly ask you to start your 10 minutes
presentation?
Thank you, Chairman and Inquisitor.
Ladies and gentlemen, good afternoon.
It's always an honor to be here, especially
to be the first one we can get over with quicker.
So my name is Tao Lang.
I'm the General Manager at Super Micro,
responsible for marketing and HPC solution.
Today, I'd like to present to you
from a hardware designer company point of view,
what's the organic growth of the hardware solution
for the revolution or evolution in HPC world.
So talking about evolution, we must mention Darwin's theory.
It's called survival of fittest.
And I thought this picture is very interesting.
So I'd like to share with you.
Actually, I share with my son.
I say that you play a game every day.
Eventually, you're going to become that new species
on the left-hand side.
So let's come to the HPC world.
For the past 20 years, we have seen
disruptive technology emerge.
We have seen several evolutions happening in this world.
One of the significant evolutions we've seen
is in the early 1990s.
Professor Thomas Stirling and Donna Baker,
they create this bayou cluster by using commodity
up-the-shelf components, actually open the door.
Why open the door for many, many people in the world?
For example, to the world like vendor like us,
traditionally, we do not have HPC business.
Since then, we had.
Open the door also to the scientists
in the world who did not have the control or the budget
to own a supercomputer.
Now it's become possible.
So over the year, the commodity up-the-shelf components
solution, HPC solution, have been evolved
into another period of time.
We call it application optimized or HPC optimized solution.
So there are many significant reasons
that we wanted to make our system efficient, especially
in the HPC, for obvious reasons.
One of the reasons I'd like to point out
is that without the efficiency, the system, large system,
cannot be scaled.
So we were thinking efficiency probably
is a very important factor or their characteristic
in the HPC nowadays in order to continue
to grow the performance portion of it.
So talking about application optimized solution,
as Supermicro, we believe that there shouldn't
be a cookie cutter solution for any type of application.
So our methodology is called building block solutions.
That means we can put it together, all kinds
of building block, and then form an almost custom-made solution
for each customer environment or each customer's applications.
So you can see on the chart, some server applications
require IO.
Some require computing power.
So all different.
We have an appliance solution for the server.
We have data center form.
We also have HPC solution.
We also have embedded solution.
All those are different.
How can you make a general purpose server to serve them?
It might work, but there are many components on the system.
It might not be used for the entire lifecycle,
and yet they are consuming power.
So we believe that optimized solution
should be for all the customer to choose.
So how to make it happen?
It is not possible that you make every application
like custom-made or tailor-made for that.
There's business consideration as well.
So in our portfolio, we have a very large set of portfolio.
So always, customers can choose some of the parts or components
and form their particular solution for their application.
So just quickly introduce, we started from 1993, 17 years ago.
We were motherboard designer manufacturer.
So over the year, we have accumulated more and more knowledge
on the motherboard design.
For sure, in the channel, where the number one,
people say that the king of the motherboard, which
is SuperMichael.
And then we start to have other solutions surrounding
the motherboard, like power and TSC, power supply and TSC,
and also IPMI, system management.
And later on, we have blade type of solution,
and we have storage.
We have GPU nowadays, networking, and so on and so forth.
So all those expertise accumulate in the company,
stuff on the ground up and getting better and better.
So we call it home grown organic make solution.
That gives you better control and also much better integration
of the system.
So over the year, as I mentioned,
we create a large set of building block for the server solution
including for HPC.
And Darwin say that a large gene pool,
we call our building block equivalent to the gene pool
we're talking about here, is indicating a genetic diversity,
and which is very important to the survival of ability
to adopt new environment for any type of species.
If you have very small gene pool,
you have much more chance to be distinguished.
So that's what we believe.
You can see on the chart there's a 500-something motherboard
we're still maintaining up to date.
And we have thousands of TSC, hundreds of power supply,
and sub-cooling systems and so forth.
So you can imagine how many permutations
you can put together as your unique solution.
Yes, we have a large gene pool.
Then how can we make sure that all the gene in the pool
are strong enough to continue to survive for the new generation?
So we believe there are three characteristics
are very important when we create those genes.
For example, innovation, you have
to continue to innovate your solution
and keep up the step of the customer request.
Energy efficiency, I mentioned before,
this is a huge, important in these days, HPC solutions.
And the third one will be open platform.
So we still believe open platform
will give people much, much better choice,
and then they can develop their software or solution
on top of this.
So I'm going to give you a couple of examples.
On the right-hand side, you can see there's lots of motherboard.
And motherboard can create some kind of system, maybe one
you, two you, a different ending, also blade.
So those building blocks are going
to quickly show you how they evolve into today's situation.
I use the example like Tween.
Tween motherboard actually started three years ago,
collaboration with Intel.
We have that brand new DP system on the half size of the motherboard.
So therefore, you can have two nodes in one use system.
That's the initial thoughts and design.
And later on, people start to follow that.
Over the year, they've evolved to two use systems.
So you can have four nodes in two you.
And the common characteristic of those are efficiency.
You can see the design on the motherboard.
They are commonplace.
For example, VRM is high efficient.
If you have a good design of VRM to support power to the CPU and memory,
and you can have a much saving on the power consumption.
Simply change the high efficient power component.
And then you change the layout locality, number of phase, things like that.
You can save 5% easy on the motherboard in terms of energy.
So I just give you the Tween as one example.
Now let's take a look at the whole picture.
Five years ago, if you look at the computing nodes in HPC,
you would immediately think, OK, this is a 1U server.
Put it into a rack.
Each U has one node.
But it's not enough.
So we have a few seconds.
So you can see over here, we have involved into Tween-based architecture.
And Tween have a several generation on the bottom side.
On the top, they also have a blade in leveraging the Tween architecture.
So we have Tween blade.
For example, on top one, this one blade have two nodes.
So that means one PCB have two DP nodes on it.
So double your density.
In the meantime, it won't consume too much power.
And then, of course, the cost down dramatically.
The other branch you can see is a GPU system.
Also, it evolved from the Tween.
So Tween, you can put one Tween and two DP on the side.
So you can immediately have a GPU solution.
Now this is the Tween.
We're going to go back to the summary now.
So the point is we are organic growth on the ground map.
And our belief is application optimized building block solution
to serve all kind of application.
And that created large gene pool.
And large gene pool is such important as indicating the genetic diversity.
And that will allow the species to adopt a new environment,
also have the ability to survive a longer time.
And an important thing is the gene should have three major characteristics.
One is innovation.
One is energy efficiency.
And also the open platform.
So that's the answer I give everybody.
Today, our methodology, our large portfolio,
it is the answer for HPC solution, not just today, and also for the future.
Thank you.
Tao, thank you very much for your presentation.
Now it's up to the inquisitors to put their questions.
Happy will start, please.
Thank you, Tao.
I can see that in your company strategy, you want to satisfy customers
by doing tuning for a specific application.
Looking at the terms of the market of HPC that you've got mainly
general purpose machines for, say, national laboratories,
how do you see yourself getting into those markets?
Because we've got a variety of applications to satisfy.
That's a good question.
So two angle, I'd like to answer the question.
Number one, we have many HPC system integrator.
Some of the large branding you can see.
And then we have the chance to collect all the feedback from the customer
himself through our system integrator.
And all those are different angle feedbacks.
And we put it together, and we try to serve at least 90% or 95%
the customer needs by using our building block solution.
So there's no 100%.
The other perspective is that, of course, this
has to be accumulated experiences for the past 17 years.
And we will see what kind of new technology, especially HPC people,
wants.
And then we focus on that.
Usually, HPC is at the front side of all the IT, which means new technology
usually adopt by HPC people first.
And then we trickle down to the IT, to data center, things like that.
So for us, from the business consideration,
we look at HPC requirement first.
And then we come on the solution, not just serve them in the future,
serve the other IT customers.
OK, the next question is put by Gerhard, please.
Thank you very much for your interesting talk.
I appreciate the efforts from Supermicro in the past decade
to bring down complexity of the single nodes,
to get a better form factor, to be more flexible on the compute
chassis.
However, those are organic developments.
Evolution is driven by quantum leaps.
Can you see a quantum leap in the node architecture, which
could be supported or driven by Supermicro in the next five years?
For example, towards micro or nano plates,
where we have a single chip, a few memory chips, an InfiniPen chip,
and all those put together in a very small form factor at an aggressive price.
That would be some kind of a quantum leap in evolution.
Well, thank you for the hint.
So we see a good market on that direction as well.
So again, there are some customers actually request for that.
So you never can serve everybody happy.
So there are some business considerations as well.
So to answer your question, yes, we have that consideration.
You put everything on the board, make it small.
And even some customers ask for a memory build on board.
There are many reasons for that, too.
And there also are the reasons that prohibit us to do so.
Maybe it's because the yield rate or the chip have some failure
and the whole board is die.
So you have to have some balance in between.
So we are in the planning for that.
Thank you for pointing it out.
We have some kind of solution that put it into very small board,
only serving the CPU power and the memory,
and then put it into the small form factor.
So we have a much dense solution.
Actually, that's not just for HPC.
For cloud computing today, they also
request for that kind of solution.
Thank you.
Let's thank once again our speaker, Dr. Tao-Ling.
Thank you a lot for coming.
Our next speaker is Richard Draycott, General Manager,
High Performance Computing of the Digital Enterprise
Group of Intel USA.
Richard, it's yours.
Thank you very much, and good afternoon, everybody.
Thanks very much for your time.
Appreciate you all coming in here.
I actually decided to change what
I'm going to talk about today after the presentations
and the data came out yesterday on the top 500.
Because I thought some interesting questions
were raised somewhat thought-provoking,
and I thought I'd literally peel the onion on some
of the data in a different way, and maybe try and shed some
light on the answers to some of those questions.
Before I do that, I will do the shameless brag sheet.
We're very, very pleased with the results of the top 500.
We've now been able to increase our position in this list
every time since 2006.
Of note, we now have already seven of the brand new Xeon 5600,
also known as Westmere, and two of the 7500, also known
as Nahalami X. So they are both on the list
within literally a couple of months of being introduced.
The 5500 series, lovingly known as the Nahalami P,
already at 184 systems.
So we're very pleased.
That means that the Nahalami architecture is approaching
40% of the list in less than five quarters
after its first introduction.
Also interesting, if you look to this in terms of not just
the listings, but which ones were new listings,
Intel actually had 93% of the new listings.
So this really does show that the preponderance of the HPC
community is selecting the Intel architecture as their choice.
We're also taking a good stand in the top of the top 500.
There were three new listings.
Two of them were marked grades, one new system.
Two of those used the new Westmere technology.
So we're very pleased with that.
And of the five new listings in the top 30,
there actually wasn't that much movement.
We got four of those five.
So I think a very good representation
of how you as our customers are embracing our newer
architectures.
So what's about this onion?
Some interesting questions yesterday, I thought.
Based on changes in the trends on the top 500,
potential areas of concern that were raised,
there was a lower than average performance increase.
I think you said it was actually lowest ever,
a drop in the total number of new listings.
The no new technology shift underway.
You showed the core count shift from one core to two core
to four core.
And there it sat, where four core and seems
to be nothing else happening with higher core counts not
really making a showing yet.
Has more laws stopped delivering?
Is that underlying part of the problem here?
And then finally, do we need to move away?
And this one wasn't so much from the top 500 discussions.
It certainly has been a sort of a pervading theme
as I walked the corridors.
Do you need to move away from a general purpose CPU
in order to maintain the exponential growth
rate that we see projected on the top 500?
And simply put, my assertion, and what
I'm going to convince you all today,
is that the answer to all of these is really no.
So let me start with the first one.
Was there really no new technology transition going on?
The core count trends were what was
used to show that there's really nothing happening
over the last few lists.
And the thing is, I'm going to convince you,
is that there is actually a very significant shift underway,
a big one and a fast one.
And that is the shift in the hailem architecture.
So we had the hailem EP, the 5500 Zero Xeon,
announced about a year and a bit ago.
And then, as I said, the two newer versions
just in the last couple of months.
If you look at the rate at which that architecture has
been adopted over the last three lists, 7%, 19%,
and then to 39%.
I didn't have the curves available, the charts
available, but had you plotted it on there,
I think you'd see this is almost in parallel to the shift
that we saw when we went from two core to four core.
You'd see exactly that same shape already happening.
So in fact, there is a pretty rapid technology shift going
on.
But that isn't really addressed kind of,
OK, what about the end result?
What's happening with the performance and the number
of systems and so forth?
So let me just remind you of something.
When we introduced the hailem EP, it actually
represented probably the most significant performance
increase we've been able to deliver for HPC applications
in the history of Intel.
And this is just a bit of an eye chart
from where you're sitting, I'm sure.
The blue bar there is the predecessor, the 5400 series,
also known as Harper Town.
And what you can see is that the average performance
improvement across this range of applications
in each vertical bar is a blend of applications,
weather, finite element analysis, CFD, energy,
and so forth.
The key thing that you see here is the typical improvement
is at least two.
So we actually saw a very significant performance
improvement through this family.
And the thing, what happens when you
start to combine these effects?
The so what here is that, oh, sorry.
I'll come back to the performance in a second.
So just bear that thought in mind,
you had a more than doubling typical performance
of applications.
And what you see is missing there is LIMPAC.
LIMPAC, in fact, did not increase
as we went from the Harper towns and the hailem.
There was a modest increase, but not substantial,
certainly not in the kind of range
that we're talking about here.
The other thing that's important is not just the performance,
but the performance efficiency.
And I thought this was telling, if you look at the list,
we took the high speed systems, the ones with the high speed
interconnect, whether it's Infiniband
or a proprietary solution, anything other than Ethernet,
really.
And what you can see, if you looked
at those that have a pure hailem architecture solution,
nothing else in there, the average efficiency on LIMPAC
was 88%.
And some were well north of 90%.
If you took the systems with GPGPUs,
and Dr. Stromer pointed this out,
the actual average is 42%, even though some of those machines
have a significant element of either the hailem
or Westmere in them.
So that's rather telling.
And then the other systems were about 78%.
And the question's simple, which would you rather have?
If you really want a peak performance number,
or do you really want delivered performance?
I think the answer's obvious.
What's interesting, I wanted to do the same equation
with power.
Unfortunately, the significant machines
that had GPGPUs and did not disclose their power
consumption, but I'm sure that will be an interesting
comparison once they do.
So what?
Come back to where I was.
LIMPAC only improved by a very modest amount.
But as I showed you earlier, the actual applications
performance that the hailem delivered
was typically 2x or better across the majority
of the applications.
So what you actually see is about 40%
of the list, which is the hailem,
actually delivered something like double performance.
It just wasn't represented by LIMPAC.
And so I think the improvement that you're seeing
in the top 500 in terms of what those systems
are really doing, whether they're in labs
or companies or whatever, has in fact continued
to improve at a very good rate.
It's just that they didn't get kind of the free uplift
from LIMPAC that we have on some of the shifts
between technologies.
And I think that skewed the apparent result.
So in fact, I think things are going better
than you may think on the top 500.
In addition to that, the hailem architecture
gives you very good efficiency.
We've lost that clock.
I've got time left here.
Couple of corollaries that are important.
One is core count is not the only,
nor the best indicator of performance.
There are many other things that matter.
And the other is not all cores are created equal.
Core count alone will not be the thing
that gets you there because the different cores
will give you different behaviors.
So it's a real application performance
that really tells you what's going on.
Is this gonna change?
Yes, it certainly is.
This shows a similar chart for Westmere.
The left hand here is now on a hailem EP
and the chart is for the 5600, the new one we just introduced.
And what you can see in the red circle,
LIMPAC, 60% improvement per socket.
There's no doubt that that's gonna deliver
the same kind of uplift.
In actual fact, the application's performance
is probably not as strong as the hailem EP was
relative to its predecessor.
Westmere still delivers a lot of improvement,
but when people put their new machines out there,
they're gonna get the LIMPAC kind of for free on this one.
And so I think you're gonna see a big shift
as Westmere takes on the list
over the next couple of lists.
In addition to that, Sandy Bridge is coming up right behind
and as you heard yesterday from the keynote,
that will both add cores and double the flops per clock.
So I think that's gonna help both on the applications
and the LIMPAC side.
The next thing I want to address is probably the last one,
which is, you really have to shift over
to get advantage from the different types of architecture
versus a standard GPU.
And the real answer here is that
it's all about the programming model.
It's all about the software.
Each of you probably have a slightly different priority
in terms of how you parallelize your code,
what kind of optimizations you make.
And they're partly represented here by this block diagram
in terms of the different options.
The key thing here is that with Intel Xeon
and the mic architecture that we introduced,
you're gonna have general purpose cores.
So they're gonna be good across a whole range of performance.
You're gonna get real applications improvements.
And you're gonna have programming flexibility and choice
in terms of how you mix and match these things
because this very flexible software model
will sit on top of almost any kind
of physical implementation we choose to give you,
whether it's sockets together or boards or whatever.
So bottom line, Intel's live, sorry,
Moore's law we think is alive and well
and delivering for you at Intel.
The top 500 is delivering a lot more than just LIMPAC.
And I think that should be recognized
that the list is useful and helpful,
but there is other good things going on there
to be aware of.
We'll keep that trend up with the next two generations
and the follow ones with the Xeon
combined with the microprocessor
will actually accelerate those trends.
Thank you, Richard.
Thank you.
Our first inquisitor is Thomas Sterling.
Thank you for that very informative presentation.
Your slides were very interesting.
I note you gave us a very nice detail
about the performance of your hardware cores.
And then you moved and you discussed the systems level
and you immediately switched to software.
Those who are working with
or even developing larger system concepts
where there will be millions of cores
recognize that the granularity of the parallelism
will have to become smaller or finer
in order to expose more parallelism in different ways.
My question is, is Intel considering
in future generation directions or products
additional support of the hardware core architecture
for the system-wide global parallel execution?
Interesting question.
The short answer is yes.
The slightly longer answer is
I can't really tell you much about it.
We certainly recognize that there's things
like synchronization and stuff like that
that go on both across the CPU itself
between the cores and then throughout the system.
So we are looking at what we do
both at the processor core, the uncore,
the subsystem around it,
and working closely with key systems customers
to figure out what do you need to do to help this all work.
Not just scaling in terms of application performance,
but also stability, right?
As these machines continue to go in size,
they're gonna be much more susceptible to error rates
and natural failures that will occur.
So how do you make the systems more robust?
How do you make the software more robust
in the system software, middleware,
as well as the applications themselves?
And so we do recognize there's a hardware element, of course,
but we are also very cognizant
that the software challenges here are immense.
And that's one of the reasons I really wanted to focus
on that with this particular presentation
in terms of the tools that are already available
and that are already being enhanced
to take advantage of those future directions.
Thank you.
The next question comes by Donna Crawford.
Very interesting presentation, thank you.
Slightly related to Thomas's question,
has to do with feeding data into these high-powered chips,
and are you considering user-controllable cache
or gather scatter for the memory controllers
to feed the data into these systems?
Yeah, this is something we do get asked
by a number of customers, typically the ones
that are really more sophisticated
and really digging deep into the architecture.
The challenge we have at the moment
is when we expose some of the controls
into the cache architecture, as you describe,
unpredictable behavior can happen.
And so we're thinking about how can you expose any of that
without exposing some of the underlying logic
in a way that might be unpredictable.
So we know that there's things
that people wanna be able to do.
So we're trying to think of other ways,
whether it's cache locking and things like that
that we could do, or working with some customers
to figure out other different things
we could do in the caching algorithms
for certain kinds of workloads.
So there are some ideas there,
but exposing that to end users is something
we've not been able to figure out
how do you get in a manageable fashion.
I think your other question was how do we feed calls?
I apologize, I forgot the second half of the question.
Yes, we are looking at different memory technologies,
both in terms of eventually having
that kind of scatter gutter capability.
The reality is that that kind of thing
is actually gonna require support
out of the memory technology as well.
So I could do whatever I like, but it's not gonna work.
And I've got a memory vendor I can work with
who's willing to invest in the same
kind of complementary technology.
And so yeah, that is one of the areas of exploration.
We have also looked at different kinds
of memory technologies for power management.
Right now you have to turn on the entire memory segment
in order to get data out,
but using more smart power management mechanisms
on the memory subsystem,
having that managed by the CPU
as it makes the memory accesses.
So we're looking at it both from a performance point of view
as well as from a power management point of view.
Thank you again, Richard, for your presentation.
Our next speaker will be Mark Hamilton.
Vice President HPC Sales Support Oracle USA.
And I asked Michael Kagan from Mellanox
and Phil Murphy from Coologic to enter our speaker's cache
to speed up the procedure to prepare you
for your talk with the microphone.
Thank you, Hans, and thank you for inviting me here.
It's great to be sandwiched in between Intel and Mellanox
in the speaker cache because they're both great partners
of Oracles.
So I thought I would start off my presentation
with the question since I was already asked
by Michael Feldman in his ISC preview of HPC wire.
He said, well, you know,
the question people should ask, Mark,
is what are you doing here
and what have you done with my Sun servers?
So I'll try to answer that in the presentation.
Certainly one of the Sun systems
was one of those top 10 Intel and the Halem clusters
that Richard talked about.
So thank you very much to our customer Sandia
if they're out there and congratulations for moving up
into their new top 10 position.
I thought I would go through and give my own statistics.
I know everyone has talked about the top 500 a lot
and I'm not sure I could add a lot of value
in cutting the statistics differently.
So I thought I would give some different statistics
I haven't heard a lot of talk about,
but that's how HPC and the web is coming together.
And certainly some of the sessions talk about HPC
and cloud computing, but from a pure sort of web
or social networking status,
there's over, I just did a Google search on ISC 10
and there's over 63,000 Google articles on ISC 10
or references, so that's pretty good.
Now I'm actually somewhat addicted to Twitter myself.
I'm not a big Facebook fan, but I do like Twitter
and in the last 24 hours, there's been over 100 references
or links on Twitter to ISC 10.
So you might wanna go out and check out some of those.
Now on Facebook, apparently ISC isn't too popular
on Facebook because only 89 people have liked
the Facebook page for ISC, but that'll be an interesting
trend to look at next year and see how many more people
are using social networking.
So at Oracle, they're trying to figure out
what to call high performance computing
and the name sort of changes from week to week,
but right now we're trying to focus on the name
Oracle Technical Computing.
I'll talk a little bit why I noticed Microsoft
was actually using technical computing
in a number of the different HPC literature
and it's certainly meant to imply that besides working
on some of the big Sun systems in the top 500,
we're really our focus as well on enterprise uses
of high performance computing and in fact Sun is,
or Oracle has partnered with Intel
and a number of other companies in a very interesting
initiative that's referred to as the missing middle.
And part of the missing middle is really meant to say
that middle space that's in between what you can do
on your desktop or your laptop, even if it has a GPU in it,
and what you can do in a top 500 system
because the vast majority of the manufacturing,
financial services, just about any industry in the world
is not driven by a top 500 system.
It is not driven by something that runs on a PC or a laptop,
it is driven by that missing middle.
So we're working with Intel and others
on what we can do in those areas.
Of course, I've got a nice disclaimer
and since we're in Germany here,
we actually have not completed the legal integration
between Sun and Oracle in Germany,
but of course no one really wants to hear about that.
But one of the big reasons that Oracle acquired Sun
and the focus for Oracle's work with Sun hardware
is really to transform the industry
and really to do that by bringing to market
complete engineered and integrated systems.
And really innovate across the stack.
Now clearly in the classic enterprise stack,
you start with storage, you start with servers,
you add a virtual machine, you add an operating system,
and then you add Oracle's classical software products,
their database, their middleware in their applications.
But when you think about it,
and you think about what runs on an HPC system,
and especially what runs on an HPC system
when it's in a cloud environment,
almost every single customer I've talked about
with here this week is using databases
somewhere in their HPC environment.
And almost every customer is using middleware.
So there certainly are, I think,
direct applications of this stack in an HPC environment
before you even start to get to some of the classic
HPC applications.
Really, if you think about it,
one of the big differences between Oracle's approach
and between previous approaches taken in years past
towards delivering complete stacks
is when I think of, and when someone comes to me and says,
I have a complete solution I'm gonna sell you,
I typically think of two things.
I think, well, it's probably gonna be expensive,
and that complete probably implies
some level of proprietariness.
Well, that's the big difference with Oracle,
is first of all, it's not proprietary.
If you look, everything we're doing in the HPC space
is based on open industry standards,
open industry components like the x86 microprocessor
or Mellanox and Finoban chips.
An open source operating system, Linux,
and open source software up and down the stack.
But what we'll focus on is really not only delivering
top 500 systems, but really industry solutions
in that missing middle.
And those won't be delivered 100% with Oracle applications.
In fact, in most places, most of the application partners
we're working with is not Oracle software,
but is other third party software providers.
NASTRAN, ANSYS, Schlumberger, you name your favorite,
HPC ISV.
Finally, one of the hallmarks that Oracle will bring
is really, I believe, a higher level of customer service.
I think this is really valuable in two areas.
Certainly, anyone who's deployed a top 500 system
has struggled, most likely,
with the implementation of that system.
It's a little bit natural
as you develop leading edge systems.
On the other hand, if you're running a business,
you don't want to just run a top 500 system,
but you want to deliver a service.
You want to manufacture an airplane part.
You want to do crash analysis of a car part.
You really don't want to be focused on building up
the components of the cluster.
You'd like to have that complete solution.
So I just wanted to focus, change from the product side
and the stack side, to really talk about,
again, address why is Oracle interested
in high performance computing?
Because that's one of the questions
I've been asked a lot.
Well, certainly, Kirk answered a good part of that
in his keynote yesterday.
So Larry Ellison, our CEO, has been up on stage
and said very directly he wants to compete
head on head with IBM.
He's been very vocal in the press.
And I don't believe, and he doesn't believe you can do that
by ignoring 25% of Intel's server CPU market.
So that's certainly one motivator.
A second one is, and Kirk talked about this yesterday,
about how technology in the HPC space
doesn't start at the bottom and build up,
but really starts at the top end in a big way and moves down.
And one of the innovations that Oracle
will be able to do by focusing on a complete stack
is really drive those innovations
from top 500, top 10 systems down
into everyday technical computing clusters
at a much faster rate.
A great example of this is something
that Oracle did even before the acquisition was complete.
So in June last year, at ISCO 9, we
celebrated the EULEC Europa System,
being one of the first Intel and the Halem systems
and one of the first QDR Infiniband systems
on the top 10 list.
Less than six months after that, Oracle
introduced a product to their mainstream commercial database
market, the Exadata database machine,
using Intel and the Halem CPUs, using Linux,
using dual-rail QDR, even one level more reliability that
was in the EULEC cluster, and using the Flash DIM technology
that was first introduced as well on the EULEC cluster.
So a great example of using HPC technology,
delivering that innovation to market in a complete solution
much faster.
And I already talked about the missing middle part.
Followed just in my last minute, talk a little bit
about Oracle Enterprise Linux.
When I went out on the show floor
and I tried to find one of the exhibitors
that was actually using Oracle Enterprise Linux,
and outside of the Oracle booth, I didn't actually find one.
So that's actually a bit of a challenge for my team.
I just wanted to spend 22 seconds here
to sort of say, a lot of people don't know
what Oracle Enterprise Linux is.
It actually tracks the Red Hat distribution.
And every time Red Hat issues a new package,
Oracle takes that package, does their own bug fixing,
and publishes it, releases it as part of Oracle Enterprise
Linux, then takes any change and passes that change back up
to Red Hat to be included in the regular Oracle kernel.
To give you an idea of exactly how compatible it is,
since 2006, Oracle has shipped thousands, literally
hundreds of thousands of products
that have been deployed on the Red Hat operating system.
And Oracle doesn't use a single copy of Red Hat internally.
We do all of our Red Hat development,
all of our Red Hat production work
is done on Oracle Enterprise Linux.
And we've never had a bug filed by Red Hat or an ISV
against that.
So with that, if anyone has any other questions from them,
I'm sure there are some other good questions
on the panel.
I'll do my best to try to answer them.
Thank you, Mark.
The first question will be put by Gerhard Welland, please.
Thank you very much for this interesting talk.
I was really excited when I saw that Oracle
is going to talk about its vision of HPC.
And personally, I'm representing a computing
center from a university in Germany.
And we are exactly in this middle market,
between a PC and a top 100 installation.
However, the workflow of our users is very simple.
They submit a job, and they return a number.
They give us a budget, and we have
to maximize the number of nodes for a reasonably stable system.
Your talk was very industry-centric.
And you mentioned a lot of features,
which are of not major importance for us,
like virtualization.
Is Sun Oracle going to stay in this very competitive low margin
market, HPC in academia?
Well, I think every company that hasn't gone out of business,
and there certainly have been a number of HPC companies that
have gone out of business, is in the business
to make margin and to make money.
So we certainly look at every deal
as an opportunity to make margin, whether that's
low margin or high margin.
I think the difference is we're not
focusing on the margin for an individual component
or the cost of an individual component.
We're focused on the total cost of ownership.
And again, I think this is one of the areas I see changing
in the world of HPC.
Five years ago, if I came here and asked a center director,
how many of you pay for power as part of your budget?
Almost no one would have said yes.
They said, no, power is paid for by someone else.
Today, more and more center directors
are actually being charged back for power.
So again, I think the speaker from Supermicro
talked about power efficiency.
If we can lower the total cost of ownership
by putting more efficient power supply in there,
is that worth anything to you?
So that's really what Oracle is focused on,
is lowering the total cost of ownership
versus lowering the cost of individual components.
OK, the next question is put by Thomas Sterling, please.
The assimilation of Sun by Oracle
was one of the major events in high performance computing
this year, and certainly even broader than the HPC field.
You've made a compelling argument
for the invention of MPC, or middle performance computing.
And no doubt, a valuable market.
When Sun was standing alone and contributing to the field,
it was a technology innovator, a technology
advancing in the DARPA High Performance Computing Systems
Program.
It was a successful competitor in two of the three phases.
I think the community as a whole is very interested to know
in the long term, will Oracle continue the trend of Sun?
Or will it abandon it of technology advancement
for the high performance, not the middle performance,
but the high performance computing field?
I think it's only fair to comment
on what Oracle is doing today versus going in and trying
to predict the future.
But one of the most crowded sessions here at ISC
was the Silicon Photonics session this morning.
And I'll note that Oracle is actually the only tier one IT
vendor that actually has a DARPA-funded contract
for research in Silicon Photonics.
And we're actually continuing that.
So that clearly is one example of being
at the very high end of high performance computing.
But while I talked about the focus on the missing middle,
we actually don't think that the technologies there
are that different.
In fact, the flow down in the Exadata example,
in less than six months, flowing down from a top 10 system
into that middle market.
We absolutely, I believe, need to continue
to focus on that high end if we're
going to play in the middle market.
Thank you very much once again.
Our next speaker is Michael Kagan.
He's the CTO of Mellanox Technologies in the USA.
Hi.
So thanks for having me here again.
One of the things on the title of the slide of this talk
in the program, it says Roadmap to the Periscale Computing
and Beyond.
So it's obviously a typo.
We are well beyond periscape computing.
There are three periscale computers
on the top five fanart list.
Two of them are Mellanox and FinEvent.
OK, so going further with the supercomputing,
we started the supercomputing.
We debuted the supercomputing list
on the top five fanart list in 2003
with the Virginia tech cluster.
Basically from the pure silicon provider or very low level
hardware provider, we are stepping up
on the stack all the way up to providing full end-to-end
solution, not only on the hardware, which is adapter,
cable, switch box level, but also on the management
and the network provisioning utilities.
So basically from Mellanox, you can get the full end-to-end
solution.
And we believe that's the key to go
to the next level of performance and the next level of the scale
to the hexascale computing.
This is the slide that shows the trends on the top five fanart
list.
Basically what you see is that for the last two years,
we started our presence in the top five fanart list in 2003.
And we constantly grew the number
of clusters built with the Mellanox and FinEvent
from year to year.
And basically what you see here is
that we are very close to the crossover point
versus other standard interconnect, which
is ethernet.
And basically we can predict the trend.
On the right-hand side of the slide,
you see that 64% of the top 100 machines are in FinEvent.
And as we all know, when time grows,
basically the first 100 machines will slide down next time
around.
So we have a very good reason to predict
that Mellanox interconnect, in FinEvent interconnect,
Mellanox will be a major interconnect on the top 500
list.
And actually, this picture explains it all.
It's all about efficiency.
It's all about the ability to get cycles
for the applications versus cycles for the networking.
And you actually see here three representations.
One is the ethernet.
That's the yellow part.
So basically, you see that ethernet clusters are roughly
50% efficient.
So you need to buy twice as many computers
to get the application performance you
are looking for.
The blue dots are in FinEvent, and the highest one is 96%.
So there's up to 96% efficiency.
That's what you can get from the in FinEvent cluster.
And there is another dots, which are Cray.
What you see here is that proprietary solutions like Cray
are somewhere in between standard
in FinEvent-based solution.
And if you look at this, rolled out the time back a few years,
there was another proprietary solutions
on the top 500 list.
And basically, they were all squeezed
by the two industry standards, in FinEvent and Ethernet.
So basically, what you see here is
that there is no apparent advantage
in the utilization of the system going
to the proprietary interconnect and proprietary solution.
And basically, it means the standard will win.
So the hyperperance computing challenges,
it starts from the raw bare metal performance.
It's like measuring latency.
Now we measure latency in the nanoseconds,
not in the microseconds anymore.
And the bare bandwidths of the wire,
and what you can get out of the node, the message rate.
In order to scale up, you need to be
able to utilize many cores.
And these days, computing power or computing infrastructure
that you are using actually becomes
more heterogeneous, in a sense.
Obviously, Intel x86 CPUs are the standard machines
for general purpose code.
But there is also following point accelerators,
like GPU, coming in.
And in order to be able to build the next wave,
or next step in the hyperperformance computing,
you need to be able to operate within this environment
very efficiently.
The number of nodes are growing, and computing
becomes much more and more distributed.
Basically, a while ago, there was
very fine tuning between the upper layer
software and the practical or particular network topology
and the location of the nodes.
Now, this notion basically going away
in the favor of flexibility of application locations.
Once the size of the machine goes up,
the reliability challenge becomes even more interesting.
And the way to overcome the reliability
and deliver the performance, the reliability
needs to be pushed down to the lower level,
to the network, to the network layer, and not to the software.
And at the end of the day, it all boils down to efficiency.
After all, people are buying or building machines
to get the application performance.
And therefore, all the operations
which are related to the network or close to the network
are better off handled by the network layer
and not by the general purpose CPUs.
One more thing is that with a larger machine size,
you need to take care also of the physical location
and the power so you're better off
being able to run multiple services, not only clustering,
but also storage and the general network on the same wire.
That's what Infiniment provides.
It must be energy efficient in a way
that it will be practical to build very large machines.
We talked about the CPUs and GPUs.
And we need to be able to get the network as close
to possible to the application layer of both CPUs and GPUs.
And last but not least, ecosystem
is a critical part of this.
So once you go for the standard solution,
the economy of scale drives the cost down
and enables you to build the large scale systems.
So three parts of memory application and the protocols
needs to be handled directly by the application.
So the ability to deliver native RDMA,
so basically you can share the memory in a way
across the network.
And GPU Direct Technology, which
is a relatively recent innovation
that we have done together jointly working with the NVIDIA,
enables GPUs much more efficiently and effectively
bring network much closer to the application layer in the GPU.
On the application layer, beyond the transport,
which is on the right hand side like protocols,
there are some other application level network-related
operations that are done by the applications.
And the Core Direct Technology basically
enables to synchronize and offload
the collective operations and other MPI operations
to the network and not spend precision CPU cycles
or reduce the OS jitter while running the MPI applications.
So this is what is really needed
to get to the next three orders of magnitudes
from petascale to exascale.
So right now, the three key components on the machines
is the x86 compute node, floating point accelerator
as a GPU, and large scale switch systems
with the management layer on top of it.
These are the three key components
that are enabling to build larger and larger machines.
And basically, Mellanox are delivering the full end
to end solution, enabling network
to utilize much more efficiently and get the raw application
running on the CPU.
This is the nutshell of the Core Direct Technology.
Basically, we are saving the memory access
and getting the network closer to GPU.
That's where GPU directs.
So basically, GPU has more direct access to the network.
And we see 30% difference performance
boost on the applications running or using GPU direct.
In Finiment, Roadmap, Raw Performance Roadmap
is the most aggressive ever.
We are delivering 40 gigabit solutions for more than a year.
Actually, a couple of two years.
And the next wave of speed bump is going to be available soon.
That's basically the conclusion.
So in terms of raw performance, InfiniVent
basically is the best both on latency, bandwidth,
and MPI messages per second operations.
Scalability enables to bring the network directly
to the application.
Network is very reliable, enabling
to build 7,000 and 8,000-node machines.
And it becomes very efficient in both of network usage
and compute power.
That's it.
Thank you, Michael, for your talk.
Before we have the first question,
may I kindly ask Victor Savietov to move into the speaker's
cache in the first row.
So the first question is asked by Donna Crawford.
Thank you.
You explained how InfiniVent has played a key role in expanding
the role of high-performance computing in the top 500.
And I wondered if you could tell us a little bit more,
in particular, about improving InfiniVent's network
reliability and power efficiency as scales to exascale.
OK, so in terms of network reliability,
InfiniVent trade association, there are more and more.
So going to the pure performance rate.
In terms of reliability per se, we
are looking at the end-to-end solution
from the one PCI Express to another PCI Express.
And we do have fairly substantial work
being done on the link layer to improve bit error rate
so the network will not lose the packets.
Our solutions are guaranteed to have a bit error rate
of the 10 to the minus 17, although the standard
calls to 10 to the minus 12.
So that's five orders magnitude higher.
And that's on the nutshell on the low-level layer 1
technology.
Besides that, InfiniVent, as a standard,
has a transport layer built in the network,
alternative path, and being able to route
around the hotspots of broken links.
So that's it.
Thank you.
The next question comes by Happy Sitoli.
Thank you, Michael.
I was impressed to see that you are looking at your Mellanox
products in speeding up or getting efficiency of CPU
and GPU.
My question is basically, when one considers
any architecture and interconnects,
now for the centers, we also look at the balance
in terms of application.
Are you also looking at this efficiency
in any scientific applications where this can be sort
of a benefit?
The numbers that you showed, which
appears on the top 500, is mainly the efficiency
from the impact performance.
But I think we'd like to know what
is your plan in terms of the applications beyond the impact.
So the numbers we are showing based on the impact performance,
because basically, that's the top 500 statistics.
We are very much involved with the more commercial HPC
community.
And there are different applications
that are being measured there.
I think a couple of years ago, or maybe last year,
we have showed the example of efficiency
running the LS dyna or fluid dynamics, which
is also way higher than Ethernet-based machines,
both on the point and on the scaling up.
So short answer to your question is basically,
we are looking at the broad number of applications.
We believe that the requirements that
are on the wider, more commercial market
are very applicable to the high performance computing
to the top notch.
And we basically learn the needs and applying it
on the both directions.
OK, thank you once more, Michael.
Our fifth speaker this afternoon is Mr. Phil Murphy,
Vice President Engineering, Coologic, USA.
Phil, please start your talk.
Thank you very much for having me back again this year.
I'm very excited to have the opportunity
to let you know what Coologic's been doing with respect
to high performance computing, especially
around our recent release of the InfiniBand Fabric Suite 6.0.
But just to give you a quick perspective on who Coologic is,
Coologic is a company of over 1,000 employees today
headquartered in Southern California in the United
States.
The focus of Coologic has always been
on high performance networking, whether that's
data networking, storage area networking,
or high performance computing networking.
And the other point to make here is the very financially
stable company.
Company has revenues of over $500 million
last fiscal year, a lot of money in the bank.
And what this means to you is that Coologic
is a company that's here today and will also
be here tomorrow to support you.
Just to give you a perspective on some of our products,
we actually focus on three areas.
The first one is actually a new area
for Coologic around converged networking.
And in this case, this is the technology
that we would use to go after the convergence of ethernet
and fiber channel in the data center.
But the most interesting technology for this group
is our high performance computing technology.
And we should provide an end-to-end solution.
This would include all the hardware and software
components necessary to put together
an efficient and high performing network for high performance
computing.
And then, of course, our last and our largest
area for Coologic is our traditional area, which
is storage area networking, hardware, and software
products.
You've already probably seen enough analyses of the top 500.
So I'll try to be brief in this area.
But as we know, the performance on the top 500
continues to grow exponentially.
We did a one petaflop system as an industry in 2008.
And we're talking about already exascale
in the next 7 to 10 years.
It is kind of interesting to see how these solutions are
being delivered.
And when I first started, a very small percentage
of the total top 500 capacity was
delivered by high performance clusters.
Just over the last seven or eight years,
it's already grown to almost 85%.
And of course, with a high speed interconnect,
with a high speed for high performance computing cluster,
you need a high performance interconnect.
And you've already seen this a few different ways as well.
How has that been changing?
Today, even in the composite, you
can see that Infiniban actually holds
207 of the top 500 systems.
And in terms of performance, it's even more dramatic.
Again, looking at the top 500, but in terms of flops
instead of systems, it has just about half
of the total performance on the entire top 500 today.
But I think it's an interesting question to ask.
Well, how did it get to that point?
Because I was really with the Infiniban development
from the very beginning in the late 90s.
And in those days, it was not targeting
high performance computing, but rather actually
trying to solve a problem that was
perceived around the PCI bus in those days, a parallel 32-bit
bus, also maybe a way to unify Ethernet and fiber channel
in the data center.
And so really, it didn't have an high performance computing.
But it did have great latency and great bandwidth.
And it did have an ecosystem that developed around it.
And because of those things, it actually
had the ability to enter the high performance computing
market and over time drive the other proprietary competitors
out.
But to go forward, that's really not sufficient.
And so QLogical offers what we call our true scale
internet for high performance computing.
And at its heart, of course, it's Infiniban.
It's got the Infiniban speeds and feeds and protocols
and so on.
And we wrap around that, the standard industry software
comes from the Open Fabric Alliance.
And that's the basic Infiniban that you can get from us
or from others as well.
But we do layer on top of that a series
of extensions that are very specific to high performance
computing to take Infiniban from its roots, which was really
the enterprise data center target market,
to the high performance computing market.
And that's the true scale cluster.
And this is some of the extensions
that I like to try to cover quickly this afternoon.
The first one is around the host interfacing.
The traditional approach for host interfacing Infiniban
into a high performance fabric would
be to basically run the NPI layers over verbs.
And you can certainly make that work.
That's been working well for some number of years.
But it's really not the optimal way to do it.
If you take lessons from past technologies that really
targeted high performance computing,
there's really a much better way.
A connectionless stateless approach is far more scalable,
especially as the CPU performance increases.
And just to give a quick snapshot of what
that would look like, I'm going to skip that
and go right to this slide.
One example of how this can matter is in messaging rate.
Messaging rate is very important for many applications.
And as individual node performance continues
to escalate, this is going to continue
to become increasingly important.
This perspective is on a single node,
talking to another node, 8-Way Nahalems.
And it's a micro-benchmark.
So really all that's going on here
is you're generating as many messages per second as you can.
If you look at one core, one core
can generate about 2.5 million messages.
If you double that to two cores, you
can get over 5 million messages a second.
But in the connectionless stateless approach,
as you continue to scale up the number of cores
in that server, the message rate continues
to scale linearly, where traditional approaches actually
hit limitations that cause them to have issues in this
benchmark.
So that's one key difference.
Another one is in routing.
So again, when the thin-and-ban was defined,
it was defined as a data center network.
It was modeled after fiber channel.
And it was defined as static routing.
And of course, if you're expecting
a small number of messages per second,
you may not have any problems.
But certainly in high-performance computing,
that can be an issue.
And so it's really important to be able to detect
and eliminate congestion.
Because if you don't, you could end up
with a situation like this.
Here's another micro-benchmark.
On a small cluster, there's only 64 nodes.
And what you can see from this chart
is that without some kind of dynamic routing,
even when you're sending very large messages,
you may end up with only 60% or less
of the total fiber capacity.
And of course, if you're spending
a large amount of money for your cluster,
the last thing you want is some of your inner-switch links
to be extremely busy while all the ones are idle.
So how do you remedy that situation?
Well, there's actually two approaches
that QLogic is putting out in our IFS 6.0 release.
The first one is what we call our dynamic switch-based
adaptive routing capability.
And this is for non-random traffic patterns.
When I say non-random, I just mean
that there is at least some persistence in the traffic
pattern.
It can certainly change hundreds of times a second,
but at least there's some predictability.
And when you do that, using our IFS 6.0 release,
it's very possible to actually see
that we get right to the maximum and have
a perfectly balanced system when the message sizes are large
enough to congest the entire fabric.
And the second one, which is a similar nature,
is our dispersive routing capability.
And dispersive routing is used for highly random environments,
which we do see in some of our customer environments.
In this case, we simply are able to spray all the traffic
across all the available links.
And while we can't quite get to the same level of efficiency
that we can get for the predictable, more pseudo-static
patterns, we do get about 85% of the total,
very useful for random traffic patterns.
So another one, and another one really focused
on fabric efficiency is what we call virtual fabrics.
And anyone who's familiar with VLANs in ethernet context
or VSANS in a storage area networking context
will be familiar with virtual fabrics.
It's really all about the separation
of different traffic classes.
So let's just say that in the beginning,
the administrator, before any jobs are running,
will set up a set of virtual fabrics.
He might get set up one for his compute traffic.
And he sets up the capabilities to say, I care about latency.
I don't care about bandwidth so much.
But he might have another virtual fabric
that he sets up that says, this one cares a lot less
about latency, but cares a lot about bandwidth.
And you can set up as many as 32 different virtual fabrics
using our technology.
When you do that, you can see that if you run a job
without virtual fabrics, you might be able to see
as low as 1.2 microseconds of latency between two nodes.
But if you add a storage job
that's trying to get 3.2 gigabytes of traffic,
all of a sudden that works and it will get its 3.2 gigabytes,
but it's very possible to destroy the performance
of your compute job.
So if you actually run them on different virtual fabrics,
what you'll see is you can actually maintain
the 3.2 gigabytes of data while only minimally affecting
your latency of your compute job.
And so this is really the value of this approach.
We also offer the capability for supporting
advanced fabric topologies, which we're seeing come up
more and more in the context of either large clusters
or certain codes that are three-dimensional in nature.
And we do support both 3D tours and mesh topologies.
And the last thing to point out is our capabilities
around our tool set.
Anybody who's installed, configured, managed, monitored,
try to diagnose a large cluster understands
that there will be problems, they need to be resolved,
they need to be resolved quickly.
And our IFS 6.0 includes a very powerful tool set
to address those issues.
Thank you.
The first question is put by Thomas Sterling, please.
Again, thank you for a very interesting, informative talk.
Focusing on one of your areas of interest
in storage area networks, the analysis of very large
data sets is becoming increasingly large challenge
both as systems grow and as problem sets grow.
Could you identify what Peculogic considers to be
perhaps the greatest challenge they face technically
to the evolution of scale and capability
of feature storage area networks?
So when you say storage area networks, Thomas,
are you referring to traditional storage area networking
or in the context of, or in some other context?
Traditional context, untraditional solutions.
So unfortunately, my main focus is on high performance
computing and I'm not really focused on the data set
efficiency with the storage area networking side of things.
So my apologies, but that's a question
that I don't feel comfortable addressing.
Do I get a second chance?
Sure.
All right.
Then focusing on high performance computing
and future networking challenges,
there's increasing interest and attention
in the broad area of message driven computation,
active messages, and so forth.
How do you see that trend being supported
by the kind of technologies you're discussing?
Right, so in our current generation of technology,
we don't have any direct support for,
as you mentioned, active messaging.
We are actually doing a significant amount of research
into one side of communication paradigms,
which we think are gonna become increasingly important
for computing paradigms,
especially as the complexity of the problems
continues to escalate.
And so in our next generation of solutions,
specific focus for handling the so-called one-sided paradigm
will be of utmost importance.
Okay, Gerhard, please.
Okay, thank you.
For a very interesting talk,
you did focus a lot on software.
However, I've got a question concerning hardware.
InfiniBand did a great job to go far beyond ethernet.
However, if we look at the balance of peak performance
of modern compute nodes,
as compared to the network bandwidth,
we are now still back at the time, eight years ago,
of gigabit clusters.
And I have the feeling that processor performance
increases much faster than network capabilities.
Can you give us hope that this gap
at least doesn't widen in the same speed
as it did in the last eight years,
or do we have to accept that we will have
loosely coupled single-node systems in the future?
That's a great question.
You know, I certainly agree with the observation.
As we try to move to exascale computing,
one of the observations that I think is inescapable
is that certainly the node performance,
individual node performance is growing quite rapidly,
but not, unfortunately, at the exponential scale
that would be required to actually achieve
the goals of exascale computing
in the next seven to 10 years.
And so the upshoot of all that
is that that basically means that the clusters
have to continue to grow in size.
And of course, to grow in size
and to maintain efficiency and scalability
requires, as your question implies,
an ever-increasing capability in the network,
in the high-performance computing network.
We have a roadmap that's pretty clear
what that roadmap is right now.
We have two next-generation technologies, FDR and EDR.
They are substantially higher in bandwidth,
but of course, bandwidth is only one small dimension
of all the things that can make a network
far more efficient.
I would say that the really important items
would be to make sure that high messaging rate is achieved,
very, very low latency at scale is achieved.
And then on top of that...
Right, so message rate, low latency,
and support for the single-sided communication paradigms,
I think, together with offloading
of some of the NPI processing,
together can make the network efficient enough
to achieve the next-generation goals.
Thank you, Phil Murphy, once again.
Our next speaker is Viktor Sovietov,
Chief Architect and Business Development Director
from TE Platforms Group in the Ukraine.
Hello, everyone, nice to meet you here.
Now I have to speak to my colleague, Igor Zakharov,
and I will join him with answering questions, thank you.
So my name is Igor Zakharov, I am from TE Platforms,
and as Viktor said, I will be vocalizing this message.
Viktor is our Chief Engineer from Massive Computing.
He invented it all, so that's why he will answer
all the questions, and I'm just saying,
what is the message, our initial message?
This?
No, that was the switch off button.
While we solve the technical problems,
I will tell you what we are going to tell you.
And this is about scalability of the operating system
into the petascale and exascale.
This is to go forward, thank you.
So we base our message to you on this wonderful experience
of having delivered half a petaflop system
at the Moscow State University.
This is number 13 on the latest top 500 list,
and this is where we have actually employed
and used some of the techniques
I am going to talk to you about.
This half a petaflop system consists of 5,000 nodes.
Now imagine if you go to more performance
and everybody we are heading towards the exascale,
how many nodes you are going to employ?
We are talking about 10,000s, 100,000s,
or millions of nodes.
Of course it will depend on what is inside the node,
how many cores and all that, but you understand
that it's not a single node anymore,
it is 100,000s of nodes you have to manage.
And what challenges do we actually face,
do we have to address with the operating system
when we do that?
Now, I am like just vocalizing, this is the chief,
and he's like Moses and I'm like an Aaron,
and I'm just saying this is the first of the 10 plays
which is going to happen to you when you go to that scale.
Now, let me, there are many of them, there are 10, okay?
Here are nine, and I want you to focus on the first one,
just to start with.
Now, reliability of the system.
Do you think that you can keep all the nodes
and all the bits and pieces available
and running at the same time, all the time?
We know that at that level, the failure rate,
just your statistical sort of movement will be high.
And so, we are facing a very sort of flexible resource
which we have to manage.
We better actually acknowledges as a fact,
we don't have a stable hardware all the time.
When we plan for an operating system which manages it all,
we have to admit, we have to care about
inherent unreliability of the system, okay?
And then, if we go one by one through all those,
you'll understand that there are challenges everywhere,
and I understand that my distinguished colleagues
try to solve them in one way or another.
And everybody will come to you and say,
okay, this is solution for this, this is solution for that.
But then, the last play which I'm going to display now
on the screen, does it scale?
Because this is what we have to face after all
with all the tools, techniques,
and everything which we apply.
Can we scale to this better access scale level
which we actually, where we want to go with our installations?
Now, we want to come up with a definition of the solution.
So, patience, right?
It takes some time to program and to go there.
Now, let me tell you first what is not a solution.
We are, we know all about traditional,
by now, traditional operating system like Linux, Windows,
and in my past it's Iris and everything you know very well.
Now, I must tell you, this is actually
not the solution we are looking at
to solve the management problem.
For us, if we consider the level of 10,000,
100,000 of nodes, it becomes sort of a microcode
we employ on a kind of node
to pass control to the application.
We are not considering that at all
in our best for solving the possibility
to run a system at petaflop and exaflop scale.
We want a new thing, we want a cluster operating system.
It is a management which allows us
to handle the resource at large
with all its components being computational,
being infrastructure network, everything.
Now, how do we do that?
First of all, we must know what is going on there.
That's the first step.
If you blindfold yourself and you go in,
you will, you are bound to hit the walls.
We must get information from the system.
Now, currently, the information is heavy weight.
What we need to do is very lightweight,
real time information from the system
at the level of about 100 per second,
100 parameters per second.
That's sort of the average we found
which is necessary to control the systems
at that scale out there.
Now, you have to give control
to the operating system about which we are talking,
that operating system to handle things for you.
If there is a failure on the cooling system,
it must be able to switch down the components
which depend on it to prevent the meltdown, right?
So, we are talking about real management,
real operation, real time operation
based on the information gathered at real time.
And what you want to know as a user,
how do you run actually your applications there?
So, we want to have an integrated resource management
which would allow the applications
to start on the nodes which are available
and see them completing
and delivering resource to the end user, right?
So, this is actually, we are getting there.
We are getting to this solution
to the problem of running the resource at large.
Now, I can tell you after five years of work,
we have managed to deliver an operating system
which does scale to better and extra floor dimensions.
How is it done?
First of all, we start with here, architecture, right?
No single point of failure of the components
which belong to the management network, okay?
Nodes can go up and down,
infrastructure elements can go up and down,
but we take a very careful consideration
at the elements which are there to measure things
and those elements which are there to manage things.
So, that's the first thing.
How to make it scalable?
Well, make it a three level information flow, okay?
We put elements, all redundant,
all no single point of failure,
which cache information for you or for the system, right?
To make it scalable and build up to any dimensions we want
because at each level, there is some processing
of the information which we gather.
And this is when I'm not talking about applications, right?
I'm talking about just operating system handling that all.
Non-intrusive data collection,
you don't want your huge computer to stand there
and consume electricity and just be used
to collect data about the computer.
You want to run applications.
So, lightweight data collection and at the right rate
to be able to manage events in the system
is key to it all and we actually made it.
We made it by rewriting all the necessary services
which go into this.
So, over the years, we as a community,
we have accumulated a lot of tools to deal with things, right?
In the Linux operating system, you have this and that
and you have great things like DHCP
which we use also at this conference
just to distribute the IP addresses.
No, this thing doesn't scale, but what scales?
Now, that we have rewritten.
What matters for the services,
we have rewritten them all and this is why I say
that was a great effort to do.
And at the lowest level of it all,
we have put in a database which keeps on the configuration
of the system which is always available no matter what, right?
And this is the basis sort of for all,
everything we do inside the system.
Now, at this time, I am not just talking about eventualities
and what we are going to do in the future.
Of course, we continue developing this.
Of course, there is a lot of work.
The people are working all the time.
We invent things.
We go to possibilities of life migration of work.
Now, imagine that you have a note about to fail
but we have real-time information flow.
We can measure.
We know it's going to fail.
Now, let's migrate work from that note to a different place.
That's what we are about to complete now.
But at the same time, we have a true product to look at.
There is a Clustrics operating system.
Parts of it are Clustrics watch
which we can employ on any hardware, on your clusters.
We can go and actually do it with any hardware round.
And Clustrics save which allows us to switch systems
off in case of predictable events which can happen there.
So this is all available for your clusters today.
And to conclude, the Clustrics operating system is here.
It's scalable.
It is a reliable next-generation operating system
for petaflop and exaflop computing.
Thank you very much.
Spasiba, for your talk and for being in time.
And the first question comes by Happy.
Thank you, Victor.
Listening to your talk, first time I thought
new operating system and you correctly opened your talk
by reminding us of our nightmare
as a system administrator, all the things that you said.
But you didn't seem to be bringing a solution to me.
A new operating system now,
how do we deal with interoperability?
Are you going to be an island as a T platforms
only producing your systems
and we should dump our old systems?
And I see that you also mentioned monitoring system.
And I still don't understand what new thing do you bring
from what we already know in the market like MoiApp.
So I've answered the first question.
Suppose you develop your application under SUSE 10.1.
And this is your requirement for your application.
Now, when you submit the job, you tell us,
okay, for my application on 1,500 nodes,
I need SUSE version 10.1.
And we actually install it
because we have dynamic provisioning on those nodes
which resource manager has selected.
We install the operating system you need
and you run there, fine.
At the same time, your colleague may be developing
under Windows and in another 900 nodes,
based on his requirement,
we just install that operating system on the nodes
and run his job.
So pair system, we are not looking at those Windows, Linux,
any type of combination of versions
as on the operating system.
For us, it's a microcode.
You just use, you happen to use a new application.
That's it.
You see, so we do solve for you the possibility
to run according to your requirement
on the petaflop scale machine and hexascale machine.
I don't care.
Do you agree with me here?
On the monitoring side,
this is a much more technical question
and I will pass to my colleague, if you don't mind.
Yeah, about monitoring.
Imagine then we introduce a narrow system for the cluster
because we speak about real-time monitoring.
It's critical for dense systems introduced nowadays
because as Igor said, we have to manage heat excursion
larger than 60 kilowatts of heat per rack
and we can't monitor such kind of system every minute.
We have to look our metrics each second at least
just to deliver the good viability of the system.
Next question by Donna Crawford.
Thank you.
We've also written our own cluster
high availability operating system or chaos software.
So I'm interested to do some comparisons.
But in particular, in our experience,
large heterogeneous systems are susceptible to chatter
or jitter and getting out of sync
with latency across the nodes.
And this noise is a significant source
of performance problems.
And I wonder how you suggest jitter be controlled
in a tightly coupled heterogeneous environment.
Okay, thank you very much.
That's really...
So there were two parts to this question,
two parts with my answer.
Hopefully I can make it in this time.
To start with, the operating system itself, you see,
it doesn't have to take care
of the performance of the application.
You're talking about performance of the application
which suffers from the clock jitter on the processors.
So you can go and actually,
like we do with steep platform solution,
where you can employ a special network
which synchronizes clocks
and this reduces the clock's jitter.
So this is a hardware solution, but you have something else.
But with having a real time monitoring system,
you can implement adaptive schemes
to manage a large set of nodes.
So if you have not, global time synchronization
in your cluster, it's still possible to write some rules
for monitoring system to try to manage the situation
each second.
Thank you very much again, the dual speakers.
Our next speaker.
Our next speaker is Dr. Motori Okuda,
his executive architect technical computing solutions unit
Fujitsu, Japan.
Please start your talk.
Okay, thank you, Chairman.
So, talking about high energy efficient
for classical computing, we should think about
not only power consumption of the system,
but also other important issue, such as efficiency,
not power, performance efficiency,
and also high productivity.
That means how can we reduce the burden
of application implementation,
or high reliability and availability of the system
and the flexibility and easy operation.
So Fujitsu are now developing several technology
for energy efficient for classical computing.
So first one is the new CPUs.
It's a HPC enhanced spark v9 architecture based CPU
with high performance, low power consumption.
And also this chip has a capability
of high efficient rating between core.
And new interconnect, new innovative interconnect
with 6D dimensional mesh and torus network.
And we adopt direct water cooling technology
for CPUs and ICC.
And other software are based on open software
with developing operating system
and software tools and file system.
So these technologies apply to the Japan's
next generation supercomputer project,
which targeting impact and petaflops.
Today I'd like to focus in the CPUs
and the interconnect and mathematical library
as a software environment.
So CPUs are spark 64 8FX processor.
There are several HPC enhancement in these CPUs.
It's eight core CPU and with SIMD extension.
And also the floating point register
is extended to 32 to 256.
And there are inter-core hardware
synchronization capability to realize
the high efficient rating between cores.
And user controllable cache is one of the distinguished
feature of the new chips.
And the performance is 128 gigaflops
and the power consumption is only 58 watt.
So right-hand side upper graph show the history
of performance and power consumption of spark 64 families.
And power consumption is almost half
and comparing from previous one indicated by red bar.
And performance gain is almost three times improvement
comparing with previous chips.
And we adopt direct auto-encoding technology
to achieve the low power consumption
and the high reliability.
And these chips are also designed
by same design concept as mainframe chips.
So achieve a very, very high reliability.
Right-hand side show the chip photo
or area covered by loss functions.
So how are these HPC enhancement affected?
This is a preliminary major data of new chips.
First one is SIMD.
Using the SIMD instruction set from 1.6 to 1.8 performance,
improvements are achieved in the sample groups.
And size of the floating register
is a very important issue for compiler optimization.
So use right-hand side graph compared
to 32 registers performance and 256 register performance.
In this case, almost 1.67 performance improvements are achieved.
In this case, it's a very complex loop.
So user can control the cache by compiler directly.
So cache divided is a two sector.
One is a sector zero in the normal cache
and the sector one is a pseudo-memory cache.
So the user can assign or locate some specific data
to the sector one to avoid data replacement.
On the right-hand side graph shows some example
of this performance gain.
So let me explain a new interconnect.
The each computing node has 10 links
and each links have a five gigabytes per second
bidirectional performance.
So total looped performance of this network
for each node is 100 gigabytes per second,
very high value.
The topology of the new interconnect
is six dimensional toras mesh in the physically.
But from the user's point of view, application point of view,
it is a logical, simple 3D toras network.
So let me explain how this 6D toras mesh network configure
and how they work.
So basic unit of this interconnect is the node group unit
and consists of 12 nodes linked by two by three by two.
And this unit start as 3D dimensional.
On the right-hand side shows 3D connection of each node.
So how the logical 3D
topology are mapped in the physical six dimensional
toras network.
So in this case, six by nine by six logical 3D toras network
are mapped on the three by three by three
node group unit cubic.
So right-hand side graph show the how they route.
It's a y-axis routing and this is the nine rings.
And this is a x-axis, next one the y and z-axis.
So the logical 3D toras network mapped on physical
six dimensional toras mesh network.
So why six dimensional?
So there are three major advantage of this network.
One is high performance and second one is operability.
So comparing from the traditional
conventional toras network, new interconnect
can be assigned several jobs with that interference.
And also the average hope count is almost half
of that of conventional toras network.
And third one is the fault tolerance.
So comparing with conventional toras network,
a new interconnect network can be configured
3D toras network if some node cause failure.
And if the system operating in the redundant mode,
that mean we assign the redundant node in advance,
we can assign the same size of the toras network
if with my node, one node cause fail.
So the performance and the operability
and fault tolerance is the advantage point
of the new interconnect.
So, Fujitsu provides several HPC solutions
for petascale computing but the question
how to reduce the burden of application implementation
is still open question.
So how to utilize merge core CPUs
and how to reduce the burden of two stage parallelization.
That mean MPI threads threading between core.
So mathematical library which optimize the merge core CPU
and hybrid programming model may be one candidate solution
for this question.
So Fujitsu I would like to provide this library
with collaboration and contribution
with HPC communities and as a open source.
So Fujitsu libraries of Europe in located in London
will be coordinate this activity in EU area.
So as a conclusion with collaboration and contribution
to HPC society, Fujitsu committed to provide
petascale solution and also targeting exascale
and user centric and application centric computing.
Thank you.
Thank you Dr. Okuda and Gerhard Welland
will start the questions.
Thanks a lot for your interesting talk
and I appreciate your way and strongly support your way
to build up such a system from scratch
with not having a history of 30 years
of binary compatibility or programming model
which we don't know in high performance computing.
However, will it be at once in the lifetime event
or is there a plan, a roadmap to commercialize this solution
in the cluster market with 100 nodes, 1000 nodes,
10,000 nodes or is it just a petascale, exascale project?
So of course Fujitsu are providing several HPC solution.
Today I introduce some high end, top end solution
using Fujitsu property, CPU and interconnect
and some operating system.
But in the lower end we provide some several
Ix86 based solution.
So from the user's point of view,
if we provide some hybrid programming model,
it's I mean MPI plus threading between core,
user can easily to scaling up in the source code level.
So this is our solution for a wide range of HPC customers.
Okay, Thomas Sterling please.
So the world has watched with interest
the drama in Japan with respect to exascale
and I can say without fear of contradiction
that Fujitsu has emerged as the hero of this
and very pleased to see Fujitsu take on the challenge
of trans-petaflop scale computing
with its continued history of excellence
in engineering and design.
I have a slightly strange question to ask
but it is relevant to the class of machine
that you're looking at and it has to do
with the user interface to the supercomputer.
It is increasingly of value in many cases
to move from a batch mode of operation
to a dynamic directly interactive steering mode
for large complex applications.
How do you see that interface requirement
reflected in the design of your system
hardware and software architecture?
In the new system is a completely
a kind of batch based system
because such kind of interactive system
will cause some kind of what to say.
What to say so interference between several jobs
and of course in IO system.
So in that case we need some jump for operating system
and also some kind of middleware
which support direct memory access
for the jobs are running or such kind of tools
visualizing the system result.
For example, the Fujitsu customer
develop some special program
which directly access the memory
which a program are running.
So it is very useful the user can access the program
without modification or debugging tool.
So in such kind of special tool
maybe essential for this kind of huge memory
and the huge computing powers machine
without application modification is important I think.
Thank you Dr. Kuda.
Our next speaker is Dr. Donald Grice.
Distinguished Engineer Technical Executive,
IBM Systems and Technology Group, IBM USA.
Donald.
Good afternoon, thanks for inviting me back again.
Today I'm gonna talk to you about distributed
RAID groups and disk failures.
In previous talks I've told you about some
inflection points that we've hit
and what our research groups were doing.
We hit an inflection point where leakage current
was equal to what we were doing with computation
and so I told you some about what we do
with power management and things.
Then we ran into core frequency plateauing
and I told you what we were doing about scaling.
We seem to be now at a new inflection point
in terms of the number of disks that we think we need
to run the file system.
And so our research group has taken on a challenge
of trying to figure out how we scale up
so you can get the throughput that you need
and cope with the disk failure rates
that we're gonna be seeing at that kind of scale.
So I'll take you through some of the high level points
of what we're trying to do and how we're accomplishing this.
We're set to deploy it on our big blue waters system
so it's in operation.
And so I'll start with some of the fundamentals.
I don't think this will be new to anybody, right?
But all of the big systems are being used
for large scale problems and the large scale problems
have large scale data sets associated with them, right?
And so we keep putting continuous pressure
on the bandwidth out to the IO system
because as I put in red, right,
anytime you spend just doing IO you're not doing computing
and so your efficiency went down.
So we get continuous pressure to make sure
the file system performance keeps up
with the peak performance of the system
and that poses interesting scaling issues, right?
So that's the fundamental reason we have this pressure,
right, and so what we have as a set of requirements
for instance on our big HPCS or PERC system
is the need to do six terabytes a second of file IO,
right, into 100 petabyte file system.
And so six terabytes, as one of the research guys said,
that's about the bandwidth of the entire internet
and so we are doing that out to our file system
on every machine we put together, right?
That's a lot of disks and a lot of bandwidth in the cables.
We also need a tera set of files, right?
So that's a petabyte just of metadata.
So we are dealing with all this striping
and we're dealing with an increase in the number of disks
so that the mean time to data loss is about two months
if we only did this with RAID 5.
We can get 56 years with RAID 6
and it's 130 million years
if we go to the next level of RAID, right?
So we were trying to figure out what we do here
and one of the really critical points about the failures,
right, is now at the bottom.
That in this parallel file system,
we wanna make sure that we're getting this bandwidth
out to the application
which means you have to be running
all the disks in parallel.
Now, just like the Jitter problem,
if one of the RAID groups is being rebuilt,
the entire system slows down to the slowest group, right?
And so what we are trying to figure out
is how to cope with the rebuild time, fundamentally, right?
And with the mean time to data loss.
And the way we did this because of the,
well, so let me, one more point to bring up
in terms of trends, right?
So what we have is the top 500 list,
clearly we know the trend there, it's exponential.
We're gonna hit a exo-flop in 2018
and it's gonna be a thousand times
every 10 or 11 years after that.
So that's what the performance is doing.
That means that's what the disk bandwidth has to do also.
But the individual disk bandwidth trends,
just like the core frequencies are bending over, right?
And so to get the kinds of performance you need,
you're gonna see a big uptick
in the number of disks that you need.
And the pressure for cost effectiveness
is driving us to commodity parts
that actually have less reliability over time
than we currently see with some of the higher price spread.
So the failure rates are growing faster actually
than the disk rates.
And so what do we do with that, right?
And the way we are gonna approach this
is with a software declustered rate approach, right?
The big systems that we build,
we already have IO nodes
that are actually what's in contact with the file system.
And they then interact with the compute nodes.
So what we are going to do is actually put
all of the error checking and the rate checking
in software on those IO nodes, right?
And so we now have the flexibility
of deciding how much rate you need
based on the size of your system, right?
So as I said, with RAID 5,
it'd be about four days to data loss
or 100 years or 130 million years.
So you can pick how often you'd like to lose your data.
That's really just a matter of
how much disk you need to put out there, right?
In addition to that,
because we're reading it into the IO nodes,
we can actually put CRC checks on the data on the disk
so that the individual streams off the disk are checked
and then the data across the RAID group is checked
so that even with the volume of disks we have
and the silent or not so silent failure rates,
because we're doing this in the IO nodes,
we can make it as robust as we need to
and you can increase the robustness over time
or over system size, right?
So fundamentally, we're going away
from a hardware-controlled set of checks
to software-controlled set of checks
because the IO nodes have plenty of horsepower
to deal with doing this checking.
So you add a little bit of latency,
but you don't actually impact the bandwidth, right?
And so it looks like the right way to go forward.
It's a fairly simple approach
with a standard RAID group,
you would have the checking code in one of the disks.
Now to read in the data, to rebuild it if a disk fails,
you fundamentally have to read each of the other disks, right?
So the time to read it is actually
the time to read one disk.
If you stripe this thing with declustered RAID,
what we're doing is taking the RAID groups
and pseudo-randomly spreading the pieces
over all the disks in the system
or as much as it's practical in a declustered RAID group,
right, so you can pick that too.
So now you have the checking pieces for your dead disk
spread over all of the disks in this cluster.
And so the time it takes to read in the data
is now reduced by the ratio
of the number of disks you have striped
compared to the RAID group that you had striped, right?
So you have control over now
how much time you want to spend rebuilding.
So this is the read part of it, right?
And so it's pretty clear that
since you're doing it in software,
you can actually bring this in,
stripe it any way you'd like.
But you also then have to repair what you had
and put it on the spare disks, right?
So with a conventional RAID system,
you would have a spare drive.
And so you read in the data,
like we just discussed on the previous slide,
and then you write it on your spare drive.
So now on the right side,
you're limited by the performance of a single disk again.
So what we'll do is we'll actually put spare space
on all the disks and not actually have spare disks.
So you can see this white space
that's equivalent to that spare drive
is now spread over this whole declustered system.
So when you want to write back the repaired data,
you can actually use the bandwidth
of your entire set of disks as well, right?
So you've improved the write back time by a factor of N.
And that means that your disruption of your bandwidth
of real data coming back in, right,
is reduced by a factor of N.
The other secondary piece that's not quite as important now
but it will as the number of disks grow
is the actual mean time to data loss gets better
because you don't have a vulnerable RAID group
waiting for something else to fail, right?
You've reduced the vulnerability time
of this group with a failed disk
because you've rebuilt it quickly, right?
And so you actually take advantage of this in the jitter
and in the reliability and you can put end to end
on the disk.
So this looks like a way for us to cover a magnitude
of all kinds of issues, right?
So in conclusion, large number of disks, right?
You have to get across all these guys
to get your bandwidth.
It's just like the explosion of a number of cores
only it's our four file system guys that are doing this.
Our reliability trends are gonna make it worse.
So this distributed RAID looks like it can actually
reduce the impact of the failed rates
and control the amount of time
we're gonna spend in rebuild.
And the CPU cycles on the reading node
doesn't actually impact the performance
and certainly not as much as you would see
if you didn't fix the rebuild time.
So we see it as a win across the board.
Thank you for your talk.
First question by Donna Crawford.
Good afternoon, Don.
I applaud IBM's continued leadership
in high performance computing.
And this looks actually very interesting.
And so it somewhat relates to my question.
As you know, Learnmore has chosen Lustre
over GPS of GPFS for all of our systems except purple.
Oracle's position on Lustre notwithstanding,
what are you gonna do to convince me
to use GPFS in the future?
Even or especially if my next system
is not an IBM system?
So I guess I could convince you
because the last machine I helped deliver was purple
and you use GPFS then, right?
So it's an interesting question.
This is a whole organic community issue.
And we could certainly have a discussion
on the pros and cons of the file systems.
I think this actually in addition to-
And dollars too.
Oh.
My problems.
So in addition to the robustness, I think,
of the performance and the attributes of the file system,
I think this kind of thing is gonna be required
actually for any file system, no matter whose it is,
to actually make it, I mean, when you look at the number
of disks that are in these things,
you have to do something like this, I think.
Okay, next question by Happy.
Okay, thanks, Don.
My concern is similar to Donna's concern.
I see that you've got a product
that you consider rate of the challenges
that will be coming with periscale computing,
but I sat and asked myself,
what percentage of storage will I be able to get
with my budget if now I'm going to look
at this software-controlled rate?
Ah, so right, so this actually addresses that problem
because, yes, the amount of the budget
is increasingly going into storage
and increasing even faster going into power, right?
I mean, so we're trying to get all of those down
so you can spend your time buying compute cycles.
But what's important here is that a big piece
of the cost is actually in the RAID controller,
so you don't need the RAID controller,
and you can go to less expensive disk technology
because the disks themselves don't have to be as robust.
Right, so we didn't actually add anything
in terms of cost to the disk subsystem.
We actually took away the cost
and gave you a more robust failure coverage.
Okay, thank you again, Don.
Now, last but of course not least,
we will have the presentation of our friend,
Dr. Steve Scott.
He is senior vice president
and chief technology officer of Cray USA.
Steve, please.
Thanks, Hans.
Great to be here again.
I thought I would try to take this 10 minutes
and give you a little bit of an update
on what's been going on at Cray
since we were last here at ISC in 09.
As the chart says, it's been a pretty busy time.
These are a sampling of some of the recent orders
we've had for large Cray systems
shipping all around the world,
and I'm happy to say that we're continuing
to grow our presence here in Europe.
We now have three European systems from Cray
on the top 30 list of the top 500,
and I guess we just announced this morning
a new contract with a new customer,
KTH University in Sweden.
Of course, when we start thinking about top 500 lists,
I can't fail to mention that since we were last here,
the Jaguar system at Oak Ridge National Labs
took the number one spot in the top 500 lists.
They actually have two systems at Oak Ridge
now above a petaflop in the top five,
but I'm not gonna dissect the top 500 lists,
and I'm not going to talk about efficiency on LIMPAC
because I don't care about efficiency on LIMPAC.
What I care a lot about is efficiency on real applications,
and I'm particularly proud of the fact that
when they installed Jaguar about a year and a half ago,
within one week of porting applications out of the machine,
eight different teams ported their applications,
scaled them up, and set new world records,
and there are currently three scientific apps
that are sustaining over a petaflop.
This has been true for some time now,
and I think it's the only machine
that has scientific apps running at over a petaflop,
and the most efficient application
is actually not LIMPAC.
It's one of the scientific apps,
so I think that's a testament to the machine.
This last year, we launched the XT6
and its little brother, the XT6M,
providing about 2x the compute density
over the current XT5, which is instantiated with Jaguar,
over three times the memory bandwidth,
so that was kind of a big event.
This is the XT6 compute blade.
It has eight of the new AMD Manicore processors.
This is the Optaron 6100 series.
They've increased the memory bandwidth,
so it's gone from DDR2 to DDR3,
and up to eight channels of DDR3 per dual socket node.
It's plug-compatible, so it fits into the same infrastructure
like we've always done,
being able to upgrade machines in place.
It's currently shipping with our C-star 2-plus network,
but just early next quarter,
we'll start shipping the new Gemini Interconnect,
and then next year, it will be socket-upgradable
on the same compute blades
with the new Interlogo series processors from AMD.
This is a picture of the Optaron 6100 dual socket node,
and you can see that it's a very tightly connected,
very beefy two socket SMP.
Eight channels of memory bandwidth,
up to 24 megabytes of cache,
over 200 gigaflops of computational power,
and they're all fully interconnected.
The sockets are actually two chip MCMs,
and there's a full, all-tall Interconnect
using hyper-transport among them
for very nice, scalable SMP.
They've done some nice things
in the cache coherence protocol as well to make that scale.
The XT5 was actually the greenest high-end supercomputer
around x86 supercomputer,
at over 250 megaflops per watt on Linpak,
and again, the real story
is the sustained application power efficiency,
where the story looks even better,
and then going from the XT5 to the XT6 compute blade,
that goes up by about 35%,
so very, very power-efficient computing.
Another thing that we recently introduced
just earlier this spring is our third generation
of the Cray Linux environment.
Cray has done a tremendous amount of work
to squeeze the noise out of Linux
and make a Linux kernel that scales very effectively
to very, very large system sizes, getting rid of OS jitter.
What we've recently added,
which is tremendously exciting to us,
and I think especially interesting at smaller scale,
but also in big systems,
is what we call cluster compatibility mode.
You can now dynamically load in more OS functionality,
including a full cluster-compatible version of Linux
with each application,
so you can have a machine running at extreme scale,
and then if you want to run some ISV applications,
shrink-wrap straight out of the box,
you can run them,
and the full OS functionality gets loaded dynamically
with the job and torn down after the job,
so very, very nice productivity tool
for the system administrators.
We were also happy just one week ago today, literally,
at our user group in Edinburgh, Scotland,
to reveal the new XE6.
This is a pretty big deal for Cray
because it's the first major step forward in interconnect
since we introduced the XT3 back in 2004.
The XE6 system is really designed
to improve message rates and other attributes.
I'll say a little bit more about it in a moment.
One of the things that was kind of nice
is that normally you try to line up a sale or two
that you can talk about at the time of launch.
We actually had, in advance,
over five petaflops of XE system orders
going to several large sites around the world,
so pretty exciting, pretty big event for Cray.
The XE6 can be upgraded.
You can upgrade an XT system to an XE system
simply by removing the mezzanine card
that has the network chips
and replacing it with the Gemini mezzanine card
that's shown here.
As I said, this will start shipping next quarter.
The topology remains a 3D torus,
so you can keep your same cabling
and the rest of the system infrastructure
and just swap in the card.
When you do that, you get a pretty dramatic improvement
in the network interface.
50 times improvement in the MPI message throughput rate
per node, dropping the latency
from about a factor of four from where it was.
The C-star interconnect that we have in the current systems
has relatively high unloaded latency,
very good latency at scale,
and this dramatically improves that in the Gemini system.
There's also hardware support for one-sided MPI
and for global address space,
so gets and puts in atomic memory operations
launched at very low overhead,
overlapping throughout the system.
Quite a bit of work has also gone into resiliency features.
We'll have a fully resilient MPI communication protocol
as well as a lot more hooks
for better diagnosing the machines.
Another announcement that we made just this spring
is the new Cray CX1000.
CX1000 is the big brother to our desk side CX platform.
It's a cabinet that you see here,
a standard 19-inch rack.
I'm pointing at the screen in front of me.
Standard 19-inch rack, you can't see this anyway.
A standard 19-inch rack with different sorts
of compute blades that can be placed into it.
The dash C option gives you a 7U chassis
that's got 18 dual socket Xeon blades
with an integrated Ethernet or Infini-band network.
Very tight integration for high throughput
or distributed memory capability processing.
Then there's a dash S, which is a one and a half U,
four socket SMP unit that could be scaled together
up to 16 sockets in a tightly coupled SMP
with 128 cores and a terabyte of memory.
And there's also a dash G,
which provides nine accelerated compute nodes
with a pair of Zions and a pair of Teslas in each one.
So with that, we now for the first time as Cray, Inc.
have a full product line offering
all the way from around $15,000 with the CX1 and CX1 IWS
all the way up in overlapping bounds
to a system as large enough
as your imagination will let you consider.
So a very nice situation for Cray to be in
for the first time in our 10-year history.
Now going from the small end up to the high end,
I wanna say a little bit about our thinking about the future.
I personally and many people on my team
and throughout our R&D organization
have been spending a lot of time thinking about Exascale.
This last December,
we launched an Exascale research initiative here in Europe
and that's focusing largely on applications
and their interface to the system.
The R&D group is spending a lot of time
thinking about the hardware technologies
and software technologies
that we'll need to get to an Exascale
and really concentrating on four different areas.
The first one is power.
We're working on efficient power and cooling infrastructures
to get a good power usage effectiveness rating.
We're working on accelerated processor architectures.
We're working on power efficient interconnects.
Concurrency, working very closely with the application teams.
It's a tremendous challenge to get applications scaled up
to the Exascale.
We're working on building latency tolerant architectures
and programming environments that can dynamically exploit
all the different types of parallelism concurrently
in applications in order to get at that concurrency.
The concurrency and the microarchitecture changes
due to power are both gonna complicate life for software.
And so we're spending over half of our R&D energy
on software and specifically working
on making the programming environment easier to use
through compilers, run times, tools, et cetera,
to help map that concurrency onto the hardware.
And lastly, resiliency.
This is perhaps the most daunting issue.
We're building our OS with redundant services
and robust communication mechanisms
to make the systems fully resilient.
And then we're working on making the application resiliency,
which is a community effort,
but there's also some interesting research going on within Cray.
So I'm here to place a stake in the ground
that we are going to continue our tradition
of being the first company to break through
a sustained, pass a sustained milestone
for the next factor of 1,000.
In 88, we had the first sustained application gigaflop
on the YMP.
In 98, the first sustained application teraflop
on the T3E.
In 2008, the first application sustained at a petaflop.
And in 2018, we're gonna have the first application
sustaining an exaflop on 10 to the million processors.
Thank you, Steve.
Thank you.
Thomas, please start your final question.
I'll start and finish it, Hans.
After that talk, I feel embarrassed
to even ask you a question.
Steve, you're one of the premier
supercomputer architects.
You've developed more than one breakthrough machine.
You have described some of the real challenges
and some of the directions that Craig wants to get.
But I'd like, you're talking about the stake in the ground,
but I'd like you to sharpen that stake just a little bit.
If you could think, forgive the cliche,
outside the box for a moment,
the machines over your career have changed
in their logic and their look.
What will you imagine that change continuing to be
in logic and look over the next 10 years or so?
Wow.
Well, I think in terms of the logic,
the responding to the power challenge
is the dominant change that's gonna happen
over the next 10 years.
We're going to have to do,
the standard multicore x86 path
isn't gonna get us where we need to go.
We heard a little bit from Kirk Scoggin
about some things they're doing.
The other processor vendors are doing the same.
We're gonna have to change the microarchitecture
so that most of our work is being done
by energy-optimized cores
rather than latency-optimized cores.
And that's gonna cause some pretty significant changes
to the execution model of the machines.
We're gonna have to pay a lot more attention to locality
and manage the way the data is placed
and the way data is moved.
And that's gonna put more burden on the compiler
and the runtime systems.
So pretty dramatic logical changes.
In terms of look, it'll probably look like
a whole bunch of refrigerators lined up in a row
with a really nice mural on the front.
We used to build some really pretty machines in the past
and it's all about parallelism today and in the future.
So I don't see a big change there.
And the very final question is asked by Donna, please.
I just wish I could buy one now.
Thank you very much.
You talked, of course, about power being a big problem
and you're working on addressing that.
Can you tell me what you are guesstimating
your ExaFLOP machine
and ExaFLOP sustained performance application?
What kind of power are you projecting that to be at?
And how many places in the world do you think
we can cite one of those kinds of systems?
Right, so it's interesting to note
that the largest data centers today
are on the order of 100 megawatts in power
and they're much larger than the largest HPC centers.
These are the big internet data,
internet cloud data centers.
So it's possible if you've got the economics to back it up
to build and field systems on the order of 100 megawatts,
no one in the HPC community really thinks
that that's a viable approach.
So the number that we are sort of targeting is,
can we get to an ExaFLOP and hold it to 20 megawatts?
That also happens to correspond
with the very aggressive power efficiency goals
that Bill Herrod in the DARPA UHPC program has set
of a 50 gigaflops per watt sustained benchmark,
but it's smaller scale.
So that's kind of what we're targeting.
At the 20 megawatt scale,
I think that there'll be a number of sites today.
You can think of the largest sites today
conceivably being able to do that.
It's getting more and more uncomfortable, no doubt.
Thank you very much, Steve.
This is the end of our hot seat session.
Just a couple of minutes later, we ended and then scheduled.
I would like to thank the nine speakers cordially
for their presentations and for answering the questions
by the four smart inquisitors.
I think especially the inquisitors.
I also thank my colleague,
Aunt Bode for doing the job again with me,
moderating the hot seat session.
And I thank you, the attendees here in the lecture hall,
but also the attendees in the internet.
I don't know how many, maybe not one billion, that's a lot.
We expect you a quarter past 5 p.m.
again here in the lecture hall.
There we have the Tuesday keynote address
by Professor Merkel.
He now lives in China and he will give
the highly interesting talk on how IT
and the internet revolution has changed the Chinese society.
This is especially important since I think
that China currently is doing a great job
in high performance computing.
Thank you, have a short break.