I'd like to introduce my inquisitors this afternoon
as Professor Frank Perrant on the left,
next to Cechton Guerra, Professor Cechton Guerra,
Dr. Feng Peng Ling and Professor Gabriele van Voigt.
And without any further ado,
I'd like to call the first speaker, we had a change.
The first speaker is Scott Misage from HP.
He is the R&D director for HPC technical program
and he's coming from the US.
Thank you, Eric.
So first, before we get started here,
I have to explain why exactly we did this change in speakers.
Many of you know Dr. Bethke here in the front row.
He's been speaking for HP in the hot seat
for I don't know, Frank, a long time,
10 years maybe or more.
And last year, Frank was finally indoctrinated
as an ISC fellow, I think one of five ISC fellows.
And I've told Frank before we started here
that when you start to get the word fellow next to your name,
that's maybe a sign that you're getting old
and that you should step a little away
and let somebody with a few more hairs on top of their head
come in and give the presentation.
So that's the basis for the change in direction here today.
So I'm gonna spend a few minutes talking to you
about HP's method or approach
for the high performance computing market.
And I'm gonna leave you with what I hope
are three key messages.
First, I'd like to get the point across
that HP understands high performance computing
all the way from the low rate systems
all the way up into the petascale.
And as a proof point for that,
I'm gonna spend a few moments talking to you
about a recent win that HP has announced
over the last couple of weeks together with NEC
and NVIDIA for a petascale class system
at Tokyo Institute of Technology called Subami2.
And the third point that I'll be making
as we go through the presentation
has to do with how HP is approaching the problems
of the next generation system,
the exascale class systems.
And I'll talk to you a little bit about the research
that we have underway on those systems.
So with no further ado, let's get on to the next slide.
So nice catchphrase here.
Daniel Golden who worked at NASA for 25 years
was the inventor of this phase.
I think this is the mission
for all high performance computing vendors.
In fact, I think this is a mission that our customers,
whether they be you guys as users
or your customers as application inventors and scientists,
this is the mission, better, faster, cheaper.
We have to work on all three of these things.
Many folks say you have to choose
two of these three things.
No, that doesn't work in high performance computing.
We have to be able to pursue all three of these things
and we have to deliver against all three of these.
And this is in fact what HP tries to do.
Next slide.
So what does that mean from terms of a system architecture?
These are really the hard problems
that any systems vendor like HP has to deliver
in order to be successful in the marketplace,
whether that be at the low end
or the extreme of the high end.
Memory and IO bandwidth just absolutely dominate.
I don't think there's anybody in the audience
that would disagree with that point.
And we understand this and we know how to build
this class of capability into our systems and our servers.
Of course, interconnect bandwidth is hugely important.
Bandwidth and latency, necessary to be able to deliver
application efficiency to our end customers.
Absolutely must have it.
Programming models, as we move into the multi-core
environment or Intel's new phrase,
the many core environment, we need to be able to adopt
our programming models to be able to deal with
the changes in the underlying processor and technologies.
Of course, one that we're always up against
is power and space constraints.
So we think these are the four major areas
that vendors have to really work on
in order to be successful in the marketplace.
So HP has a series of product infrastructures
that we have brought to the market over the last 10 years.
I'm not gonna spend a whole lot of time
on the pizza blocks class systems on the left,
DL class systems, many of you have those
in their designs today.
In the middle we have C class, most successful system
ever in the history of the top 500.
We have today on the top 500 nearly 170 C class systems.
This is what drove HP's presence in the top 500.
Unprecedented, this has never happened
in the history of the top 500 list.
Nobody has ever had that kind of success
with a single kind of system.
So this was kind of the proof point for HP
in the high performance computing market.
Last June at this show we introduced
a brand new class of servers called the SL,
stands for scalable line.
That class of servers is purpose built
for high performance computing.
What did we do?
We took the features and technologies
that we had in C class that was really flexible
and really good at being able to be a heterogeneous system
and we have narrowed that down
to be a more homogenous system.
And we know that customers are gonna be pumping out
rows and rows of the very same kind of machine.
We've taken that technology and built that into C class.
Or into SL, I'm sorry.
What does that really mean?
It means we've taken some things away.
So we certainly didn't take away performance.
We certainly didn't take away the betterness,
but we took away some of the cost.
So we've in fact made it cheaper.
And when I say we've made it cheaper,
I mean we've made it cheaper both to purchase
from a capital expenditure point of view
and we've made it cheaper
from an operational expenditure point of view.
We have designed incredible efficiency
into this SL class of machines
right from the bottom of the infrastructure
with power efficiency on the transforms,
power efficiency on the VRMs,
power efficiency on the disk drives,
the ability to control every quarter of a watt
that goes through that system.
That's very tightly integrated
into this SL class environment.
And that has made us very effective
in terms of power efficiency.
The SL class holds for nearly a year now
the top position on the spec power benchmark
around the world.
And many, many new systems have come in below it
trying to unseat it and haven't been able to.
So it was delivered to the marketplace last June
very highly efficient.
We have a next generation version of that
coming out later this summer
that's gonna again approve the power efficiency
and effectiveness of that class of servers.
So that SL design was really sort of the culmination
of taking the learnings that we had from C class
and applying this model of better, faster, cheaper.
Okay, so let me change gears now
and talk to you a little bit about HP's
petascale solution and more specifically
the example for Tokyo Institute of Technology.
Switching to the laser pointer here.
So this is a drawing of the system
that will be implemented in Tokyo later this summer.
It is of course a GPU based system.
So let me start down here in the lower left hand corner.
Talk a little bit about the nodes, okay?
So there are approximately 2,400 nodes in this configuration.
Intel West mirrors, okay?
Which are gonna be, each node is gonna have
three GPUs in it from NVIDIA.
The M2050 class GPUs.
And we're gonna put those into SL class systems.
So I just finished talking to you about the SL.
This will go into a new node from SL
that will be out later this summer.
And West mirror processors.
Each node is gonna have two QDR links coming out of it
for incredible bandwidth and performance
across the interconnect.
Dealing with that IO latency question that I talked about.
And each node is gonna have significant PCI bandwidth
to deal with the transfers between the processors
and the NVIDIA co-process, okay?
So we really designed this new node for SL
from the ground up to be highly successful for CPUs.
At the fabric level, we're of course gonna have
an Infiniband configuration.
As I mentioned, it's gonna be a dual rail configuration
to allow sustained application performance.
At the storage level, we're gonna have
a Lustre implementation.
Again, using Best-in-Class server, I'm sorry,
storage technology from our partner DDN
with the Lustre file system layered on top.
And we're gonna have another class of system
delivered from HP for home directory space, okay?
So we're really gonna be delivering
a fully integrated configuration from top to bottom.
Let me move gears now and talk to you a little bit
about the next generation of the XSCL class systems.
And this really requires that we develop
new fundamental technology from the ground up
in order to be successful.
I have two examples coming from HP Labs.
One is based, as it shows here,
on our silicon integrated photonics.
This is necessary in order to both scale
the power efficiency to even greater degree
than we can today, and to deal with the incredible
high bandwidths that we're gonna need
between all elements of the system.
Many folks have talked about the memory barrier
for a long, memory wall for a long, long time now.
And really to overcome that memory wall,
we gotta drive the bandwidth up.
And light is really the way we're gonna do that.
So we're developing fundamental technologies
that will allow us to do chip-to-chip interconnects,
board-to-board interconnects, and server-to-server
interconnects, or server-to-storage interconnects.
All three of those will be done by light in the future.
Fundamental building, new building block,
invented at HP, the fourth circuit element, if you will.
This was a theoretical element until a couple years ago
when a very senior researcher at HP Labs,
Dr. Stanley Williams, invented this
and not only invented the,
or delivered on the theoretical proof of the concept,
but in fact implemented the concept
in standard CMOS technology.
That fourth circuit element is called the MemRester,
and we really do think this is gonna revolutionize
the way that storage and memory technology
are gonna be implemented in exascale class systems.
So again, a fundamental building block.
Okay, so that's my 10-minute pitch.
Again, just the summary points.
HP understands the basic core things
that need to be done for high-performance computing.
We're demonstrating that in a petascale class system
at the Tokyo Institute of Technology,
delivered later this summer.
And we're working on the fundamental building blocks
that will allow us to scale that technology
up into the exascale class, okay?
Thank you very much.
49 seconds to go.
Thank you, Scott.
That's true.
The first question of the afternoon
is gonna come from Professor Bieran.
Sorry, one of the overall topics,
thanks for the presentation.
One of the overall topics we are facing today
and we will increasingly face in the future
is the question of energy efficiency
of these compute systems and compute centers.
You mentioned that the current systems you are delivering
and the upcoming systems are pretty energy efficient.
I do understand that this is not only based
on the way the air conditioning is running the center,
but also starting on the chip or on the board level
up to the higher levels.
What are your company's concept
for upcoming exascale systems
with respect to integrating energy efficiency
and the watts per flop or whatever concept
just into the design from the chip really
from the way all the new technologies you just mentioned
will come to life in the upcoming years?
Absolutely.
So I think the one way that the industry thinks
about this is by measuring the total efficiency
of the data center by taking on the numerator
the amount of power that's utilized total
in the data center, including everything
for power transfer and cooling as well,
and then dividing that by the amount of electricity
actually used for computing.
That term in the industry is called the PUE.
This is a term that HP pays a tremendous amount of attention
to now in our designs for service.
So we have a concept that's selling today
called the Performance Optimized Data Center.
This is a closed loop containerized data center,
if you will, that allows us to completely manage
the power and cooling of the entire ecosystem
in a very tightly coupled control system.
This will allow us to take a PUE efficiency measure,
if you will, that is pretty commonly around 2.0
in most enterprise data centers, maybe around 1.8
in a typical HPC environment
where we've paid more careful attention to energy.
We can lower down to about 1.2
in today's Performance Optimized Data Center, okay?
So a savings, if you will, of six euros
out of a significant 60% or so reduction.
With next generation designs, the kinds of designs
that will be necessary when we're
in the EXEC scale class system,
we're looking at introducing free air cooling
into that Performance Optimized Data Center.
We believe with that tight coupling sensors,
control systems, all the way from the component level
all the way up to the major cooling systems
in something like a Performance Optimized Data Center,
we can drive PUEs down to about 1.1, okay?
Getting below that level of efficiency,
going below 1.1, requires a really,
truly significant investment.
You are kind of approaching
the acetotic curve, if you will, okay?
There are technologies that you can go further, in fact.
Folks are looking at, you know,
fundamentally looking at water cooling
or floor inert or other inert cooling
directly on chips and on dyes, okay?
We really think that that's kind of crossing over
into that point where you're at the acetope of the curve
and you're trying to get just that little bit more
to drive yourself down below something like a 1.1.
In some cases, that may be necessary, that may be wise,
okay, but for the general case, probably not.
We're investing along those paths.
I would go on with this topic.
Do you work on software solutions as well
in order to reduce the power consumption you have?
Absolutely, so the question was,
do we work on software systems
in order to drive down the power consumption?
And the answer is absolutely yes, okay?
So HP has a couple of technologies in that space.
Can you say a little bit more than just yes?
Yes, I was fully intending.
I got a minute and 16 seconds.
I'm gonna use it all up, okay?
So yes, we do, we have a couple of technologies
that I think are really important.
One is called sieve sensors, okay?
So we have on a typical server today,
we have over 20 sensors that measure things like
air pressure, temperature, power utilization,
and we have a feedback system, if you will,
that deals with the integration of that sensor information
and tightly manages the power consumption.
An example of how we do that, okay?
Many of our servers have multiple power supplies.
Power supplies have an efficiency curve.
The further you are up to the 100% utilization,
the more efficient the power slide becomes.
The software technology, if you will,
it's actually firmware technology that we deal with,
that manages the supplies,
drawing down the amount of supplies that are running
when you're not consuming the full load on a server
so that you're running at the top end of that efficiency curve.
That's one of the tricks, if you will, that HP uses.
Let me give one more example,
because I think it's really, truly unique.
HP owns gates on Intel and AMD processors.
Nobody else owns gates on Intel and AMD processors
besides HP.
What do you think we use those gates for?
We use those gates to uniquely implement
controlled feedback systems on those processors
with HP's upper layer software technology
to manage power and cooling.
Nobody else can do that in the industry.
Why can we do that?
We consume more Intel processors
than anybody else in the industry by far.
So we have a very tight coupling
all the way down to the processor level.
I'm at zero. How do I do?
Right in time.
Thank you very much.
So our next speaker will be Keith Chatlin,
who is Principal Product Unity Manager at Microsoft USA.
And the title of his talk will be
The Future of Technical Computing at Microsoft.
In addition, I would like to ask the other speakers
to come here to the front row
to our so-called speaker cache, please.
Hello, ladies and gentlemen.
Thank you very much for having me here today.
I'm gonna go ahead and talk with you today
about programming models,
focused on domain specialists and developers
delivering solutions for HPC environments.
So about three weeks ago, Microsoft announced
its technical computing initiative,
which really consists of three things.
First was TC-focused tools and applications
that allowed domain specialists to go ahead
and develop and prototype models
in a set of easy-to-use tools
that were not normal development languages
that catered more to their personal needs
as scientists or domain specialists.
We are going to focus on allowing them
to go ahead and enable these applications very quickly,
and then go ahead and prototype them and test them,
first on the workstation, then scaling to HPC servers,
and then as the need arises,
also being able to go ahead and move these models
to off-premise systems in the cloud.
As part of this, we also know that some of these systems
will need to be recoded
in much lower-level development languages,
so go ahead and give them the power, performance,
and efficiency that they need
to get to answers very, very quickly.
So we're also focused on developing and delivering
a set of programming models, runtimes,
and tools for developers to go ahead and use
to do this in a quick and easy fashion.
For the rest of the talk, I'm gonna stay focused
on developers and talk a little bit
about what we've already shipped in Visual Studio 2010
to go ahead and aid developers in producing HPC-based,
our technical computing-based applications,
and I'll talk a little bit about the direction
that we're heading in the future.
There are really one premise, one primary mission
that we were trying to go ahead and enable for the developers
to go ahead and make it easier for them
to go ahead and express parallelism in their codes,
and then to go ahead and deliver those solutions quickly
to make them much more productive
than they've done in the past.
To do so, we produced a series of programming models
that are much higher-level abstraction
that makes it easier for them
to express parallelism in their code.
They interoperate with MPI
and allow developers to go ahead
and exploit the many core aspects of their systems.
In addition to this, we deliver tooling
that works at the same abstraction layer
as the programming models.
This basically means that developers,
instead of having to go ahead and fall back
to very low-level system details
to debug their applications,
are able to go ahead and work at the same abstraction level
to go ahead and do debug and profiling
and performance optimization of their application.
And the last thing is that through our models,
we actually provide mechanisms for developers
to make good choices.
With all low-level language,
there are many, many different ways to go ahead
and implement the same algorithm,
and many different ways to go ahead
and implement similar kinds of patterns.
We provide models in order to try
and help developers do so successfully
and to, in some ways, limit their choice a little bit
so they're making the right choices.
This is an example of the things
that we delivered in Visual Studio 2010.
At the top, you see, hold on for a moment,
let me get to the, there we go,
two sets of programming models that we've delivered.
One is called the parallel patterns library.
This is a task-based model that allows developers
to go ahead and easily take independently-schedulable
pieces of work and encapsulate them
and write them in their codes.
On top of those task primitives,
we actually build parallel loops
and also a full set of parallel template libraries.
So things like MapReduce, for example.
In addition, if people are familiar with Erlang,
we have a actor-based model that we went ahead
and produced as well.
This is called the agents, the asynchronous agents library.
With this, we provide developers a mechanism
for isolating particular pieces of code
to ensure that they're not sharing data
in ways that they should not be.
And this we do, we change state
or deal with state transitions through message passing,
a very lightweight message passing system on the node.
In addition to that, you can go ahead and use this model
to create networks, data flow, and control flow.
All of this sits on top of what we call
a concurrency runtime.
Concurrency runtime has two major pieces,
a scheduling system to go ahead and schedule
the concurrency that's been expressed by the developers
and a resource manager to understand
what kind of parallelism is available on the platform
and to effectively map the parallelism expressed in code
to the resources on the platform.
You can see here, also, we deliver a full set of tooling,
as I said, that works at the same abstraction layers
as the programming models.
We have a new parallel debugger,
and I'll show you some pictures of this.
And we also have a concurrency profiler
that developers can use for optimizing
these programming models in their codes.
The parallel patterns library,
we've been working with Intel on this
probably for the last two years.
We've been working very closely with them,
code designing the APIs.
The APIs for this are the same in Intel's TBB libraries.
Both the Intel TBB and OpenMP libraries
also work synergistically with our concurrency runtime
and the resource manager.
In addition, the Intel parallel studio
is cognizant of the programming models that we have
and interoperate, and you can use those for tooling as well.
So you might ask, why did we focus on this actor model
or this asynchronous agents library?
The reason was is because not all programming patterns
fit well into a task-based model.
Things such as pipelines, state machines,
producer-consumer patterns need the ability
to go ahead and create data flow networks
in order to go ahead and have them
optimize and work effectively.
The attributes of agents, as I said,
is encapsulation and isolation of data.
This, we can help developers prevent things
like race conditions in their code
because they actually cannot encounter them.
This lightweight messaging system to go ahead
and communicate state transitions between components
and we manage, the runtime manages
actually control flow and data flow.
So the benefits of agents, safer by design
because developers cannot get themselves into trouble.
States coordinated, not shared, so no race conditions.
And it's a very teachable and learnable programming model.
I wanted to show a little bit about our tooling.
Here you can see some of the windows,
new windows that we ship in our debugger.
This actually gives you a view of the kind of
concurrency that's being expressed.
So in this window here you can see
that there are several tasks that are running.
You can see the relationship between the tasks.
And by actually hovering over any of these,
it'll give you full information.
In addition, by clicking on any of these functions
in the task, you can actually go in tool directly to source
so that you can actually navigate from windows
and the tooling to the source
to go ahead and correct your problems.
In addition, we give you information
about the current status of tasks.
So for example, in this one,
you see that we have a deadlock condition.
The tools are also able to identify these conditions
and enumerate them in the visualizations.
In addition, we have a new profiling tool.
This is a picture of the profiling tool.
Here you actually see over time
what various threads are doing.
Threads in green are actually in execution mode.
In addition, we give you call stacks
to go ahead and tell you where exactly time is being used.
From these call stacks, again,
you can navigate back to source
so that you can easily go ahead and see, for example,
where the call stack is actually in your source
and what's taking time in particular situations.
As far as where we're going in the future,
I think the things that we deliver in Visual Studio 2010
have been a great first step in making it easier
for developers to express parallelism.
But still, there are several different models
that developers have to know.
For example, if they're writing a distributed app
and they want to go ahead and leverage parallelism,
multi-core parallelism at the same time,
they have to learn MPI, which is pretty difficult to learn,
and then they also have to go ahead
and learn other low-level primitives
for exploring many-core.
In the future, we're going to go ahead
and try and produce a more scalable model.
We have a model called Link Language Integrated Query.
Language Integrated Query was introduced
about four years ago in .NET.
Since that time, we've actually produced P-Link,
which is parallel language-integrated query.
This is a declarative syntax,
similar to SQL, for numerating over data.
We do automatic parallelization of this
underneath the covers.
In this way, developers don't need to know
anything about parallelism.
They don't need to know anything
about threads, MPI, or other things.
We can go ahead and just express the parallelism for them.
Microsoft has been working for the last two years,
and if you go to our research site,
you'll actually see something called Dryad Link.
Dryad Link also encapsulates P-Link.
This is a distributive version of this programming model.
The idea is that we're going to go ahead
and leverage this programming model
so that developers, whether they're building
client applications, or HPC applications,
or cloud applications, can go ahead
and quickly and easily enumerate over large data sets.
Thank you.
I'm out of time, and so the three messages
that we want to enable developers
to extract top performance through
programming models at scale,
that we want the tooling to go ahead
and work at the same level that the programming model does
so developers don't need to learn new things to debug,
and that we want a distributive runtime
that works from client to cluster to cloud.
That's it.
Thank you very much.
Thank you.
Questions will be by Fang Pang-Ling and Chek Dongara.
Thank you for the presentation.
I understand that the current approach is more horizontal.
That's what technical computing claimed.
But in the practical world, actually,
we require, if you want to really realize something,
accomplish some work, you need a vertical integration.
So then you will get involved,
also collaborate with, incorporate with some other APs.
So I would like to ask you how,
from this paradigm, all these tools,
how to incorporate with other APs
in a high performance fashion?
When you say APs, can you?
Application.
And so typically, this is just the development stack
that we've been talking about.
There's a full runtime stack.
In addition to the programming models,
there are things like the STL,
which are low-level libraries for expressing
particular patterns in code.
And then as we move beyond that,
each one of the domain specialties
typically have a various set of libraries
that are optimized for applications.
The focus is to go ahead and get these libraries
also based on the same programming models
and inter-operating it with the same models.
Therefore, you have this library layer
that actually provides the vertical integration
between the application and the programming model
and runtimes for the system.
I don't know if that helps.
Yeah.
Thank you.
Keith, thank you very much
for that nice presentation on technical computing.
My question is a little bit more
towards high-performance computing.
Sure.
Microsoft has been in the area
of high-performance computing from, I think, 2005.
I sort of remember Bill Gates giving an opening talk
at a supercomputing meeting,
sort of the kickoff for the HPHPC program within Microsoft.
If you take a look at the top 500 list today,
there are five machines which claim to use Windows
as the operating system, Microsoft Windows.
So my real question to you is
how much longer do you think Microsoft will be allowed to,
how much longer do you think you'll be allowed
to continue in high-performance computing
given the numbers really haven't taken off
the way perhaps people had hoped they would?
Well, I think if you're using top 500
as a measure of success, that's one measure you could use.
But there are many smaller HPHPC systems
that Microsoft actually is deploying in commercial space.
So where we may not have been as successful
in the academic space,
those who have enough money to invest
to go ahead and actually achieve top 500 performance,
we have been doing very, very well
in the commercial space with smaller configurations.
And so we are making a business out of HPHPC.
We are focusing more on the low-level systems.
We are focusing more on the commercial space
where they are writing new codes
rather than in the traditional academic space
where we have entrenched codes
that have been around for a very long time.
And through this mechanism,
I think we'll actually demonstrate
that we'll be able to go ahead and grow our share.
Thank you.
Thank you very much again.
Thank you.
Thank you very much again.
Thank you.
So I'd like to call the next speaker, Michael H. Jackson.
He's president and co-founder of Adaptive Computing.
And Norbert Eicher in the cache, please.
Welcome.
So Adaptive Computing.
Many of you may have known our company formally
as Cluster Resources.
Other people know us from our products
more than they know us from the company name.
Products such as Moab,
which is a workload management
and intelligent automation tool,
as well as technologies such as Torque Resource Manager,
Maui Scheduler, Gold Allocation Manager, and the like.
But today, what I want to talk about
is looking at some of the evolutions of the complexities
that are hitting high performance computing
and the organizational level issues
that we're trying to help solve with our technologies.
But before doing so,
I wanted to kind of get the picture
of who uses our technology.
We are, if you look off on the right,
that's actually last year's supercomputing
or international supercomputing conference.
And we had approximately 10 of the top 20 systems
we're running the Moab technology.
And when you look at industries,
very broad set of industries,
it's really not specific to one.
It's a very horizontally applicable solution.
So whether you're oil and gas,
whether you are government research,
or whether you are pharmaceutical,
it has a very good applicability of our technologies.
So we are typically used more towards
leadership class systems
and those that are trying to have
more advanced systems within their environment.
So part of that included having
the world's first petaflop system was managed.
That was Roadrunner at that time under Moab.
And then the world's first two petaflop system,
which is the Oak Ridge National Labs Jaguar system,
was also managed by our product Moab.
And we're currently working on five petaflop systems,
working in that design,
as well as even working towards the design
of 20 petaflop systems.
And so as you have the need of mass scale,
we certainly have been the historical partner
to provide the automation and workload intelligence
on top of those technologies.
So what is Moab?
Unified automation, workload management.
It is also considered to be the brain.
What is its role?
At first our technology began as a simulator.
A simulator that was trying to decide
what is the best decision we can do within your environment.
But then people said, well don't just simulate it,
make it do it.
Have this tool implement this best decision
to get me my better performance.
And so then it moved off to application optimized
because it was with my heterogeneous set of applications
and resources, help me have each application
reach its best performance.
And we'll get into a little bit more of that
further in the presentation.
And the next set of things is not just to adapt
the workloads but also adapting the infrastructure.
If the application does not fit your infrastructure,
can you change your infrastructure?
Can you right fit your infrastructure
to what you're trying to achieve
and ultimately how do we help you make and enforce
the best decisions within your environment?
So looking back at kind of the evolution,
starting with homogeneous clusters.
What is the role of intelligence and workload management
within this space?
It's very simple.
It's trying to achieve the 90 to 99% utilization
and how much work can you pack in there.
It's also trying to get, oh I gotta point the right way.
It's also trying to get rich policy controls.
Next we began to see, well I have applications
that need high memory.
I have applications that are on my old system
versus my new system and they have different
processor speeds.
I've introduced GPUs to be able to optimize
certain applications.
But as you introduce these new applications
with their unique resource requirement,
you can either have a whole bunch of separate silos
that are a lot of work to manage
or you combine this into a shared system
and then you have a tool such as ours
come in and do application optimized placement
of workloads.
So it understands that these apps need the GPU
or these apps prefer the GPU but if they can't use them,
then I still wanna run them in a high memory system
or I still wanna run them on these particular resources.
So it's the ability to introduce the many complexities
of whether it be shared memory systems,
whether it be GPUs, high memory systems,
allowing you to get a single point of management
that is over all of those complexities
and helping you get in all of your workloads
and intelligently place it in such a way
that you get the optimum application performance
for each of the unique applications and environments.
So what is some of the next complexities?
The gentleman that was on here right before me
is part of the introduction of that complexity.
They're creating new opportunities for applications
that were not formally within the high performance
computing space.
They're taking many of the Windows desktop engineering
applications and moving them into high performance
computing so many of us are going to have that need
of dealing with the many C applications
that are not currently HPC that will be moved in
and so now even if it is only 5% then 10% then 15%
of our environment, how do we set up separate silos
or do we help unify this together and allow
our organizations to once again have a single system
that they're submitting to that is able to have both.
Now in this case it is not just allocating
to the Windows side or the Linux side.
It's going beyond that.
It's being able to get a single shared system
that dynamically changes the operating system
in the space to match the workload
that you're trying to accomplish.
So it's no longer allocation, it's dynamic provisioning,
dynamic repurposing of my environment.
Now it's not just Windows, it's also multiple flavors
of Linux that are application optimized
and it's also not just OS's.
This is also having to hit the energy problem.
So as we're trying to solve energy issues
within our environment, how do we get at the beginning
and the end of our cycle is really when we have
some power down issues of where we have idle servers.
In the middle period of our lifecycle of our systems
we don't, they're fully saturated
so we don't have to worry about that.
So power down strategies at the beginning
and the end of the lifecycle, in the middle
you do what's called thermal balancing.
Your cooling systems kick in when something
goes past the threshold.
So if you can intelligently understand
where it goes above that threshold
and when you can push it down onto a cool resource
instead of the hot resource, you can get up to 25%
of savings and sometimes within the energy optimization
policies from a software perspective.
Not just energy, it's also network
and storage and other such issues.
For example, with Voltaire, we connect into
the unified fabric manager and we're able
to have topology aware placement.
We understand that we can place workloads
specifically within a network location
so that you have the fastest application performance.
Or if it's too big for that, well let's put it
in the next topology aware location.
Not only does it do topology aware,
it's also able to set up a VLAN dynamically
around a particular workload, remap the connections
for that workload in particular and they've seen
with that remapping and the optimization,
they avoid the congestion and in some cases
getting up to 2x improvements in network intense
workload applications.
One example is Synet, it's a Canadian consortium
that had over 30,000 cores within their environment.
They had multiple operating systems.
They had Scientific Linux, they had CentOS,
they actually had multiple resource managers,
they had Torq, but then they powered a power system
from IBM so they wanted Load Leveler.
So we're the ones that help them unify that complexity
across multiple architectures, help them have multiple
types of operating systems and also do the power
management and Moab, its cost was paid for itself
just by the energy savings within the first couple weeks.
Even before they finished acceptance,
we had finished what was needed there.
So now we get into other types of complexities
that we help solve, whether it be a local area grid,
a wide area grid, as you're trying to consolidate access
across multiple different locations.
What is the next gen?
The next gen that we're, and this is not expecting you all
to be hitting this generation right now.
This is one to 3% maybe hitting this next gen.
Breaking down one more silo, and that silo is connecting
HPC with the rest of the business.
It's connecting it with the data center.
It's connecting it with business processes.
It's helping you understand from end to end,
what is that which I need to do to be able to get
my aircraft to market, to get my drug to market,
to be able to get my research project accomplished,
and ultimately it is that work results driven focus.
So we try to be the vendors that will focus,
of course cloud is on the horizon.
Cloud has significantly more impact to the data center
than it does high performance computing
because we already have many of those attributes
already natively, but as cloud hits the data center,
that is gonna pull in HPC into those sets of processes
as we're seeing it with some of our leading customers.
So I'd say that's the picture of the solution sets
that we're helping to provide and organizationally,
not just tactically, but organizationally
solving your problems.
Thank you for your time.
Thank you very much.
Questions are gonna be from Professor Von Voigt
and Dr. Lin.
Michael, thank you for your presentation.
You spoke about cloud computing,
and this brings me, of course, to grid computing.
What are your developments in order to tackle the problem
of energy efficiency in grid computing?
Are you working on that area as well?
If you want to go into grid computing?
Grid computing versus cloud computing in HPC?
Yeah.
Okay, we had some good conversation this morning
with Intersect 360 on the same topic.
Of cloud computing, grid computing,
what is the difference?
Some of that comes down to the business models
that are implemented with it.
Some of it comes down to the technologies used
to whether or not you're doing a paper consumption model.
Some of that comes down to are you having some
of the web type interfaces to it or not?
But there's actually a fairly thin set of differences.
The other nature, which we're showing adaptive HPC,
that's a precursor with adaptive HPC
being the infrastructure adaptation.
That's a precursor to HPC cloud.
So we have, for example, Rocky Mountain Supercomputing Center
is the HPC cloud that we provide our software on,
and so we're providing a composable computing environment
that is able to have this repurposing of the environment
according to the application needs
so that as they become the service provider
to their end customer, they are able to rapidly
repurpose things, have something that allows them
to have remote access into it, able to do billing accounting
that is associated with it, allowing them
to deliver service levels that are specific
to the requirements of the organization,
handle multi-tenancy, handle the security.
There's a lot of different little flavors, but again,
it's not a tremendous amount different than grid
other than the richness of the controls,
some of the business models and technologies.
But in energy, was that your question?
Did I miss part of the question?
So the energy optimization, certainly with the cloud,
they're gonna be employing, well,
if I don't have a tenant using it,
why do I have it powered up?
Have it dynamically be powered down?
If I happen to, the things that we're working on
from a R&D perspective is within the cloud,
can we actually work with an Intel chip
or other such chips to dynamically change
the processor speeds that are being used
to power optimize for the particular application?
Can I do the thermal balancing to do energy optimization?
Can I place things, we've had some customers
ask us to actually place workloads in such a way
that they can even turn down one of the cooling systems
once we have consolidated workload onto a smaller set.
So there's a number of different technologies.
Thank you.
My question, I read recent news about your very
daunting task in entering the finance service arena.
That's a very kind of a tough challenge, I believe,
because there's high demand on speed, security,
and also accuracy as well.
So my question is, what's your product?
Is kind of out with the other products
in terms of entering the business of the finance service?
So how is our technology best applied
within the financial services?
Within financial services, so adaptive computing
by nature of its name is showing about adapting
computing environments, whether they're HPC
or whether they're data center.
We're certainly deployed in high performance computing
environments within the data center, I mean, sorry,
within the high performance computing space,
but you're right, the workloads are much more transactional,
so you can't touch each workload.
You begin to have to treat it as a pool of transactions,
and you try to control the borders of how much
is dedicated as a service for one set of transactions
versus another set of transactions.
However, we've had greater uptake of our technology
from a data center cloud perspective with HPC
within the financial markets.
We have multiple banking clouds,
private clouds that we've deployed.
One of the customers that we're dealing with
has 14,000 applications, and so you really have to have
a tremendous amount of scale and intelligence
that makes a composable environment for them
to right purpose it for all of those different
application needs, allows them to have the billing controls
of who is consuming what, and so our value there
is from an organizational perspective,
as opposed to trying to move a particular transaction around.
That's better for a load balancer.
Thank you, Micah.
Thank you.
Thank you very much.
Thank you.
Our next speaker is Dr. Robert Eicher.
He's a researcher at the EULE Supercomputer Center
and the Powertech Cluster Competence Center in Germany.
He will give a talk on MPI scalability
for petascale cluster and beyond.
In addition, I would like to ask Mr. Tenner
to come to the speaker's cash please.
Yeah, so, thank you.
Yes, Thomas already told,
So I'm from Bülich, and I'm talking here
for the Parastation Consortium,
and I'm really grateful to Patek,
given the opportunity to present the progress
that we made concerning MPI scalability
during the last months on our vehicle
we have in Bülich, which is the Europa cluster.
Let's start with a little bit of motivation.
During this conference, we already heard much
about the scalability issues that we have to face
in order to go to exascale.
And so some of them are always jitter
or the question of resiliency,
so how to build a reliable compute platform
upon unreliable hardware,
the issue of collective operations and all that.
There are various projects try to tackle these problems
that we have to face during the next years.
So for the Yulich site, during the last months,
we funded two labs, so one together with IBM
and one together with Intel and ParTech,
which was announced just on Monday,
in order to face these problems
and try to circumvent the problems.
But today, I want to talk not too much
about the far future, but about the problems
we already see today when we try to use MPI
on really large clusters.
And yeah, so today I want to talk mainly about
two issues in MPI scalability.
So one is the simple problem of MPI startup
and the other one is the memory footprint
that MPI takes from the application.
So by preventing the application to using this memory.
And this work is done as part of the
Geropa development project, which we are currently in,
where we try to morph this cluster system
that we set up one year ago
into a real scalable supercomputer.
So and our target is to, by say the end of the year
or sometimes next year, really,
facing all these problems
and trying to solve these problems.
And we do this together with the partners
that we have chosen to set up the system.
And yeah, so pretty much of the details
you were already able to hear this morning
during the Geropa breakfast.
So, but let's go a little bit more into detail.
So MPI scaling.
So the very first problem you have to face
is just by doing the application startup
of a huge MPI application.
And so what sounds so simple
really is a complicated process.
So first of all, you have to start all the processes
but you're not done with that.
But afterwards, you have to make all these processes
enabled to talk to each other.
So, and also this is a more complicated process
since you have to exchange quite much information
about the processes and how to reach them.
And then you have to establish all the connections.
And yes, as we will see later in this talk,
it is really a good idea to delay the last step.
The experiences we made in Julech
is based on the power station MPI.
And so the reason for that is that,
yeah, Julech is part of the power station consortium.
And so we really want to use our software
where we have control on,
and which we can modify in a way,
we want to do it and to adapt it to our needs.
And I think we are pretty successful with that.
So for example, with power station MPI,
in the meantime, we are able to start processes,
to start applications of the whole machine,
which means more than 25,000 processes
in less than five minutes.
And yeah, I think this is quite a pressing result.
So as you can see here,
these are the scalability numbers
for various versions we tried out in between.
And you can see, so that was the starting point
where you already hit this five second wall
for 2,000 processes.
And you have to keep in mind,
so these are logarithmic scales.
And so now we are really at five minutes
for the startup of the whole applications
on the whole machine.
But nevertheless, once you have started the application,
you might run into trouble with memory footprint.
The reason for that is basically a kind of design flaw
in the InfiniBand protocol.
And this is due to the fact that
in order to set up a connection,
each connection needs some private memory.
So in the case of power station MPI,
with the standard setup,
it takes you about half a megabyte of memory
just to set up a connection.
And if you think about the largest job we currently allow
on the system on a day-to-day basis,
which has 4,000 processes,
then already two gigabytes of memory are occupied
just by the MPI buffers,
which means since you have just three gigabytes
of memory per core in our setup,
more than half of the memory is used for that.
So this, of course, is not the reason
why we bought all the memory,
so that we don't want to use the memory
just to do communication,
but of course in order to do computation.
So the first solution is quite easy.
Since most of the MPI programs
don't really need every connection.
So, and the feature we implemented here
is we call it on-demand,
so you just start the connections
that you are really needing.
And yeah, as you know,
we did an impact run on the system,
on the old system, which was a pure MPI LIMPAC,
so we had 25,000 processes.
And yeah, so there has to be a possibility
to do that without using half a megabyte of memory
for each connection.
And yeah, and we just did it by this on-demand connection.
So this really reduces the amount of memory
required by the application significantly.
Of course, there's a backdrop,
and the backdrop doing this strategy
is that if you have an all-to-all type of communication
late in your application,
your application might crash
because it runs out of memory.
So this is a reason why we don't have
this feature switched on
as a standard on the system,
but it just switched on for the application that needed.
So yeah, so I already mentioned
that there are applications that do all-to-all communication.
So which means applications that really require
to set up all connections.
And for that, we have the possibility to tune a little bit
the amount of memory required by each connection.
And so this picture gives a brief impression
how this works.
So if you have the standard set up,
you see that already for about 8,000 processes,
you run out of memory.
So you have used up all the three gigabyte of memory
per core to use that.
But if you choose some other values
as a number of buffers,
you see that you might get even to almost 20,000 cores.
So that's a basic strategy we currently do in ULA.
So if there's no all-to-all used on demand,
otherwise try to tune the buffer sizes.
And well, if you need both,
so if you need the whole memory
and still want to do all-to-all,
you have to wait a little bit
until we really take the new development
we have into production.
And this is changed from the standard reliable connections
to the so-called UD connections within the MPI.
This is pretty much like switching from TCP to UDP.
And this means, so you just don't have to use
the full buffer for each connection,
but you have one big buffer,
so say about four megabytes
plus some bytes for each connection.
Which means that you're from the memory footprint point
of view, you can scale to millions of processes.
So unfortunately we run out of processors before,
but at least in principle this works.
So if you look at this very new numbers
that we were able to measure with this new system,
you see that, okay, there is some impact on bandwidth,
there is some impact on latency,
but at least it enables you to use the system
for your MPI application as you want it.
So, thank you.
Thank you very much.
Thank you.
Thank you very much.
Thank you.
Questions will be by Jack Dongara and Frank Bieran, please.
Oh, thank you, Norbert.
Thank you, Norbert, for that interesting talk.
It's nice to see the practical uses of software
on large machines like that.
I'm interested in exascale systems,
and with exascale it's, one of the paths have us
at roughly 100 million processors working.
And if I do, if I look at your numbers,
you were taking five minutes to start 25,000 processors,
so that would say it would take roughly 13 days
to get them all going in that scenario.
So, what do you think, what are some of the challenges
that you will face just in terms of process startup
for MPI tasks, going to exascale?
Yeah, so first of all, I'm not really sure
if we will have really, so if you have millions of cores,
if you really have millions of processors.
So, I'm not really sure if the MPI programming paradigm
really holds for this scale of systems.
So, I'm, yeah, I would more assume that we will have
some mixed programming paradigms.
So, some part are threaded, some parts are really processes,
which means that you really reduce the amount you have to do
in order to start up your application significantly.
So, since you just have to spread your processes,
and the spread of the threads is done really in parallel.
So, this is just a local operation,
which will help you very much in order to tackle this problem.
So, strategy you mentioned to overcome
this memory requirement is going to this different type
of communication setup with higher latency
and some other downturns on the other side.
A lot of people did spend and do spend a lot of money
in getting low latency networks integrated in the systems.
And now, finally, the solution
for a kind of obvious problem we face here
is that they may be even forced to buy faster networks
in their compute systems to offer the headspace needed
for this kind of communication.
So, are we riding a dead horse here
and do we need to find a new one?
Is this really a solution to go to this kind of
lower quality communication to overcome
a principle design problem?
Okay, so first of all, these are preliminary results.
So, this is a type of kind of prototype.
And we are still convinced that we can get back
some of the additional time in this latency area,
that we can get it back just by improving it,
by tuning the protocol and so on.
Well, on the other hand,
so we have to do this basically
because there is a problem in the InfiniBand protocol.
So, and I think that there is no principle problem
with redefining this protocol in a way
which is more tuned towards latency and still scalable.
For example, so when we started the Duropa project,
we actually took a look at the quadrics interconnect.
And so, quadrics really had solved this problem already.
So, unfortunately, quadrics went out of operation
before we were able to install the system.
So, we had to go back to InfiniBand.
So, the quadrics example just shows
that there is no principle technical problem.
So, this is just to work around for some kind of design flaws
that are in the InfiniBand protocol.
Nevertheless, so if you really look at axis scale
and if you look at how the compute power in the node evolves
and compare that, will the latency evolve in the same amount?
So, compared to our Duropa system,
just to give you a sense of how we can do that.
Duropa system, just talking about accelerators,
you see a factor of 10 in compute power in your nodes.
And in order to get a balanced system,
you would expect that you want to have
a factor of 10 better latency,
but I cannot really see how to implement that.
So, at least currently we have to go through
the PCI Express bus and the PCI Express bus
gives you most of the latency that you see
in nowadays fabrics, high-performance fabrics.
So, yes, there are some problems to face, of course.
So, there is no boulevard to axis scale,
but we have to pay some ways.
Thank you very much again, Robert.
Our next speaker is Dr. Robert,
Dr. Oliver Tenet from TransTech Germany
and I'd like to call Bjorn Anderson in the cache, please.
So, I'm Oliver.
Good afternoon, everybody.
I'm from TransTech, I'd like to tell you what,
I'll tell you some things about our company,
what our answer to the HPC questions, so to speak, is
and to introduce what we have to do with HPC.
The title is 30 years of experience in scientific computing
in fact, TransTech is celebrating
its 30th anniversary this year.
We were founded in 1980 in Tübingen,
small university town south of Stuttgart.
At that time, when TransTech was founded,
there was, it's a nice anecdote,
a small startup company in Bellevue, Washington
called Microsoft developing MS-DOS 1.0.
This is the time when Tübingen,
when TransTech was founded in Tübingen.
And in fact, it's fair to say that we have 30 years
of experience in scientific computing
because TransTech at that time started
as a reseller for deck equipment, digital workstations.
Later on, Sun, Spark, Clarins were added to the portfolio,
we extended our portfolio and included IBM R6000 systems.
Those were the, at that time, those were the systems
scientific computing in the academic world was run on.
There have not been any HPC clusters at that time.
But when they arrived, TransTech was the first company,
well, not the first, but one of the first companies
in Germany in offering HPC clusters to customers.
So in Germany, it was the end of the 90s,
beginning of the 2000s.
And TransTech was among the first providers
of HPC cluster solutions.
Its experience was strengthened in 2003
when we acquired another Tübingen company called Dr. Koch.
And well, since then we had around 400 HPC cluster
installations.
Some of them found their way into the top 500 list.
The latest news is that since April this year,
we are an authorized reseller for the CrayCx1 system,
which greatly enhances and strengthens our portfolio.
As I said, we were founded in Germany.
In fact, we are located not all around Europe,
but in seven countries, UK, France, Switzerland, Austria,
Belgium, Ireland is covered by UK too.
We see ourselves on the one hand side as a manufacturer
of hardware, so to speak, but not only of that,
but as a provider of comprehensive HPC solutions.
So when we leave the customer, so to speak,
we don't just ship him several tons of metal,
but when we leave the customer,
the customer has a turnkey solution
and the cluster he buys is up and running productively.
So what we try to focus on is we provide turnkey
HPC cluster solutions.
On a highly individual base, we take very much pain
into consulting the customer before and design
and size the cluster solution to its exact needs.
We've got two large customer segments.
One is the academic world, that's for historic reasons.
Of course, on the one hand side,
TransTech was born into the academic community
because of its reselling deck equipment history.
And the other thing is, as we are located
in the south of Germany, where there is much
automotive industry and engineering customers,
the other customer segment we are focusing on
is the industrial customer segment,
the engineering automotive companies,
but also life science and pharmaceuticals.
So what would we say are our values?
We ask what the customers want and we provide
the solution to what do customers want?
They want, well an HPC cluster solution
is a complex beast underneath.
Our goal is to hide this complexity and provide
users, administrators, and customers with easy interfaces,
with the needs of management, even for HPC systems,
where everybody knows are normally complex beasts.
The customer wants to increase his productivity
and wants an efficient solution, so we try
everything we can to provide customers with not only
a performing system, because this is what HPC is all about,
piece for performance, but also with a highly productive
and efficient system.
Customers do not like to spend much time
with administrative overhead.
They want an easy to manage solution.
Also, customers want to be prepared for Windows
in the HPC world, even if they prefer Linux nowadays.
I mean, of course, Linux is still dominating
the HPC world, might change in the future.
Customers want to be prepared and we provide
a solution for that too.
Customers want an easy to manage HPC storage solutions.
Customers want to do GPU computing.
GPU computing is very important nowadays,
and last but not least, customers want a well engineered,
high quality solution bundles which they can rely on.
So, what are our answers?
First of all, within the Linux world, every cluster we sell
in the Linux world, so Linux HPC cluster,
is Intel cluster ready.
That's for two reasons.
The first reason is Intel cluster ready.
On the one hand side is a tool for us to check
if the cluster has been set up correctly.
It's an independent, say, quality standard, so to speak.
Independent, meaning it's not from us,
it's defined by Intel, so it's not too wrong,
so to speak, to rely upon Intel.
The other thing is the standard, Intel cluster ready
defines more or less a standard for HPC clusters,
and a customer can be assured that if the HPC cluster
is certified Intel cluster ready,
a certain list of applications is defined
to run without problems.
So, it's a way to define an HPC standard, so to speak.
In the Linux world.
I said customers want to be prepared for Windows
even if they prefer Linux in the first place.
For that, I mean, you've heard the talk of Michael Jackson
before, or the one talked before.
We include the Moab adaptive HPC suit within our portfolio.
The background is, I mean, we see ourselves
as a concentrator, we're a small, medium-sized company.
We're not exclusively doing HPC,
and we don't want to reinvent the wheel
if there are perfect solutions or solutions
that we think are the ideal solutions
to problems that customers have out there.
So, we see ourselves as concentrators
of providing customers with what we think
are ideal solutions from one point of contact.
We have a single point of contact
and giving customers solutions and tools
that are already there.
So, for the problem of, or for the challenge,
the task of dynamically deploying resources,
dual booting clusters dynamically,
providing Windows nodes if use case requires it,
and the rest runs with Linux is perfectly, in our opinion,
covered by the Moab HPC suite from adaptive computing,
which is why provide customers with that.
The other thing is, as I said,
customers want well-engineered bundles,
well-engineered HPC hardware, which they can rely on.
So, since April this year,
we included the Kray CX1 into our portfolio
and have a recent agreement for Germany.
So, we are one of the two only German manufacturers
that resell the Kray CX1 system,
which combines the power of an HPC cluster
with the ease of use and seamless integration
of a workstation.
And we see it as a perfect bundle solution
for desk-side HPC within the industrial sector
and the finance sector.
Customers want a scalable HPC storage solution as well,
which is why we chose PENAS's HPC storage
as the storage solution for high requirements
as regards performance to the storage.
And as you might know, PENAS is more or less,
well, contributing in a very strong way
to the forthcoming PNFS standard.
We are, of course, NVIDIA TESDA preferred provider,
and our cluster management, ease of management also,
includes Tesla hardware into our unified management.
What I do to focus on, or what I try to tell here too,
is for us, providing a customer with an HPC solution
is not just selling things, selling the customer solution,
but also we take part from the very beginning
in this process of consulting customers.
In a very early way, we size, actually size the solution
correctly as the needs of the customers are,
test the systems, do benchmarks, deploy the cluster.
When we ship the cluster,
the cluster is pre-installed, pre-configured,
and we integrate the solution on-site,
on the customer side, into the customer's environment.
We even go so far, we even go so far
as if the customer's needs services
because of lacking administrative power,
we could do it for him.
Of course, we need strong partners at our side.
As I said, we see ourselves as concentrators,
providing, from a single point,
ideal solutions that are out there to the customer.
This is why we need strong partners.
So, thank you very much.
Thank you, Dr. Tenet.
Yes.
Questions are from Dr. Lin and Professor Von Voigt.
Okay, thanks, Oliver.
Thanks for the presentation.
I would like to ask, could you give us an example
of the most successful case that you have been,
kind of, you've been done?
And also, can you answer if such success
can be repeated in the future
when people talk a lot about data computing,
exact computing?
Okay.
We did not name the project as we can explain.
What, in our opinion, was an ideal project case,
so to speak.
We had the invitation to,
it's not finished,
should it be finished or not finished?
I'll take a finished one.
We had the invitation to an institute
in the southern, south of Germany,
because the mouth-to-mouth propaganda was saying
that we are ideal HPC partners, competent people.
And so we went there and we into, well,
influenced the wording of an invitation
to tender exactly as it fits to us, so to speak.
And from this early phase of the project
until the actual installation and integration,
everything went, it just went well.
The cluster extensions is forthcoming
within the next couple of months.
We added more and more and more storage
to this customer as well.
So it's just an ideal case where we,
no mistakes, I mean, of course, sometimes things happen,
but this was an ideal case, in my opinion.
And another very successful project,
from my point of view, which is ongoing,
it's not yet finished, is, for example,
that in our Austrian subsidiary,
they got a call from a university in Slovenia,
somewhere in Slovenia,
where we actually have not done any business yet.
But the mouth-to-mouth propagandists said that,
why don't you give TransTech a call
asking for an HPC solution, asking for bidding at a tender?
And, well, it makes us very happy, so to speak,
that even in new countries where we haven't been present yet,
our reputation is very high, so.
Okay, thank you, Oliver.
If I run a computer center, I do run one, yeah,
but just by coincidence.
And if you do the consulting,
our major problem at the moment is power consumption,
basically, from all the machines,
especially these clusters.
So if you do consultancy,
what would you recommend to me
how I can tackle that problem?
Tack of the problem of power consumption.
I mean, that's an-
You said you would do consultancy in every-
I mean, there's one thing,
we don't develop our own hardware,
we're not hardware developers.
So we have to take whatever the market
or the manufacturers give us.
Something that has to do with power consumption
has something to do with efficiency.
Customers want to maximally utilize their cluster.
So what we try to convince the customer of is
we can't really do anything
which has to do with power consumption
because we can't change the hardware.
Hardware consumes as much power as it does.
What we can do for the customers,
we can try to maximize the utilization
by intelligent scheduling, for example.
As we have mow up within our portfolio,
which is among, I think, the cluster management
and scheduling systems,
one of the highest reputations that are out there,
it would help the customer to understand
that it makes much sense
if he maximizes the utilization rate of his solution.
So that's what we can do for him.
We can't actually develop hardware
which reduces power consumption.
That's, we are not a chip designer, for example.
That's not what we are.
Thank you, Oliver.
Thank you again.
Thank you.
Our next speaker is Bjorn Anderson,
who is the director of product marketing
at Bluark in the USA.
And the title of his talk will be
When Demands Change Quicker than Your IT Infrastructure.
Thank you.
So I'm very happy to be here today.
And what I'm gonna talk about
is maybe a little bit different
than what you've heard before.
I'm starting with a customer case
and then at the end,
I'll go into what we do for that from Bluark.
So if you haven't heard about Bluark before,
we deliver basically accelerated
and scalable network storage solutions
with a full portfolio of software for management
and ease of use as well.
We're based on standards.
So we deliver NFS and SIFs in ISCAC today.
And we're working on a PNFS solution also.
We're one of the leaders in the implementations
of PNFS going forward.
And by the way, we have a buff tomorrow
if you want to go and listen to that.
So what I wanted to start with
is really what we have seen in many customer cases lately.
Customers have been investing in their IT infrastructure,
but things start to change really fast.
New applications, new signs coming out.
And we deliver shared storage environments.
So we often have multiple sciences
using the same storage environment.
So if you add up all of that,
you often see sort of the meantime between publications.
It's usually in the order of weeks when you add it all up.
Applications change in that mixed environment
maybe in the order of months.
Both of that is much, much faster
than you can change the hardware
or change the IT infrastructure.
Best case, you can look at procurements
in the order of quarters.
But really what's driving the timeframe here
is the investment in terms of money.
So the depreciation of the investment
is usually in the order of years.
So if you would take a look at that
and sort of, I'm sort of a graphical guy,
so I like pictures.
So if you look at that and sort of graph
the problem space that you're trying to solve.
And this doesn't have to be your storage.
It's sort of in general.
And then over time, start to track what's happening.
And in this case, maybe you have based your investment
on a couple of benchmarks that I just think are reasonable
for your environment.
But as things start to change over time,
you start to go beyond that problem space.
The first couple of times, it's probably fine.
You can deal with it.
When you start to spend more time outside of the space
that you're designed for,
then inside it, then it really starts to hurt.
So my point is really that you need to look
at a much broader picture
and preferably actually run your application sets
instead of benchmarks.
And this goes back to an experience that we have at Blueark
is that when people actually try our systems
with their applications, they usually buy.
So when you can do that, that's often the best.
So focusing in on more storage,
you can span up the problem space in this case
with bandwidth and IOPS as two sort of parameters.
And I've just plotted six applications here
where they would fall in this problem space.
And it really doesn't matter where they fall at this point.
The point is that if you base that on, yes,
sort of one static picture in a point in time
and just a few benchmarks,
you will be surprised as these started to move around.
You add more applications on your compute cluster,
you get new data sources,
you get more compute resources that put more load
on the storage and so on.
So if you have based your purchase on
just a few benchmarks that may have seen
like a good choice at that point in time,
you may be in for a surprise going forward.
So the obvious choice here is actually that
expand your view, test with your applications
and really try to be at the point
where you have a more balanced performance.
That way you're more prepared
for the changes going forward.
So let's take a look at what Blueark is doing about this.
So just a couple of words about our architecture.
We have accelerated file servers
and behind that we have a SAN
and we can attach different types of disk technologies
to the SAN.
We have different choices in terms of disks
from different vendors.
In this case, we used Hitachi disks.
We also have LSI and DDN on the backend.
And I'm just gonna point out
a couple of different things here.
So if you go down that first column,
that's really the read and write mix.
And if you look at the performance numbers,
you can see that they're pretty stable
regardless of the mix of operations.
The same thing with the block sizes.
So again, a very robust type of delivered performance
from our systems.
And the last thing is that we can actually scale also.
So you have a really robust stable base.
Then as you add more disks, more spindles,
you actually scale to more performance,
a very scalable solution.
I mentioned in the beginning that we do standards.
So NFS is one of the protocols that we serve out.
And many people sort of think about NFS
as not the highest performance file protocol out there.
So what we do is that we really deliver high performance
through that standard.
In this case, we ran it with just one of our cluster heads.
We had five Solaris clients putting load on this.
And with the mixed 80-20 read and write workload here,
we got basically 1.5 gigabyte per second
in bandwidth out of this.
Another example, in this case,
we did the end disk behind it and two heads.
And as you can see, we basically scaled linearly
to about three gigabyte per second.
And the other thing to note here is
if you remember from the last slide,
that had 64-kilobyte block size, this is 32.
So usually you go to bigger block size
when you want to get bigger, higher performance numbers,
when in our case, we are not forced to do that.
I also wanted to focus in on actual testing
that we've done earlier in the year.
So at Blueark, we have gone sort of one step
beyond benchmarking and actually testing
with customer applications.
In this case, it's in the life sciences space
where we partnered with Illumina
with using their application set.
And this is sort of a typical type of setup
in sort of modest to bigger size installations
using their software and their instruments.
So on the left-hand side here,
you have the instruments themselves,
basically dumping terabytes per day
or data per day asynchronously,
not timed or synchronized between the instruments.
And then you have a computer cluster
and then on the right-hand side,
you have interactive access from researchers.
So we've actually went out and rented a computer cluster,
loaded the software, got data from our customers,
and really recreated that environment
and did some in-depth testing with that.
What we found is that using the application set from,
in this case, Illumina,
it really put sort of a random load
on the shared storage.
So a couple of things on this slide.
This is what's called the pipeline set
of applications from Illumina.
So data sort of flows from top to bottom.
And the characteristics of the data change in every step.
So that's one of the variables, if you will,
that is changing the workloads on the shared storage.
The other one, if you look at the left-hand side,
is the data from Illumina.
If you look at the left-hand side here,
when we tested this, they were in the process
of going from 1.5 to 1.6 of their software.
And that really changed the workload as well.
In this case, that first application, Firecrest,
moved inside the instrument itself.
So that workload disappeared from the shared storage.
In this case, that was a heavy use of long sequential reads.
So that workload didn't matter at that point in time.
So it really changes sort of the way
you look at the shared storage.
Makes it much, much more random.
Just selected one of the graphs that we produced from this.
This is the second one of the applications out there.
The key thing to take away from this
is that it's really spiky performance,
with sort of low average and pronounced spikes.
But you have sort of one thread of these applications
running for each experiment.
Then you have multiple experiments running in parallel.
So again, producing an extremely random type of workloads.
This was some post-processing codes
that we actually got from our customers.
And we also used that to analyze what
you need to do on the storage side
to really have a good solution for these environments.
The last one here was actually one
that finally was able to put some load on our systems.
We had a really hard time stressing our storage
system in this environment.
But this one actually did that.
And I'll touch on that later on.
So bottom line is that it really adds up to a random workload.
And I just wanted to focus in on that stress situation
for one of our systems.
It was read.
And what we did was that we enabled
what we call dynamic read cache.
And with that, we were able to scale linearly
to 3 gigabyte per second.
This is our lineup of our systems
from the low-end Mercury 50 up to the high-end Titan 3200.
And basically, the difference is the scale and capacity
between all of them.
So just to sum up, what you see from Blueark
is that we deliver really maximized performance
for all types of workloads.
And we have the ability to optimize
with cost and performance with our open back-end and disks.
And the size of our systems are really,
from mid-range to high-end, you can choose the right-size block
and then easily scale and capacity.
And we do this with a single unified namespace.
So keeping simplicity for the user,
but all the control knobs for the administrator.
So thank you.
Thank you very much, John.
Thank you.
Questions are by Frank Beran and Jack Dombara, please.
Yeah, thank you for the presentation.
Question was when demand, or the statement was when demands,
change quicker than the IT infrastructure.
Question is, even if you have a balanced system to start with
and then suddenly puts you off balance,
is there any different solutions than buying new hardware?
And that was the first part of the question.
And the second part was moving from standard disk drives
to solid state drives.
Does it change anything with respect to the strategy
that you're looking at?
So a good question.
So just the first one first.
Yeah, you can never test everything from the start.
But you need to create a system where
you have flexibility and robustness
in the system itself.
So what we've seen with our setup, for example,
if you run into a situation over time where maybe
the performance is OK, but you need to add more capacity
because you have underestimated the inflow of new data.
In our architecture, it's very easy
to keep the number of file heads, if you will,
and you have to add more disks.
And you can do it the other way around, of course.
So that's one way to address unexpected types of changes
going forward.
And then, of course, we have our software stack also
to help with moving data to where it makes best sense
in terms of using different types of disk technologies
for performance or cost and so on.
So if you set it up right from the beginning,
you have that scalability and flexibility built
into the architecture.
So that helps a lot.
Then with regards to the question about solid state,
already today we're working with Texas memory instruments
for SSD devices into our architecture.
And going forward, we're integrating that technology
even more into our core architecture.
So for example, for metadata operations,
that's a good target for where you use the SSD
type of technology.
And you'll see that from us as well.
Our base system is really, really good at metadata
to start with.
So this just adds another level of performance for that.
Thank you.
So I have to admit, I'm not really a file system guy.
So maybe my question is a little naive.
How does your file system compare
to something like Lustre?
Lustre is used on maybe 15 of the top 30 high performance
systems on the top 500.
Can you say something about the comparison between Lustre
and your system?
Definitely.
That's also a good question.
The question we get from many of our customers.
On one hand, so what we do today is based on standards,
the current standards.
So it's NFS.
So it's not the clustered file system today.
We do have a cluster behind the scenes.
But what we export to users is NFS.
And then going forward, we're working on PNFS.
And PNFS is really our answer to Lustre going forward.
With our current system, we scale up
to multiple gigabytes per second in terms of throughput,
depending on how many disks you have behind our heads
and so on.
With PNFS, we actually get to sort of unlimited scalability,
much like what Lustre has today, but with the standard.
And the key thing there is that on the client side of this,
we expect the client to be just a regular part of standard
Linux and standard Windows out there.
So you don't have to deal with the complexity of the client
side, but still get the scalability
that you can see with specialized systems
today, like Lustre.
So thank you very much again.
I'd like to call the next speaker, Dr. Rudolf Fisher
from NEC.
And I'd like also to call Dr. Goh into the cache, please.
Yeah, ladies and gentlemen, thank you for being here today.
Last year, I had a difficult time
because NEC just had caused quite some concerns
in the community because we stepped back
from the national project, the Japanese national project.
And people were surprised about this.
So yeah, well, it was difficult to present.
And the last slide I used last year contained these statements.
And today, I want to tell you in what way
we are acting on these promises to you.
So first of all, NEC is profitable now.
Well, not to the extent that we would want it to be,
but it is the year of turnaround now.
And in the end, I do not really believe
in loss-making business.
So that's a good news.
And in particular, it's a good news
that we in Europe, the NEC, the Deutschland division, HPCE,
is profitable and exceeded a very interesting target.
We made a 50% increase in scalar revenue
on a year-to-year basis.
And we hope that we can continue like this.
We might be small, but let's continue like this 10 years.
That might be OK.
So how are we doing?
We see the market trends, as you all do.
There are these systems built from standard components.
They do a pretty good job.
Intel and AMD provide very interesting technology.
But it seems that there are limits and we're hitting walls.
And to my opinion, the hype around the GPUs,
which are interesting technology by themselves as well,
shows us that there is something that
is needed which is currently not really covered.
My personal conviction, but also the one of NEC,
is that we will see some kind of merging
of these kinds of technology.
It will all be some kind of SIMD.
And I think the best person to ask is Intel.
If you look at their future plans, well, this is just true.
A little remark from my side.
I still wait for the real vector register,
as we know it since days of Cray 1,
to return because that is a good thing to hide latencies.
That is a register which will provide data, not for one cycle,
but will provide data for a sequence of cycles.
And that way, even exploiting SIMD architectures more
than we are knowing this today, I
think the latencies on the chips will call for the sooner
or later and are waiting for the first vector register
on an Intel CPU.
But for today, in order to do business,
we really need scalar and vector machines.
And beyond that, we need to provide complete solutions.
So first of all, yes, we have a scalar roadmap
and we have a scalar product lineup, which
ranges from entry-level things to machines which are really
used for operational purposes, which
means to have a high reliability and resilience feature set.
And this is also something you might
see from other vendors as well.
We have a complete building block scheme
to really address the customer's needs in the most flexible way.
Again, you might see this from others,
but we try to differentiate.
And how can we do this if we are doing what others are doing
as well?
There are a few examples.
For instance, you see LXFS.
This is a Lustre-based file system.
And it happened that some customer needed something
like this.
And we did this for the customer.
We optimized it.
We integrated it to monitoring tools and such thing,
optimizing it also with regard to quality
of service and InfiniBand.
These are things that we do if a customer wants this.
And if you have a particular problem, please call us.
There are other examples like this.
For instance, you have perhaps seen the HLIS Voltaire NEC
presentation about optimizing InfiniBand traffic.
That is also one of our contributions to the scene.
But beyond that, we still believe
that there is a room for a vector
and that this will return, definitely.
And you see what we had until now.
That was what we call now conventional SX series.
And we are going to continue.
People didn't believe that last year, but it's really true.
We will have the compatibility in programs.
Source code compatibility is at least necessary.
I'm personally not a fan of binary compatibility.
It will be a vector architecture.
For the reasons that I just mentioned,
I believe that this is the future anyway.
It will feature large memory bandwidths.
We have heard this quite often because since decades,
this is already a bottleneck.
It will have a high performance core.
Please note the word core here.
You will see soon why.
Which means that we want to address scalability issues
also by means of providing a very fast single process.
And I have heard this in many presentations here
that the best way to address scalability issues
is by having a fast processor.
Then, we will look at the shared memory,
but it will not be as large as in the past.
On the SX9, the memory architecture was very, very complex.
And a huge amount of cost and power consumption
went into this very complex memory architecture.
16,000 banks and 16 very fast processors.
And for that reason, we now think we should go back
a little bit, make a smaller SMP.
You will see how much SMP this is,
but at least we believe that that way,
we get a very high sustained performance.
Again, this should help the user.
The high memory bandwidths,
we believe that we have to keep a certain level.
Over time, this degraded all the time,
but we want to excel on memory bandwidths
and we are targeting a one byte per flop memory bandwidth
and that way, we are definitely faster than other vendors.
This for me is honestly the key to efficiency.
That is not enough.
We also have to think about data localities.
I will come to this later.
So now you'll see with the SX6,
we for the first time had a single chip vector CPU.
This evolved over time until the SX9,
102 gigaflops per chip.
And now we are looking at a single core in the same range
like a SX6 nodes has before.
That is, I'm allowed to mention the 64 gigaflops per core
is our target value that we want.
And from there we go multi-core.
As you can see, it will be more than two
and definitely it will be less than eight
and it will be a power of two probably.
So our target currently is four cores.
Having said that, such a chip would correspond
to an SX8R system.
So I think that's a nice case of having a fast machine
which did quite a good job on a single chip.
That is our current plan.
That is the machine that we have in mind.
And then you see, we will combine this by an IXS.
We always call this IXS
and we have quite some technology in-house to do that.
And this machine will then have this vector cores.
It will be an SX architecture.
We will have this ADB,
consider it a software controlled vector cache.
So that way we are exploiting data locality.
In some cases, this has helped a lot.
In some cases, we were a little bit disappointed
but we learned the lesson.
In the SX9, it was our first implementation.
Now it will be our second implementation.
And there will be enhancements
which are quite promising for efficiency.
As I said, it will be a multi-core chip
in excess of 200 gigaflops.
We are targeting 256.
Another point is similar to other vendors.
The memory controller will now be integrated.
That is a big part of saving electricity.
We don't have any routing chips
or crossbar chips in between anymore.
There will be a direct connection
between the chip and the memory.
Beyond that, the high-speed interconnect
and the important part is also our network interface
is directly on the chip
so that we can have a very efficient MPI implementation.
On top of this, we have our own MPI.
You might know Hubert Ritzdorf to a guy beyond two meters.
This is our MPI guru and he is one of my colleagues.
In the end, anyway, I believe it comes down to people.
So if you want to differentiate
and it is what we are doing with this LXFS
and with MPI and such,
it all is about people who have skills
and who are committed to the job
and I'm proud to be a part of such an organization.
And now I find out that I was way too quick.
Thank you, Dr. Fischer.
The questions are gonna be Professor Von Voigt and Dr. Lin.
Thank you very much for your presentation.
Yes, you spoke about this all comes down to people
and that of course brings me to our users.
They do have applications
and they are written by loads of people
of not anymore in the institute
and they are not really specialized on a vector architecture.
So what do you do for these users
so that they can really make profit
out of the vector architecture?
First of all, yes, we provide trainings.
Secondly, honestly, I believe that vectorization
is becoming more and more prominent.
When doing some little exercises on the Intel chips nowadays,
I find that you have to vectorize there as well.
With future versions of these chips,
this will be even more the case.
And beyond that, if I look at alternatives
that we currently see,
I think that the users can well deal with vectorizations.
One has to help once in a while
but if I compare a CUDA port
with a vectorization of a certain sub-protein,
I think I know well what I would prefer.
Definitely it is something that the user has to do
and definitely there are codes
which are not so easily vectorized.
Therefore, I personally believe
that NEC has to provide both architectures
and in the end, I still believe we have to build them
in such a way that you can use them in a hybrid fashion.
That means, let's say via MPI space,
have part of the code running on one side
and the other part on the other side.
Did this answer your question?
I think I'm following some of the questions.
There's many users from my acquaintance
and they actually are very curious,
what's the NEC's vision of the future of vector computing?
Because obviously vector computing is not in the majority.
Could you elaborate more?
I would say it's not yet the majority
but it's going to come.
Do you know any chip
which does not have a SIMD instruction set anymore?
Perhaps there are some gurus here who know about such chips
but IBM has it, Inte has it, AMD has it,
Sparks now in Japan, they have it.
So I think the vectorization is rather mainstream
and the only question will be,
what is the average vector length that you need
in order to become efficient?
And on the scalar side,
this average vector length is going to grow up to AVX, AVX,
plus this is going to grow what you need.
And when we do our experiments, we have some thoughts.
That's the reason why I'm asking for the vector register
because for the same reasons
that you are doing vectorization now
by having one instruction dealing with several data,
for the same reason, you also need this vector register
in order to have one instruction dealing with even more data
but you don't want to reduplicate 8-fold, 16-fold
your pipelines.
For that reason, I think vectorization is not the past.
Vectorization is the future.
Also for reasons of power consumption,
there are reasons why SIMD is cheaper.
The reason is that you don't have to deal
with that many instructions.
So I think users will have to do a little bit
but I believe that we are where other people will be.
Combine GPU, from user point of view,
GPU programming and scalar programming,
combine this, then you will get something
where we are already,
which means the user has to learn vectorization
but I'm certainly willing to help.
Thank you, Dr. Fischer.
Our next speaker will be Dr. M. Lingo,
who is CTO at SGI USA.
The title of the talk will be
Managing Massively Large Datasets
with Breakthrough Technology at ICR.
Thank you.
Thank you, everyone, for your time.
Just switch the projector over.
Yes, today, unlike last year,
I won't be talking about exascale,
even though we are co-designing an exascale system
for standing up by 2018 plus and minus.
I would like to talk about the front end
to that exascale system.
Most supercomputers have front ends,
but how do we build a front end system
that can service and support an exascale system
at the back end?
At the same time, make it easy to use,
just like a scientist's laptop.
And this is where my marketing people have suggested
that I don't try and do,
but I thought it would be interesting to try.
I logged in into the US using this cable, the PC.
Essentially, to just understand that this is a PC
and works just like a PC,
I type in a Linux command,
and it tells me that it's running an Intel processor.
And then I ask for it to be,
what operating system are you running?
Basically, I say U name minus A.
It says it's running a standard Linux kernel, 2.6.32.12.
Basically, you just download it
from the LinusTowelskernels.org website,
load it onto this PC, and it works.
If you want a distribution, you choose one.
In this case, you choose one,
and I've loaded the SUSE rate hat,
SUSE slash 11 distribution.
Now, if you ask the next question,
what happens if I try plugging in a graphics card?
And I did a turbo VNC software on this PC,
and it comes over.
Yeah, it's a standard graphics.
If you really want to be sure, load open office
and make sure it works, yeah, okay, it does.
The speed is really the network speed,
which is pretty good, yeah.
But the next question you ask, therefore,
is what's so special about this PC?
Let's ask about this memory, memory info.
My laptop has two gigabytes,
most PCs maybe have four gigabytes,
eight gigabytes, 16, 32, I hear 64, 128, 256, 512.
Well, this PC is a small one for us.
It has a total of a kilobyte, megabyte, gigabyte, terabyte.
This PC has 5.2 terabytes of memory.
We can go up to 16, we didn't fill up all the slots.
The only reason why it stops at 16
is because the Intel current core processor
has a 44-bit of physical address space
and 2 to the power 44 is 16 terabytes.
So when Intel improves on that, we will grow with it.
I think 64 terabytes is our next hope.
Next question is, how many CPU is there in this PC?
It has a total of with hyperthreading switch on, 2,000 cores.
It can go up to 4,000 with hyperthreading switch on.
So basically what happens here is
that we are trying to build a laptop-like system that
works just like your laptop for scientists and engineers,
but it is of a much higher capacity.
We have tested people writing their code on their laptop,
compile it, and then FTP-ing the binary to this PC,
and it runs.
The result of this is that you will dare for scientists
and engineer in MATLAB or in your C programming
to even think of a program like this.
It looks simple, but it is very insurmountable today.
Typical engineer or scientist would
like to create a model that has an x model with an x
direction of 1,000 points, y direction of 1,000 points,
z direction of 1,000 points, and 1,000 time steps.
A model of a heart could be that.
But you will realize that this, with 4 bytes,
is going to be a 4 terabyte array.
Anyone want to try compiling this?
If we dare, yeah.
Even more daring, anyone wants to try and run it here.
It will run.
It's just that I'm not going to sit here and wait for it
to finish here.
So basically, this is designed to look just
like a PC, but of a much bigger capacity.
And what I'll do is I'll just finish this session
by showing you the VNC again, and you
can have a look at what we can do with it.
I'm just going to use a smaller array in this case, A dot C,
because it will take a long time to finish at 4 terabytes.
I just trim off 1 0 at the end.
I'm just going to run this program
and show you the implication of managing 2,000 cores
through one kernel and so much memory.
I put it as O3, A dot C, compile it, and I run it.
The kernel decides where to put that thread.
It decided that this processor was the freest at the time.
It put that thread there.
This is the turbo VNC thread.
And it has locality to know to load this amount of memory,
starting with this is one socket, right?
To load this socket memory first,
and then to load this nearest neighbor socket memory.
And then when this is done, it should do it either here
or there, right?
It should not go crazy and just load it somewhere else.
This shows you the total bar, right?
And it started.
So it's pretty smart.
And then if the scientist and engineer
wants to get interesting with the tools it has,
by the way, it did the memory recovery.
All the scientist and engineer needs to do is add one more line.
And we all know this is OpenMP, the OpenMP line.
And now I just compile.
Since this is GCC, fOpenMP, right?
Minus 0.3, a1.c, compile it and run it.
So what happens now?
And now I get multiple cores to help me.
There we go.
All the threads are running.
All 192 cores.
And the memory is haphazard.
So I did not control where the threads go.
This is just like the Linux kernel just going crazy, right?
No, I shouldn't be saying crazy.
It just goes in and loads and decides which threads to run.
So if I want to control a bit more
and have a bit more order in the system,
this is doing memory garbage collection right now,
the scientist and engineer just needs to do say, OK,
I want to control the number of threads to 64.
And then I would like to place the threads very carefully,
explicitly, all on cores 0 to 63 only.
And don't go crazy.
And now you'll see very orderly.
And I'm not using the virtual cores.
And the memory is placed.
So in conclusion, basically what we have done here
is looked at seriously as the world looked at exascale,
to also think of building a system that front end is
big enough to handle and be the front end of an exascale system.
An exascale system could be churning out
hundreds or tens of petabytes of data.
You need a system big enough to deal with it.
And we're starting to get there by 2018 with this front end
system.
So we hope every exascale system gets one of these front end.
To prove there is a real PC, we went out to iTunes,
bought a bunch of VNC software, free, $5, $10, $35.
So we found that a $35 one worked the best.
I guess you get what you pay for.
And you can now also operate on this directly.
Just later, those interested can come up, have a look.
And to close off this presentation, I just go to ICR.
In addition to it being a front end,
we're starting to see interest in institutions
like instill cancer research.
Dr. Chris Barkow is investigating how he,
that's how he describes to me, how cancer cells actually
migrate through the body.
In fact, that's how they become destructive,
by clawing their way through cells, other cells.
And he's figuring out ways to try and prevent that clawing
by modeling it.
Dr. Roon Linding is trying to model cells
based on cellular logic to try and convert them
to logic gates.
And they therefore need global optimization
of these massive graphs that need terabytes of memory.
And this is yet another case where this big PC, a PC
just bigger type concept would be useful to them.
On this note, thank you.
Thank you very much.
Questions by Chek Dongara and Frank Bader, please.
That was a very impressive demonstration of your,
I'm over here, to your left.
To your left.
Sorry.
That was a very impressive demonstration.
I'm a big fan of productivity, and I
can see where this would really enhance a user's productivity
in terms of using it, especially if it were tied into tools
such as MATLAB, as you pointed out.
My real question is about the efficiency, however, of this.
So if we think about running it as you did,
you were doing something which was embarrassingly
parallel, but if you had something
which had more tighter coupling between the processes,
what kind of inefficiencies would you
see over doing it, let's say, by a detailed MPI implementation?
That's a very good question.
In fact, there is no magic here.
It's just a lot of hard work, and there
are latencies in the system.
If you deliberately take the step
to do MPI coding, where you are then
forced to do domain decomposition,
that will definitely run better, because you
have put in the code, the discipline,
to control the way communication is done.
Then run MPI on this system, because there
is a library to intercept the MPI calls
and make them fast shared memory calls.
However, this really is for scientists and engineers,
where they are developing their code in the early days
in conceptualizing their ideas.
We realize from talking to a lot of scientists
that they conceptualize on their laptop,
and then when the laptop runs out of steam
to do an actual size model, they have
to take a very big step to look for a programmer
to help them take that step.
Some can do it themselves, a lot can't.
So we thought, although we are still selling those MPI
machines in the back end, we should
try and help these scientists to try and delay
having to take that step sometimes,
because they have not completed their thought
and experimental process.
Don't break that thought and experimental process.
As such, we thought, then these scientists
can then, so book time, say 10 hours on this machine,
and continue the experimentation in the usual way.
There will be efficiencies, but at least you
can still run the two terabyte model,
otherwise he couldn't have or she couldn't have,
except for going to that big step.
So that would be the place for this machine,
not so much to take over the MPI world, no.
As I say, this really is designed
to be the front end of an MPI machine, or a UPC machine,
or whatever the access scale step is.
Thank you for that question, because it is important
to position it correctly.
MATLAB or similar systems have been mentioned repeatedly.
What would happen if you move something
like a classical, non-modified MATLAB to the system
compared with the MATLAB, which then is fine-tuned
for this parallel world in?
Are we talking about an order of magnitude and performance,
or less, more?
So we just shipped this machine on Friday out
of our 20, 25 customers.
The first one got it on Friday.
I haven't got it yet.
Maybe it's on its way.
And I've asked for my application team
to look at MATLAB.
So I don't have an answer to that.
There are parallel MATLAB applications.
We use them, too, that allows you to declare a large array
and then punch it through an MPI interface
to run on a cluster.
If you're happy with that, go ahead using it.
We sell those back-end machines, too.
But this one provides an alternative.
And that is your current MATLAB that
is running on your laptop, if MATLAB allows it,
I mean, I don't want to break any rules here.
They actually just FTP the MATLAB program over
to this machine.
It was just to the MATLAB, just looks like just another PC.
It is the convenience, basically.
You can get up and running right this minute now.
Someone gives me a MATLAB CD.
I'll load it right now.
I figure a way to load it right now.
You can use it right now.
So that's it.
And you don't have to deal with library links and all that.
It works just as is.
Just so happened that you have five terabytes of memory
behind that.
This was the goal of that design.
In fact, it was so difficult to make it.
It took us six years.
Mark Varenche, I must really appreciate
him giving us the resources to complete this and ship
on Friday.
The difficulty all these years isn't so much
the complexity of the system, but the difficulty
of hiding the complexity of the system,
and the difficulty of convincing the Linux community
to accept all these changes so that the Linux is also
standard.
So that's the goal.
Make it look really simple, just like your laptop,
for the scientists and engineers.
Thank you.
Thank you very much again.
Thank you.
So the last speaker of today is going
to be Jean-Pierre Panziere from Bull, who is going to look ahead.
Sorry.
Hello.
Bonjour.
So the title of this presentation is Beyond the Petaflop.
So why Beyond the Petaflop?
And maybe the next slide, how do I do this?
OK.
Next slide will explain the title.
You probably have seen this announcement that there are 100
projects.
If you have not, here are a few figures.
So first, there are 100 is a result
of a collaboration between CEA and Bull.
It is deployed, as we speak, at CEA.
It qualifies as a petaflop because the peak performance
for the full system will be one and a quarter petaflop.
It's made up of 4,300 nodes.
These are nodes that are fat nodes that are developed
with Bull technology.
And some of these nodes are actually fatter
since they will be including our ASIC chips that
extend the node capacity to 16 sockets, 128 cores,
one terabyte of shared memory.
Globally, you'll have an excess of 140,000 Intel Nail MEX
cores for a total memory of 300 terabytes, 20 petabytes
of disk storage, and QDR, InfiniBand,
quinteconnecting all the nodes, and a global file system
throughput of 500 gigabytes per second.
So this is what we have in mind when we talk about petaflop
and what we want to do beyond that.
So why beyond that?
Actually, when we spent a lot of time talking with users,
HPC users, and usually there are very strict requirements
about what they, if you can go back,
to what they mean by more performance.
And they express it in flops, in bytes, in memory bandwidth.
But most of the time, they express it in their own language.
They do want a higher resolution for their problem.
And if that resolution is a factor of 10,
because your model will be 3D, that's already
a factor of 10 to the cube, 1,000.
If you're doing time step iterations,
the time step will be smaller.
So it's another factor of 10.
Obviously, you want your application
to be more and more accurate.
And so the numerical method will be less and less approximate.
And so that will result in more computation per element.
The next thing is they want it to go faster.
And so when you take into account all these parameters,
you're really in excess of a factor of 1,000 performance
improvement with respect to today.
So if we are a bit of flop, the next step is the exit flop.
But let's take this factor of 1,000 with different parameters.
One caveat is that even though this requirement
and this performance will be increasing
by several orders of magnitude, the budget will not.
People think that they might get more budget,
but certainly not an order of magnitude,
certainly not orders of magnitude.
So what it means for us, system builders, system architects,
is that we'll have to build the systems which
will be much more powerful with about the same number
of components.
And because of the price constraint,
they will have to be cheap, quote unquote, components.
Another important constraint is consumption,
the power limitation of any site that we know.
I don't think that I am aware of any site that
is using more than 10 megawatts.
And I don't think that we'll see a lot of these
in the coming years.
So these are what the users are requesting.
And I did push the wrong button.
There we go.
Sorry about that.
So this factor of 1,000 improvement, what it will require,
well, how do we get there?
If we rely on technology evolution,
I'm not very good at this.
If we rely on technology evolutions,
that's about 10 hops if we think that every technology
hop is a factor of 2, 2 to the 10 is 1,000.
But given the pace of the technology evolution,
that's probably going to be 15 to 20 years.
And the average user is much more impatient than that.
So what it means is that we need a dramatic performance
improvement, and so that's where the use of accelerators
come in.
So another important factor that I mentioned
is power consumption.
So to get to this factor of 1,000, what you need
is also could be a great reduction
in the power consumption of all the components.
So you will be getting super green.
And actually, what you need is both components
that are much more performant for a power envelope much
smaller.
So I mentioned the accelerators as a way
to get a much better performance.
Actually, this needs to be qualified.
So this is on the top left.
This is magic.
It's going on by itself.
On the top left, you have a picture of our current blade
system, where we have the capability of capacity
to put two CPUs and two GPUs tightly interconnected.
And so it's very easy to compare side by side CPUs and GPUs
and to see where the different parameters are.
If I compare, for example, the gigaflop, the peak performance
or floating point of the GPU over the CPU,
you have this ratio of seven.
So that's exactly the key parameter that's exciting.
Everyone, so that's why everyone wants to get faster,
so that's why they get to GPUs.
Another important parameter is that the memory bandwidth,
the local memory that is embedded in the GPU,
is also much higher.
It's this factor of 4 and 1.5.
It's not quite the ratio in performance, but it's close.
And there is this nice ratio between flops and memory,
or the same ratio between flops and memory in the GPU
and the GPU.
The consumption, actually, however, on the GPU
is rather roughly the double.
And so that mitigates a little bit the interest of the GPU.
Now, if we look at the memory size,
the memory size on the GPUs is actually an eighth.
So when you combine all this factor,
actually, you realize that the application that
will be successfully accelerated with this kind of setup,
with GPUs, are actually some stringent constraint.
The first thing, and this is a constraint that may go away,
the kernel that you will accelerate have to be small.
Currently, the tools that you are using,
even though there are these open CL tools,
and required that the developers intensively
work on the kernels.
So the smaller, the easier it is.
As I mentioned, the memory size on the GPUs is smaller,
so that will only work if the model can fit in the data
set, can fit in the GPU memory, or if you have a very good data
locality.
Finally, because this GPU interface
to the rest of the world through a PCI Express bus,
you have a bottleneck for the communication.
So an application that do a lot of communication,
obviously, it's not going to work.
Nevertheless, the list of applications
that developers are able to accelerate with GPUs
is increasing day by day.
And so you see here a list of domains of application,
graphic rendering, seismic modeling, imaging,
molecular dynamics, astrophysics, financial simulation,
electromagnetism, gene sequencing, and more.
So the next slide here, we've plotted
the usual evolution of the performance.
But here is the magic is gone again.
The performance of the different components with respect
to the consumption per megawatt.
So the exact measure is a picowatt per flop,
or megawatt per exaFLOP.
So you want to go to get this factor of 1,000
from petaFLOP to exaFLOP.
You want to get under, let's say, 10 megawatt.
And if you look at the current trend of CPUs,
you see that you'll be well above that.
Don't forget that the vertical axis is a log scale.
So there is no way with current evolution of CPUs
you get there.
So the GPUs are the pink squares.
And you see that they started on a good, very nicely, actually,
because between the last version of the current one,
you get quite a nice factor.
The main reason for that is that since here I'm
comparing double precision exaFLOPs,
there was the current generation as much more powerful in terms
of double precision.
But overall, if we look at the longer term evolution,
it will follow the same line.
Now, if we look at the extreme limit for this power consumption,
it's given to us, I believe, by what
we see with the embedded microprocessors.
And you see that's well below.
And actually, that will enable you
to get to under the 10 megawatt per exaFLOP.
However, we know that these embedded microprocessors
what you get on your phone is not going to scale.
They want there.
I mean, there is no good memory interface.
I'm not talking about interconnection or anything
like that.
So the minimum that we're hoping for
is somewhere in between the GPU line and the embedded
microprocessor line.
OK.
So beyond how much?
Zero.
So this is my last slide.
So I'm happy that I have no time left.
So briefly, and I can expand on this if you have any question,
we see that scaling a system beyond the petaFLOP
is going to be a great challenge.
For that, we'll need better components integration
than the system.
We'll need to minimize the PUE and also take
care of the scalability of the software management tools.
But the greatest challenge in my mind
is for application to get beyond the petaFLOP
and even before that to get to the petaFLOP,
because currently we don't see many applications running
at that level.
And for this, we'll need to improve on our development
tools and to make sure that they are scalable.
Thank you.
Thank you, Dr. Fischer.
And questions are going to be from Dr. Lin
and Professor Berant.
Thank you, John Beer.
Thanks for your presentation.
Bool actually is, as I know, is a leading European HPC company.
So my question is, what's the key technology that
can keep Bool at the leading edge?
What kind of technology?
Key technology.
Well, first I want to reemphasize what you just said.
I mean, Bool is the leading HPC company in Europe.
And, well, probably the only one of us is a certain size.
The technology that we have in our portfolio
that can enable us to get beyond the petaFLOP
and maybe toward the exaFLOP are not
any different from what you can find in other companies.
And I've seen the presentation of the previous speakers.
In our company, we have ASIC expertise.
I mean, we have people able to design ASICs.
And, for example, we have been expanding the fat node
capabilities beyond the usual four socket to 16 sockets.
And we stopped there not so much because we couldn't go beyond,
but mostly because we didn't see a commercial interest
for building larger systems than that.
There are some use for this, but not
enough to justify the investment.
That's what our thought was.
Secondly, we have people designing boards and also
integrating and building infrastructure.
I mentioned that in the conclusion slide
that we are working on that actually
is desirable to get an optimal PUE, a better PUE.
So currently, we see systems where the PUE
is in the order of 1.7.
That means that for every watt, you
spend another additional 70% of that to cool the whole system.
We believe that with technology that we are putting into place,
we can go down to almost the holy grail, the free cooling,
whatever that means, but certainly less than 1.2.
And then finally, in terms of software,
we have a team of about 100 engineers dedicated
to R&D for software.
And these people are obviously taking care
of, in the first place, the management and the administration
and the monitoring and all the software tools
to deploy such a larger system, but also the application
development tools, for example, MPI application to scale
to the 100th and more 1,000 task that you'll have on your MPI,
but also to reduce the footprint of these MPI jobs
so that it doesn't take too much of your precious memory.
I think I have a second question.
Sorry.
The abstract of your presentation
mentions that a disruptive approach
is needed, encompassing all aspects of hardware, software,
and application design.
I didn't really get what kind of support
the user can expect with respect to application design.
You mentioned a number of limitations.
But what kind of help do you deliver?
Can you offer today to the user?
I put it there because it's on my agenda.
And it's certainly one of the challenge
that we would like to face and actually tackle.
And certainly, it's something that, well,
let's say that bull for most of his software stack
relies on open source.
And we do focus on certain point where
we think we can make a difference.
And I mentioned the MPI library.
We're also doing a lot of work around luster.
And the next topic probably for us to invest on
will be helping the developers to deploy,
to develop their scalable applications.
And so that will encompass performance analysis tools.
And also, I would say maybe not compilers,
but way to implement large scale applications.
And so we are starting some research projects of this.
But it's probably too early to talk about this.
I'll come back.
Thank you, Champierre.
Thank you very much.
So this brings us to the end of the hot seat session number two.
Please let's thank all the speakers again.
Thank you also to the audience for attending the session.
We now have a short coffee break.
And we look forward to an exciting keynote speech at 5.15.