Good afternoon, ladies and gentlemen. Thank you very much for coming back for the second
hot seat session. The inquisitors today to turn up the heat on our candidates are going
to be from the left, Professor Frank Berant from the University of Technology in Berlin,
excuse me, Berlin, Germany. Dr. Feng-Peng Lin, who is a director at the National Center
for High Performance Computing in Taiwan, Professor Thomas Stirling, who already gave
a talk yesterday, and who is a professor at Louisiana State University, and Professor
Kathy Yellick, who is professor at the University of Berkeley and the director of the NERSC
Supercomputing Center. And without further ado, I'd like to invite the first candidate
in the hot seat, which is Robert Murphy from Sun Microsystems, and he's going to talk about
open storage for HPC. Robert? Thank you. Next slide. I just tell you which slide to go to,
and that's how we do it. Is there a clicker for me? Ah, super. Thank you. I'll just use
this. So, that's a picture of the Beatles. This is Hamburg. This is where the Beatles
became a group and became famous. And I use this picture not just for the connection to
Hamburg, but if you note, I took a photography course a few weeks ago, and this is done from
a famous photographer, Bill Epridge, who you've seen his work. He did the picture of Robert
Kennedy when he was assassinated. But what I want to show you here is see all the professional
photographers and where they're positioned, and the picture that they got during this
scene. And then he moved to a different position to get a different perspective, and obviously
a much more valuable picture. And what I wanted to communicate was, particularly in this community,
in addition to following the processors and how fast CPUs are going. The other dimension that I
wanted to communicate to keep a track of for this community is IO and also storage. Here, if you
look at Moore's law, processing doubles every two years, but that's not the case in HPC. We use the
unit of a rack now in HPC in 2005, 42 rack servers, 500 gigaflops in a rack. Our halon rack,
a constellation rack, has nine teraflops now. And if you just look at the roadmap for some of these
other processors, using Sandy Bridge as an example from Intel with 32 flops per core,
eight cores per socket, four gigahertz, you get to 172 teraflops per rack. Just to put that in
perspective, the 200 rack machine at Oak Ridge, Jaguar, would be eight racks in this time frame
in just a couple of years. So processing is getting faster and faster, and data is also getting more
and more. This is IDC showing the geometric data increases over time. Things that are driving that
in HPC are increased sensor resolution across L domains, CT scanners, confocal microscopes, genome
sequences, et cetera. This happens to be the biggest data generator in the world. This is the LHC
collider at CERN, 27 terabytes per day when that goes online. But that's not really the problem. The
problem is this is an example of desktop, laptop machines inside labs and hospitals around the
world. That on the left is a genome sequencer, on the right is a confocal microscope. Both of those
generate one terabyte of data per day. And just like computers, what has happened, these devices
are proliferating because they're getting lower costs, and instead of having one per university,
now there's one per department, one per lab, and now one per investigator. So this is what HPC is
dealing with moving forward. Not just the acquired data, but also simulated data,
higher resolution simulated data. So if you were to take those two parameters, and if you didn't
do anything else, you have the time to load data, time to compute, and time to store data. In 2011,
the HPC workflow will be consumed by moving data on and off of storage, and very little of it will
happen in a processor. So Sun has a various number of technologies to mitigate this. I think
we're at the forefront of understanding this, and we have three types of storage, actually four,
parallel storage, our famous Lustre file system, network storage, our new Amber Road series 7000
file servers, and then also archive storage to move huge amounts of data in and out of the
processor complex of a HPC system. Just a shout out for our
computing part of our system. This is the new Constellation system with NahelaMinute and onboard
QDR and Finoban, and also it has onboard flash drives there too, very, very high performance,
and also very high ability to get data on and off the processor. I want to spend a little bit of
time on this. I took the top 500 list and I plotted the top 10 supercomputers in the world
on efficiency versus the manufacturer. These are the top 10 systems here, and you can see
the Hewlett system QDR and Finoban has a efficiency of 0.89. Our older machine, the TAC
machine with DDR and Finoban is 0.75. The LANL Roadrunner also uses DDR and Finoban, 0.76.
And then, Dr. Goh, I noticed the SGI machine was very efficient at 0.8, and so I did a little
investigation in its dual rail and Finoban, and also the NPT you use to get that high efficiency
number, and you should be acknowledged for that. And then the Cray machines at the mid-70s
at Oak Ridge. So one of the things I wanted to point out is this new era of high efficiency with
the QDR and the Halem combination, and to challenge the community, because if it's not
doing LIMPAC or actually computing, it's just generating heat in your bids and RFPs and
proposals moving forward, to challenge the vendors with an efficiency requirement on LIMPAC
and set it at a high bar, like 0.9. And I think that would be something to consider moving forward.
So how we mitigate this at Sun is using Flash. Flash, we've integrated into a lot of our
computer products, our server products, and our storage products that gives tremendous
I.O. versus hard disk drives, and also generates very little power. Here's an example. We did ANSYS,
which is a MCAE application, most popular MCA application. We did two SSD drives and then two
SATA hard disk drives. Blue is the hard disk drive. Orange is the SSD drives. You can see
that we're able to cut up to two to one the completion time just using SSD drives,
and the same processors. This is our new Halem servers here. And then the whole server suite,
whole benchmark suite, 32 percent cut in time to solution. In addition to Flash,
we know we have our Lustre storage system, 42 percent of the top 100 use Lustre. And again,
we have this technology that can get data in and out of the supercomputer as these processors get
faster and faster. And then finally, we have a new NAS product that takes SATA drives for
high capacity, low cost, and fronts them with SSD technology for high IOPS and high sequential
throughput, combining the two with our ZFS file system that can automatically migrate data back
and forth between the slower drives and the higher SSD. And that gives us industry-leading
performance, 280,000 IOPS, at a fourth the price and 40 percent of the power consumption of the
existing NAS systems that are out there. And then finally, we have, as you know, a very
industry-leading tape archive capability. And I think tape is about to make a renaissance again,
mostly because of its green attributes. It's the lowest power consumption online, near-line
storage on the planet. And this is able to migrate data in and out of a Lustre file system using
our SAM storage archive manager product and provides a very cost-effective and very reliable
way to store data up to 30 years. So I started in the computing side of HPC. It took me many years to
have my mind, change its perspective to see how important storage is. I don't expect that to
happen in just 10 minutes here, but I did want to share with the community this idea of how
important storage is moving forward. I think computing, for the most part, has been solved
by the manufacturers that we source these things from. So once our mind is stretched by this new
idea, it never regains its original dimensions, this quote from Oliver Wendell Holmes. And I think
if people were to consider storage and IO moving forward, I think it would change a lot of the
results that we're seeing. For example, Jaguar, if they had the QDR and Finneband combination with
the Halem, they'd be number one. If NERSC had that combination, you'd be in the top 10. And
I think that should be in people's minds moving forward. Thank you.
The first inquisitor of the afternoon is going to be Frank Bieland.
Yeah, thank you very much. I do have a less technical question, but you stated that each
researcher produces roughly a terabyte of data per person on various kinds of equipment, various
kinds of experiments in your company, and many other companies are interested in storing all
this data. But he also had this little picture demonstrating that retrieving the data and
restoring the data takes more and more time. Is the technological answer of making storage
faster, bigger, really the real good answer? Or would you consider also other ways of dealing
with this waste amount of data? Yes, in many of those institutions, the first rule, it's currently
not being done, but I think it's going to be a reaction, is managing that data in the workflow.
Right now, everything from the device goes into the institution's data pool. And what you want to
do is only send the stuff that you really need in there and try to cut that off and only send the
only important data. Because once data, what we found at least up to today, is once data gets into
the storage, it stays there. And the best impact on workflow or the amount of data,
amount of storage you need to buy, is to call that data before it goes in. Because once it goes in,
it's very hard to call it afterwards. So workflow management is the answer, I think, for that.
Please allow me to follow up on this question. Do you have any concept, or does your company have
any concept on supporting scientists in doing this workflow decision, how to deal with original
data? Because workflow is an interesting term, but nevertheless, something has to be done in
this data process. Not our company, but the software that these people use, the accelerates,
the biotech analysis companies, that's their biggest growth industry moving forward, is
workflow data management. So the manufacturers of these devices, and also the scientists,
the software they use to analyze this stuff, that's a critical component of those manufacturers.
Second inquisitor today is Catherine Yellick. I'm wondering if you could say a little bit about
how you see the role of Sun within the HPC community growing or shrinking under the new
model as part of Oracle? Yeah, unfortunately, I can't answer that. Okay. Can you say anything
about the architectures that you think are best supporting data analysis? In particular, some of
the Sun systems such as Niagara and other sorts of... Well, yeah, so there's a... That's also a
touchy topic, and I don't think I could... I talk about the Niagara roadmap. I can't comment on that.
So yeah, that... Niagara and chip multi-threading is beautiful for data analysis and those types
of applications. And we have a strong existing product line that's multi-billion dollars a year
using the current Niagara chips. And there's some unconfirmed rumors or statements going around
the future of that. And I can't comment on that, but you're right. Chip multi-threading is a fantastic
mitigation technique against the hierarchical changes in memory access, and particularly for
data analysis. And Niagara is... Any chip multi-threading processor is really good at that.
It's been shown to be good at that. But I don't know if I answered the question that you didn't
ask, but there's a... The roadmap for that product is now kind of in the press, and unfortunately,
I can't answer anything about that either. Okay, thank you very much. Thank you very much. It's
a pleasure. Thank you again. So the next speaker is Mr. Raphael Wong. Mr. Raphael Wong is director
of product management at Supermicro in the USA. And he will give us a talk on energy efficient
computing challenges and solutions. Good afternoon, ladies and gentlemen. My name is
Raphael Wong. Today, I'm going to talk about the energy efficient service solution for HPC from
Supermicro. First, I'm going to talk about the history of Supermicro in delivering energy efficient
HPC solution. And I'm going to talk about the latest product offerings from us.
Supermicro was founded in 1993 in California, Silicon Valley. As you can see, we've been
profitable since day one. Our NASDAQ symbol is SMCIs. If you like, you can check out our
stock performance. Since day one in 1993, we've been shipping the server board. And a couple
of years later, we're in the chassis market. And then we developed energy efficient service
solutions back in the 2002 timeframe. And recently, we have got a lot of industry leading
solutions. For example, two GPU in one U, twin based solutions, two U twin solutions with four
independent nodes in two U space. I'm going to talk about more of this solution later on.
Let's look at our history, our involvement in HPC. Perhaps a lot of people are not
aware of our existence. But let's look at the performance, our involvement in HPC area.
We shipped over 15,000 one U four way, one U four way servers. We shipped to all these
major national labs like Lawrence, Livermore National Lab, Sandia National Lab, Los Alamos
Lab. And we have thousands of this kind of solution, for example, deployed in a tri-lab project.
And if you look around in the exhibition hall, you notice that there are a lot of
Supermicro products being demoed. At least 10 books you can find a lot of Supermicro products.
And if you look at the top 500 list, in fact, a lot, even though Supermicro is not listed as the
vendor, but the inside system, a lot of the system are actually using Supermicro solution.
For example, the number four ranked SGI supercomputer is actually using Supermicro motherboard.
So you see that involvement. And we've been working very hard to deliver the most
cost effective, the best performance, but what the most energy efficient
solution follow up the HPC vendors. In fact, a lot of the top tier data center or EPC solution
provider, search engine company, big data center actually using our Supermicro solutions.
Certain, the Large Hadron Collider Project, for example, are using Fringer Plus Superblaze
in the very, very important, highly visible project. And in fact, over half of the
high-end medical systems all over the world are using Supermicro solutions.
So you can see that we've actually been in the HPC market for many years.
What are the challenges? Let's say from the hardware side. As you can see, the CPU core
can increases. Just a couple of months ago, the core can is only four cores per CPU,
but today it's six core from AMD Istanbul. AMD is going to have eight core and 12 core coming up
in early Q1, and Intel will be having similar offering in this kind of multi-core solutions.
As you can see, the core can increases, density increases, power consumption increases,
so it is very important to deploy energy efficient green solution. It is of particular
importance for the HPC market. And there is a lot of things that we as the hardware vendor
can do to help accelerate the deployment of green solution. We have to look at the firm most
solutions. We have to look at the reliability of this kind of high-density solutions. I'll show
you a little bit more about what we're doing. Let's look at the industry-leading recommend
solution. Supermicro is the world's first vendor to deliver this kind of twin solution.
We have two independent nodes in one new space. Two years ago, we shipped this kind of solution.
We called it one new twin. And recently, we delivered this kind of two-year twin
square two solution with four hard-plug independent DP nodes in two-year space.
Each node got its own hard-plug three-and-a-half-inch drive in the front. And we also have the two GPU
in one new solution launched recently with two full-in-full high performance GPU. Each will give
you one teraflop with two GPU in one new space. You've got two teraflops in one new space. It is
the fastest one new in existence on the planet. So whoever would like to use this kind of GPU
system for HPC application, it is the best solution because if you look at the GPU solution,
each GPU card gets its own PC Express Gen 2 by 16. Not by 8, but by 16. So we've got
non-blocking performance. Let's look at the industry leading blade solution from Supermicro.
We've been shipping Superblade for close to two years. We have different kind of application
optimized blade solution. We have optimized solution for the personal computing. With 10 blades,
you can have super quiet blade, 50 dB with 10 dB nodes. Or we have the data center optimized blade
with 14 blades, 7 new space. We ran LINPACK and managed performance. We achieved over 350
gigaflops per kilowatt with 14 blades and networked together with a gigabit switch. Just gigabit
switch. We assume that the performance will be a lot better if we use infinite band. It's just
some kind of simple experiment that we've done in our lab. We will have the so-called twin blade
solution coming up. It is the world's best highest density blade. We can achieve up to 28 dB blades
in 7 new space in what chance, QDR, IB switch, et cetera, et cetera. And the power supply
efficiency can go even higher than 93%. We will have a 94% gold level power supply for our twin
blade solution. So what's especially about efficiency? So let's look at the 93% efficiency
offering from Supermicro. As you can see from a chart, if you look at the tier one, the best
you can get from tier one when there's roughly about 90%. But from Supermicro, all of our solutions
I mentioned earlier can give you over at least 93% efficiency. It's not trivial. If you compare the
interest average, you get probably about 65% to 75%. But if you look at our power efficiency chart,
we achieve over 90% as a very light low, around 20% to 25% range. At this kind of light loading,
we achieve 90% plus efficiency. And it's not that easy because we have to do a lot of experiment,
we have to put a lot of expensive component into this kind of high efficiency power supply.
And not every vendor is willing to invest that much money into this kind of high efficiency
system. Why? The bottom line is that save money, right? Let's compare our 93% efficiency to an
average, interest average, one year of red money solution, whatever, that will give you only 75%
efficiency. So if you look at the chart, I'm using the electricity cost for the US market,
not Europe. It costs about 9.4 cents per kilowatt hour. So in summary, with four twin blades in one
7U chassis, you save close to $6,000 in three years. The energy cost in Europe and Asia is at
least double this number. That means that you can easily get one blade system for free in three years.
So you can see that there's a lot of reason for you to use energy efficient competing solutions,
particularly those from Super Micro. Let's look at some of the challenges that we face every day.
Look at the back of the rack. It's messy cabling. It's very, very difficult to deploy system to
debug and to maintain. Poor air flow, obviously, because you have so many cables behind the rack,
messy cabling, that means reliability problem, thermal problem, and high energy cost.
What would be the best solution for you? It's a blade solution. With blade, you've got the cable
reduction benefit. And if you look at the solution from Super Micro, let's look at the four-way blade
from Super Micro. It is beautifully designed. It's so symmetric, thermally balanced design with four
CPU and 16 DIMM. In this model, we use high efficiency VRM, high efficiency power supply.
We use the PWM fans. We would let our user to remote control the thermal and everything
over the internet. So with the twin blade, we achieve 16 teraflops with today's solutions,
2.8 gigaflops Istanbul, with a twin blade solution in Q1, with the 12 core, we achieve 24 teraflops.
What does that mean? You can easily use only 50 racks standard rack solution to achieve one
petaflop solution. And we've been winning a lot of awards because of this. And because of our
achievement in the green solution, CERN selected our blade for this. So you can see a large
LSC project. So in summary, we have different kind of solution that will give you the best
density, best energy efficiency. And again, and again, so please visit our booth, number 310,
if you have any questions. Thank you. I think I'm out of time.
Thank you, Mr. Tawong. First,
first, and visitors, Thomas Stirling.
So some of these numbers are very impressive. You talked about your energy efficiency and you
emphasized the role in the power supply design. Many of us appreciate the fact that
there are active solutions within the computing system itself that also could be applied
to significantly reduce the power expenditure. Does Supermicro take advantage of any of those
approaches? Actually, in fact, we are looking at different ways to further improve the energy
efficient system. For example, with just regular air cool, fan cooling system, we can achieve close
to 24 teratops with just one rack, 42-year rack, not proprietary rack. But if we use liquid cooling,
water cooling, that kind of solution, we can achieve even higher. So we're looking at different
ways to further improve the energy efficiency. Thank you.
Thank you. We know that a different component of the system actually consumed power differently,
consumed energy differently. So my question is, you only used power efficiency as a matrix.
I wonder why you choose these metrics to do the comparison. Isn't it possible that you can
introduce other metrics? And please explain the reason as well. Thank you.
So power efficiency, it is one very obvious way for us to understand the benefit. Of course,
there are so many ways to do it, but it's something that's so common and obvious for us
as the hardware vendor to do it. There are so many ways for you to further fine tune
the system through software, right, from a storage site, the demand of package you can
fulfill the channel, that kind of thing. But right now, we're just focusing mainly on the
hardware side. In fact, because we are the hardware vendor, but at the same time, we also further
strengthen our software department. So we're going to have a lot of new software coming up
in the next six months, I would think they would help in the software area.
Thank you very much, Mr. Wong. Thank you.
Our next speaker is nobody else than our very own Frank Pitke,
who is the global director for HPC at Hewlett Packard.
Good afternoon, ladies and gentlemen. Pleasure for me to be here, talk about new developments
and trends in scale-out computing. And as you know, our focus on high-performance computing is
a bit different from other vendors, so I would like to put the emphasis on that today.
The idea behind a new organization before it was in HP was to look beyond classical HPC,
and to embrace also back to zero environments and enterprise scale-out. And of course,
embed this into this new trend, this new technology well-known and discussed here called
cloud computing. And I want to make clear here, and it came out in the discussion yesterday,
we are not looking at it that is very, very similar. The image here should visualize,
there are some overlapping areas, and what we want to do is leverage the overlapping areas as much
as we can. Now, before we dive deeper into it, I'm proud to announce, of course, that in terms of
entries, we continue to lead the top 500, and the philosophy of HP is to do this
by sticking to standards, to standard components, standardized designs, but a rich portfolio of
designs as much as we can. And I'm also pleased to say that our blade architecture, the C-Class
blade architecture probably is the most successful architecture ever to enter the top 500 with 207
C-Class blade systems as of today. So this is a very nice success story for a single blade
architecture. One of the reasons for that success is the rich portfolio. I don't want to go into
the details here. Basically, you know, there are notes for all kinds of requirements, lots of
storage, less storage, very high density, large amounts of RAM, and a rich size of interconnects
and storage options. So I would like to look at options that are of key interest for the HPC user,
and of course, if you use a standard approach, you ask yourself what differentiates such a
solution from a more specialized HPC architecture, and of course, interconnect technology is one of
those topics. So what you see here is a comparison of a couple of interconnect options. We continue
to evaluate very carefully those options to check whether a standard-based approach is appropriate
for high-end architectures. Then you can see that InfiniBand QDR on the right side, in terms of
latency, the green line and bandwidth, is doing quite well. So at this time, we feel
that we don't need to go to proprietary interconnects and continue with a standard-based
approach and offer this in the context of our blade architectures. Does it mean that we
no longer focus on classical rack-based servers? The answer is no, definitely not. BL is a blade
series. This is a model, and I'll just show you one, of a classical rack-based server, and yes,
there are customers who want to stick to the more classical form here, not going to blades,
and they actually make good news of this new technology called accelerator. So what you're
seeing here is a Tesla-based server which primarily goes into the oil and gas market.
It's a very successful product, and we are sure it will also show up on the top 500.
Now, this is a very special way to go. What else? Thinking about massive scale-out and thinking about
lowering energy requirements, and this is basically looking to a different direction. It's
kind of a bare-bones server. So if you have massive scale-out requirements and you want to get rid
of everything you definitely don't need, and of course, reduce the price, then this is a different
approach. This is the so-called SL series, and what you're seeing here is a two-by-two server,
and it's a member of a new family that got just announced a few days ago in June.
What we did here with this new series, we took off everything that is not really needed,
that includes also TIN. So the internal code name of this system for quite a while,
while it was under development, was skinless. Now, of course, the server can't be completely skinless,
and I don't want to go into the details here, but you see here is components with very little
hardware and TIN. You see a lot of holes here, and then this goes into a tray, and you see the power
supplies over here, you see the fans, and you see again a lot of holes here to reduce weight,
and of course also to save material. Now, in certain areas you can't save weight. What is
the reason for that? The reason is you have to observe fluid dynamics because the airflow has to
be extremely efficient. Why? Because if it is not sufficient, you're wasting energy again.
And it has been said here again and again, you want to make sure that the system is efficient. Yes,
the power supplies we are using are extremely efficient, and we do a lot of research and
actually got a lot of patterns in that area. Here, you continue to build up a server, so there
are specified bulk rails which allow us to build up a rack-based environment, but you can also use
standard racks for this environment. And at the end, you end up with racks of the new SL lines,
and we have customers who have ordered hundreds, and we are expecting thousands of those to be
deployed shortly. Now, I said initially we are somehow looking at HPC, but simultaneously also
the VEP area, and of course cloud computing, and this is always a very interesting theme here, and
Richard Kaufman came up with this slide yesterday looking at trends. You know, if you Google trends,
if you look at who is looking for good computing, distributed computing, high-performance computing,
that's a green line, and cloud computing, and you see cloud computing is very, very
in fashion today. So what does it mean and what is our approach towards cloud computing?
If you look at typical HPC nodes today, that's the one that made it to the top 500,
of course you see that the majority is InfiniBand connected. There are classical HPC users that don't
need low latency requirements. They are fine with Gigi. The top 500 today is not just us,
it's dominated by blades. Integrated InfiniBand simplifies cluster layout and can reduce total
cost of ownership, but those servers aren't really hot, and the power virus of course is
linpack, and there are many applications that are bandwidth and floating point intensive,
and that makes the system hot. About 250 watts per server to socket server, the average cloud nodes
are more or less, and actually at this time 100% Gigi connected, tin cans as shown, and about 160
watts per or even less per two socket server. Now if we compare a current, as of today, a current
cloud with an HPC data center, we see differences and similarities. Both exhibit massive scale out,
both have very few node types, but with HPC you have a lot of power per act, more modest on the
left side, and the interconnect is massive more towards the HPC side and more modest towards the
cloud side. And here comes a point I want to focus on at the end of my talk here. Node reliability is
a key feature here, because on the HPC side node reliability is very, very important, definitely,
and the cloud and web world is willing to accept some trade-offs here, and finally this is more
an optical thing, the typical cloud data centers are ugly, and the typical HPC centers and many
heads of those centers are here are very, very nice. So what is this area of, is it okay to accept
errors and faults? Cloud applications want the cheapest node, and they're willing to accept node
failures, and even wrong answers up to a point, and so there's a trend to implement reliability
features in software with less need for reliable hardware. HPC applications definitely need a
higher node reliability, and wrong answers are not only bad but unacceptable for obvious reasons.
What is the trend? The trend I'm seeing here, we're seeing, is adapt similar technologies and
do things that allow to recover from errors in the software environment, but system level
checkpoint restart is no longer a solution. So I believe future applications have to include
tolerance at the application levels, and the solution we are also providing, having
hardware fault tolerance in the enterprise space is not acceptable to the HPC world,
HPC customers will not pay for it. Thank you very much. Thank you, Frank.
Kathy Yellick, please.
You described a model of putting together these integrated HPC systems and also cloud systems
that highly leverages commodity technology components, and I'm wondering what you see as
some of the major technology challenges in the next 10 years, and in particular whether you see
this model as being an effective way to get to, say, an exascale.
That is a good question, I'm tempted to say. Does it work to go to exascale? My first answer is
what I expect that that technology will carry us for sure deeply into the petascale range.
Yes, and what I expect actually is technologies I did not show today here, for example,
the ugly data centre approach, where you put those servers into pods, into containers,
you actually move the whole data centre to areas where electricity and power is cheap,
and basically run it remotely. So what I expect centres that look like a cloud centre today,
but are actually serving typical and classical HPC customers, and I can imagine that those,
again, I'm making these ugly centres here, a garage of containers, will be somewhere at
remote places. Whether this technology will allow to go to exascale, I believe there are a
couple of barriers to overcome, and the biggest barrier I'm seeing here is not a technological
barrier, it's the economic barrier, who is willing to pay for it, at the vendor side and at the
customers. Tom Sterling. Good afternoon, Frank. Thomas. There are so many questions I would like
to ask, but I have to represent the people in the audience. Let me key off of my colleague's
question, something more modest than exascale. Your competition, Cray and IBM, have both deployed
petascale systems. HP nonetheless has the largest deployed base of any single corporation.
When will HP enter the petascale arena?
I think we have a sales rep waiting outside.
The answer is, as soon as we have customers who are willing to work together with us on a petascale
standard-based architecture. It has been set several times. Of course, we have groups that
provide special hardware, special setups, and as we do with web hosters, as we do with
cloud hosters, and I would like to come up with one example that is one and a half years old,
that for a while was the Asia number one machine. We worked very closely with the
Tata Engineering in India to define the topology of a system. It's InfiniBand and it's 460c
blade-based to come up with a machine that is the 100 plus teraflop range. We would, Thomas,
to answer your question, follow exactly that pass for a system that goes into the petascale range.
Well, I don't usually have follow-up questions, but then, since you didn't want to answer my
question, I'll ask you another one. When will HP produce a product that someone will want to pay
for at the petaflops level? The critical word here is when.
I'm in the situation, as the previous speaker said, I cannot give you a date for that.
Yes, but what I wanted to say, yes, we are definitely looking into petascale systems,
and we are talking, as we are here, with a couple of prospects or customers,
how to design such a system based on the technology we have available.
Thank you, Frank. Thank you very much.
So, our next speaker is Dr. Eng Lim Goh, who is the Senior Vice President and Chief
Technology Officer of SGI in the USA, and the title of his talk will be
Scalable Architecture for the Manicore Era.
Yes, I apologize to my desktop. Thomas, this is the first time I'm here,
and now I know why they call this a hot seat.
Thank you. A few years ago, when we were embarking on our next generation HPC architecture,
called Autovilot, as Chief Engineer, moving up to CTO, I had a heavy responsibility to ask myself,
what are we going to spend this money on? It's the order of the $100 million program, and
credit to Christian Tenasekiu, actually based in Germany, we sat down, worked through application
performance, and decided that this is where we're going to focus the R&D on. Essentially, the fact
that the world is moving from clock speed improvement to Manicore. We believe going forward
this will cause problems for some applications, and eventually more applications as you are forced
to scale your codes. Let's look at one example that we looked at in the early days. This is fluent
over a standard model of a 40-foot truck container, and this is not us, it's just a standard code
running over 128 cores, 256, 512, and 1024. Then we did an analysis of what went on in those
runtimes, and realized that the dark blue area was communication, and the light blue actual
computation. Essentially, by 1024 cores in this model, you are spending more time talking than
you are working. That's another way of putting it. If you scale that to 100%, the same chart,
just 100%, you can see that at 1024 in this model, for this application, you are spending about 70%
of your time in communication, and the remaining 30 computation. Further analysis shows us that
blue area are essentially dominated by MPI collectives. Then further analysis beyond that,
we realized that there are lots of micro imbalances in the threads, where the reduce or the barriers
are waiting for other unbalanced threads to finish. But net-net, there is still a problem to
solve. As the world forces you to scale your codes to 1,000 core this year, 2,000 core next year,
4,000 cores the following year, and so on, because you have no more clock speed to help you out,
we believe this will become more and more a problem for more and more codes. As such,
it's the ultra-violet program, where we focus on partnering with microprocessor vendors, where,
for example, Intel focuses on compressing the light blue, then our responsibility is to find
a way to compress the dark blue for you. All reduced looks just like this. You collect all
your 10,000 threads, sum it all up, and then you broadcast the result back to all of them. A typical
system, if you did gig E, order hundreds of microseconds to do that all reduced. In Finoban,
orders of tens of microseconds. The question is, can we solve it instead by using a communication
accelerator that accompanies every two microprocessors, say, to do these accelerations in
hardware, do this MPI communication in hardware. This gives you a very high-level diagram of this.
As a technical guy, I would want to spill all my beans, but I've been told this is an open session.
We'll do it in a separate session privately for individual institution if you are interested in
the details. Essentially, what we do, the first thing is we give every of the chipset that we
build as an ASIC 53 bits of physical addressing so that that chipset will be able to see every
blade's memory. So essentially, every blade you plug into the system, its memory joins a collective
and is visible by all chipsets in that system. Essentially, this system becomes a globally
addressable memory system. We believe this approach will first cut down lots of chit-chats
that is required to complete an MPI transaction because the receiving thread can actually see
the data already. It has global addressability. The next thing we did was we shrank this chipset
to 90 nanometers, and in there, I told the engineers, somehow you're packing an MPI offload
engine for me. We were 2X off two years ago, and I told them, don't come out of the room. They
didn't spend two years in the room, but don't come out of the room until you pack it all in.
They worked it out that they could put a barrier in logic, in hardware, a reduce,
a synchronizer, and a point-to-point, a get-put engine, and a gather-scatter engine in there
with, as I think I've mentioned, a synchronizer. The goal here, therefore, is for every two sockets,
I've only shown one, for every two sockets, you get one of these MPI offload engines to do your
MPI communications for you. We'll build an MPI library to which your straight MPI code will sit
on top of, and then you'll automatically exploit this hardware. It is not just the MPI offload
engine. It is also the global addressability that makes the MPI engine works in a very
shortcut way, when it does a send-receive, when it does a reduce or a barrier.
The way we'll do it, Dan, is you have a bunch of rack, a rack with a bunch of blades. The
blades will have two Nihalem E-axis with the MPI offload engine there. That chip called UV Hub
will contain a high-level feature. One, a global addressable engine, 53 bits to see every blade's
memory. Two, an MPI offload hardware logic. That's one rack. That's one blade over there
on the right, and then that's two blades connected. Those blade pairs that you see,
this red circle actually holds those blade pairs, and that's how one rack is connected.
And that's how two racks are connected, four racks are connected, and that's how four racks
form 1, 2,000 cores, 16 terabytes of fully cache-coherent globally addressable memory,
globally shared memory. And then if you want to scale beyond that to petascale, you would then use
a 2D torus from a factory, and every gray dot is one of those four racks. To give you the
bisection bandwidth, we'll repeat that 2D torus 64 times. And what will happen here is that for the
one quarter million cores in this system, when you do a send-receive barrier or a reduce
with a straight MPI code, the MPT library will take over from that and do all the barriers in
hardware and so on, with global addressability capability. So you have one Linux operating
system running in each four racks in a gray dot, and you just have multiples of those Linux operating
systems across those 16 by 8, but albeit that you have 16,000 moles MPI offload engine to do all the
communications for you. So the summary here is that, yes, we need computation accelerators,
as we have spoken of them a lot, but our belief is that the many-core world will force us to need
accelerators of another kind, and they'll be called communication accelerators,
and in this particular case, we have, you know, wrote the API to make it run as an MPI offload engine.
Thank you.
Thank you, Mr. Goh. First, thank you, visitor Fang-Pang-Ling.
As known that we cannot assume that current software module, in particular for multi-core
SMP, will scale up to many-core. So as a software developer, I would like to ask
in this kind of architecture, how you will convince the current multi-core software
developer to migrate to this architecture? In fact, we don't intend to. Our plan is as follows.
That's a brilliant question, by the way. Our strategy is we don't intend to convince them.
The idea is we will just go to all the MPI codes out there, and then we'll ask all those that say
that who have their codes that have stopped scaling and would wish to scale them further,
and don't wish to go into the R code to change their codes because they don't have the time or
resources. For those customers, they say just run it on this system without any code changes,
and perhaps if you are scaling to only 4,000 cores a day, this system can help you scale to 8,000
without doing any further work, just instead of spending the money on software developers,
spend it on this machine. No, I shouldn't say it that way, but it could be a split between the two.
This gives you a bit more of a head start, a bit more of a push. I hope that answers your question.
It's a tough one to convince ISVs and software developers to enter that process.
Second front, Bernd.
Essentially, part of your statement is that the users will have to further adapt to
the challenges resulting from the increasing number of cores in their systems, but wasn't
computing about helping people, solving problems, and not bringing more problems to them.
Essentially, most of the scientists in this room, at least to non-computer scientists,
are more focused on the physical problems as they want to deal with the chemical problems,
and so on. The question is, to what extent do you expect that in the future development of this
multi-core, in what color and taste to ever architecture we are looking at, and to what
extent new algorithmic developments will be able to assist the final user, people like me who like
to solve energy-related problems and not solve problems in a very deep level of a processor?
To what extent do you see that those people deciding systems like your company and people
talking or thinking about algorithms will come back together in a more efficient and a better way
so that the end user will be able to do what he likes to do? Tough question, yes. It reminded me,
as you were describing this, about scientists being scientists, right? One day, the day I met
Professor Stephen Hawking, I remember him painstakingly keying this in and saying this,
right? We are scientists, not computer scientists, right? So we don't want to do computer science
work in order to achieve our science. It's a tough problem. Why? Because power, because of power.
You're not going to get any more clock speed improvement because all these years you had it
good, we had it good, right? Clock speed kept improving. You scaled your codes to four cores
and you just folded your arms and sat down, and every year you just hoped that those four cores
became faster. Not anymore, right? The reason was all these years we managed the lower voltages
to keep the power in check as we increase the speed, clock speed. And the voltages have gone
down from when I started was five volt for microprocessor, five volt, three volts, two volts,
one volt, zero point. I told my engineers zero will be the minimum, right? And we are almost there,
right? So when you have no more voltages to help you, every time you touch the clock speed,
the power goes crazy. That's it. Unless something very profound changes, right, in the next few
years, I think it won't be at least in the next two or three years, hard to say after five years,
something profound changes in the way we design chips, silicon, right? We're going to have to
live with it. Yeah, it's my opinion, right? And what we will do, have to do therefore is to
cooperate. Therefore we believe that's the main reason why we went with this offload engine with
the idea that if you can change your code, do so if not, we'll help you. Let's drag the time out
until we reach maybe five years where something profound happens. And I suspect it will. But I
can't see that far. Three years, we have to live with it. I hope that's satisfactory. Any other
questions? Thank you very much. On time. Thank you. Our next speaker is Michael Cregan. He's CTO of
Mellanox Technologies. Hi. Great speaking here. So I'll talk about basically covering topics that
were touched by previous speakers, both HP and the SGI is what needs to be done in order to
utilize better high performance computers, power and money, and how do we move forward to the
cloud. So basically, at the nutshell, the high performance computing is now more and more about
clusters, how to make many, many computers, many, many cores do a job or multiple jobs at some
particular characteristics. And basically, cloud computing is using very similar technologies. And
what we see right now is there are some sort of a waterfall of the technology and the products
from high performance computing to the cloud computing. We at Mellanox is all about delivering
the interconnect. So application that uses standard application interfaces will talk to us and they
will make sure that whatever is needed for the remote peer or IO will be delivered to this
application. So sort of starting right from the bottom line, it's all about return on investment
and it's all about efficiency of getting back what you paid for. And if you look at the
just statistics of top 500 list, InfiniBent is the most efficient interconnect to create a cluster.
If the most recent cluster on the list, which is a Julia machine, actually exceeded its goal
in the performance at almost 92% of efficiency. If you go to the other interconnects like Gigabit
Internet, basically about half of the money that you spent on your machine is thrown away. It is
either heating your room or just staying idle. So bottom line, it's all about efficiency. It
starts in the high performance computing. It's obviously relevant for the cloud computing where
IT becomes a competitive advantage. So Mellanox as the provider basically deliver end-to-end
solution from the API on the server to the application through the software stack adapter,
cable switches to the other side. And right now we are more than a year as of now with the 40
gigabit per second, actually we've introduced our first 40 gigabit solution end-to-end in this
conference in Dresden last year. This is the statistics of the top 500. Basically, InfiniBent
is experiencing steady growth. It is almost 60 machines, I think 59 machines out of the top
100 machines using InfiniBent interconnect. And it's basically InfiniBent growth on expense of
S&P machines proprietary interconnects and Gigabit Internet pretty much stays flat.
This is just yet another illustration of what happens on the top 500 list. So again, InfiniBent
gaining a market share, it's covering of the top 100 machines on the list. So what needs to be
done in order to scale out further and further above petascale? Actually, the petascale machine,
the first machine that broke peta-flow barrier is also based on InfiniBent. So the interconnect is
a key. As a previous speaker mentioned, there is a lot of time and inefficiency when interconnect
doesn't do the job right. And it means that interconnect needs to take care about itself
and it doesn't need to burden CPU and it needs to scale in a way that resources that are needed
on a single node are growing with the power of the node and not with the number of nodes you're
connected to. So the scalability and ability to deal with the networks on its own, starting all
the way from the transport level, going down to the failure, recovery and resilience, it should
be done by interconnect itself without involving any cycles on the computer. This has to do with
the ability to recover the failures and the ability to allocate resources and allocate and
use them on demand rather than allocate them up front and then use when we need them.
Last but not least, power becomes an issue. It is a hot issue. So the green high-performance
computing is actually based on InfiniBent. So if you look at all the characteristics that are
required to build a very efficient high-performance cluster to run HPC jobs, a single
job on every machine, then basically you need a very adaptive network. You need to make sure
you can support multiple topologies. You need to have very reliable hardware. You need to offload
certain operations and the previous speaker actually touched a very sensitive topic about
the offloading of operations from the CPU to the hardware. You need to be able to deliver the
quality of service because you will run multiple type of IO operations, either high-performance
computing, and therefore you need to make sure that each type of IO gets its own share of network
resources. And last but not least, you obviously need a very high bandwidth and low latency just
to deliver performance. If you look at the key characteristics that are needed for what's called
cloud computing, they're pretty much the same. There may be a different focus on different
aspects of these solutions. There may be some other things that are in the core of the network,
but at the basic level it is the same characteristics that are needed from the IO interconnect
to deliver high-performance cluster are required in the cloud. And actually what we see today is
that high-performance, the infiniment that started in the high-performance computing
debuted in 2003 on the top 500 list is now actually very popular in what's called technical
computing, technical HPC or commercial SPC. And it's going now to the data centers. There's some
financial institutions on the Wall Street are building or basing their data centers on the
infiniment. So in terms of the concept of the cloud computing, basically what consumer sees,
it sees the utility computing. It sees the computing and the storage as a services.
What IT manager or data center owner sees, it sees multiple capabilities that are required
to deliver this type of solution. And basically at the end of the day, what we are looking for
is to be able to deliver computing and storage services in a very similar way as their power
is delivered today. Basically you never buy a new power station, a new generator when you buy
a new refrigerator, you just plug your register or any other appliance to the wall and you get power
and you pay as you go. Similar things is trending with the storage and the compute services and the
interconnect is the key to deliver it. So in summary, end-to-end solution from the application
interface all the way down to the cable, to the electron, is deliberate by Merlnax today.
If you see on the cost and energy savings, it's more than half of the dollars you save
when you buy infiniment-based clusters. Total cost of ownership is about half. And with that,
the performance goes up by about 2x by doing the infiniment clusters. Again,
comparing it to the Internet, it's about 60% utilization of your machine with the Internet
and it's above 90% utilization with the infiniment. So the last topic is we are proud of member of
HPC Advisory Council that actually there are 80 members. There are clusters that you can
try. You can actually get the time on the clusters from multiple vendors and various
platforms that you can try out your applications. And with that, I will conclude my talk and
get to the questions. Thank you very much.
Tom. Well, I have to admit, I've never heard the word cloud show up so many times in 10 minutes.
And I routinely watch the weather channel. My life is dull.
I am confused. I have to admit, I tried very hard to understand. And I know cloud is a buzzword,
but one problem I have is not understanding why, you know, vaporware is a bad thing,
but clouds are good. I mean, to me, they're both the same thing. I'm not even sure who
your story was targeting. If clouds are essentially the same thing for your
system area network, then they're simply remote systems that are being accessed remotely.
If you're using a broader notion of clouds where aggregates of systems
unbeknownst to the requester are being aggregated, then your network is not being used as a single
integration, but rather there are there are wide area networks that are tying them together.
So perhaps I'm the only one in the room that doesn't get it. But what is the significant
impact of Mellanox to cloud computing that isn't directly handled or related to the high
performance computing or cluster computing itself? Well, Mellanox delivers the interconnect that
enables you to virtualize hardware, both compute hardware and storage hardware, and deliver
computing and storage as a services. At the end of the day, when you run your application or you use
a very thin client to access compute services or your data, you don't really care on what piece of
hardware it runs. And you need to provide the ability to your service provider. You need to
provide them technology. So if hardware that application runs fails, it can be migrated
transparently without disrupting the service that you are getting. So the main characteristics of
the interconnect that are required are actually delivered by InfiniBand. This very same
characteristics are needed for the high performance computing where you basically
bundle together multiple hardware instances and run a single application on them. And these
characteristics are actually water falling. These products are water falling down to the data
centers where people are looking at delivering computing and storage as a services. I hope I
answered your question. Kathy? So I guess I was wondering if you could say something about the
kind of unique features of InfiniBand relative to Ethernet and in particular things like
RDMA accesses and whether you see that as an important distinction for HPC relative to the
cloud. Is that something the cloud would use as well? And I guess part of what I'm trying
to understand here is it seems like you have this challenge of trying to figure out whether
there's a significant set of differences relative to Ethernet and in which case it's worth paying
for InfiniBand rather than paying for Ethernet. And I'm just trying to understand what the
added benefit is. Okay so the Ethernet is the layer for network that has some certain
characteristics and delivers some certain network services. InfiniBand is a broader scope. It
actually starts from like layer five which is RDMA all the way down to the network. So and this
is actually a very layered architecture and you can see the RDMA can be done over Ethernet. It can
be done over InfiniBand. We actually demonstrate RDMA over Ethernet in our booth if you are
interested. So the RDMA is basically the way to offload the network related operations from the
host to the adapter to the network hardware. InfiniBand defined this type of offloads and this
type of architecture how to enable this offloads end to end. So that's why InfiniBand is in the
unique position in terms of other technologies. Ethernet is a layer two technology that has some
characteristics that people need and people will use. InfiniBand is a layer two technology. The
layer part, the layer two part of InfiniBand has some other characteristics that are required
in particular to build very large clusters and basically and we at Mellanox enabled to mix and
match capabilities which is what we call channel IO. Basically having a single adapter that can
be virtualized on the host and being able to connect it to either Ethernet or InfiniBand
switches per type of network services that are required to the applications.
That's what we call virtual protocol interconnector VPI.
Thank you, Micah.
Our next speaker is Dr. Marcus Fischer who is senior software architect at Miracom
in USA and he will give a talk on Miracom 10G 10 gigabit Ethernet with a supercomputing heritage.
Thank you, Thomas. Over the past five years, our CEO Chuck Seitz took the opportunity to
take the hot seat session but this time his scheduler was too tight. So he told me to,
this time it's for me to get the heat. Nevertheless, he told me to send you his regards
and he might be even attending this presentation online and all the other ones as well.
Just to give you an idea about what this talk is going to be about, let me start by giving
a little bit of background. MiratnG, which actually was introduced at ISC05, is the
first generation of Miracom products and it's a convergence at 10 gigabit data rates of
Miratnet with Ethernet. So Miracom did this decision quite a few years ago, visioning
that there is going to be one network in the future and this is going to be Ethernet and
not Ethernet. MiratnG is based on 10 gigabit Ethernet files, layer one, and it's a standard
10 gigabit Ethernet cabling, also available on copper and fiber. And at the layer two,
the MiratnG NICs actually support both Ethernet and Miranet network protocols at the data link
level. And one of the key features about these NICs is that they include the processors and
firmware for offload and kernel bypass operations. However, what we also of course provide for a
complete solution for either HPC or storage and Ethernet of course is that MiratnG switches
retain the efficiency and scalability of layer two Miranet switching internally. So we complete
our solution with of course also delivering the switches and they can have a mix of 10 gigabit
Ethernet and 10 gigabit Miranet ports externally while using very efficiently Miranet on the inside
for lowest latency routing. So these switches of course do scale very efficiently up to 8192
ports keeping all the premium features like full bisection bandwidth and so on. Miracombs
10 gigabit Ethernet NICs, they are the most significant new developments over the past years
and they include basically the Alani 2Z8 ES chip which is an integrated S-Firm chip and it's fast
and it's programmable. It's a low power controller between PCI Express 8 lanes and two 10 gigabit
ports and also was first successful in silicon. It started appearing in the MiratnG NIC products
in October 2008 and from what we see in the market these are the fastest, lowest cost,
lowest power 10 gigabit Ethernet on the market today. But we not only focused on this hardware
over the last time, the software team also has been developing additional firmware accelerated
software distributions for demanding networking applications in markets such as video storage,
security and financial trading because this is what we are now bridging. We are bridging on one
hand the Ethernet market with tons of opportunities and also of course still working in the HPC market.
So just in an interest of time, the next five slides are going to show you the most popular
NICs we are delivering today. So this is a MiratnG NIC with two SFP plus ports for failover.
The chip, the Latai Z8ES just takes three watts and for this one with the failover,
basically one port is active and the other one can take over just in a matter of microseconds.
But we also support the blade systems of course. We see two pictures here. The left one has two
ports for failover. This is for the ABM blade center H and also only using three watts. Of
course all these NICs provide the wire speed rate and the right one actually includes two chips
and gives you a total of four ports, two of our performance, two for failover. So actually this
aggregates to 20 gigabits per second and with two chips there it's taking six watts.
But let me remind you also these chips here are the fastest, lowest, cost, lowest power
Ethernet expansion cards available today actually for the IBM blade center.
But they're also great for MiratnG with MX.
So this one is another one, another NIC offering two ports for performance.
This is also with the SFP plus connectors very much tailored for 10 gigabit Ethernet.
And again these are the fastest two port 10 gigabit Ethernet NICs available. So just as a
reference with the Linux bonding for example you achieve 19.8 gigabits with a 9k
kilobyte MTU but also because of the efficiency of those stateless offloads also for a smaller MTU
of just 1500 bytes we also achieve 18.9 gigabits per second. But let's not forget about HPC.
This is IC after all. So this is the preferred one port NIC for HPC clusters.
Very efficient design and many of us will remember the cluster challenge benchmark
at SC08 in Austin and Indiana University and the Technical University of Dresden they teamed up
on this one and it turned out to be the best performance NIC within a limited power budget.
And these NICs were using 3.3 watts on an IBM iData Netflix system
in combination with a MiratnG switch. For HPC we also have the two port NIC for HPC clusters
because more cores require more bandwidth and this NIC also has two ports and even also for
HPC applications we can aggregate the bandwidth and actually I'm going to come to this a little
bit later. So what kind of software do we support or offer for these NICs? So basically all the
software you know from HPC like various MPI implementations of course we do support.
It's all based on Miratnet Express and so the low level message passing system here
offers kernel bypass and it also offers this in combination with 10 gigabit Ethernet switches
because these NICs are now very versatile you can use them for your 10 gigabit Ethernet
installation or for Miratnet and keep in mind that we showed the MX over Ethernet
two years ago in Dresden achieving also a very efficient latency of 2.4 microseconds
just by using also Ethernet. So of course MX 10G includes the TCP IP, UDP, software sex,
improved sockets, metalware and here as a unique feature also the NIC bonding and teaming
at the HPC level with MPI. So basically what this means is that with n NICs or n ports you nearly
see n times the data rate at the same latency. There's more examples on that later. I don't
have to forget that MiratnG is a software distribution for these NICs and it's for
conventional TCP IP and UDP and storage for example. So a lot of the blue gene installations
have these NICs installed and this software distribution for 10 gigabit Ethernet is included
with all the MiratnG NICs. It's all available of course on the modern operating systems. It's
vehicle certified for the Windows drivers but it's also in the Linux kernel stack which kind
of gives you on benchmarks how standard it is. And as for the firmware these are stateless
offloads and they integrate very efficiently with the CPU and operating systems and don't give you
any trouble like the state full offloads. One of the very important added value here is special
firmware developments for the MiratnG NICs. So one is to mention the video pump for IPTV,
for UDP streaming because now you have to keep in mind that you are no longer in a cluster
environment where you have Mirinet on both sides so you have to be more flexible. Another very
popular firmware is the Snephyr 10G package which is good for packet capture for inspection.
So DBL is datagram bypass layer for low latency UDP, very good for pop-up applications and of
course FCOE is provided as well. Bonded NICs have the good capabilities of providing new aggregated
bandwidth but they can also be used for failover. So in our booth actually we're showing two ports
for performance dual NICs bonded together achieving actually true 40 gigabits per second
data rate. And this is a screenshot here from the booth and showing you that the sustained
performance is 39 gigabits with a latency of two microseconds. You can actually also enlarge this
vision for a quad rail system and we can talk about this later in the booth. Thank you very much.
Thank you, Markus. First is Frank Bierndt. Yeah, Markus, from the previous talk we learned that
InfiniBand is kind of ruling the HPC business, then we have gigabit Ethernet, we have the old
Mirinet products, but somehow it seems that the original Mirinet concept gets a little bit squeezed
between the improved gigabit Ethernet or 10 gig Ethernet and InfiniBand on the other side. So
where do you see the future of what was one of the pretty good contender in the field for market
share? So actually I see a very bright future in this context. What you don't have to forget is that
nowadays under the Ethernet, a couple of installations will just not be mentioned in
the top 500. So we have equipped the top number one system, the route runner, with storage,
including 800 plus NICs, but we also provide switches. For example, the top number 16
system is including a 10 gigabit Ethernet switch, which some people just announced, but we actually
already have this in production in the top 500 as the backbone, actually connecting about 30,000
cores together. Also for the scalability of the switches, there is no chance of having
another, you have to have a fast backbone connection. So 10 gigabit Ethernet, the ports you
see on the outside, but the efficiency and scalability is on the inside. So Mirinet is
there by use for all these low latency switching and the scalability, of course.
So we believe that right now there's just another last wave of a specialty network
coming out right now with a claiming 40 gigabit connectivity, which boils down to 32.
Peace Express buses can only deliver 25 right now. But we believe, and we already have
some orders there, that basically in the future we're going to see the converged enhanced Ethernet
taking over, and basically we're going to see the low latency Ethernet strategy kicking in.
And we have been working on this and have mature software for that since two years,
that you can actually already, as of today, have HPC installations with low latency Ethernet.
And so the 10 gigabit Ethernet market is actually increasing quite dramatically right now.
It's going to lower the cost on the NICs and so on. And so we're definitely going to see these
also then getting back into the HPC market. And then we can talk about this question also one more
time, hopefully. Second from Pang Lin. In addition to your HPC performance, I'm really
impressed of the video performance, video streaming performance. We did some kind of
long-haul video streaming, 4K video streaming. So I would like to know what's the performance of
this mirror 10 gig for the long-haul network, if I can provide more than 10 gig optical fiber
network to you, and do you have best practices in your case? Could you talk some of it? Thanks.
I'm not sure if I picked up that question correctly. What is the
interconnection using in your technology for video, very high-end video streaming.
And you mentioned it in your presentation. So I would like to know if there's a
cross-institute or even a cross-ocean connection. Can you have a similar performance
with the router and switches company with it? Oh, I see. Let me try to answer this. So basically,
here with the video pump software, we leveraged the kernel bypass techniques, which we have used
within MX, and put them at a different level. So the operating system is much more efficient now,
handling also protocols like UDP, and basically much better getting data out of the system
in a very efficient way, in combination with the stateless offloads and the direct programmability
of the processor. So it's actually a firmware development kit for these processors available,
so you can kind of design your own feature. But across the ocean or so, we did not have
any benchmarks. So this is really for, I don't know, the in-house reception and the more streams,
of course, you can handle the less servers you need, because also the CPU load on these streams
is just very, very small, and so you have those multiple streams. Thank you very much.
Our next speaker will be Philip Murray, he's vice president for engineering for QLogic.
Good afternoon, everyone. First, let me say that I'm very, very excited to have the opportunity to
provide to everyone a little bit of background on QLogic and QLogic's contributions to the HPC
interconnect space. I'd first like to say a couple words about QLogic in general. Many of you may
know QLogic from a different marketplace. QLogic has established itself very well in the storage
area networking market, very well known for host bus adapters that support fiber channel networks,
recently has expanded out into iSCSI and the emerging fiber channel over ethernet capability
as well. However, it's probably a little bit less well known that QLogic also several years ago
invested heavily in getting into the high performance computing market. QLogic acquired
a couple of companies back in 2006, and has ever since then been applying its capabilities in the
enterprise space to this emerging market. It's a very stable, financially stable company,
it'll be there when you need it. Now, the topic of my talk today is really all about scaling
and really what is that problem statement? The problem statement is as you add more and more
compute power, you really want to get 100% of that compute power realized. And in many cases,
this can be limited by the communication infrastructure. Now, we had seen in the past
that we've seen actually from previous speakers and it's sometimes negative to actually go this
deep into the session because you've seen a lot of this already. However, one of the trends that
we're saying is in order to be able to continue to add additional science into our algorithms yet
keep the time of execution to a reasonable level, there's really only one major cost-effective way
to do that, and that is to deploy high performance computing clusters. And as we can see, clustering
has really been pushing out all other types of approaches except for the MPP. And to deploy
successfully a high performance computing cluster, you need a high performance interconnect.
So as we had earlier from Frank that two tin cans and a piece of string can actually handle some of
the embarrassingly parallel applications in which there are really very little stress on the
interconnect. However, for highly communicative applications, high performance computing network
is required and over time, InfiniBand from the number of systems deployed, InfiniBand has really
been pushing everything else out of the marketplace. If you take a look at the same chart instead of
the number of systems deployed in terms of performance, you can see that the difference
is even a little bit more dramatic here. The proprietary interconnects still have some
significant piece of the market, but InfiniBand from the standards point of view has taken really
the high performance piece Gigabit Ethernet is dropping in that case. Well, why is this so? Well,
InfiniBand has got a very robust roadmap. InfiniBand started deployments in the 2002-2003
timeframe with single data rate 10 gigabit per second links. We installed the first large
double data rate cluster of 20 gigabits per second per link at the Army Research Lab in April of 2006.
And QDR is in the market today and that's a 40 gigabit per second solution. I would expect that
the next generation of InfiniBand, which is already in the works, will be again a doubling
of that performance level and you'll see that in the next couple of years for sure.
So, InfiniBand is winning. That's great. Well, what are the problems that we're facing going
forward in order to be able to continue to scale, to try to get more linear scaling as we have
compute power, especially when it comes to the interconnect, which of course is where
QLogic focuses. Well, clearly we need the hardware components to deliver the performance
that we need. That really goes without saying, but what does that really mean? Well, performance is
really all around low, but not just low in a micro benchmark sense, but low in a scalable sense,
meaning if you ran the HPC random ring benchmark, for example, you still need to see low latency
across the entire cluster. High consistent bandwidth, meaning even when you're utilizing
your network to an extreme degree, you still get the bandwidth that you're expecting. Another one
that's sometimes overlooked is extreme messaging rate. It's really, really important that as
applications scale, message sizes get smaller and the number of messages therefore go up, that you
have to have extreme messaging rate without driving the power through the roof. And then these all
have got to be embodied in systems that allow you to deliver these very, very large scale clusters
cost effectively. That's half the story, but the other half of the story is around the software
piece. You absolutely have to have superior management capabilities. Some statically routed
networks, InfiniBand included, can run into unfortunate situations in which there is congestion
that could actually be avoided should the routes have been described differently. That's a really
important addition that'll be required. Also, configuration, management, monitoring, diagnosing
of any failures in the fabric as these fabrics become big is continuing to be very important.
So, how is KeyLogic approaching this? Well, KeyLogic is bringing its rich heritage of enterprise
class developments in both the fiber channel host box adapter area as well as the fiber channel
switching area to bear on this. We're applying architectural and advanced development capabilities
that have been grown in the company over the last 15 years. For the first time, there is a second
set of ASICs that are actually available to support the InfiniBand market. KeyLogic has
got its own host channel adapter and switch ASIC. We've applied that into systems. We've completed
the host and the fabric management software. Another part that sometimes gets overlooked
is the investment required to put together a very robust test organization and test capability.
We've got a heavy capital investment in that as well and are pulling together ecosystem partners,
ISVs, and our NetTrack development center to improve interoperability with all the important
codes in the marketplace. Just a couple quick words on our host channel adapter. I've already
mentioned some of its key attributes. Most importantly, it has extremely low latency.
Actually, we can get under one microsecond back-to-back MPI latency. Simultaneously, we can get
30 million messages per second, which is a significant improvement over current technology
at the lowest power in the industry by a significant factor. It is a stateless,
connectionless design, meaning that it can be a very small design with little state. It also makes
failover very, very straightforward. Let me talk a little bit about the switching ASIC. It is a 36
port device, and it's really focused, again, on extreme message rate, 40 million packets per second
per port, which means that in total, the chip is able to handle 1.4 billion packets per second
at very deterministic latency, even as you load it up to 90 percent or greater of the total
bandwidth available. We've packaged this switch device into a family of systems. One size does
not fit all. Basically, the size of the cluster will determine the specific systems that you need.
We can scale today anywhere from 72 ports to only up to 648 ports in a single chassis,
all at full bandwidth QDR capability. We also have a second capability, which gives you 33 percent
additional ports. The bandwidth is shared, so if your application is not bandwidth intense,
it's an outstanding solution to get QDR link performance, QDR latency at a much better cost
performance. We also have some edge switches, which are outstanding. We have purpose built
edge switch for HPC applications at a very low cost, very small form factor, very easy to manage.
So moving on into some of the software areas, one of the most important factors to consider is when
you're installing your fabric, configuring it, trying to get it to work, do you have the tools
that you need? QLogic has invested over the last six, seven years in a capability called our fast
fabric toolset, which is essentially able to find any flaw in the fabric, whether it's an HD that's
installed improperly, a cable that's acting up, a cable that's missing, firmware that's at the wrong
level. When you're starting to talk about enormous systems that can have many, many things go wrong,
having a very robust toolset that can find these needle in a haystack, very, very subtle problems
is extremely critical. I mentioned a little bit about the routing requirements. Infiniban, as
defined by the IBTA spec, is a statically routed network. The PES are identified ahead of time,
and you can run into some unfortunate circumstances in which inter-switch links are, some
inter-switch links are underutilized, others are actually saturated. And so QLogic has decided to
go after this problem based on its fiber channel experience and put together a hardware-based
adaptive routing solution, hardware-based in the sense that every single switch has got the
knowledge of the working inventories, all the queue depths, and time to the next hop, and it
can make dynamic decisions, and it's very scalable because it's done at the switch level, which I
think will give us a significant advantage. So just in summary then, I think what we can see here is
that the demands of the applications are continuing to put significant pressure on
high-performance computing applications. This is in turn putting pressure on the interconnect.
QLogic is trying to contribute to this HPC market by delivering next generation of
high-performance computing solutions with very, very low bandwidth at scale, and that's the
important message here. High consistent bandwidth at scale, extreme message rate,
advanced routing functions, all with the lowest power consumption in the industry. Thank you.
Thank you, Philip.
You talked about the high messaging rate and the increases in messaging rate that
you've had, and I'm wondering as we grow the number of cores per chip and also perhaps with
that grow the number of MPI processes that are running on a chip, whether you see that as
something that you can continue to support from the networking side, especially given that the
size of those messages may be shrinking and the number of other processes they're communicating
with will be growing. So where is the limit of when you can support that, and do you see any
challenges at the software level, the low-level software level, in terms of getting access to
that bandwidth from all those cores that are running in, say, shared memory space, so that both
the software side and the hardware side in terms of that message injection rate? And then finally,
I guess the final piece of this is, do you see any of this moving on to the processor chip?
Those are all great questions. Absolutely. I think that the trend in the industry, as we see it right
now, is that the nodes themselves are becoming more and more powerful. They're becoming fatter
nodes, so to speak, which actually increases the demand on the interconnect dramatically,
not only from a bandwidth perspective, but from a messaging rate perspective, because, as you said,
it's now very possible to run more and more and more MPI processes on a single-fat node
and drive the messaging rate, actually, through the roof.
Qlogic has taken a very unique approach here. One analogy is, over the years in I-O development,
where my background is, we've tended to offload, onload, offload, onload, depending on the
particular point in the cycle that you were in. Maybe five years ago, it made good sense to try
to offload from a 486, some of the processing that was being done on the 486, on to a co-processor.
Today, the world is very different. Today, what's happened is, instead of having a single 486 core,
we've got an eight-way nehalem driving a single interconnect, and so it's very, very difficult
for a small microprocessor to try to keep up with the demands of that. Therefore, it's really
important to be able to put a little bit of that back on the CPUs so that they can scale to these
enormous messaging rates. And that's the approach that Qlogic has taken.
Tom? You mentioned that you can achieve sub-microsecond latencies with some packets. Could
you expand on that a little bit, especially as it relates to lightweight messages? This is an area
that may have increasing importance in large global address spaces, where gets become increasingly
important for informatics applications, and where message latency sensitivity can be very
high to the overall both performance and scalability of applications.
Right. So, we have already demonstrated sub-microsecond latencies. Basically, we're
running the OSU MPI latency test to achieve that. That is, unfortunately, a point-to-point test only.
And so, I think our experience has been recently that some of these micro-benchmarks that generate
these so-called hero metrics are not necessarily an excellent predictor of application performance
going forward. But more so, what we see is that some of the HPC challenge benchmarks, for example,
like Random Ring, in which you get a much better mix of messages, can really show you the latency
that you get. And what we can find with our approach is that the latency in those environments
is low and stays low as you scale the cluster. Thank you.
So, our next speaker is Mr. Frank Lafourche, as a senior engineering consultant from FORCE 10 at UK.
And the title of his talk will be, High Performance Non-Blocking Reliable Switching for HPC.
Hi, everybody. I would like to talk first a little bit about the history of FORCE 10 and the HPC
environment, tell you a little bit about our success here and why we're successful in this
area, and then show you what the latest development in FORCE 10 is for HPC computing.
We have been in the HPC market since the beginning, since we're shipping equipment. The
HPC market for us was the first market we sold our E series product line into. That was 2002.
We have been very successful in this environment, and we are happy to say today that all the big
computing leaders built their systems around, or otherwise, the other way around, they built
their systems and including our switches for the switching infrastructure. And you find IBM, Dell,
Sun, T platforms, all the leaders for big HPC computers. The development we've seen over the
last couple of years is that Ethernet wins more and more importance into this
HPC space. If you look at the top 500 list from November last year, you'll find that 400 out of
the top 500 clusters are based on Ethernet. 284 are gigabit Ethernet. 71% of the top 500 are
using gigabit interconnects. And out of these top 500, what you find is that a large amount is
based on FORCE 10 network infrastructures. We're anchoring 30% of the fastest 40 supercomputers,
including number one, four, five, and six. We've been part of the first petaflop supercomputer,
the Roadrunner. And like I said, we've been used by most of the biggest supercomputer builders
around the world. The areas where we used are the Sun, the user interconnect, the management fabric,
but also in certain areas you find us in the processor interconnect. The reasons why
the likes of Sun and Dell and T platforms are using us is really the stability of our product,
the non-blocking performance, as well, of course, as the high port density. Now, the most important
point for our partners is really the reliability of our products. Reliability is something
which doesn't come just by accident. You have to build that into the architecture of your product.
What I would like to do is show you where the reliability of our product comes from. First of
all, what you see here is an example of our E-series product line. And we have a very unique
architecture in that product line. We're separating routing, switching, and the management plane into
three separate CPUs. This provides reliability to the product. Why? Because you have a
protected memory space for each of the three areas. And if you have a problem in the code,
it is limited to that memory space. Therefore, the switch is protected. But it goes further.
We have an ASIC in which we do all the forwarding performance. This provides the line rate
performance no matter what the packet size is. Even at the packet size of 64 bytes,
the small packets, we can reach line rate performance on every single port. The reason
why we do this is because we have a distributed forwarding architecture. We're forwarding the
packets in ASIC on every single line card. The CPU of the root processor module is not involved
in the individual forwarding of the packets. Furthermore, we have an independent data and
control path, meaning the back plane provides a clear path for the data. And a separate path
which enables the root processing module to communicate with every single line card and
update the forwarding tables. And they're separate from each other. And therefore,
there's no blocking of update messages because a port is running at line rate.
Furthermore, there are protecting mechanisms on the RPM. The RPM is responsible on the
CP chip to terminate all the protocols on the switch. Now, in order to prevent a single CPU
to overload, what we do is before we send traffic to the CPUs, we shape the traffic to a certain
fixed rate which is set in hardware. Furthermore, we prioritize traffic. So there's three queues,
one for each of the CPUs. And in these queues, we prioritize the traffic going to the CPU.
Furthermore, what you find is we constantly monitor every CPU for utilization. And when we
reach 85% of utilization of the CPU, we back pressure towards the queues and rate limit the
traffic even further. So this is preventing the CPU to run into 100% utilization and therefore,
to a situation where the switch gets unmanageable or it blocks, it just reboots,
which you see with a lot of other vendors. On the line cards, we have additional security
mechanisms. Like everybody else, we support ACLs. So that allows you to filter traffic and make sure
that only the traffic you want is passed through the line card or towards the CPU. And we can
support up to 1 million ACL rules on the system. What I've shown you so far was the hardware side
of the switch. So, of course, when you develop a switch as big as ours and you want it to be
reliable, you have to have the reliability built into the hardware architecture. But without a
stable software foundation, that doesn't count for anything. So you also need a reliable software
basis to run on the switch. And in our case, the operating system is called the Force 10 operating
system. It's a modular operating system. Why is it modular? Because, again, modularity ability
brings stability to the system. Every single protocol is its own process. The processes are
completely isolated from each other. And they communicate over an inter-process communication
bus with each other. So, again, when there is a problem in one feature, it doesn't affect the
rest of the code. Furthermore, when you do updates or you bring in new features into the code,
the software developers don't have to touch the rest of the code. Which, as you know,
biggest problem in software or code writing is human error. And the less the developer has to
touch a code, the less he can break. So it's a good thing. Our operating system is based on
NetBSD. It's a Unix-like kernel. And the great thing here is we run the same operating system
across all our product lines. That is the E series, the C series, and the S series.
We've taken the experience we made over the last couple years on TerraScale, which was our
current E series product line, which we shipped into the likes of Uleak and Roadrunner in the U.S.
And we took it a step further for the next couple of years for the new bigger head of flop type
HPC installations. And we came out last month with our third generation of our product line,
of the E series product line called Exascale. And with Exascale, we doubled the slot capacity
to the previous product line, which is now up to 250 gigabits full duplex.
We increased the line rate port density of the switch. That is for 10 gigabit, up to 140 ports.
And for a gigabit, 1,260 ports per switch. At the same time, we reduced the power consumption
on the per 10 gig port from 75 watt to 36 watt. So pretty much half the power consumption.
And we increased the maximum lag capacity from 160 gig to 640 gig. At the same time,
we kept the proven operating system we had on TerraScale, and we have been shipping since 2002,
and made the switch ready for future applications like virtualization, automization,
40 and 100 gig. It is MPLS ready, and ready for a converged fabric.
And with that, I'm done. Thank you. Thank you, Mr. Lafourche.
First is San Pang Ling.
I did check your product on the websites, and I'm impressed by your success.
Yes. Thank you.
Congratulations. But I found your product has a very, has a level, say, ExaScale,
which is very daunting. So I would like to ask, the first one is,
what's your plan for future Exa computing? Four?
For future Exa computing. Obviously, it is in the plane, and you are very aggressive on
that direction. This is the first one. The second one, you mentioned already,
actually, the power consumption has been reduced. Yeah.
Okay. A lot. But could you talk a little bit more for general energy efficiency of your system,
how to improve it? Right.
I have to admit, I still haven't understood the first part of your question.
Well, I mean, we are pushing forward for Exa computing, right?
Exa computing, yes. From PETA to Exa. Right.
What's your plan? I mean, how do you match up with that direction?
Right. Clearly, the requirement for Exa computing will be even more ports.
What we see lately is the increase of 10 gigabit ports and HPC computing increases
dramatically, actually. We've just finished the upgrade of Yulich, and we're close to 1,000
10 gigabit ports in that system now. So the way we're seeing it is twofold. First of all,
the density of 10 gig ports in the system have to get higher and higher.
I've mentioned here in the presentation that we're now running 140
10 gig line rate ports in a single system. The second half of the year will come out with an
oversubscribed card, which will have 40 10 gig ports on a single blade, which will get us to,
what, 520 10 gig ports in a single system. So that would be the first step. But clearly,
the 10 gig port density is just the first step. The next step clearly has to be 140 gig.
So we're seeing the 40 gig in the computing market probably being
earlier available than 100 gig interfaces. The standard should be rectified mid next year,
and we expect to see products on the market sometimes in that time area.
In terms of power consumption, I already mentioned we came down quite drastically with
the power consumption of the 10 gig ports. In general, the new system uses about the same amount
of power, just a little bit higher. We were at roughly five and a half kilowatts at the old
system. We're at six kilowatts now, but have double the density. So the power saving we did
on the new development is quite drastic. Second front period. Please allow me to start with a
statement. Counting all the percentages of various companies presenting this afternoon,
having the top 500 network in business, it's a little bit beyond 100%. So it's always
interesting to see to what kind of part of the system the percentage relates to. And
latest with your presentation, we definitely climbed above 100. Nevertheless, from an application
standpoint of view, we are climbing up to pretty large systems now. We already mentioned and
discussed that with respect to the cost and the programming models and the way we implement our
physical problems into the computer environment, we are running into, or we already did run into,
a number of interesting problems, which will be with us for a number of years. Do you see,
expect, or forecast similar problems with respect how communication has to be dealt with from the
application point of view in the upcoming years? Or do you see that it will be, besides the fact
that the percentage of communication may get up, that this communication part will be hidden from
the user? When you talk about communication part, you talk in general communication, not HPC related?
Difficult to say. I have to admit, I'm not an expert on the application side, on the HPC,
we're doing the network. I mean, clearly, what we can see is we're doing more and more faster
and faster computers. The amount of data which has to be computed across these supercomputers and
therefore passed over the network gets more and more. I mean, the indication is there with the
number of 10 gig ports we're now installing. So yeah, it definitely is growing. And coming to your
statement, why more and more people claim to be part of the top 500 list, maybe one word here,
clearly, if you're looking to you, as an example, we claim to be in there, yes, because we're
part of the SAN, we're part of the user interconnect, but there's a fiber channel
network in there as well. So if you talk about the same installations, there's
parts of the network you're having and other parts are owned by other companies sometimes.
Thanks again, Mr. Lafar.
The last speaker for this session will be Dr. Rudolf Fischer. He's the CTO for NEC Germany.
Ladies and gentlemen, thank you very much for waiting for such a long time. I want to explain
a little bit about NEC's future direction. I guess this is okay. And I guess that you have heard a
few things recently which might have made you worry, but don't worry, we have a future. Perhaps
some things have been over-interpreted. First of all, let me explain the status that we currently
have, especially now for Europe. We installed quite some SX9 systems successfully at very renowned
customer sites. We are proud of this. We achieved a lot there. This was since last year when I was,
when I made the hot seat session. Apart from that, you recently learned and you found it in the new
top 500 list, the Earth Simulator 2, which is an SX9-based system. This already shows you in
what direction we are thinking. It is about leading efficiency. We are close to 94 percent
and I think this is remarkable and this is what NEC's supercomputing strategy is all about,
to keep efficiency. Then we finished the design of the vector part of the overall Japanese national
project. But then, and that you might have heard, NEC decided to pull out. As much as we were
hit by this decision, I can only understand our CEO, Yanosan, because that was a tough decision
for him for sure, but the stability of the company was more important and he had to make
quite some tough decisions. So I think that was reasonable to do. But please don't over-interpret.
That's not the end of NEC's supercomputer series and not the end of our overall business in that
range. In the meantime, Rekin is reconsidering the concept, so they will continue and they will be
successful and we deliver the technology that we developed for that to them. I'm very sorry
currently. We cannot provide complete plans or roadmaps. We are working on this heavily. We have
several options to continue to move forward, but we need a few more months and our target is to be
able to show this roadmap in Portland, which means on the supercomputing this year.
Now, what is NEC's primary goal that is deliver real application performance? And our good customer,
Tohoku University, Professor Kobayashi-san, he measured the HPC challenge results, yes,
and he could rank top on 19 out of 28 measurements. So this is what we have in mind, because we believe
that the HPC challenge benchmark reflects real applications much more than, for instance, the
pure LINPACK run. On the other hand, we are aware that 19 out of 28 means that we are not scoring
best in 9 out of 28, so that is what we have to work on. The other direction that we have in mind
is that we think that not only one size does not fit all but also one type does not fit all. And
for that reason, we already since quite some time are thinking about integrating on a cost
granular level different architectures into one supercomputer system that can tackle different
kinds of applications for which this would fit. For instance, fluid structure coupling, you might
have a structure code which is very scalar, and you might have a fluid code which is running quite
well on vector. This we are doing with our partner, HLIS, and in doing that we installed a vector
system, which is a cluster, which is ranking 77 in top 500, by the way. The features that we need
to address there is, for instance, a shared file system, it is clear you don't want to move files
around, and a heterogeneous network to couple those two systems and have applications running
across both. And there is a lot of work to do. You have to think about programming models and
methods. You have to figure out what applications fields would really benefit from such a structure,
and this is for us a test bed to go forward and to figure out how we have to design our future
system where we also have hybrid architectures in mind. Now, I always was claiming that
vector will return, and the truth today is vector is there and it's everywhere.
You know this, you know this from Intel's SSE, and you know this from IBM's AltiVec,
and all these things are SIMD, to my opinion. For me, it's not really vector yet.
The other trend that you see is that hybrid parallelization, MPI and OpenMP is increasingly
important, and we heard this in presentations here. We have done this since 10 years, and
it does not always work, but sometimes scalability issues are quite well-addressed by having an OpenMP
under an MPI process. We see this increasingly. It's not yet overall, but we see this increasingly
in our own benchmarks. Then the real problem since many, many years seems to be memory bandwidths.
The problem is only memory bandwidths is damn expensive, and therefore the hardware somehow
has to cope with memory locality, which really exists in applications. We can see this,
and we have to have some hardware to address this. We did this with the SX9 for the first
time, and this is something we will also further evaluate. I already was talking about praise. I
really like the diversity of praise, because I think it's a very good idea to have many different
architectures available to the scientists in Europe in order to address their needs,
rather than have one big petaflop system, so every scientist can address what suits his application
best, and in that direction, we would also want to think with our whole product strategy.
Future technology enhancements. What is going on there? I remember the presentation of
Andreas von Bechdel's time, and he was claiming that in order to get to exascale,
we would need to have 15 percent growth in operation per cycle in the future in order to
really address this. This is what I would like to explain to you a little bit more.
How can we expect to get this? If you look at an SX9 architecture, it could also be an SX8
to quite some extent, our architecture was already reflecting things that you see today
since quite some time. For instance, nowadays, Intel is talking about 128-bit or 256-bit wide
registers. We always produce with each individual floating point unit eight double precision word
results per cycle, so in a way, we had a 512-bit register already working. This is now common sense
in a way. I think this will continue. Everybody is claiming that we need more BAM or bandwidth.
With SX9, we have 256 gigabytes per second, and our developers in future are telling me they could
do better than that problem. There is, as I said, it comes at a cost, because in the SX9,
70 percent of the cost is in the memory system. This is not just chip, but the whole architecture.
That also leads us to we have to use memory locality. This is also common sense. Everybody
is doing this with the cache. We had a sort of a programmable cache. I think there is quite some
efficiency to get to really use such things and such hardware efficiently. I think that is something
that will be more common in the future as well. We are experimenting with it. Mask registers,
as well. I saw this, that Intel is going to have this. I think I found it in Intel's AVX.
Well, Gasser Sketter, Intel is also talking about this. This is something, because unstructured
codes are becoming more important. This is something that should become more important
in the future, and I think there we have to learn a lot. By the way, memory locality is also nice
there. Then there is something that I do not see yet in other vendors' plans, but I think it is
very useful. That is the vector register. The vector register is really great because of the
following. Imagine that you keep your pipes busy for several cycles after you issued an instruction.
These cycles can be used to execute other instructions which are not contributing to
the offloading port operation count. That is normal. If I look at an assembler code for our
system, I see that there are many scalar instructions accompanying those vector
instructions. These scalar instructions, only if you can hide them effectively,
only then you will achieve a good efficiency. Therefore, I think vector registers are great.
Moreover, we really have to control things better. I do not like so much those GPU ideas because
that means that you do not control from the scalar machine what the GPU does.
In our case, the accelerated vector pipes are tightly integrated. That makes a lot of sense.
We will develop a future vector product. We have several options to do that. We definitely know how
to build a system. Beyond that, we will have a diversified product portfolio. You will see us
already by now that we are offering more in the scalar market for scalar projects. We try to
differentiate there based on software, stack, and service and support. The integration of vector
and scalar product lines, that will be a major goal as we are already proving now by working
with HLAS, our dear customer. If expect future products from NEC, especially future vector
products, expect that we will expand our market share in the scalar HPC and that we will offer
complete hybrid solutions. Once again, please be patient. In Portland, we will tell you more.
Thank you very much. Your machines are beautiful. They are a work of art.
I would refer to them as the BMW of computing. I own a BMW. It has a problem that BMW has too,
and that is cost. Can you address that issue? Yes. I think the prevailing part of the cost,
as I said, 70% are in the memory architecture. In this case of SX9, it is 32,000 banks. That is
needed in order to achieve this bandwidth. That is really what we have to address. In the national
project architecture, we addressed this a little bit by in fact reducing memory bandwidths per
flop. I think it is going there, but that cannot be defined as an answer. I think that addressing
memory locality in hardware and having clever ideas there going forward, this is the way to
address this cost issue. The other thing is a purely economic effect, and one of the options
is also to cooperate with some partner. The economic effect is that if you build a CPU which
you build, let's say, 10,000 of them during the lifetime of the product, that is a principal effect
that you cannot really deal with. That is the reason why some of the options are cooperations
to really tackle a broader market. What we were perhaps not good enough yet is to figure out
how to use these architectures in markets other than HPC, because I could imagine and perhaps
the sell chip is a proof for that, that such machines could also be, let's say, embedded
systems in tomography or something like this. Perhaps we have to be more smart in that respect.
Frank Perrin. Yeah, I would like to pick up on the same point and develop this a little bit.
Essentially, the ethics theory has got a little bit out of control with respect to
balance of cost between certain subsystems, and especially the memory part you mentioned yourself
is kind of prohibitively expensive nowadays. Looking into applications, we will see an
increasing need of high memory parts in computers. You mentioned that you undertook
first steps in addressing this. Do you really believe that first steps will be enough to make
this a viable pathway? The second point is processes one thing, but also the networking
in between, the transfer between the memory and the processes will be of increasing importance,
especially with increasing memories. Can you address this topic, or is this also,
let's wait for Portland part? Perhaps we can address a little bit. Number one, the memory issue.
The idea for the National Project was really a system where you have a very big part of like
ADB, which is sort of a software-controlled vector cache. We were addressing this there,
as I mentioned. Was this enough? I cannot tell you. That is also an economical question. One
would have to figure out, but for sure, you can make the memory cheaper by having smaller
SMP nodes, because it was all multi-core now. I think that in the future, also for us, it would
hold that all memory interface has to be on the chip and the memory directly attached. That will
reduce cost for the memory. I think that is one step that we have to still make, and that we should
investigate. The other thing is having smaller SMP nodes, the SMP just the multi-core, which makes
it easier to code instead of having two hierarchies of OpenMP, then you need more interconnect.
And you see things that in interconnect technology, we can really do something different. That is one
of the parts where we think that we should differentiate in the future, which, however,
is not really easy, because if you are, let's say, microsecond or sub-microsecond latency,
that's great, but talking exascale means you are talking about nanoseconds, hopefully. A nanosecond,
according to Einstein, is this. You are talking about geometrical latency or something like this.
I cannot tell you how far this will go, but for sure you are right. In the future, the interconnects
will become a dominant part when evaluating performance of real applications. But how to
address this, well, we are looking at it. Thanks again, Dr. Fischer. Thank you.
So this concludes our hot chair session. I would like to thank all the speakers for their
contributions and the inquisitors for their questions, and wish you a good rest of the
conference.