Thank you very much Hans. Thank you for the introduction and for the flowers. Of course
I had only like 20 minutes to look on my laptop for better pictures of myself and worse pictures
of Hans 25 years ago and I had to realize that I probably was going to lose that competition
anyway, so I gave up on that one. Now for the top 500 itself, the new top 10, you already
heard some of the systems, shows one new system on the number two, the system in Japan
called Nebulé. It is a system built by Dawning and installed at the very new computing center,
the national supercomputing center in Shenzhen, which is currently being built just in the
finishing phases. The number one system is and was before the CHERGO system at Oakish
National Lab and the differentiating factor between two systems is architecture as well
as performance. CHERGO has about 1.7, 1.8 petaflops. Nebulé reached just barely 1.3
petaflops, but in terms of peak performance, Nebulé is actually ahead of CHERGO. In terms
of raw peak performance, the biggest system, the largest systems we ever saw in the top
500 with about three petaflops peak performance. The difference in technology is clearly that
the Chinese system is an accelerated system. It uses NVIDIA Tesla cars to achieve its peak
performance and to achieve its Olympic performance. We are going to talk a little more about this
change in technology you see here. What it is to have a system in China that high on
the list is clearly a demonstration of the Chinese commitment to supercomputing and the
financial support and backing to actually achieve that. It is even more impressing the
Chinese position if you look down to number seven, which is a second system in China in
the top 10 at the national supercomputing center and which had been on the list before.
So China actually has two systems in the top 10. That's new. It has the highest position
ever. It has two systems, and both systems are actually accelerated system. The Tin Tianye
1 system is actually accelerated by AMD GPUs, so very similar architecture, but a difference
in the details and history. The only other changes we saw in the top 500 since November
is the ones in red, which is basically NASA Ames upgraded their SGI system, play yardies,
and has now a little higher Olympic performance and moved one spot up because of that. The
other change was that the Red Sky system, Sandia, they missed the November deadline
of the last measurement by only two days, so they didn't make it. With this measurement
in the November system, we acknowledged the measurement this time. It actually allowed
them to move from position 11 to position 10 because those two systems at Sandia and
the tech center in the U.S., they're virtually tied. The difference between those two systems
is less than a tenth of a percent. That shows you the competitiveness in the top 10 and
how hard it sometimes is to get into the top 10. So as I said, the big news in the top
10 is two Chinese systems. The other big news is actually that there wasn't anything else,
and that's very unusual because typically within six months, we see a turnover between
three and six, seven systems in the top 10. This time, we only had one. That is actually
reflected in all kinds of performance levels across the list and in the different type
of indicators. One indicator is simply the replacement rate. That is the number of systems
which fell off the list because they don't make the cutoff anymore. They're simply too
small. That's 143, which is one of the lowest ones we have seen for quite a while. So there
is less turnover. One minor reason for that might be that Hans's conference is almost
four weeks earlier than usual. But the main reason, of course, and the main speculation
here is that it's an impact of bias behavior, that there is less spying because of the economic
situation and maybe even something else is going on. Another way to look at this is,
of course, look at the overall performance development. That's a historic slide. We show
pretty much every time. What you see here if you look to the very right of the graphs
is that the overall sum after accelerating in accumulated performance is actually leveling
off a little and the same happened at the end at the level 500. There is a little bit
of slowdown in performance. If you analyze that a little more carefully, we can do that
by looking at the annual increase in performance list by list. You get the following picture.
You see that the increase in performance compared to last June is actually the lowest
one we have ever seen in the top 500. So the growth rate in terms of performance of the
top 500 has a historic minimum. Part of that is we have a low turnover. And again, the
question is, why is the growth of performance so slow? Is there something serious going
on? Is it economy? What's the reason for this? In red, I marked here Moore's law and
for the existence of the top 500 on average, we pretty much enjoyed in HPC the growth and
performance growth of our system sizes larger than Moore's law. Now we are clearly behind
Moore's law, at least as far as the recent history is concerned. Now, if we project our
performance curves out into the future, of course, the last two or three data points
on this graph are not going to influence the projection tremendously. So the projection
is still on course for a number one exoskeleton somewhere around 2018, 2019. It's going to
be interesting to see if in the next one or two years, if the current trend slowdown in
performance growth is going to continue and if this projection has to be changed or not.
For now, everything looks as planned, as anticipated. Now, if you break down, if you do the typical
statistics we do pretty much every year into different categories. We can look at countries
and I already mentioned that China for the first time has two number one systems in the
top 500. Now, if you look at the breakdown in terms of countries where the systems are
installed, you notice a few things. First of all, United States traditionally always
has been by far the biggest consumer and it still is, nothing has changed there. The
next couple of countries there have been two changes. UK, France and then Germany and China
in a virtual tie both have 24 systems in the list. The big changes here is Germany is on
the decline again, used to be much higher. So Germany is going down and China has pretty
much tremendously increased, it's present in the top 500. So China is coming up for
a tie now. So that's the two changes in this graph. And if you look back, you can see that
if you look carefully in this slide and that's the way we typically present it as a percentage
of the total list in terms of systems or in terms of performance. If you like to analyze
in a little more detail how China actually developed over the last couple of years, that's
the little red slice up here, if you want to analyze that in more detail, it is actually
much more appropriate to look at performance and how the performance of all the systems
installed in China, how it has increased over time compared to other countries. So I'm going
to show the slides now without the normalization 100%. That's the performance increase, the
performance growth of all the systems installed in the United States. That's an exponential
growth rate and it has a few hiccups but nothing major. You compare that to Europe, it's pretty
much parallel. Sometimes Europe falls a little back and catches up again. Those are both
geographic regions which have a long history of high performance computing. There is a
strong commitment of government with funding. There is a lot of industry and a lot of installations.
So they pretty much go in parallel. The third geographic region historically is Japan and
Japan actually has seen quite a bit of change during the last 15 years because of, mostly
because of the economical situation, that the number of systems and especially the aggregate
performance in Japan has not kept up with the U.S. and Europe. Japan keeps falling behind.
We have seen them for a couple of years. Now these three curves, of course, the interesting
thing is now if you compare that to China, this is China. Obviously it's on a much larger
growth rate. It's a little more jagged and you might actually also notice that I started
my graphics only in the year 2000 and the reason for that is that before 2000 China
was basically absent in the top 500. It was not there at all. So what you see here is
the first example of a country which has obviously an interest in high performance computing
at the very largest scale and it puts its dollars or its currency where its mouth, it
also supports its centers and its industry the same way. That has led to China now overtaking
Japan and actually in terms of countries now being the second, being the country with
the second highest number of performance installed in the top 500 just behind the U.S. and ahead
of the European countries by themselves. So China, and this clearly shows that it has
a big interest in HPC and has a commitment to see it through and it's getting there.
Of course, this you have to take with a little grain of salt. A lot, more than 60 or 70 percent
of the performance of China installed in China is actually accumulated in the two systems
in the top 10. So those two systems carry a big share of the installed performance.
But for instance, if I would plot Germany, which I'm not going to do, it's going to have
a similar, there's going to be a similar situation with the big Eugene system capturing a very
big portion of this performance installed in Germany. Now if you look in a little more
detail because we are in Germany, we are in Europe at the European situation, you see
in terms of number of systems the steady decline of Germany over the last years and while there
was a little bit of a recovery three years ago, it's still going down. UK has increased
and is now the number one in Europe ahead of France and then the other European countries
which come and go according to their funding situation, I guess. Situation in Europe.
Now if we break it down in some of the other categories, one of the races we have watched
for a couple of years in the top 500 is by companies which manufacture the hardware between
IBM and HP. The last couple of years, the last two years I believe, HP was able to again
on IBM and actually be the number one in terms of number of installed systems. This edition
of the list, IBM actually managed to get back in the lead and the clear lead in terms of
number of systems, now with about 40% slightly ahead of HP. The real news here is an old
news that it's still a market of two players and only at the very high end like in the
top 50 other companies like Cray play a major role. Those are more players which are really
focusing on the very high end of numerical computing at research centers and that's
where Cray and SGI play a big role while in the broad market overall, that's pretty much
shared between IBM and HP. In terms of performance, IBM has held the lead for a couple of years
and is still doing so, nothing really has changed there. If you go on and look at technological
indicators of what goes on in the top 500, one of the technological features we analyze
is what interconnect for instance is used in the top 500. I have shown this graph also
a couple of times. It shows the interconnects used in anything which we label as a cluster
and you see that nowadays it's pretty much a race between gig internet and InfiniBand.
Mirinet as a specialized product has faded away and the company is actually now manufacturing
gigi equipment and InfiniBand equipment. It's a race between those two. The interesting part
is that gigi actually seems to be going down after reaching a peak. It has to be seen if
that is really a continuing development or not because at the very high end, especially in the
top 50, is very clear that you have to have a high end interconnect system and for us that nowadays
that's InfiniBand or that's custom systems which are not listed here. InfiniBand is still on a
strong rise and that's in my opinion likely to continue for a while. Now another technological
indicator which we look at is an indirect one is simply by analyzing performance more carefully.
What is the performance and the performance efficiency a system achieves in the top 500?
So what you see here is the ratio between limpeg performance and theoretically peak
performance of a system and you can see there is obviously a big spread.
Most systems are somewhere between 50 and 80, 90 percent and even for the naked eye they clearly
fall into two groups. One group is down here which is in the 50 to 60 percent range and the other
group of systems is up here in the 80 plus percent range. That differentiation is actually and I
colored it now by interconnect. That's why I'm showing this slide after the interconnection
slide. The blue systems here are all the systems connected by GIGI as they're in the database.
So if you have a GIGI based systems your efficiency on the limpeg benchmark
and presumably the efficiency on most HPC and most scientific computations is going to go down
because of the lower efficiency of the GIGI compared to here. Now if you look at the slide
you might notice that it looks like there are some outliers, there are some data points up here
which probably is just a mistake in the database and there are also some data points down here
especially at the very high positions in the top 500 and those are worth looking into and if we do
that we actually find that those four systems which have a lower than we expected average
efficiency on the limpeg benchmark, those four systems are the accelerated systems, the big
accelerated systems which use GPUs to achieve their peak performance and their limpeg performance.
So the systems which have GPUs in the top 500 have a substantially lower efficiency on the
limpeg performance despite the fact that they have a fast interconnect network. That's the number two
and number seven. There is another NVIDIA accelerated system I believe number 19 in China
and there is at around position 64. There is the system at GSIC from Matoshi Satoka
which is actually a very heterogeneous system, it has clear speed accelerators as well as NVIDIA
accelerators. I point out this system for two reasons, one reason is that even so it is a very
heterogeneous systems and it is accelerated that they actually managed to get their percentage
in the limpeg efficiency up to a level of the competition so it is not insurmountable if you
use NVIDIA or any other GPUs to accelerate your system, the good good limpeg performance but you
have to spend a lot of time and you have to know what you're doing and it just happened so that
this morning Satoshi announced that they're going to build they're going to they just acquired a new
system with NVIDIA GPUs for acceleration, a 2.4 plate developed system which they're going to
install later in this year and if you want any details about that you can talk with him at his
booth. Now another technology category we are looking at is of course what type of processes
are you using and what are the features of these processes. So here is a more detailed breakdown
in the type of processes you see the whole zoo and history of Intel processes intermixed with
some of the products from AMD as well as Intel in terms of number of systems as well as performance
this is number of systems. You see towards the end of the list which is up here very small
percentages you see some of the nearest processes like the Westmere processes coming into the list
by large we see actually still what core processes in the list. So even so the Westmere is
very low in terms of number of systems or hasn't increased very much yet if you break the if you
break the list down in terms of performance the same processor is already substantially higher.
The same is true for the newer offerings of AMD with the six core chips and other and other
new chips. One difference here in the market or a clear indication of what kind of a clear
this is a clear indication of one special feature going on in the HPC market. The HPC market tends
to innovate from the top down so it is not like a new product enters the market in a small system
first and then later on large system adopted in the HPC market it works the other way around
in the HPC market new technology is typically adopted at the very high end so the very high
end is going to have the new chips it's going to have the new architectures and it's going to
trickle down from there that's different from other markets and the breakdown in terms of
processes has that shown for a couple of years that you always see the new processes carrying
a lot more performance because they are employed first in the very big system and then it trickles
down. Now if you look in a little more detail we can actually group these processes a little bit
by generation of processor and a very rough generational signature is how many cores
does a system have per socket. So if you look at that you see the majority of the system
85 percent right now is quad core based processes those have been in the list for a while they are
still are a few remaining dual core and single core systems and there are a couple of systems
which now have more than four cores per process is like six core eight nine and 12 core systems
those are just starting to enter the list especially the six core systems the Westmere
processes and such and we expect to grow these to grow more rapidly in the future obviously.
If you actually look at the history how this core count per socket changes over time you get the
following picture and what you see here is that we had a rather rapid change from single core the
dual core which then rapidly were replaced by quad cores but what you actually notice is what's
missing on this slide. If we really assume that nowadays we use we utilize the transistors we
gain by Moore's law for increasing core counts you would expect that core counts increase with
Moore's law and that has not happened during the last couple of years. If it would follow Moore's
law we would have probably half of the systems in the top 500 should have processes with six or eight
cores by now and that's clearly not the case. So there is a delay here that delay can mean multiple
things can simply be an economic delay because people are not buying new systems can actually
be what I consider a much more severe technological delay. It could be that chip manufacturers use
the increased transistor count to implement other features which we don't see in the statistics and
which might or might not benefit HPC performance or in the worst case it could actually be an
indication that something is wrong with Moore's law itself. So in the past we have had
in the past we have had numerous questions as to whether Moore's law is in trouble or if
we ever saw a slowdown in Moore's law if there is anything like that going on and in the past I
always said that's clearly not the case because you cannot just from one or two data points
extrapolate something like that. Now what's different this time around is that we actually
have a situation where we have two technological indicators which indicate something is slowing
down or changing in the market. The first one is that performance does not grow as rapidly as it
used to be. So that's the first indication and the second indication is here something is changing
in the underlying technology as well. So this is a situation where we're clearly going to have to
look more carefully in the underlying technology if it's really a technological reason or if there
is another reason for the slowdown in overall performance. Now one of the things we started
looking at the last couple of years is power, power consumption of systems in the top 500.
It has become the last three, four year power consumption of our systems, of our technology
in general has become a major concern at all levels of performance. So the first question we
ask simply is what is the power consumption of the system in the top 500 here in kilowatt
over the rank of the top 500. And you see the number of systems which we know use more than
one megawatt is actually not as high as we feared a couple of years ago. You see about 20 systems in
the top 500 and if you know that we only know about the power consumption of about 60 percent
of the systems you can estimate that there are going to be about 40 systems which have a power
consumption over a megawatt. So that doesn't sound as bad but still it's worth looking into
more detail into this situation. If you look into the power and power efficiency now according to
different technologies one of the things you find here is if you look at different processor
generations again enumerated by cores of processors that if you go from two to quad core to six core
to nine core and higher core systems that power efficiency tend to go up. Now that is simply and
that's expected that because that is simply generational change this is older technology
this is medium age I would say and this is newer technology. New technology seems to have higher
power efficiency that's what we were hoping for that's what we are working on. So this is
it's good to see that we see that but the question is is that enough to really stop the increase in
overall power consumption in our systems. We can also look at this statistics of power efficiency
with different technological indicators and one of these indicators for instance would again be
what type of interconnect network do we actually use in our systems. So if you again break it down
by infinity bank gigabit ethernet and custom interconnect at the very high end we see that
because this is not a generational division this is really a technological division that interconnects
which require more money and more effort to build that they actually deliver the higher not only
performance but also power efficiency overall in the system. So custom built systems with custom
interconnect like the Kray systems and the IBM blue chain systems they tend to have a much higher
power efficiency compared to infinity band and especially compared to gigabit ethernet. Again
that's two reasons. One reason is the difference in the limpback efficiency but the other reason
is actually the difference in technology. Now another statistics which I'm going to show for
the first time ever here is that we now we have three years of history of power data in the top
500 so we can actually start to look at the history of how power changes over time. So what
I did here is I plotted again the power average power consumption of a top 10 system top 50 system
and a top 500 system. A top 10 system a top 50 system top 500 system for the last three years
there is five editions of the list and I actually put their own numbers below
because if having only three years you shouldn't try too hard to analyze the data because there
are there is still a certain lack of data points but what you clearly see is all three curves
showing trends that power consumption overall of the system increases especially the number 10.
The number 10 is a little tricky you have to be very careful because 10 is a small number to make
a statistic to start with. We don't have power consumption for all 10 systems so there might be
a selection bias in there that centers or system manufacturers owners who have higher power
consumption don't report it to us and stuff like that but nevertheless if you look at it overall
you see there is a an indication that overall power consumption in the centers is going up
and a lot of center directors actually confirming that that with new procurements they're more and
more on in this situation that their old machine rooms are not only too small in terms of size
but they're actually too small in terms of power feeds and things like that they have to increase
their power power feeds. If these trends would really hold as they are on this slide I would
be very worried and the reason for that is that the top 10 curve is so much steeper than the
overall top 500 curve because that would be a clear indication that the top 50 systems in the
big research centers that they're going to run into much more power problem than the average
customer in the field. I'm not quite sure that the difference you see here in growth rate is as large
as it is as I said I think there is a good chance that some of that is a selection bias like if you
eliminate this data point the growth rate is much smaller but overall there is a growing there is a
trend that overall power consumption grows and there is where some if you look forward and if
you start extrapolating these curves which I have not done at this point. Now the good news of course
is if you don't look at power consumption itself but if you look at power efficiency
if you look at power efficiency power that that is Olympic performance divided by power
and to the same average for the top 10 system top 50 and top 500 system we see again a growing trend
and this one is actually a good one because it means we are getting more efficient in how we
compute the question still is or the question is going to be in the future is this increase
sufficient to hold our overall power consumption in check or not the last three years as far as
it's reflected in the top 500 is a clear indication that it's it's not sufficient
power efficiency increases but not nearly as much as power consumption overall as performance
overall and things like these we have to spend more effort we have to spend more
engineering and getting into the and keeping or getting this situation under control.
So this is all I have prepared so far we are running a little bit ahead of time
and I'm not sure if you should take question at this point or if you want to go on
questions so if there are a few questions
I guess it's too early.
Because I'm not sure the next speakers are here.
There's a question.
Do we have a mic?
On the peak usage is this peak usage data average power usage on the efficiency or is
this total center power averaged over the year?
Neither. So what we do and there is actually going to be a both session tonight at 5.20 where we
talk in detail about how we do it and how we're going to want to go forward in talking and
analyzing power data what we do is we decided in the top 500 the top 500 is a list which is
really focused on individual systems so we cannot really look at a center at a machine
room or anything like that as an aggregate because that brings in the environment and
doesn't work with this so we try to come up with rules which clearly divide what belongs to the
machine room what belongs to the system and we just try to account for and list everything which
belongs to the system and then as a second step we say a machine plate power is not what we want
to list that might be important for the machine room layout in terms of how you have to provision
power but as an as most important now power number we actually consider the average consumption
so the rule is basically you have to run limpac and you have to measure power consumption while
you run limpac we did some studies two years ago as to the influence of how much the actual
workload influences how much power system draws and there are some differences but those tend to
be at least two years ago those were secondary effects so HPL is not a bad proxy for scientific
workload it's very close some system components like CPUs are utilized more heavily they draw
more power some other system components are utilized less so they're all a little less power
so the top 500 lists basically our average power draw during a limpac run and we divide that by
limpac performance so it's actually two measured numbers that's one differentiating factor between
the top 500 and the green 500 if you look at their data they have data for more systems
but some of these or a lot of these data are nameplate data peak power consumption stuff like
that and we decided not to list those and not to use those because peak power consumption sometimes
is very much higher than average power consumption so we don't do that so this is actual power
consumption this is what really hits your budget in terms of the electricity budget if a vendor
has a system which draws a lot of power when you hit the on button that's a different issue
it's also an issue
any other questions
there's one back there too
there's one yeah
have you ever considered collecting data on system utilization after one year of using
each system because in the case of more exotic architectures limpac is a sort of stance on its
own yeah i mean we had the limpac discussion obviously we had had that for many times
many years the truth at the end of the day is it's hard to get by any more serious numbers
if you would be serious about this we would do a real scientific workload try to measure that
performance and utilization based on that to really capture all the features of a system
that's what you have to do at the end of the day if you buy a system you cannot use the top 500
and the numbers we report to buy a system if you do that there's something wrong you have to use
you have to do a workload analysis you have to compile your own benchmarks for the large systems
run those benchmarks and do your analysis based on that why we hang on to limpac is and not just
use peak numbers is basically for two reasons the limpac is is a first order correction to peak
uh for two reasons one is if if your system is in boxes in the basement or is faulty you're not
going to be able to get a limpac run through especially not at the high end where limpac runs
can easily take 12 hours or more a limpac is numerically very sensitive to uh to any errors
in the system if your bits change while they're on the wire bit going between cores you're going
to get a wrong result in limpac so that's one nice feature a feature of limpac benchmark
it's a real hard verification benchmark for new systems that's one reason and the other reason is
even as it is a very mild benchmark it's still a benchmark and you still see that different
technology achieve different efficiencies and you still see some effect of technology in limpac
performance of course we all know the average scientific calculation is going to achieve much
efficiency performance efficiency than the numbers i have shown here so nobody should
take that as a guidance buddy so last year when we ran limpac on jaguar it took about 18 hours
this year was about 17 hours and we've talked with professor dongera and some of the other members
of the top 500 team about looking at new ways to perhaps limit the time that it takes for uh for
limpac to run can you say anything about what the committee has discussed on that
so yeah we discussed a couple of ideas because at the very high end it really is becoming a problem
you can't just take a system which costs 50 or 100 million dollars and take it out of commission
for two or three days because you want to take limpac numbers and every run is going to take
you a full day so how can we improve that situation and we tossed around a couple of ideas on how we
could abbreviate limpac runs for the very high end systems there are there are ideas how to do
that and still get a reasonable number it's probably going to be a solution where if you're
on the full benchmark you might get a little more performance than if you're on the abbreviated
benchmark but there is something in the work and so the scheduling here at is becoming more and
more tricky check was here before i came he left when i came he's going to come back tonight or
tomorrow and he's actually going to give a talk i believe on thursday where he's going to address
some of those issues so i don't know if you're going to be here on thursday but there is
definitely something in the works uh there is going to be something which is only going to be
available for the high end systems the very large ones so if your limpac runs less than an hour on
your system we're just going to ask you to run the whole thing you're not going to try to to reduce
limpac to a couple of minutes because the the nice thing about a six hour round or four hour
round is actually you have to have a stable system that is often issue too for very new
systems they are not quite stable i had a question in the third row i don't know where the mic is
or is the mic back there okay let's take the question back there i've got a question regarding
capabilities of power utility companies nowadays can we just approach it differently and figure out
if we can get the maximum peak performance because of the limits of distribution of power
to those clusters or are we not hitting that limit yet so i'm not quite sure i understand
the question the question basically is is power consumption is is the power distribution in the
system already at the level that it's a limitation for what we can achieve in terms of performance
and is it going to impact more slow directly yes and no i i think overall it's probably not
an issue but if you if you're a little more carefully if you look into this turbo modes
for instance in the processes then the answer clearly is yes if you have a turbo mode which
only kicks in if the power or consumption and the temperature in the chip is low enough
then you are limited by how often you can turn that on and the hpl is actually very good in
stressing that so for those systems in a way yes power is already a limiter and because you
cannot turn on turbo mode all the time across all the cores because you're going to melt your chip
so in that respect yes it is already an issue in terms of laying out cables and getting power to
the individual components i don't think that's an issue yet yes okay i'm from the institute of
computing technology chinese academy of sciences i suspect the systems equips was accelerators
especially gpu gpu we will increase very rapidly so i'm very interested in the statistics
informations of the architecture model and vendors of them so could you tell us more about this
so uh for about i would say two years uh we have been asked the question what about GPUs
what about those accelerated systems i would say the first accelerated systems recently which we
have seen was actually the the ibm products with the the power cell architecture with the sony
playstation architecture uh inside those showed up in a fear number and are still in the list
and we started watching them and analyzing them the cheap use they had a lot of rumor
early up up front and it took them a little longer than we expected to actually show up in
the list all right now i'm still only counting four system which have GPUs as compute engines
in the in the top 500 that's the four i mentioned so that's still a fairly low number however having
said that those are four systems which when they showed up they were clearly top 20 systems
those are four large systems and we know that they're going to be fairly soon a lot more uh
a lot more of those i think there are still technological reasons in the hardware as well
in the software which which are holding those systems back and we just have to overcome those
once those are overcome i think the number might increase rapidly much more rapidly
but right now of course it's it's a challenge it's probably not just the leading edge it's
probably the bleeding edge to some extent if you have one of those systems
are there any other questions okay then thank you Eric for the interesting talks
and for the opportunity to put some questions