development at the Computational Information Systems Laboratory at NCAR, U.S.
And I think we have had some interesting titles before, like climate supercomputing on the
grid, how far does it carry it?
I think now we will learn something about more than a big machine, and for the final
panel discussion, especially why we need multiple breakthroughs, is an interesting question
which will perhaps be answered now.
Okay.
Well, this is always an enviable position.
I'm at the end of a very long session.
Everybody's tired and jet lagged.
Microsoft is failing, and there's an alcohol party starting right after I finish speaking.
So I'm in a – also, all of my slides have been presented by the previous speakers.
So everything is a bit unusual in this kind of situation, but I'll try to go through
the points that haven't been talked about, and hopefully wake you up and get a little
more discussion going.
So let me go – this picture's already been presented.
This is basically the simple point that clouds have this ridiculously small scale in which
they operate.
Some people have noted that we could keep going downscale all the way to molecules.
It's bad enough that a single cubic kilometer of cloud has roughly 10 to the 18th water
droplets in it.
So if you were to just interact water droplets together, you already have an exascale problem
in a single cloud.
So at some point, we obviously have to cut this problem off and say the rest of this
is phenomenology.
Just one of the things I want to really poke a hole in here is the notion of the feasibility
of doing a one-kilometer global atmospheric model.
And so I've done some calculations here, very simple.
The area of the Earth in square kilometers, taking the number of levels at 100, the number
of chemical tracers at 100, the number of seconds in a century, giving you a flop budget
of 5,000 flops per time step to do all the calculations required in this.
That may be algorithmic dependent, but it's not a completely unreasonable number.
And the number of flops per century that you end up with now ends up being scaled by the
spatial discretization in kilometers and the time step in seconds.
The interesting thing here is the numerator is roughly a mole of flops.
So an exa-flop is 10 to the 18th flops.
That's actually pretty small compared to a mole.
So when you actually think about computing at 10 to the 18th flops sustained, that really
means you need 100 exa-flop peak supercomputer, roughly 10 to 100 exa-flops peak in order
to have a prayer of doing the calculation, and when you're done, you still won't have
met Sylvie's requirement that you be able to complete a century in a month.
Also this will horrify climateprediction.net because that's, of course, one ensemble member
that will come out of a computer basically the size of the planet or something.
So this is a pretty scary thing to contemplate.
So it's a scary thing to contemplate.
The basic problem is, of course, that we're sitting quite comfortably at 100 kilometers
or so out here, and we have to jump over two orders of magnitude through a kind of no man's
land on the scale of around 10 kilometers where things are rather difficult to represent.
So you can parameterize cleanly at 100 kilometers.
At one kilometer you can start resolving convective scales or cloud scales, but at 10 kilometers
things are in a kind of ambiguous situation.
So it's really this deep end of the pool that we're looking at, this factor of 100, especially
when you raise it to the third power that is the source of all the basic problems here.
So as been mentioned earlier, we don't have all the time in the world.
We don't want to find out the answer to this after the poles have melted and it's too late
to do anything about it.
So the only way to do this is to get an investment in a really extraordinarily large, extraordinarily
large machine way before it's really feasible to do it.
We can't wait for Moore's law.
It will be all over by the time Moore's law catches up with this factor of 10 to the seventh.
The problem, of course, as other speakers have noted, is that Amdahl's law is essentially
a formidable opponent.
Multiple things like trying to synchronize a billion threads is similar to getting the
entire population of China to snap their fingers at the same time or something.
It's practically impossible.
It takes a very long time to do that.
If one of those threads is busy doing something else, everybody is waiting, which is wasting
machine resources, of course.
We already see the effects of these kind of things on machines with 10,000 processors.
Once we start getting to millions or even a billion threads, this becomes extremely
difficult to imagine working.
The other point that I don't think has really been made clear here is because we are only
parallelizing typically in the horizontal, the time step dropping off as we increase
resolution means that our integration rate will drop lower and lower and lower.
So by the time you get below, let's say, a few kilometers, you'll no longer be able
to simulate climate at a useful rate with any kind of reasonable machine.
The final point here, I think that I should mention, is everybody would right now, most
people when they're doing these simulations are actually using too few levels.
And so adding levels and adding physics and complexity to the physics is only going to
increase the amount of work that the threads have to do.
So we all know that the threads aren't going any faster, and so this creates a real problem
because we simply cannot scale our way out of this problem with the machine anyway.
This is just a picture that everybody knows that takes 30 years.
This picture was just presented that shows us flattening out.
This is an interesting picture in that it shows the fact that parallelism is just coming
more and more common in systems every 18 months or so.
We seem to double the number of processors.
We all know that now.
I want to show you possibly a good example of an answer to one of the questions that
came during the last talk, and that has to do with this notion of why do you need non-locality.
So this is a thing we call the higher order multiscale modeling environment or home, and
it's a code built on a cube sphere, which is a quasi-uniform mesh.
It's very local.
It's a very accurate method.
It has been modified by people at Sandia to be conservative in terms of mass and energy
and so forth.
That model has very good locality properties and is explicit.
When you look at the scaling curve here, and this is five years per day, which is one measure
of scientific productivity in these things, so we simulate five years in a day.
By the end of a month, you'll have a century and a half.
You can perform one of Sylvie's experiments, and you still remember what the question was
that you asked at the beginning of the month.
This shows 50 kilometers, 25 kilometers, and 12.5 kilometers atmospheric models running
in a mode that's called aqua planet, so it's an idealized ocean underneath an atmospheric
model.
You can see that at 50 kilometers, we can easily do this below 10,000 processors.
We can easily get to five years per day.
At 25 kilometers, we can just make this at close to 100,000 processors.
This happens to be on a blue genial system at Livermore.
However, once we get to 12.5 kilometers, this model cannot approach this.
We seem to have some issues scaling here, but even if we could extrapolate this as being
scalable out to that level, we would require more like 200,000 processors to possibly get
to this five years per day.
The point here is that the higher the resolution, the harder it is going to get to that integration
rate.
The exascale, how do we get there?
I think I'm going to make some high-level points rather than explaining how we get there
because the short version of the answer is I'm not too sure how we get there.
I do know that there are certain elements of what we've managed to accomplish right
now that show you how it's going to have to go down.
The first thing, we have to have large interdisciplinary teams.
This is a listing of the names of people that are currently involved between the Department
of Energy and NCAR and a variety of other organizations, people who are actually involved
in doing these simulations, developing the software.
There are quite a few software engineers, quite a few scientists, a large number of
computer science people and mathematicians involved.
You see that it requires a great deal of different grants, all sorts of computer times
on a variety of systems between craze and blue jeans, primarily the systems we focused
on for doing high resolution work because of their scalability properties, which are
quite nice for this kind of stuff.
Second point is, and this has been made I think by Balaji and a couple others, these
systems have grown into complex systems.
I often have had a vendor say, well, send me the kernel from the climate model and we'll
optimize it.
There is no kernel.
It's 1 million lines of kernel.
The amount of time spent in any one component is typically not that big, even subcomponents.
It's fairly evenly distributed, so it's not something like a signal processing code or
a DNA sequencing code where you might have a very small kernel that you could rewrite
and assemble, for example.
So all these interacting components typically are written by different people.
You have to be careful that they don't have interoperability problems or different units
assigned to physical quantities.
So absolutely a predecessor to this is that you have to, before you can contemplate doing
this kind of work, you have to have a flexible modeling framework that allows you to build
it or a series of frameworks that allow you to build these sort of things and reuse as
much of the infrastructure as possible.
The other thing that I want to particularly emphasize is the fact that we are short of
people.
We're short of brain power on this problem.
We don't have enough young people who are interested in high performance computing to
begin with, but we certainly within the sphere of high performance computing don't have enough
people on this problem.
And one of the things that I have gotten involved in is this notion of having students come
to NCAR during the summer and work on real HPC problems and get involved in the challenge.
This was the first year.
This is the second year.
We've basically doubled the number of summer internships at NCAR over the last three years
trying to ramp this up.
I believe that this is only a small piece of what needs to be done here, but we certainly
need to find programs, build programs that involve many, many groups of people.
The other thing is that we have found it particularly useful to have routine access to a bunch of
processors.
If you're going to run on 100,000 processors, your largest machine that you develop on cannot
be 16 processors or 32 processors.
You have to have several thousand cores that you can get without having to have a center
director give you a special permission slip.
In particular, you have to allow young people who are liable to make mistakes and learn
the hard way, they have to have access to these kind of resources.
So one of the things that we did was write a proposal that brought in an experimental
blue genial frame that we have used both in teaching and in parallel scalability research.
Now since the threads aren't getting any faster, we are going to have to figure out how to
get these models to actually integrate faster.
Now using that simple-minded one kilometer analysis with the mole of flops, you can
see that there's a couple issues with just trying to brute force this problem.
Now if you look at the scales report from 2004, you'll find that in other disciplines,
in this case magnetic fusion research, at least in that case, they were able to demonstrate
that the scalability from hardware improvements was matched and even exceeded by algorithmic
improvements to the code.
In other words, they got as much improvement to the simulation rate from doing math as
they did by sitting around waiting for Moore's law to actually make the things better, faster,
cheaper.
Combined together, this resulted in the kind of improvements that we're looking for out
of the climate system.
So this is the basic problem that to get the algorithmic accelerations means that we can't
really have systems that can only do explicit methods well.
This goes back to the question was asked a little earlier.
We need to have systems that can deal with the interplay between mathematics and computer
architecture in a way that gives us some wiggle room to develop some better algorithms.
So when I say things like this, I mean consider computational efficiency.
Well you can do, you can look at computational efficiency as lowering the number of flops
per unit of spacetime volume.
But how do you do that?
Well, you can make the time step bigger.
You can have fewer horizontal grid cells or you can have a more efficient algorithm
to begin with.
Now one way to get fewer horizontal grid cells is to do AMR.
But AMR, of course, is an irregular communication pattern that can give certain computers a
great deal of problems.
You can take a bigger time step, but that usually involves an implicit method in which
one has to have a solver.
Now solvers, of course, are non-local.
You look at a CG solver, GM resident kind of Krylov space solver, they have inner products.
So they like to add up all the numbers in the machine.
And that in itself is a scalability issue.
So it's inevitable that the mathematics that we're going to try to use to accelerate these
machines will lead us away from localized explicit methods to non-local methods that
stress the interconnect on the computer.
To see how bad existing computers are, this is the HPL benchmark from, I didn't have the
latest numbers from here, but this is from supercomputing.
And you see these XT5 machine at 902 teraflops.
But if you look at the best numbers brought out by the GFFT number, they're substantially
lower.
In fact, they're roughly 170 times lower than the best numbers that you get on LIMPAC.
So we have these machines which are very, very sensitive to non-locality.
So we have to really have a strong interplay between the architectures and the mathematics
so that we can find some kind of modus vivendi between the mathematics and the computer architectures
to get some kind of improvement.
This is just an example showing a multigrid method that's being developed by Randall and
others at CSU.
And it looks very good at 10,000 processors.
But if you take the Amdahl piece of it and you extrapolate it out to 150,000, you see
that it tails off, or it is projected tail off.
So even things like multigrid, which are thought to be very scalable algorithms, have the potential
of being limited to 100,000, 150,000 processors before they run out of gas.
Now the other issue that's coming along, I call it the flops archipelago.
This is to deal with accelerators.
The accelerators, we have tons of flops, but like in an archipelago, they're little islands,
and they're cut off from each other by these alligator-infested waters, shark-infested
waters.
And so when I want to move between one of these cards and another one, I'm going to
run into some problem with the communication.
Now the basic problem with this is illustrated here, and I think Andy mentioned some technology
solutions to get around this particular problem, which is we're projecting here the number
of pins from the ITRS roadmap report, the number of pins on a packaged chip between
now and 2023.
And this is about 3,000, this is about 6,500, so we're looking at only doubling the number
of pins coming out of these chips.
But we're adding flops because we're dividing this zygote of these cores every Moore's
Law half-life, this thing divides, and all we have is kind of linearly improving pin
count.
So we have things like Cell that look a lot like this, we have things like the NVIDIA
GE9800 and its successors that are going to look more and more like massively parallel
systems cut off from the rest of the world.
And the thing that we're going to have to learn to deal with in decomposing these problems
onto these sorts of systems is this concept of computational intensity.
And of course this is the relationship between bandwidth, which is characterized by basically
how many pins you can pull off of this thing, times the computational intensity to actually
get the number of flops that are coming out.
Now we've done some measurements, this is from our computer science group at NCAR.
This is taking an algorithm, I won't go into the details of it, but it's showing the computational
intensity of the cell chip, and you very quickly saturate, down here in the bottom I should
say is computational intensity, this is actual performance.
You saturate very quickly, so this is a bit like a vector length curve, but the cell turns
out to be extremely difficult to program.
On the other hand the NVIDIA is easier to program, but it saturates at a much higher
computational intensity value.
So this might be the number of floating point operations per byte, which is transferred
on and off of that card.
So what we're looking at right now is we don't have, that I can see, an accelerator that
is easy to program and relatively resilient with respect to the computational intensity
required.
So that makes it difficult to adopt such systems in the application space that we're dealing
with where we have lots of code, it's somewhat malleable in the sense that it will change
periodically because scientists want to improve the algorithms.
So what's a possible compromise that looks doable at this stage?
There's a thing called superparameterization.
It is an intermediate step on the way to a cloud resolving model.
If you look at the relative cost of this sort of model, you essentially represent subgrid
scale, you run a regular climate model say at a few tens of kilometers and then you have
a subgrid scale cloud parameterization, which is perhaps one kilometer cloud model but
doesn't have the physics or the rest of the dynamics at that scale and then it couples
back and forth.
And this sort of model is suitable for moving on to graphics cards.
So if you look at the relative cost, a day of simulation of a cloud resolving system
is a few years of, is equivalent to a few years of a superparameterized model and is
comparable to a millennium long simulation of a traditional say one or half degree model.
So this is an interesting concept.
The details are here that basically have already said this.
The way you could look at it is you could have a conventional core as a front end, you
have some atmospheric cloud columns and you have a bus that connects to an accelerator
card and then you have the expensive cloud models distributed across the processors in
the GPU and you get your acceleration that way.
If you have a system that's built out of accelerators and regular CPUs, you could obtain
simulation rates that are comparable to what we're seeing with regular atmospheric models
at this point with accelerators providing the boost for such a simulation.
And that's something that we're basically working on now.
So let me just summarize this and then we can get to our panel.
I see this, you know, when we talk about investment and we really sort of say, well, suppose we
had the money to build an exascale computer, is it really wise to build an exascale computer
or perhaps invest the money slightly differently and mindful of the fact that Andy pointed
out this morning that by 2020 a pedascale system will be one rack.
So about as much as we were standing next to in that one photo, it seems to me that
we would be better off investing in the people for algorithmic research, for software engineering
and for the computational science research that will allow us to make this jump between
where we are and a factor of 100 in a way that is sensible.
We also need to really find an architecture which involves a usable accelerator with a
reasonable computational intensity in one half, if I can use that analogy to those of
you that remember vector machines.
And finally, we have to find modeling schemes that take subgrid scale physics and actually
can apply it to these kind of architectures in a way that is scientifically useful.
And so I can say a couple other things.
One, we have been to 100,000 processors and it is doable to run on 100,000 processors
with atmospheric models, but we have to go to those kind of resolutions where the integration
rate starts to get into trouble.
So we need faster threads at 100,000 processors in order to really make significant progress.
And that is my last slide.
So I will be happy to take any questions.
Thank you.
APPLAUSE