And so we are looking forward to you as kind of an advocate of the – well, we used to
name a grid, maybe it's the cloud now or something like that.
Okay, thank you.
Miles, up to you.
Thank you.
Okay, so it's rather odd to be speaking at a supercomputing conference because I'm probably
best known in the climate modeling community as somebody who never uses supercomputers,
in fact, the largest computer – the largest project that I'm involved in, which Reinhard
just kindly mentioned, is the climateprediction.net project, which made its name essentially
by running climate models on personal computers, on the smallest computers – well, not perhaps
the smallest computers, but the sort of workhorse desktops that people use every day.
But so here's the question, why am I talking at a supercomputing conference?
I'm very grateful to Reinhard for inviting me because I – and I agree with Julia and
Joachim, I assume Joachim agreed with himself when he was sort of calling for a revolution
in climate modeling, that we do need a revolution, a computing revolution in climate modeling,
but I'm not sure that I think it's the same revolution that Julia and Joachim are
calling for.
So perhaps this talk will give you some ideas to argue about in the panel discussion.
Please don't take away from this talk that I'm disagreeing with Joachim and Julia that
we do need a quantum advance in the science of climate modeling and that computing plays
a central role in that, but I think we do need to think quite hard in calling for this
revolution this crucial question.
How do we know when we've won?
How do we know when the revolution of the climate and climate modeling is over?
And sort of, you know, before you invade – before you invade anywhere, it's a good idea to
ask yourself, you know, how are you going to know when you've won?
This is a lesson which some people have learned rather painfully over the past few years.
So before we embark on this grand challenge, we need to, you know, ask ourselves what are
the success criteria, you know, how do we know when our revolution is over?
This one – if we said the revolution's over when we have a 25-kilometer resolution
global climate model, we're kind of nearly there already.
It's hardly even a revolution.
Joachim was showing you results from 60-kilometer resolution models.
You might say, okay, what we need is a one-kilometer resolution global model.
This was also put forward at this Reading summit as a criteria for, you know, when we're
going to be done.
A more sort of scientifically-oriented criterion is when we don't need to parameterize clouds.
Joachim was explaining to you how at some scale we always have to represent small-scale
processes and climate models with approximations, and these approximations are fraught with
uncertainty.
And so clearly, the more resolution we can pack in, the less we have to deal with small-scale
processes by parameterization and the more confidence we can have in our results, okay?
But of course, as Joachim pointed out, there's been no coordination of these talks, but it
fits beautifully with the previous talk, actually.
As Joachim pointed out, you know, to avoid parameterizing clouds, you really need to
go down to one-kilometer resolution to avoid parameterizing ocean eddies.
I mean, some of those structures Joachim was showing you in the ocean are well subkilometer,
in fact, in terms of the filamentary structures in the ocean.
This was also suggested, actually, in a nature editorial after the Reading summit that we've
only really won when we have a bigger computer than the guys who design weapons.
That's an interesting criteria, but I think you need to justify it a bit better than that.
This is my answer, which is a little different from the answers – some of the answers given
at the Reading summit.
And my answer goes like this.
No matter how much – how we perturb our climate models, we get the same distribution
of future climates, okay, the same range of uncertainty and important climate parameters
in 30 to 50 years into the future, consistent with the observations of past and present
climate, okay?
So if we get to this point, I would say we were done, okay?
That would mean we have said what we can say about future climate based on current observations.
Nothing can do no more than this without more data to constrain the observation – to
constrain the forecast further, okay?
But you'll notice the crucial element here is no matter how we perturb our climate models.
The point is there's many uncertain parameters.
There always will be uncertain parameters in climate models, parameters that are tuned
to fit observations.
And the question is if we vary these parameters, if we allow them to vary over their range
of uncertainty, we'll get a range of predictions, and we'll also get a range of simulations
of the past.
And the question I would like to pose is, is there a limit to how far we can vary these
parameters and get a prediction of the future that is still consistent with observations
of the past?
We believe in physics.
We believe in the predictability of climate.
So I believe there is such a limit, but I also believe we haven't found it yet.
We don't know what range of future climates is consistent with current climate observations.
And that's a major hole, because what the policymakers really want to know is not our
best guess of future climate, but they want to know the range of uncertainty.
They want to know how bad could it be.
Some of them want to know how good could it be.
And that's the sort of fundamental question we're asked in planning.
If I illustrate this with what you might call a recent failure of climate modeling, why would
I call this figure a failure?
This is one of the most famous figures to emerge from the recent Intergovernmental
Panel on Climate Change Scientific Assessment in 2007.
On the left-hand side, you see ranges.
These different colors correspond to different scenarios for future human activity, human
influence on climate.
And the ranges here show the spread of results from the IPCC class climate models that Joachim
was talking about.
And the heavy lines at the ensemble mean, and these were contributed from 20-odd modeling
groups from around the world.
You can see there's a range of uncertainty here.
And definitely, 99% of the effort into producing this figure went into the left-hand side.
Enormous amounts of supercomputing resource was devoted to doing these runs, producing
these ranges of uncertainty.
But then if you look on the right-hand side here, you see the final punchline ranges of
uncertainty that were given by the IPCC.
And notice they're very different.
The range of the models is perhaps only half that which the authors of the IPCC report
said was the true uncertainty in warming response to this particular scenario.
Now why is that?
Well, we'll come on to why that is in a minute.
But the important point is that 99% of the impact of this figure in the IPCC report came
from the lines on the right.
All the press quoted was basically 1. something to 6.3, or I can't even remember what the
number is now.
And they were really excited that this 6.3 was higher than 6.1, which was the number
in 2001.
Anyway, the point is that most of the impact of the study came out of these uncertainty
ranges on the right.
And I'll tell you now how those arrived at.
Well, we could not use the spread of the models alone for a very simple reason.
This figure illustrates all of the models which went into this, all of the runs that
went into this multimodal intercomparison, compared with observations, the black heavy
line here in the center, for simulating global temperatures over the 20th century.
You can see that fit is remarkably good.
All the models tend to match observations quite well.
There's some evidence that they over respond to volcanoes.
There's some interesting questions there.
But broadly speaking, they get the temperature evolution over the 20th century right.
This is a rather notorious figure which shows some of these models, some of these models
in this CMIP-3 intercomparison.
It compares their response in the horizontal with the size of the driving in the vertical.
Now the reason this figure is notorious is there's no reason scientifically for these
two things to be correlated.
But that line is what you need to get the trend over the 20th century right.
So you can see why people were somewhat, there was a whiff of scandal about this figure because
it implies that the modeling groups somehow cheated by choosing low forcing for high response
models and high forcing for low response models.
Now there's been a lot of argument about this, about whether this actually undermines the
CMIP-3 ensemble as a measure of uncertainty in climate modeling.
But the key point is that when we came to write the IPCC fourth assessment, everybody
agreed that we could not use the spread of models in the CMIP-3 ensemble as a simple measure
of uncertainty because we knew that all the modeling groups were all of them aiming for
a best fit.
None of them were really aiming to explore the range which was consistent with recent
observations.
There were some models which actually gave behavior outside the envelope of the CMIP-3
ensemble which were excluded from the results.
And also the forecast spread which we got out of the CMIP-3 ensemble was much smaller
than other methods of quantifying what we call model error.
I'll mention the phrase model error a lot in the rest of this talk and this is basically
uncertainty in the climate systems response to a given driver such as rising greenhouse
gases and this is the dominant source of uncertainty in this multi-decade climate prediction problem.
So those final ranges of uncertainty, those all important bars on the right of that figure
were given by basically a lot of scientists sitting around Jokram among them and sitting
around in Norway in the middle of the summer, all of us jet nagged out of our heads and
it was sunlight 24 hours a day so nobody actually slept.
By the end of this time we were all kind of slightly feeling slightly hallucinatory actually
and then we came up with this, okay, minus 40% plus six minutes, that'll do.
I would say this is a very bad way to actually come up with the punchline figures that actually
drive policy.
Okay, perhaps I'm cartooning the situation a little bit but we shouldn't have to rely
on expert judgment in that way to come up with the range of uncertainty in climate forecasts.
So how can we do better than this?
Well if you've got a very simple model it's relatively straightforward to quantify uncertainty.
It's just a figure drawn from a recent paper which shows lots and lots of runs, see all
the little lines on this in response to a particular scenario for greenhouse gas emissions,
for carbon dioxide emissions in this case, and they're sort of colored in by how well
they fit the observations.
So you can see the ones that fit the observations fairly well are down here and there's a small
chance of a response rather higher than that.
Now that's easy to do with a simple model because you're seeing thousands of parameters
with thousands of parameter combinations of the simple model here.
You do thousands and thousands of runs.
You compare them all with the observations and you just sort of plot the results.
Very straightforward to do but how can we actually do this with GCMs, with the full-scale
models that Joachim was talking about?
We definitely want to, let's be clear here, because relying on simple models is not satisfactory
for many reasons.
One of them, probably the most important, is that these very simple models, you can
only compare a handful of variables with observations.
You can only check the model simulation of global mean temperature, perhaps it's a simulation
of carbon dioxide increase or something, but there's very few benchmarks against which
you can test your model performance.
So you're almost bound to get a very large range of uncertainty, possibly a uselessly
large range of uncertainty out of such an exercise with an overly simple model.
But to do that sort of experiment with a GCM involves running the GCM many, many, many
thousands of times.
So how would we potentially do that?
Well first of all, would we actually need to run the GCM that often and sort of how
big an ensemble are we likely to need in our GCM experiments in the future?
Well this comes down to a rather arcane debate in the statistical community, which I don't
expect you to follow or be interested in the details of, but the debate comes down to do
you know how to sample the space of possible models?
Do you know what climate models you might have come up with as well as the one you did
come up with?
And can you sample that population of possible models in a sensible, robust and objective
way?
There are, suffice it to say, strongly held but differing views in the statistics and
climate modeling community as to whether or not this is possible.
I'll show you one result from a paper in which this approach was taken where they sampled
parameters in a model, they generated around a 50 member ensemble with a GCM.
This is from a Hadley Center study led by James Murphy in 2004.
And then they fitted their models to observations and so what you see here is a sort of raw
histogram.
The reason it's smooth is because they sort of interpolated between the model runs between
the members of their ensemble to give a nice smooth distribution and then they weighted
by the fit to observations, they came up with that red histogram there.
But the crucial problem here is that you can sample parameters in a different way, sort
of relatively innocuous looking differences in the way you sample parameters can make
an enormous difference to your results.
And then of course you get into very heated debates about whether or not the difference,
which sampling strategy is right and so on and the whole field starts to look a little
opaque to outsiders.
The other problem with using what I would call small ensembles, that is ensembles of
size 50 to 100, is that dealing with all the different sources of uncertainty in climate
modelling requires you to adopt rather complicated methods, very complicated and often rather
opaque methods of combining results from different ensembles.
This is a figure from the recent UK Climate Impact, the UK Climate Projections Report
which was released last week and it shows what they've had to do, they've had to do
one ensemble of ocean and sulphur cycle models, another two ensembles of the atmosphere model,
they've combined all these together with something called a statistical emulator to try and synthesise
what they would have got if they'd have been able to run a multi-thousand member ensemble
and all this effort and all this complexity basically arose because they couldn't do this
with their supercomputer.
I'll tell you what this is in a minute, but this is in essence what they would have liked
to have done if they had not, in order to avoid doing all this sort of complicated combination
of different ensembles.
So how large an ensemble do we need?
The Bayesian approach, as I say, requires this robust sampling from the space of all
possible models.
If that's possible, and the jury's still out, then you can probably get away with ensembles
of size 50 to 100, but you still need some kind of complicated statistical post-processing
in order to get useful answers out at the end.
Why should you care about these kind of arcane statistical arguments?
Well, if you can't sample parameters objectively and robustly in these climate models, then
more robust approaches to quantifying ensembles require even larger ensembles.
Here's an illustration, again, back from that simple modeling paper, which I used as
an illustration earlier.
Don't worry about what the actual quantities are I'm talking about here.
Each dot on this diagram corresponds to one run of the simple climate model, so height
shows the relative likelihood of the goodness of fit to observations.
So the high ones fit the observations well.
This is a sort of standard statistical technique called likelihood profiling, which allows
you to work out basically what's the range that's consistent with the observations at
a given confidence level and, crucially, is not sensitive to how you sample parameters.
You get the same result with this approach, whether you have a lot of models down here
or only a few, because only all that matters is essentially the outline of the distribution
here.
So finding that outline requires you to run, potentially, a very large ensemble.
There are clever ways around that, but we can come back to that at the end in the question
times if you want to.
And the problem is, if you need, you know, there are many thousands of points on this
diagram here.
And so if we want to try and do that with a GCM, we potentially want to run our climate
models many thousands of times.
And this has profound implications for climate prediction system design.
We've been trying to do this in Oxford.
This is under the climateprediction.net project.
In particular, we've been trying to do this under something which was called the BBC Climate
Change Experiment, which has been running for a few years now in Oxford, where we essentially
invited members of the public to download versions of the MetOffice climate model, essentially
the same one that was used in the last IPCC assessment with a slightly lower resolution
ocean.
And this is the sort of basic experimental design you see in front of you here.
The crucial point was that we actually got people to do these simulations for us on personal
computers, which meant that the ensemble size we were able to do was essentially, well,
compared to standard modeling groups, was essentially unlimited.
So far we've done about 23,000 160-year model simulations, and, you know, they keep trickling
in.
Here are the first 500 runs, so this is the figure I showed you earlier, which shows you
global temperatures from the first 500 members of that ensemble compared to observed temperatures
over the past 50 years.
Now what's striking about this figure, okay, clearly we're having some outlier models here.
When you're running models on people's home computers, all sorts of strange things happen,
which you have to be very careful about interpreting.
When people sort of, obviously the software, the guys who designed this software, it was
a very substantial challenge to basically port this model so that it could be run on
somebody's home computer, where it didn't really matter whether they unplugged the computer
or switched to playing video games or whatever halfway through the run.
You know, it had to tidy itself away in a nice, robust manner and so on.
So it was a considerable software engineering achievement to make this figure possible,
but the computing you're seeing here is essentially gigaflop scale computers, each one of these
lines being produced on a gigaflop scale personal computer.
We then evaluate what we call this likelier profile, basically the goodness of fit to
observations.
You don't particularly need to worry about the details here, but here's some results,
some of the early results from this experiment, and so here now in the vertical it's how far
away we are from observations, okay?
Those red dots there, those are the CMIP ensemble, that ensemble of runs from many modeling groups
around the world, and you can see that the spread of the so-called likely range assessed
by the IPCC, as I pointed out to you, is much larger than the spread of the red dots.
So those IPCC experts sitting around scratching their heads in Norway in 2006, I suppose it
was, assessed that there was much more uncertainty than suggested by those red dots.
In fact, if we look at the corresponding likely threshold applied to the climateprediction.net
ensemble, you can see that the range below the likely threshold is actually rather disturbingly
similar to the range assessed by the IPCC experts in Norway.
So the IPCC experts perhaps did very well, and I can assure you that this was the first
version of this figure, so I haven't sort of fiddled around with the figure to make the
two ranges agree.
There are some interesting things going on up here, and we'd like to know what those
models mean over there, but the crucial point for supercomputing 09 is that each of the
dots on this figure represents a 160-year climate simulation with a general circulation
model.
Now if getting out this kind of information on the range of uncertainty requires you to
run this sort of size of ensemble, then it makes a profound difference to the kind of
computing resources we need in order to address the climate challenges of tomorrow.
I've stressed that the most important source of uncertainty in these long-term climate
objections is model error.
Forecasts based on just a single climate model are largely useless unless you have some range
of uncertainty, some way of estimating that model error term.
There's a lot of debate about how you can estimate model error, but if you're going
to do it directly from full complexity models, you need either to run 100-member ensembles
if you can somehow solve the parameter sampling problem or 1,000-plus member ensembles if
you can't.
These are results I've shown you from using distributed computing, using results from
people's personal computers.
You may be wondering, well, you know, is this actually a technology that could be applied
to the more advanced resolution models that Joachim was describing?
Well, we have some other projects going on, and here's one of the projects we're running
which uses an N144.
That's about equivalent to T159 resolution version of this atmospheric model.
And as you can see here, it's not as good, this is a zoom-in on Europe, you can see sort
of written there, okay?
So you can see that that's sort of observations after fashion, and that's the model simulation.
So you can see that although we haven't got quite the filamentary detail of the figures
Joachim was showing you in Julia's talk, you know, we are capturing real weather in this
kind of model, and again, this is being done in a distributed computing environment.
So here's the potential.
Modeling groups have been requested to perform of the order of 5,000 simulated model years
to contribute to the IPCC fifth assessment.
This is the approximate throughput of the climateprediction.net global distributed computing community
every day.
Okay, so that's what we can do.
And we are a very small component of what's called the Berkeley Open Infrastructure for
Network Computing Community.
This is something set up by David Anderson at Berkeley, built on the old SETI at Home
project, and that's basically how the Berkeley, the so-called Boink community is currently
spending its CPU, okay?
They're still spending a third of it looking for little green men.
They're spending about another third looking for gravity waves and interesting astronomical
phenomena, and only about less than 4% is actually being devoted to climateprediction.net.
But that 4% of this petascale computing facility is already enough to give us a throughput
in terms of model years much higher than most national modeling centers.
It's worth saying that the reason we're only 4% is because we can't come up with experiments
fast enough to keep this, for the public to actually keep the public any busier than that.
We wouldn't want to be bigger than 4% if it was just University of Oxford here, because
we wouldn't have enough experiments for people to do.
Okay, that's to give you an idea.
That's the sort of throughput of that Boink community at the moment.
It sort of averages out at a roughly 200 teraflop capability at the moment.
So those are my conclusions, and the key one is that climate modeling is traditionally
focused on showing what is possible in the future.
For example, famous result coming out of the Hadley Center, the Amazon rainforest could
die out by the middle of this century.
To really pay our way, we need to start showing people not only what is possible, but we need
to start telling people with confidence certain things are not possible.
For example, within the next decade, will we be able to rule out sudden Amazon dieback
in the middle of this century?
That would be a really, really useful result, because we'd be able to tell policymakers,
no, this is not going to happen.
Helping them with things that might happen was useful up to a point, but to really pay
our way, we need to be able to credibly reassure people, no, the thermohaline circulation is
not going to shut down next week.
Joachim does a good job of reassuring people on this one, but we need to build up the range
of things we can reassure people that they're not going to happen.
This means addressing the problem of model error and the challenge I would like faced
by the climate modelling community is essentially rethinking how we set about climate modelling.
Typically, climate models are designed to be run perhaps ten times on the computing
architectures available.
The kind of experiments I'm talking about, which really feed through into policy because
they give uncertainty ranges on our climate forecasts, require models to be run hundreds
to thousands, possibly tens of thousands of times.
That requires a fundamentally different computing architecture.
It doesn't mean we don't need a global climate modelling effort, and perhaps I could easily
digress on why we still need a global climate modelling effort, because you need coordination
in the science, you need coordination in the way these modelling experiments are performed,
but you don't need to focus it all on a single petascale computer.
On the other hand, in this sort of question, I was thinking about these questions as I
came over.
Perhaps the ideal would be a million gigaflop computers, a thousand dedicated teraflop computers,
and because you can get that million gigaflop computers for free, as we've demonstrated
or as the boink community has demonstrated, you can maybe still buy one petaflop computer,
which would be quite a useful adjunct to these distributed networks I'm describing.
Thank you.
Miles, thanks a lot for this interesting talk.
I'm taking one question since Hans Moyer is not in the room.
Anybody?
Go ahead.
Hans, over there, sorry, up to you.
Not sure I totally understood your argument.
You all seem to make the more frequentist argument, at least from the plots that you
showed, that you have to run a lot of low resolution models a lot of time, but that's
not how we normally proceed in science, and I mean science – I don't know much about
client science, but I know a lot about physics and condensed matter physics.
Here I think usually the Bayesian approach where we try to put invest more into understanding
models and run measurements fewer times and try to infer information usually works better.
So are you saying the climate science problem falls clearly on the frequentist side, and
that seems to me an argumentation that they shall view all over again?
The experience we've had is that efforts, regardless of whether you take a Bayesian
or a frequentist approach, you end up – in order to infer uncertainties directly from
full complexity climate models, you end up needing ensembles in the hundreds.
So nobody has come up with a way of inferring an uncertainty range about a single run of
a climate model. Now there might be a way of doing this. I'm sure Jochen would happily
give you a talk on what are called adjoint methods, where you actually look at essentially
how the goodness of fit to observations might vary as you move away from your best guess
model, but these have not been applied successfully in the climate regime, and there's some theoretical
arguments about whether or not they even could be, and they're also very expensive in themselves,
so it's not clear whether that would actually be a gain. So, you know, at the moment what
happens is people run the big model and they get a sort of best estimate of what will happen,
and then they run a much, much closer resolution model, much closer even than the runs which
I was showing you here. They run like an energy balance model to actually get the range
of uncertainty, and that's why I was saying the irony in the IPCC 2007 report was that
the headline numbers came out of a laughably simple process compared to these best guess
calculations based on the supercomputers, and that's the gap that we need to address.
We need to work out how we can design our supercomputing base to our advanced GCM climate
modeling experiments so that they come with a range of uncertainty attached to them, rather
than just giving us a best guess for the future, which on its own doesn't help.
Okay, thanks a lot again. Miles, I have to give you a gift by Hans Moijer, because you
will not be on the panel in the afternoon, unfortunately.