Good afternoon, sir, these days I see that most people are speaking about exascale, so
I don't know, maybe everybody knows how to build 10-petal float machine, I'm not sure.
Anyway I will speak about some things that are behind our upcoming system that we hope
to, will be introduced next year, and I will speak how we think it is possible to build
10-petal float with standard architecture and mostly standard hardware.
First a bit background, so right now we have 1.3-peta-flop system in Moscow State University,
and eventually this system was upgraded two times, so it was first introduced in 2009
with 420 tera-flop peak performance.
Right now this system is hybrid and it is running both CPUs and GPUs, but right now
we'll speak mostly about the system based just on the CPUs.
First let's do some calculations, so having the dual CPU node performance right this year,
about 300 giga-flops, and in 2012 it will be probably about 370 giga-flops, so we need
about 32k nodes just to build the system.
32k nodes is a pretty big amount of CPUs and we need a very high node density to build the
system in a limited space, because of course if you need more space, you need longer cables,
it will be much more difficult to install the systems.
So let's assume that we will have 256 nodes in one cabinet, so we need 128 cabinets just
to build the system, and the power consumption will be around 100 kilowatts, so based on
what we have right now in our current products, we assume that we will use a reg-based solution
with internal high voltage DC power infrastructure, and we are planning to divide the system power
in several power zones to provide more redundancy.
On the cooling side, cooling more than 80 kilowatts per rec is also not an easy solution.
We are able to cool up to 70 kilowatts just using air cooling, and you can see the system
on our booth, but we think that with increasing compute density, anyway we need water cooling,
and moreover hot water cooling just because we also need to get better power usage efficiency.
With hot water we will have several other problems, like the problem with thermal isolation
of the cabinet, because we need to prevent the heat dissipation into the data center.
We need to think that everything should be water-cooled, nodes, switches, power supplies,
and also we need to think that all the components on the boards should be cooled properly, because
with 50 degrees of water temperature, the air temperature between the heat sink and the system
board will be around 60 or maybe even 70 degrees. What we have right now, we have some working
prototype with liquid cooling, and we also think how to increase the system reliability,
because if you have 32K nodes, and for example if you have water connectors per each node,
so then you need 64K, just water connectors. The probability of leakage in this situation
is very high. What we are using in the prototype, we aggregate their cold plates, and we don't
use water connectors for each compute node. In the future system, we will implement something
similar, the technology will be different, but still it will be based on what we have right now.
For the networks right now, we are evaluating the possible network topologies, because if we are
using infinite bands, so most of the systems are built using FET3 topology, but if we will use
FET3 topology with full sectional bandwidth, so the cost just of the network will be very high.
Right now, we are looking into some modern topologies, like flattened butterfly, it will
decrease the cost of the network about 50%, just a rough estimation based on current prices for
the sweet chips and cables. Also, it's possible to use the torus topology, it will be even more
cheaper, but it will have high latencies compared with butterfly. We will use a second auxiliary
network for the data storage access and just for the remote access to the nodes. And for the
auxiliary network, we think to use just infinite band with oversubscribed FET3 and probably with
oversubscription ratio, something like 8 to 1. And the last network is just out of band management
needed to control the overall system. Of course, building the multi-petaflop solution, it's not
only to build the hardware, and we are working a lot on the software side as well. So what we have
right now in house, we have Clustrix meta-operating system. So basically, it's Linux, but with a very
scalable management and monitoring software, which is capable to run on the system with,
say, up to hundreds of thousands of nodes. Also, we have a low jitter Linux kernel as a part of
the Clustrix, but we may use some standard Linux kernel as well. For the custom kernel, we have
improved memory management, and we are working further in this direction.
For the Clustrix, it runs right now on Moscow State University system, and Clustrix is capable to
get the monitoring data about 300 metrics per one node per second. So it's near real-time
system monitoring, including everything, not only the compute nodes, but also
all communication equipment, infrastructure, etc. And for the programming support, right now we have
custom library called Dsleeve, which is something like distributed shared memory library,
and we have pretty good results on the Moscow State University system, for example, running
the Graph 500 benchmark. I believe the results will be announced soon. So basically, probably
that's all I want to say right now, and you can just come to our booth and discuss
the things you can look at our systems as well. Thank you very much.
Many thanks, Mr. Lipouin. The first question is now coming from HPC Tully.
Thank you, Andre. I would take that one of the ambitions of T platform is to penetrate the market
on HPC outside Russia. And I would be interested to hear how efficient is your design,
your cooling design at that hot water cooling of 50 degrees. Have you considered if you go
outside in other conditions, how are you going to keep that efficiency and consider the local
environment? Yeah. So actually, having high water cooling, hot water cooling design,
we solve two problems. The first one is to improve the power usage efficiency by using
free cooling almost all the time across the air. So with 50 degrees, it's enough
to cool the system efficiently, say in Russia or in Europe. Okay, I'm not sure in South Africa
because what kind of temperature do you have? Is 50 degrees enough?
And the second goal we need to solve is just to provide the possibility to utilize the heat.
And we also have some ideas on the data center level, how to get energy from
dissipated heat in the system. Satoshi, please for the second question.
Yeah, this is a similar question. So if you're HP or IBM or companies with very large volume,
then it's fairly easy to leverage the volume parts of business, which is like the cloud,
to use that volume to actually decrease cost of your system when you even build,
when you semi-customize for HPC. And in addition, companies like IBM and HP,
they are working on very similar technologies to yours, like hot water cooling that's being done
with a large Z. HP just demonstrated they have very efficient air cooling, free cooling solution.
So what would T platform not having advantage of extreme volume,
what would be your strategy? Because smaller companies like Cray are investing in their
proprietary technologies to make a difference in the market. So what would be your strategy?
What will your investment be to combat the large companies like IBM and HP in a very similar space?
Okay, so our strategy, first of all, we are relatively small company, and we are much more
flexible than big companies like HP or IBM. And this flexibility allows us to reduce the
cost significantly. We don't spend, okay, hundreds millions dollars for our R&D.
I will not speak about the exact numbers, but the R&D cost, for example, just for our
T-blade II system used for the Moscow State University, about an order of magnitude
less than a similar system from other vendors. And we are investing a lot, we are still investing
a lot, and we have some plans on future developments, not only, okay, on the system level,
on board level, we are thinking about some cheap design in the future, but it's okay,
not for next year, maybe in three years' perspective.