So, good morning, everybody.
So, my name is Cyril Mazorik, and I'm coming from France.
So, sorry for my accent, of course.
You love your accent.
Thank you, very nice.
So, I'm coming with Pascal and the Physnik.
We are working in a build company.
The build company is a company which delivers a cluster.
And the new name of this company is ATOS.
It is why you can see ATOS on the slide where.
So, we are here to present you the build environment
and mainly the MPI1, the Bollix MPI.
So, this should be the agenda.
Yes, this is the agenda of this morning,
and I think this afternoon too.
So, I'm going to, I'm beginning to present you the MPI environment,
the Bollix MPI and the Milanox tools.
And I think that this afternoon we will make a feedback
of the benchmark phase of the deal.
And we will present you some best practices
to use the Bollix MPI and the bull environment.
So, for the MPI environment,
you will find on Mistral two Bollix MPI version.
The first one, Bollix MPI underscore Milanox,
is based on an open MPI 1.6, so an old one,
compiled with Milanox tools.
Milanox tools are MXM and FCA.
And I will present you these tools later.
You will find also a second version of the Bollix MPI,
which is Bollix MPI Milanox underscore MT.
It is exactly the same as the previous one,
but we compile it with the thread multiple support.
Maybe some of them will need to use this version
just to work with the thread multiple support.
How to use the Bollix MPI?
It's very easy.
You have to load the Intel compiler,
the version that you want to use.
You have to load the MXM package, the FCA package,
and of course after the Bollix MPI package.
And you will be allowed to use the Bollix MPI.
MXM and FCA are a Milanox tool that I will present you.
This is the cells of SlideWare.
So the first package FCA is a package
which is working on the collective operation,
on the MPI collective operation.
I mean that this package will allow us to improve
some collective operation.
So if you have a lot of collective operation like MPI already used,
MPI barrier or something like that,
you can try to use FCA to improve your performance.
The second package and the main one for me is MXM.
MXM is a messaging accelerator.
So it's very coarse.
But of course this package will allow us to accelerate,
of course, your message.
And this package will allow us to increase the scalability
of your applications.
How OpenMPI is working?
The MPI layer will call several devices to use MXM,
to use FCA, to use a lot of tools.
By default, MPI will call the package CM to use MXM.
It is the default behavior.
And this behavior will allow us to use MXM
and to improve your performances.
This is variable that you can use to activate MXM,
but by default, these variables are set.
But maybe you want to use the default OpenMPI behavior,
which is OB11, to use default OpenMPI behavior.
You can activate these three variables.
You can use these three variables to use OpenIB device.
So you have the choice between OpenIB device,
which is the default OpenMPI behavior,
and MXM, which is build tuning,
which is much more performant in the measure case.
You have the choice to use several modules
to improve your collective operation.
BULL develops GHC module.
GHC module has been developed to optimize
the collective operation when you have some island
in your cluster, and when the printing factor
between island is very important.
So it is not the case on Mistral,
because on Mistral you have only one island.
So this one will not be used on Mistral.
But I want to show you.
So on Mistral you will use two main modules, FCA,
this one, which is a Milanox module,
and which has been developed to optimize collective operation,
and mainly all gave or reduced barrier and broadcast.
But you can also use the default OpenMPI module,
which is called Tune, and this is the default one.
So what is the impact of Milanox tools on your application,
on application given by DQRSID to BULL to answer to the deal?
For example, we worked on FISM.
The reference test is a test with an old BULLEX MPI.
Adding Milanox tools will allow us to increase the performance
by 6%.
And changing the version of the BULLEX MPI,
because of course we are working on this OpenMPI,
will allow us to increase the performance by 9%,
compared to the reference one.
So on Mistral, because you will work on this version,
the performance has been increased by 9% thanks to these new tools.
Cosmo, for example, we have a gain of 4%.
So 4% maybe for you is not very important,
but for us it is very, very important.
But the most interesting thing about these Milanox tools
is that both tools will increase the scalability of your application.
In blue, you have the curves without Milanox tools.
So you can see that with more than 3000 cores,
the scalability is very, very bad,
because the time execution increases when you increase the number of nodes.
With Milanox tools, the scalability is good.
It's not perfect, but the scalability is very nice.
And we are also working on EMAC,
and we can see that we have a gain of 4% thanks to these new tools.
This one?
Is the communication ratio.
I mean that 44% of the time execution is spent in MPI communication.
So you can see that for this application,
the ratio is very, very important.
Communication are very important,
so we need to work on the MPI stack to improve the performance.
And this is the ratio for the best test.
So of course, sorry, we are not working only on DKRZ.
We are working on a lot of deal, of course.
So maybe you know this company in England, Metophis.
I had this slide because it is very interesting.
The question asked by the Metophis is to run UM, unified model.
And the question is, how many nodes I need to use
to reach the target of 1,500 second?
Without Milanox tools, we need to use 500 nodes.
Thanks to these new tools, we saved 100 nodes
to reach exactly the same target.
So it is very interesting to use this kind of tool.
So do we have time to begin this part?
Yes?
Okay, thank you.
Sorry, it's me again.
So I will speak about 15 minutes
on the best practices to use Mistral.
When you are looking for the best performance,
best possible performance on the cluster,
sometimes on cluster you have to decide
which node list you have to use.
It is not the case on Mistral.
We did not speak about interconnect topology,
but the interconnect topology on Mistral is a fat tree.
It means that if you are running,
you have not to choose the node list to use.
In this case, we use Cosmo on 31 nodes,
and we submit 40 occurrences of this Cosmo on the cluster
in order to use all the node, all the cluster.
Almost, we did not use all the node, but almost.
And as you can see, the deviation is very low,
less than 4% I think.
So it means that...
Identical job.
Identical job, running simultaneously,
doing IO in the same time,
doing everything at the same time,
and you can see that deviation is very, very low.
So first of all, it is not necessary to choose your node list.
This is the good news.
Sorry, if I ask again,
to my experience with this cluster,
and you used also a cluster system for the IO.
Yes, yes.
That's very important.
This test case did not generate a lot of IO,
of course, with a big file.
But 40 occurrences in the same time,
it's several gigabytes.
So now we will try to tune the MPI part of your application.
So first of all is to know where I can find the variable
that I have to use to make the tuning.
So once ambient mount is correctly loaded,
you can try to tune MXM
and to find the correct variable to use,
the list of all available variables,
you can use this command.
And you will obtain all the variables that you can use
and the definition of this variable,
a short definition.
If you don't want to use MXM,
you will use BTL, the second device,
and to know the list of the variable,
you can use this command.
To deactivate MXM,
you should use the command below,
and you can use this command to list all variables.
In the same way,
if you want to make the tuning of the collective operation,
you can want to use FCA,
and the list of variables will be given by this command,
which is exactly the same on BTL.
And if FCA is not enough efficient,
you will go back to the tune module,
the default open MPI one,
and to know all variables used to tune collective operation,
you have to use this command.
For us, and thanks to the benchmark we ran on Mistral,
a good default environment could be this one.
First, you have to load the environment, of course,
and for us, you have to use MXM
to improve your scalability and your performance.
Then, you have to deactivate GHC,
because GHC is a module developed by BUL,
but which is performed on a different type of cluster than this one.
So, to deactivate GHC, you have to use this command.
And finally,
you have just to specify the correct and final bound device.
It is not necessary because open MPI will find the good device,
but specifying this setting will allow us to increase the efficiency of open MPI.
So, I think that this is a default environment that I gave you,
and maybe this is something that you will set by default somewhere.
I don't know.
So, for us, once again, it is the best default environment.
It is what we used for all applications of DKRC.
With this environment, you will use MXM, of course, and tune for collectives.
If you want to use FCA, you have to activate it with the correct setting.
So, now, if you want to tune to improve the MPI part of your application,
you have to know the profiling, of course.
So, maybe your application will give you the MPI profiling of your application,
or you can use an external tool which allows to make this profiling automatically.
Usually, the information given by this kind of tool is the ratio of communication,
the part of the time execution which is spent in MPI communication,
and for each MPI function, the part of this function on the communication time.
In this case, you can see that 8% of the time execution is spent in communication,
and 55% of this 8% are used by MPI or reduce.
So, we have to tune this function.
So, for some example, usually, you will see that the MPI weight is the hotspot of your application.
Maybe it is logical. I don't know.
It is something that you have to know.
Maybe it comes from a bad synchronization.
How can I change this behavior?
Bad synchronization means that the communication between tasks is not good.
So, maybe you have to change the MPI task banding to move something and to improve the performance,
to decrease this MPI ratio, and to allow us to re-synchronize all your MPI tasks.
So, as Andrew showed you, there is some option given with S-RAN,
which allows you to move automatically the binding.
So, by default, you will fill your nodes.
If you are using minus MCLIC, you will put your first MPI task on the first node,
the second MPI task on the second node, and so on.
And cyclic cyclic cyclic is to make a round robin enter an entry node.
Maybe messages are waiting because the message size is too small.
So, we spoke to this subject with someone.
So, in this case, the bottleneck is the latency of your interconnect.
I mean that each time you want to send a message, it will have a cost in terms of times.
So, if you are able to pack, to merge, to aggregate all your messages,
you will pay just one latency, but you will use all the interconnect bandwidth.
So, it is better when you can, it is not possible, of course, you can aggregate all the messages.
Unfortunately, there is nothing to do with MPI tuning itself.
It is difficult to tune your MPI stacks to improve this behavior.
There is something else to do.
Sometimes your spot is MPI barrier.
In the major part of the case, the MPI barrier is used only by developers
to create a clean output file with a good order and so on.
Maybe if you want to work on the performance of your application,
it will be a good idea to remove the barrier.
The output file, maybe the output file will not be very clean,
but sometimes it increases the time execution.
So, you can command all the barriers in the code, of course,
or you can create a library which catches the call of the MPI barrier and replace it by nothing.
So, it is a way to remove your barrier.
And sometimes it could have a huge impact on the performance.
And what happens when your bottleneck is a collective operation?
I took the example of the whole radius, but of course, it could be all two or something else.
The first thing to do for me is to test FCA, the Mellanox tool.
So, first of all, you have to activate FCA with the setting I gave you in the previous slide,
and check if FCA is activated for this operation.
I mean that with the command open MPM4 minus minus all,
you will find a variable call like collective FCA enable or reduce.
And if this variable is set to one, it means that this one is activated
and FCA will try to improve the performance of this function.
This is an extract of the output of this command.
And as you can see, for all reduce function or barrier, FCA is the best one.
I mean that if we compare FCA with tune or GHC, FCA in blue gives you the best performance
for an IMB benchmark.
IMB benchmark is a benchmark which makes only a communication
just to check the performance of your function.
But of course, IMB is not the real life because you will not use IMB,
but you will use your own application.
So maybe FCA will not be efficient on this function.
So the other possibility is to use tune, the default open MPI module.
To use tune, sorry, tune will provide you different algorithm
to optimize each MPI function, MPI collective.
For example, for all reduce, you will find this variable count.
This variable will give you the number of available algorithm that you can try to use.
And after you have just to use like that, call tune algorithm, call tune or reduce algorithm,
and you have to test the five possibilities, the five options.
And maybe one of these options will have a good impact on your application.
Just another information, when you will find a variable with,
beginning with OMP underscore MCA, it is an open MPI variable.
When you have a variable beginning with I underscore MPI,
it is a mental MPI option.
Just an information.
That's all.
If you have some question.
Yeah.
By default, there is no tuning.
Because by default, on Mistral, there is no environment which is load.
You have to decide yourself which environment you want to use.
My advice will be to use Intel MPI or Boolix MPI for small number of nodes.
When you try to use more than 100 and 200 nodes,
Boolix MPI will be better, I think.
It is not always the case, but it is what we saw.
Thomas is working on ICON, and I saw that he is using Intel MPI on 900 nodes,
on 500 nodes, and it's working well.
There is no problem.
So.
It could be fine, yes.
For me, you have to feel good with an MPI.
And if you like to work with Intel MPI, you can work with it.
It's not a problem.
Thank you.