So, thank you. My name is Thomas Marshni. I'm from Partek. I'm working in the software development
group and today I will talk about cluster reliability and resiliency and highlight the
tools of our software stack that are dealing with this. So, what does Partek do and offer
to the customers? We do consulting. We add the customer in setting up his system and
maintaining his system. That includes software maintenance, proactive system monitoring,
and cluster management. And we are also involved in on-size system operations. That means that
in case of any failures, we have to do event management. That means local trouble ticket
management and escalate issues to the hardware windows, for example. So, this is what our
software stack looks like. In order to set up a complete cluster, you start with a standard
Linux distribution. That could be Suzy, Red Hat, or one of the Red Hat clones. We have very powerful
tools for setting up the system for system provisioning. If the customer wants, we can
add a parallel file system and a batch queueing system. We provide our own MPI together with
process management and communication library. And we have a tool named Grid Monitor for looking at
vital parameters of the cluster, like temperature values, for example, or load values on the nodes.
But today in my talk, I will highlight the last two parts of that software stack. This is the
health checker and the ticket suite, because these are the parts that are related to cluster
reliability, especially. So, what is the motivation for my talk? Cluster reliability. The clusters
today and on the way to exascale are made of an increasing number of components that are of
decreasing size and thus of decreasing reliability. So, that means any component you have in your
cluster might have a very large meantime between failures. But if you put them all together, the
meantime between failures for your complete system can drop dramatically. So, if you have 100 nodes,
you have one failure a week, which is certainly okay. But if you have 10,000 or even more nodes,
then you have one failure or any type of failure an hour or even more frequently.
And you not only have hardware, but you have also software components. And the critical issue here
is that components may not only fail completely, but they may also degrade. That is, your job will
run, but it will run at a degraded performance or it will abort later. And you must ensure that
this never happens. So, what are the requirements? We start with scalable and tested software,
like our communication software. We want to have a full torrent software infrastructure.
That is something that has to be done in the next years. So, for example, full torrents MPI
is one of the key words there. We want to have redundancy. Redundancy would be best if done in
hardware. If you have every hardware component twice, you have perfect redundancy, but no one
can afford that. So, you want to have redundancy in software. If there are any problems,
you won't still have the rest of your cluster running. And, of course, you need tools for
automated error detection and error handling. After all, you also need experienced people that
have very large experience in cluster management. We gained a lot of experience with the Juropa
cluster. You certainly know. It has been set up two years ago, consists of 3,000 nodes
with 26,000 cores and a peak performance of 308 teraflops. And this system is special in that it
consists of two very similar parts of almost the same size and very similar hardware components,
but from different windows. We are running Linux on that with power station, cluster file system,
and more batch system. And the interesting thing is that we reach an aggregated utilization of about
90% outside maintenance slots. And how can we achieve that? We are doing that with the Health
Checker, our most important tool for ensuring the healthiness of a cluster. It ensures that
each node and thus the complete system is ready to run a job. So this is a test framework that is
run at certain points, for example, triggered by the batch system and prologue or epilogue time
and tests certain parameters of each node that might be hardware parameters or software parameters.
Also, the checks run periodically. For example, infinite band counters are checked periodically.
And whenever something happens, we create trouble tickets. So the issues are not lost.
We can send emails or SMS. We lock all findings. The tickets that are generated are collected in a
ticket suite. So this is very important. I told you that if you have one issue an hour or later,
even more frequently, you have to be absolutely sure that you are not losing any of these issues.
And that's why you have to organize that in a ticket suite. And you have to also
automize as much as possible. We choose track for this. This is a very popular open source
system for doing trouble management and other things, but especially trouble management
and issue tracking. And we enhance that with our own plugins. So to interface with our own tools,
namely the health checker and the infinite band checker, and also to interface with what the
windows may have as support tools. Here is the life cycle of such a problem, such an issue.
Our health checker detects a problem. And then this creates a ticket in our ticket system.
The vendor gets notified. The vendor reports a fix that causes a ticket change.
And then we have an automated tool that checks the note if the problems are really vanished.
And if the note is okay, the ticket is closed. The note is set online again in the batch system.
And, yeah, it's ready to run jobs. If not, we are in that loop with the vendor again.
So this is the key idea behind that is that as much as possible is automated. We have only one
point, and that is this point here, the ticket creation. After the ticket is created, there is
one manual step. Some of the support engineers have to have a look at the ticket and say, yes,
it's okay or not. And anything else is run automatically. So we automize as much as
possible in order to have a high throughput of jobs. Also, we can generate statistics.
This is a very important tool. If you have a large amount of tickets in your database,
you can certainly make some data mining. And we have created a plugin to the track system
to support that. You can see open tickets, ticket open or close rate, ticket residence time,
time to action, and more parameters. And you can view graphs of these statistics online
within your browser. You can compare multiple graphs for certain components. You can choose
time ranges here. You can export the data into CSV for external processing. And all that is done
on the fly. It's very user friendly. No preprocessing or configuration is needed for that.
So our experiences within the two years of Europa are that those statistics, for example,
are very good means. They provide us with objective data about the system state and
allows us to better analyze failure courses, because you can easily see which components
are the ones that are not working perfectly. We have the special situation that at Europa,
we can compare stability and reliability of the components of the different windows
that are otherwise very similar. You have even such cases where you can see
changes in vendor policies, for example, the transition from Sun to Oracle.
Initially, we had many hardware problems, especially disk and memory. The health
checker was doing a good job and created many tickets. That sounds bad, but was very good,
because we learned about the hardware problems. Nowadays, we are at a state where all procedures
are very automated, very well known to all parties. And we can say, and that is my conclusion,
that we have reached an efficient workflow by means of the power station ticket suite.
Thank you for listening. Thank you, Thomas.
First question is Frank Beren. Yeah. Thank you, Thomas, for the presentation. I'm kind of puzzled.
Title of the talk is Resiliency on the Way to Exoscale. And essentially,
you'll talk about a ticket system implemented on today's system, which is far from exoscale.
Yes. You didn't mention by word how you plan or how you expect to upgrade, to develop,
to scale your system from the challenges your software is facing with today's hardened software
combination up to three orders of magnitude further on, you will face an exoscale. So
essentially, the message I take from your talk, there will be not much to be changed. Everything
is cool. Everything is fine. So you may understand that I remain a little bit puzzled here.
Okay. So, of course, in order to prepare for exoscale, we must deal with the amount of tickets
that flow in from today's system. And if we have the tools right today to deal with the amount of
tickets that might not even be that low as we wish it should be on the Drupal system, then we
can further develop our tools. And we have certainly projects in the queue, for example,
for dealing with cases where many, many different events caused by the same failure arrive at short
time. So for doing filtering and aggregation, that is what is currently ongoing. So, of course,
we are dealing with the scaling issue. But first of all, we have to make very sure that we can deal
with what the systems of today present us. Frank Pengling.
Thank you for the presentation. It actually has similar confusion as Frank. So you didn't
mention what's the risk when you grow three order of magnitudes from PETA to EXA. But maybe I shift
to another question. Maybe you can take one of your most competitive competitor to tell me why
you have a better advantage to shift prior to your competitor to exoscale. What's the advantage
that you have? So the question was, what is the advantage? Why can PowerTag or the Power Station
software deal better with the errors, if I get the question right? I think for PowerTag,
it's the complete solution. We have experienced people. We have gained a lot of experience. We
have the tools in place now. We are very confident that we can actually scale because it is so
working well today. We are very, very confident that it's possible to scale within the next years.
Thank you, Thomas.