And let me introduce Professor Nakamura from the University of Tokyo.
He is really one of the leading experts in modeling, physical modeling of bodies.
He'll talk about that in a moment.
He received his PhD from Kyoto University in 1985.
And then he was everywhere.
He was in Santa Barbara.
He had some sabbaticals.
And then he went on to become a professor
at the Department of Mechano Informatics at the University of Tokyo,
which is a very unique thing.
I would not be aware of any other department in mechano informatics.
But University of Tokyo has been very successful
in setting this up and running this.
And of course also has been able to support the professors
in very, very interesting areas.
We have some of the professors here.
Today it's Professor Nakamura.
And he will report on supercomputing
in the loop of biomechanics and neuroscience.
So please let us welcome Professor Nakamura from Tokyo.
Thank you.
I have some trouble.
Well, it worked before.
You have a thousand witnesses, unfortunately.
Okay, now.
Don't do it now. Do it later.
All right.
Thank you very much, Alois.
And thank you for gathering in the afternoon of the third day of IOS.
And I hope you are relaxed already
and allow me to talk about one of our exciting,
but ongoing and incomplete work.
So I hope this is the opportunity to start a good discussion
about the new challenge in robotics
or further work to go.
The title of my talk is big,
but the content is, as I said, incomplete and ongoing.
Well, first of all, I have to express my thanks
to all the colleagues that contributed to this particular work.
They spend a lot of time and days for this work.
And next, I also have to thank the funding resource.
The mix provided us this opportunity.
And I am a member of the professor Takagi's team of this project.
The project is very simple.
Japan has the supercomputer.
This used to be the number one supercomputer of the world,
but it's now probably fourth or fifth.
And the Ministry of Education is helping
and emphasizing to develop scientific use of this resource.
So we are one of the many groups
developing the particular application using this supercomputer.
Well, our goal is to simulate the life of post-doc.
I'm just kidding.
Simulate the Parkinson disease.
This is our goal.
And Parkinson disease is well known
that about 1% of people over age 60
are having the risk of Parkinson disease.
And these are the pictures and photos from the very early books
around 130 years ago.
But already in the Egyptian populace or even in the Bible
the Parkinson disease was mentioned.
So it's a very long historical disease for human beings.
Well, of course, this is a disease of the blame
and the symptoms are explained in many ways.
There are several particular symptoms
like tremor or slow movement
which is known as bloody kinesia
and rigid muscles
and the impaired posture and balance.
These are the typical symptoms.
And the progression is explained.
The symptoms start from the problem of the stem of the brain
and then going to the basal ganglia
and then split all over the cortex.
So that means that the first symptom based on this paper
said that it starts from olfactory problem
and then develop little by little.
And of course, because this split all over the cortex
the symptom is very complex in that sense.
And this is the typical explanation
and this is a basal ganglia.
And the problem is that in what is named subtenia nigra
that is around here,
there is the cells, neurons, that create dopamine.
And that cell may create what is known as labey body.
This includes some particular proteins.
Then there is a problem, there will be a problem.
And dopamine, if dopamine is not created,
this brain circuit has a problem.
Actually, this magenta line from SNC,
this is subtenia nigra reticular atom.
From here to stole atom, this is the dopamine pass.
This pass will not work if Parkinson's disease starts.
So that caused many problems
due to the network and feedback systems.
And this is the particular pathway of the Parkinson's disease.
There are many other explanations and pathways,
but this is typical pathways.
And our team has different expertise.
Professor Takagi, he is a PI.
He is the expert of computational mechanics
and his group simulates the molecular level of muscle activities
and finite element modeling.
And Professor Doya of Okinawa Institute of Science and Technology
is a well-known neuroscientist.
And his group, before this work,
already tested the computer simulation of the neural system
that has as many neurons as the mouse brain.
In this group, our expertise is of course robotics
and we have algorithms and software
for simulating complex systems,
in particular the human muscular skeletal system.
And in our group, we have the expert of the Parkinson's disease.
He is from Osaka University.
So we have connected already existing software environment.
For example, Nest is developed by the next speaker in this session,
the Mark Oliver Gebancic developed the software
and we are happy to use it.
And Music is a software connecting C or C++ with Nest.
And the finite element software is developed in Fortran
and we wrote this algorithm and connected it.
So in total, we have the finite element muscle modeling
and the complex muscular skeletal network.
And the software for modeling neurons.
And we have been working on making the model
of neuromuscular skeletal system,
including the Bayesian ganglia
and some small portion of cortex.
Actually, the people in Okinawa
have been working on this blame part
and we have been working on the spine
and the muscular skeletal systems.
Okay, so there are few components we developed.
This is quite robotics.
And I just want to explain how the robotics
can contribute to these kinds of computations.
Many years ago, Professor Yamane,
he is now in Disney,
developed the very efficient algorithms of dynamics.
And Professor Takagi's group
developed the efficient FEM simulation in Fortran.
And now we are combining them
and computing the muscular skeletal network.
And we have about 1,000 muscles
and each muscle is computed by finite element algorithms.
And skeletal system is computed at least for the systems.
So all of these are explained
by, of course, the equation of motion.
This is the mass FEM motion equation.
And we have boundary condition
like the attached point of the muscles
and the joint constraint.
So all of these dynamic equations
plus constraint equations are linearized
and solved by solving large scale linear equations.
This is how we approach this problem.
Okay, so since the system is complex,
I just want to go through very laughing.
Okay, so we have 1,000 muscles
and each muscle has FEM model.
That takes a lot of time.
So muscles are computed in parallel.
We have 1,000 muscles computed in parallel.
And this is the digital body part.
The skeletal system is digital body.
And the muscle motion of equation
and digital body equation constraint are all included.
So the muscles are computed in parallel
and then solved one by one
and then gathered to the skeletal system.
That is the digital body system.
And we solve the digital body motion
and then the result will be fed back to each muscles.
So this is how we solve the large system in parallel
using the supercomputer power.
And this is some simulation result.
And we can compute hybrid system
of many finite element bodies
plus digital body skeletal systems.
This is quite a challenging part.
Actually, the computational stability is still the problem.
We have to be very careful tuning the parameters.
So the headache is the contact and collisions.
So we have many muscles and bones.
So they are colliding with each other.
And we have to compute the point of collision
and the interaction force between the contact points.
There are many of them.
So this is a challenging part.
So we have developed this kind of hierarchical algorithms.
First, we check if the big box including a muscle
is colliding with another muscles.
And if they are colliding, then we check the small ones
one after another.
So in this way, starting from big rectangle,
we can go to the detailed polygon of the muscles.
And then we can go up to the top.
So in this way, we compute the collision and contact force
and then simulate the whole systems.
So these are the highly parallel geometric algorithms
that are quite common in robotics.
This is a nice contribution of robotics
to simulate a complex system like human-muscular skeletal system.
So this is a trivial simulation result.
Of course, it can be done.
And since they are highly parallel, the efficiency is high.
Okay, now we developed the whole-body finite element
muscular-skeletal system.
And the problem we had, we have a lot of problems,
but the particular problem we had
is that nobody knows the natural shape of the muscle.
We can find some anatomical pictures,
but those muscles are already pretensioned.
We don't know free muscle shape.
It is a very hard problem.
So we then decided not to consider the shape of muscle too much.
So assuming there is a symmetric shape like German sources,
we studied.
So we computed the collision and contact force.
So our German sources collide with each other
and made this kind of network at the end.
Of course, if we carefully designed the shape,
we can come to a much better non-ugly muscular-skeletal system.
But this is the finite element model.
That has the mass and the dense of the system.
Well, next we worked on the neural networks.
Of course, modeling the neuron is a critical part.
And we are learning from the literature
and try to gather the knowledge as much as possible
and try to extrapolate for those missing.
And this is the work of Manish Sunilabasa.
He is now in the universe of Heidelberg.
And he worked with us for two years and developed this work.
And his paper has just accepted to the I2EP transaction
on neural systems and rehabilitation engineering.
And this model is quite simple.
We have only one degree of freedom system.
Of elbow joint.
And elbow joint is controlled by bicep and tricep.
So two muscles and three muscles.
And each muscle, bicep or tricep,
typically has nearly 700 motor neurons,
or maybe 200 or 300 muscle spindles, sensory neurons.
And they are modeled.
And this blue line is showing the sensory signal from the bicep.
And this is the sensory neurons.
And then the sensory signal is going to motor neuron of bicep
and coming back.
This is the stretch reflex to the self muscles.
And same sensory signal goes to the interneurons
and then go to the tricep.
And this part inhibits the activation of tricep.
Of course, the activation of tricep
goes to the sensory neuron
and coming back to the tricep to the self muscles.
And in this pathway, we are just looking at the inhibition
from bicep to tricep, but not tricep to bicep.
Just for simplicity.
So naturally, we should have one.
So this is the sensory neuron, interneurons and motor neuron pools.
The motor neuron pools are inside the spines.
And the sensory neuron pools are just near the spine, but outside.
And their number is quite well known in the literature.
But of course, distribution is not known.
So we have just randomly distributed them in the space.
Well, there is some well-known mathematical assumption
that one motor neuron controls one motor unit.
One motor unit means one bundle of the muscle fibers.
And large motor unit controls large bundle of muscle fibers.
And the smaller the neurons control the small ones.
And the smaller one activates very fast.
And the larger one is being activated very slow.
So they have this scale phenomenon.
This is very important.
And the activity of neurons is modeled in several ways.
But we just picked up the quite simple but reasonable leaky
integral and fire model.
This is just leaky capacitance.
And it just gives the current coming to the neurons.
And if the voltage comes to some threshold, it sparks and fires.
This is how it works.
And the size of motor neurons has some rule of thumb.
So we picked up the rule.
And based on the size, the capacitance, main brain capacitance is determined.
And the time constant is determined.
So in this way, we try to minimize the number of the constant, unknown constant.
And of course, the synaptic connection is the main important part, of course.
But again, in order to minimize the parameters, we just used and assumed
the Gaussian distribution.
If the two neurons are close, the connectivity is high.
And two neurons are very apart.
The connectivity is low.
And this connection is presented by this Gaussian type distribution.
Well, this is a very interesting part.
And I like this part.
Actually, this is the firing input U.
This is 1, 1, 0.
If neuron fires, the neuron provides 1 to the next neuron.
If it doesn't fire, it's 0.
And this is a discrete type of second order integration.
And by integrating the neural activities, the A is determined.
And this A is the muscle activities.
And muscle activities are typically approximated by EMG signals.
So this way, the firing neuron, the spiking neuron, is gathered and integrated
and making the muscle activities.
And very interestingly, we are looking at one type of sensors.
There are two types of sensors, Gaussian Tendon Sensory Organ,
that they are inside the tendon.
But we are not considering this model.
We are only looking at the muscle spindle.
They are scattered inside the muscles.
And the muscle spindle's firing frequency is determined by these equations.
B is the speed of muscle contraction.
L is the length.
And this is the background frequency.
So the muscle spindle has firing by these frequencies.
So to me, this is something like A2D or D2A compartments.
So the digital signal, like the spiking neurons signal,
is integrated and becomes continuous muscle activities.
And the continuous velocity or length is going to be transformed into the frequency.
So this is digital analog or frequency transformation in a sense.
But we realize that this is a very important part in the case of Parkinson's disease,
as I already explained.
The tremor is a vibration.
So vibration is not A2D type of thing.
Its frequency appears in the continuous way.
So modeling the neurons activity by this spiking neuron,
we can approach the problem of vibrations.
This is a very important part.
Well, some constant of motor neurons are also modeled by this logarithmic function.
This is also from the literature.
So we are trying to learn this kind of knowledge.
Maybe many of you already know, but it is very exciting to gather some of them.
We are trying to get all of them, but it's very hard.
Well, this is the well known heel type muscle model.
Actually, this model is applied to each motor unit.
And the whole muscle force is the just simple mathematical sum of this force.
And the muscle force is linear to the muscle activities that is computed in the previous slide.
And FL and FB are some simple non-linear function of length and velocity.
And Fmax is just a constant.
FB is very simple Gaussian type function.
And FB is again simple sigmoid like function.
Well, Manish made this model.
And using the NEST, all the neurons and sensory motor neurons are modeled.
And he did two different type of experiments for five subjects and took the data.
And for the third experiment was used for verification.
The experiment was simple.
The experiment was bending the elbow angle 90 degrees and pulling down the wrist by the weight.
And of course, bicep has contractions.
And the second experiment was that there is no load,
but the core contraction is requested to the subject.
And then blindly some object was thrown from the top of the hand.
And there is some collision and vibration afterwards.
So gathering those signal, we can simulate the muscle activities based on our software.
And then identify by optimization some parameters that we have already defined.
Well, this is the verification result.
It's hard to explain and nothing too interesting.
This is a normal case.
And this is a case with 150% of gains, feedback gains.
So this is the extreme case, but this is easier to explain.
So at this time, at time, I think t equal to 0.2 seconds, the ball is dropped to the hand.
Then this is the muscle spindle of the tricep and this is the bicep motor neurons.
And this is the sensor neurons of bicep.
And first bicep is activated.
And bicep motor neurons are making this signal.
I mean, sensor neurons are making this signal.
And the tricep is also activated.
And this is the sensor signal from the tricep.
And then by this sensor neurons of the bicep, interneurons are activated.
And then interneurons inhibit the activity of the tricep.
In this way, the loop is working and there are some sensitive vibrations.
Because this is too high gain, this case has too high gain.
Okay, so I have five minutes or ten minutes.
Well, this is quite a new work of the Kazunari Takeichi.
He graduated the master course this spring.
And he did very extensive work on the modeling.
Well, Manish Sunidopasa developed the detailed neural network of bicep and tricep.
So we wanted to extend this idea to the whole body.
This is a huge work.
But we made best effort to get all the kind of available knowledge.
But we found that only 14 muscles are known about the number of motor neurons and number of muscle spindles.
And we have 1,000 muscles.
So there is no literature knowledge.
It was very difficult problems.
But we made this kind of drawing.
The horizontal line is the number of motor neuron over mass.
This is showing how many motor neuron per mass or kilogram.
And vertical axis is number of muscle spindle.
The sensory neurons of the muscle over total neurons means motor neuron plus muscle spindles.
That means the ratio of sensors among neurons.
The horizontal line is number of motor neuron per mass.
And we put the known 14 muscles in this drawing.
Then there are some interesting features we can see.
For example, knee or arm, they have larger rate of the number of muscle spindle.
They use more sensors.
They use sensory resource.
But the number of motor neuron per mass is very small.
Head or maybe foot hand, they have very large ratio about the number of motor neuron per mass.
So if we look at the function or portion of the body, we can roughly estimate the property of the muscles.
So in this way, we put all 1000 muscles in this drawing and try to roughly estimate the number of muscles.
We know the mass of the muscles, right?
So this is the very rough estimate that we did.
And the distribution of neurons reported in the literature, of course, but it's not really mathematical.
And he spent a lot of time and he distributed all the motor neurons and muscle spindle neurons at the spines for whole body.
For example, this is the neurons of L3.
And in total, we have 128,000 motor neurons and 53,000 muscle spindles of the body.
We don't have interneurons, unfortunately, yet we have to include.
So we made a very simple self-reflex model, not inhibition type reflex model.
And of course, we can simulate and this is very simple simulation.
We just pushed the lower leg and we observed how the sensation of muscles are split over the spines and coming back to the muscles.
So this is, of course, the synapse connection is very simply made by other ways.
So it is not really quantitative.
So these are the firing of muscle spindle and motor neurons.
And this is a patellar tendon reflex.
But in our case, our simulation is pushing the lower leg, not hitting the tendon.
So the phenomenon is slightly different.
This is a simulated result.
Anyway, the complex modeling is being developed.
Of course, we have a lot to do further.
And we have many unknowns and we have to develop the learning system or training system to tune the parameters.
And just about one month ago, another master's student, Rohan, has just finished his master dissertation.
And he did very interesting work.
And one of the major symptoms is impaired posture and walking.
So we are interested in attacking this problem.
So this paper was suggested by Professor Taishin Nomura, Nomura Sensei in Osaka University.
And this is a movement of the COP of healthy people.
But this is showing a very typical difference between the healthy person and typical COP or COM of the Parkinson's patient.
Do you know which is closer to Parkinson's patient?
Actually, this is closer to the control of COP of the Parkinson's patient.
So the healthy person has widespread COM or COP while just standing still.
But Parkinson's patient has more stiff control of the COP or COM.
So these are the differences.
But this data is from the healthy person.
So of course, the symptoms depend on the person and its variety is very large.
So we want to model this kind of the posture control system.
So in order to model this, the muscle spindle is not the sufficient sensory signal.
And the Rohan spent a lot of time.
And he modeled the vestibular sensors.
So in the vestibular sensors, there are 36,000 sensory neurons.
And they are just gathered here.
And their mathematical model is given in the literature.
So using the next heap, he programmed 36,000 sensory neurons here.
And this is an interesting part.
This is the continuous sensors on the foot skin.
So these locations are quite well-defined and known.
And again, the Rohan spent some time and put all the external sensations on the skin of the foot.
So now we have the force sensor on the foot.
And we have IMU in the vestibular system.
So we want to use this neural signal to stabilize the whole body.
So of course, we don't know how to combine with the blame system that has a problem of dopamine receptors.
So the current question that our blame team in Okinawa Institute of Science and Technology
are looking at literature and trying to simulate the particular vibration or oscillatory signal of the blame.
And it seems that the tremor is around four to five healths.
And in the case of the tremor, blame also has oscillation of five or six healths.
This is one particular knowledge in the literature.
And furthermore, for other symptoms of Parkinson's disease, what is known as beta-band,
that is, as far as I understand, from 13 health to 30 health,
vibration is in the straight term in the blame.
So the question is how those signals are coming to the muscles and how the symptoms are created.
With the body, and of course, the vestibular sensation or the foot force contact force sensation,
they must be combined.
And I'm just curious how we can develop those feedback systems perturbed by the vibrations.
Well, this is the force sensing.
Well, this is incomplete work.
So I have no good conclusion, but just summarize a few things.
The building is a physiologically detailed model of muscles, skeletons, sensory motor neurons, sensory organs.
And this is very exciting part.
We have to run a lot from the literature, but the computational tool and robotics theories can help us.
And of course, the knowledge in the literature is not enough.
We have to estimate and extrapolate based on mathematical intuitions.
And of course, the parallel algorithm of geometry dynamics optimization in robotics.
Play a very important role.
This is a very strong tool for this complex system.
And our scientific challenge and scientific challenge of robotics to me
is to search for the pathways for mathematical modeling.
And simulation of the Parkinson disease symptom is a real challenge.
Of course, this is my future thought, but of course, not conclusive.
The future study in depth of cognitive skill, such as motion control and planning,
the Parkinson disease, since it's split to the cortex,
it has a problem of the cognitive skills.
So the question is how to model those.
Of course, it's very difficult.
I don't think we can solve very easily.
But the only way to approach this problem
is to use the knowledge of control system design in robotics
and cognitive modeling developing robotics.
I think we have some interesting tools to approach these exciting problems in the future.
Thank you very much.
We have time for two or three questions.
And we also have an innovation.
In the last plenary, we have a wireless microphone.
You do not necessarily have to step up to the microphone,
but if you want to, then of course, there's probably a better quality.
There's a question at microphone three, please.
Yes, please, go ahead.
So I had a question.
I'm a bit curious about your visions for accessibility for this software.
So do you think in the future that any neuroscientist
can have a high-level motor control hypothesis and use your tool,
or will you have to be a specialized PhD student in your lab?
Well, the project has not decided yet.
But as far as our software,
we have already having the collaboration with other universities.
So it's not freely open.
But if you are interested in that, I'm happy to collaborate.
But we don't have manuals.
So documentation is a problem.
But if that is okay, we are happy to collaborate with any researchers.
The follow-up question will, of course,
be how tight is your software to the K computer?
Because if I don't have access to the K computer,
your software will probably not be very useful.
So the question is, are you planning to offer any services
that I can access as a scientist?
Well, unfortunately, our software is not still tuned to K computer.
It's MPI, so it can be used for cloud computing.
Of course, to best use of K computer, it must be tuned to K computer.
That is the next step.
So at this moment, one can use any cloud computing system.
But you mentioned the MPI.
And of course, then the question is,
what is the bandwidth, for example, that you need?
And how optimized are your algorithms at this point?
And how far are you hoping to optimize them further?
Can you maybe elaborate a little bit on that as well?
Well, of course, to get the best performance,
we have to tune to the specific architecture of the computer.
That is not done yet.
So of course, it is one possibility that we tune the system
for the best performance of a particular supercomputing system.
But for wide use of the community,
I think design is another choice.
So I think there are two possibilities.
Okay, so I see no further question.
There's one.
Yeah, maybe you can now use the microphone at least once.
We paid a lot of big bucks for this microphone,
so please use it now at least once in the final plenary.
Thank you. It seems to work.
Well, it should work.
Your simulation is, of course, amazingly precise.
But nevertheless, you have to invest assumptions like
normal distribution or the German sausage assumption.
How accurate do you think will you be able to simulate the humanoid body?
Human body.
The human body.
So it's hard.
It's hard because we cannot verify.
So we are always proposing proposals,
but always reject it.
I have the good co-op later in Dekem Biosource Center.
They have engineered mice as particular genetic systems.
And they can repeat the small animal experiment.
So we are even developing the muscular skeletal system of mouse.
So by using this small animal, we can experimentally verify.
And we may be able to bring that knowledge to the human model.
So that is our scenario of the proposal, but it's not convincing yet.
I hope you are convinced.
Well, I saw some videos from that.
I was more convinced, obviously, than you are.
Anyway, thank you again.