DD support in ASPC

[Tomáš Trnka]

Hello,

the ASPC code as currently submitted for 4.6 review doesn't work with domain
decomposition. This stems from the fact that I have neither any significant
experience in MPI programming nor deep understanding of the DD core.

However, supporting DD shouldn't be too hard. For each shell, ASPC keeps a
history of its relative displacement from the carrying nucleus and uses it to
predict the new position in each integration step. There are no cross-shell
dependencies (a shell's history is only used to predict the position for that
shell). So the only thing that's necessary for working DD is that when a shell
crosses from one node to another, it needs to carry its history over.

Could anyone please provide me with some pointers how this can be best
achieved? I've tried several approaches but every time the thing got too
complicated too soon:

1) extend t_state with the ASPC history arrays - main problems with this were
that t_state is touched in hundreds of places throughout the code; also the
placement of shells onto individual DD nodes would need to be recorded
somewhere (another global index?)

2) keep the history within gmx_shellfc_t and collect/distribute in
make_local_shells() - old shell indices would need to be kept somewhere there
too to be able to do the collection or possibly the indices could be collected
from the nodes, too; I have a bad feeling about doing collection in here as it
is called from dd_partition_system, so it's rather counter-intuitive and could
break something

3) ???

Thanks for any suggestions!
Best regards

Tomáš Trnka
--
gmx-developers mailing list
[hidden email]
http://lists.gromacs.org/mailman/listinfo/gmx-developers
Please don't post (un)subscribe requests to the list. Use the
www interface or send it to [hidden email].

[Berk Hess-4]

Hi,

Keeping track of history is tedious, both for domain decomposition and
check pointing.

I assume the method should actually not be history dependent and this is
only for efficiency?

Cheers,

Berk

On 03/06/2012 10:57 AM, Tomáš Trnka wrote:

> Hello,
>
> the ASPC code as currently submitted for 4.6 review doesn't work with domain
> decomposition. This stems from the fact that I have neither any significant
> experience in MPI programming nor deep understanding of the DD core.
>
> However, supporting DD shouldn't be too hard. For each shell, ASPC keeps a
> history of its relative displacement from the carrying nucleus and uses it to
> predict the new position in each integration step. There are no cross-shell
> dependencies (a shell's history is only used to predict the position for that
> shell). So the only thing that's necessary for working DD is that when a shell
> crosses from one node to another, it needs to carry its history over.
>
> Could anyone please provide me with some pointers how this can be best
> achieved? I've tried several approaches but every time the thing got too
> complicated too soon:
>
> 1) extend t_state with the ASPC history arrays - main problems with this were
> that t_state is touched in hundreds of places throughout the code; also the
> placement of shells onto individual DD nodes would need to be recorded
> somewhere (another global index?)
>
> 2) keep the history within gmx_shellfc_t and collect/distribute in
> make_local_shells() - old shell indices would need to be kept somewhere there
> too to be able to do the collection or possibly the indices could be collected
> from the nodes, too; I have a bad feeling about doing collection in here as it
> is called from dd_partition_system, so it's rather counter-intuitive and could
> break something
>
> 3) ???
>
> Thanks for any suggestions!
> Best regards
>
> Tomáš Trnka

--
gmx-developers mailing list
[hidden email]
http://lists.gromacs.org/mailman/listinfo/gmx-developers
Please don't post (un)subscribe requests to the list. Use the
www interface or send it to [hidden email].

[Tomáš Trnka]

Dne Út 6. března 2012 11:08:08, Berk Hess napsal(a):
> Hi,
>
> Keeping track of history is tedious, both for domain decomposition and
> check pointing.
>
> I assume the method should actually not be history dependent and this is
> only for efficiency?

Well, that depends on the point of view. The whole point of ASPC is to
dramatically speed up SCF calculations by predicting the induced dipoles based
on their history. So the ASPC method is history dependent in this sense.

Currently the predictor history is not saved anywhere on disk so the first
several (1-3) integration steps are significantly slower and have worse energy
conservation. Saving the history would get rid of this but IMHO it's no big
deal.

Tomáš
--
gmx-developers mailing list
[hidden email]
http://lists.gromacs.org/mailman/listinfo/gmx-developers
Please don't post (un)subscribe requests to the list. Use the
www interface or send it to [hidden email].

[David van der Spoel]

On 2012-03-06 11:22, Tomáš Trnka wrote:

> Dne Út 6. března 2012 11:08:08, Berk Hess napsal(a):
>> Hi,
>>
>> Keeping track of history is tedious, both for domain decomposition and
>> check pointing.
>>
>> I assume the method should actually not be history dependent and this is
>> only for efficiency?
>
> Well, that depends on the point of view. The whole point of ASPC is to
> dramatically speed up SCF calculations by predicting the induced dipoles based
> on their history. So the ASPC method is history dependent in this sense.
>
> Currently the predictor history is not saved anywhere on disk so the first
> several (1-3) integration steps are significantly slower and have worse energy
> conservation. Saving the history would get rid of this but IMHO it's no big
> deal.
>
> Tomáš

I guess the history must be built up at the beginning of the run.

So how about resetting the history for each particle if it moves to
another processor, but storing it otherwise?

That will lose some performance obviously.



--
David van der Spoel, Ph.D., Professor of Biology
Dept. of Cell & Molec. Biol., Uppsala University.
Box 596, 75124 Uppsala, Sweden. Phone: [phone].
[hidden email]    http://folding.bmc.uu.se
--
gmx-developers mailing list
[hidden email]
http://lists.gromacs.org/mailman/listinfo/gmx-developers
Please don't post (un)subscribe requests to the list. Use the
www interface or send it to [hidden email].

[Mark Abraham]

On 6/03/2012 9:22 PM, Tomáš Trnka wrote:

> Dne Út 6. března 2012 11:08:08, Berk Hess napsal(a):
>> Hi,
>>
>> Keeping track of history is tedious, both for domain decomposition and
>> check pointing.
>>
>> I assume the method should actually not be history dependent and this is
>> only for efficiency?
> Well, that depends on the point of view. The whole point of ASPC is to
> dramatically speed up SCF calculations by predicting the induced dipoles based
> on their history. So the ASPC method is history dependent in this sense.
>
> Currently the predictor history is not saved anywhere on disk so the first
> several (1-3) integration steps are significantly slower and have worse energy
> conservation. Saving the history would get rid of this but IMHO it's no big
> deal.

It might be a bigger deal from the point of view of the desirability of
exact restarts. If the state is not saved in the checkpoint, then the
restart is inexact.

Mark
--
gmx-developers mailing list
[hidden email]
http://lists.gromacs.org/mailman/listinfo/gmx-developers
Please don't post (un)subscribe requests to the list. Use the
www interface or send it to [hidden email].

[Tomáš Trnka]

> I guess the history must be built up at the beginning of the run.

Yes, that's how it's done currently.

> So how about resetting the history for each particle if it moves to
> another processor, but storing it otherwise?

IMHO this is not the way to go - detecting that the shell has moved elsewhere
seems to be the hard part of the problem. When we know that it has moved, we
can easily pass the history along.

> That will lose some performance obviously.

Worse than that. Losing history will impact energy conservation (at least in
the current implementation). In the ASPC method it's the predictor that
guarantees good energy conservation. If the predictor is unable to work
properly due to missing history, the code would need to switch to iterating
the SCF solution to a high precision to keep reasonable energy conservation
(could be done, but will impact the performance a lot if this happens every
~20 steps).

2T
--
gmx-developers mailing list
[hidden email]
http://lists.gromacs.org/mailman/listinfo/gmx-developers
Please don't post (un)subscribe requests to the list. Use the
www interface or send it to [hidden email].

[Tomáš Trnka]

> It might be a bigger deal from the point of view of the desirability of
> exact restarts. If the state is not saved in the checkpoint, then the
> restart is inexact.

Sure, saving the history would be desirable for this reason.

Tomáš
--
gmx-developers mailing list
[hidden email]
http://lists.gromacs.org/mailman/listinfo/gmx-developers
Please don't post (un)subscribe requests to the list. Use the
www interface or send it to [hidden email].

[Mark Abraham]

On 6/03/2012 8:57 PM, Tomáš Trnka wrote:

> Hello,
>
> the ASPC code as currently submitted for 4.6 review doesn't work with domain
> decomposition. This stems from the fact that I have neither any significant
> experience in MPI programming nor deep understanding of the DD core.
>
> However, supporting DD shouldn't be too hard. For each shell, ASPC keeps a
> history of its relative displacement from the carrying nucleus and uses it to
> predict the new position in each integration step. There are no cross-shell
> dependencies (a shell's history is only used to predict the position for that
> shell). So the only thing that's necessary for working DD is that when a shell
> crosses from one node to another, it needs to carry its history over.
>
> Could anyone please provide me with some pointers how this can be best
> achieved? I've tried several approaches but every time the thing got too
> complicated too soon:
>
> 1) extend t_state with the ASPC history arrays - main problems with this were
> that t_state is touched in hundreds of places throughout the code; also the
> placement of shells onto individual DD nodes would need to be recorded
> somewhere (another global index?)

What's a nucleus? If it's an atom, then things should be reasonably
straightforward, piggy-backing the existing accounting for which atoms
belong to which processor. If it's not an atom, then expressing a
nucleus as being attached to some appropriate master atom and preceding
as before might be the way forward.

Mark

> 2) keep the history within gmx_shellfc_t and collect/distribute in
> make_local_shells() - old shell indices would need to be kept somewhere there
> too to be able to do the collection or possibly the indices could be collected
> from the nodes, too; I have a bad feeling about doing collection in here as it
> is called from dd_partition_system, so it's rather counter-intuitive and could
> break something
>
> 3) ???
>
> Thanks for any suggestions!
> Best regards
>
> Tomáš Trnka

--
gmx-developers mailing list
[hidden email]
http://lists.gromacs.org/mailman/listinfo/gmx-developers
Please don't post (un)subscribe requests to the list. Use the
www interface or send it to [hidden email].

[Shirts, Michael (mrs5pt)]

Adding history dependence also sounds like a job for master, not 4.6.

Best,
~~~~~~~~~~~~
Michael Shirts
Assistant Professor
Department of Chemical Engineering
University of Virginia
[hidden email]
(434)-243-1821

> From: Mark Abraham <[hidden email]>
> Reply-To: Discussion list for GROMACS development <[hidden email]>
> Date: Tue, 6 Mar 2012 22:11:12 +1100
> To: Discussion list for GROMACS development <[hidden email]>
> Subject: Re: [gmx-developers] DD support in ASPC
>
> On 6/03/2012 8:57 PM, Tomáš Trnka wrote:
>> Hello,
>>
>> the ASPC code as currently submitted for 4.6 review doesn't work with domain
>> decomposition. This stems from the fact that I have neither any significant
>> experience in MPI programming nor deep understanding of the DD core.
>>
>> However, supporting DD shouldn't be too hard. For each shell, ASPC keeps a
>> history of its relative displacement from the carrying nucleus and uses it to
>> predict the new position in each integration step. There are no cross-shell
>> dependencies (a shell's history is only used to predict the position for that
>> shell). So the only thing that's necessary for working DD is that when a
>> shell
>> crosses from one node to another, it needs to carry its history over.
>>
>> Could anyone please provide me with some pointers how this can be best
>> achieved? I've tried several approaches but every time the thing got too
>> complicated too soon:
>>
>> 1) extend t_state with the ASPC history arrays - main problems with this were
>> that t_state is touched in hundreds of places throughout the code; also the
>> placement of shells onto individual DD nodes would need to be recorded
>> somewhere (another global index?)
>
> What's a nucleus? If it's an atom, then things should be reasonably
> straightforward, piggy-backing the existing accounting for which atoms
> belong to which processor. If it's not an atom, then expressing a
> nucleus as being attached to some appropriate master atom and preceding
> as before might be the way forward.
>
> Mark
>
>> 2) keep the history within gmx_shellfc_t and collect/distribute in
>> make_local_shells() - old shell indices would need to be kept somewhere there
>> too to be able to do the collection or possibly the indices could be
>> collected
>> from the nodes, too; I have a bad feeling about doing collection in here as
>> it
>> is called from dd_partition_system, so it's rather counter-intuitive and
>> could
>> break something
>>
>> 3) ???
>>
>> Thanks for any suggestions!
>> Best regards
>>
>> Tomáš Trnka
>
> --
> gmx-developers mailing list
> [hidden email]
> http://lists.gromacs.org/mailman/listinfo/gmx-developers
> Please don't post (un)subscribe requests to the list. Use the
> www interface or send it to [hidden email].

--
gmx-developers mailing list
[hidden email]
http://lists.gromacs.org/mailman/listinfo/gmx-developers
Please don't post (un)subscribe requests to the list. Use the
www interface or send it to [hidden email].