Unified long-range electrostatics and dynamic protonation for realistic biomolecular simulations on the Exascale

Unified long-range electrostatics and dynamic protonation for realistic biomolecular simulations on the Exascale

In this DFG supported project we target a flexible, portable and scalable solver for potentials and forces, which is a prerequisite for exascale applications in particle-based simulations with long-range interactions in general. As a particularly challenging example that will prove and demonstrate the capability of our concepts, we use the popular molecular dynamics (MD) simulation software GROMACS. MD simulation has become a crucial tool to the scientific community, especially as it probes time- and length scales difficult or impossible to probe experimentally. Moreover, it is a prototypic example of a general class of complex multiparticle systems with long-range interactions.

MD simulations elucidate detailed, time-resolved behaviour of biology’s nanomachines. From a computational point of view, they are extremely challenging for two main reasons. First, to properly describe the functional motions of biomolecules, the long-range effects of the electrostatic interactions must be explicitly accounted for. Therefore, techniques like the particle-mesh Ewald method were adopted, which, however, severely limits the scaling to a large number of cores due to global communication requirements. The second challenge is to realistically describe the time-dependent location of (partial) charges, as e.g. the protonation states of the molecules depend on their time-dependent electrostatic environment. Here we address both tighly interlinked challenges by the development, implementation, and optimization of a unified algorithm for long-range interactions that will account for realistic, dynamic protonation states and at the same time overcome current scaling limitations.

Download and test our GPU-FMM for GROMACS

If you want to give our GPU-FMM a test drive, please download the tar archive below, unpack with tar -xvzf, and install just like a usual GROMACS 2019.

Our CUDA FMM can be used as a PME replacement by choosing coulombtype = FMM in the .mdp input parameter list. The tree depth d and the multipole order p are set with fmm-override-tree-depth and fmm-override-multipole-order input parameters, respectively. On request (provide your ssh key), the code can be checked out from our git repository git@fmsolvr.fz-juelich.de:gromacs.

GROMACS with GPU-FMM including benchmark systems

For running the GPU FMM benchmarks, you need to set the following environment variable:


With sparse systems as the aerosol system, you should additionally set

export FMM_SPARSE=1

for optimum FMM performance.

Running FMM in standalone mode

You can also compile and run the GPU-FMM without GROMACS integration. The relevant code is in the ./src/gromacs/fmm/fmsolvr-gpu subdirectory of the above tar archive after unpacking. Compile it with a script like this:

; in bash
export CC=$( which gcc )
export CXX=$( which g++ )
cmake -H../git-gromacs-gmxbenchmarking/src/gromacs/fmm/fmsolvr-gpu -B. -DFMM_STANDALONE=1 -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-10.0

The python script runfmm.py can be used to benchmark the standalone version of the GPU-FMM.

Please follow this link to the external workshop website for more information.

On May 19–20, a group of >50 GROMACS developers and users gathered at the Max Planck Institute for biophysical Chemistry in Göttingen to discuss various aspects of software development and future directions for GROMACS. more


Kohnke, B.; Kutzner, C.; Grubmüller, H.: A GPU-accelerated fast multipole method for GROMACS: Performance and accuracy. Journal of Chemical Theory and Computation 16 (11), pp. 6938 - 6949 (2020)
Kohnke, B.; Kutzner, C.; Beckmann, A.; Lube, G.; Kabadshow, I.; Dachsel, H.; Grubmüller, H.: A CUDA fast multipole method with highly efficient M2L farfield evaluationfield evaluation. The International Journal of High Performance Computing Applications 35 (1), pp. 97 - 117 (2021)
Kohnke, B.; Ullmann, R. T.; Beckman, A.; Kabadshow, I.; Haensel, D.; Morgenstern, L.; Dobrev, P.; Groenhof, G.; Kutzner, C.; Hess, B. et al.; Dachsel, H.; Grubmüller, H.: GROMEX: A scalable and versatile fast multipole method for biomolecular simulation. In: Software for Exascale Computing - SPPEXA 2016-2019, pp. 517 - 543 (Eds. Bungartz, H.-J.; Reiz, S.; Uekermann, B.; Neumann, P.; Nagel, W. E.). Springer, Cham (2020)
Kutzner, C.; Páll, S.; Fechner, M.; Esztermann, A.; de Groot, B. L.; Grubmüller, H.: More bang for your buck: Improved use of GPU nodes for GROMACS 2018. Journal of Computational Chemistry 40 (27), pp. 2418 - 2431 (2019)
Donnini, S.; Ullmann, T.; Groenhof, G.; Grubmüller, H.: Charge-neutral constant pH molecular dynamics simulations using a parsimonious proton buffer. Journal of Chemical Theory and Computation 12 (3), pp. 1040 - 1051 (2016)
Garcia, AG; Beckmann, A; Kabadshow, I
Accelerating an FMM-Based Coulomb Solver with GPUs
Lecture Notes in Computational Science and Engineering 113 (2016) "Software for Exascale Computing - SPPEXA 2013-2015", Eds. HJ Bungartz, P. Neumann, WE Nagel, Springer, pp. 485-504
Páll, S.; Abraham, M. J.; Kutzner, C.; Hess, B.; Lindahl, E.: Tackling exascale software challenges in molecular dynamics simulations with GROMACS. In: Solving Software Challenges for Exascale: International Conference on Exascale Applications and Software, EASC 2014, Stockholm, Sweden, April 2-3, 2014, Revised Selected Papers, pp. 3 - 27 (Eds. Markidis, S.; Laure, E.). Springer, Cham (2015)
A. Beckmann, I. Kabadshow
Portable Node-Level Performance Optimization for the Fast Multipole Method
Lecture Notes in Computational Science and Engineering 105, 29-46 (2015)
Kutzner, C.; Páll, S.; Fechner, M.; Esztermann, A.; de Groot, B.; Grubmüller, H.: Best bang for your buck: GPU nodes for GROMACS biomolecular simulations. Journal of Computational Chemistry 36 (26), pp. 1990 - 2008 (2015)
Kutzner, C.; Apostolov, R.; Hess, B.; Grubmüller, H.: Scaling of the GROMACS 4.6 molecular dynamics code on SuperMUC. In: Parallel Computing: Accelerating Computational Science and Engineering (CSE), pp. 722 - 730 (Eds. Bader, M.; Bode, A.; Bungartz, H. J.). IOS Press, Amsterdam (2014)
A. Arnold, F. Fahrenberger, C. Holm, O. Lenz, M. Bolten, H. Dachsel, R. Halver, I. Kabadshow, F. Gähler, F. Heber, J. Iseringhausen, M. Hofmann, M. Pippig, D. Potts, and G. Sutmann
Comparison of scalable fast methods for long-range interactions
Phys. Rev. E 88 063308-1-22 (2013)
R. T. Ullmann, G. M. Ullmann
GMCT: A Monte Carlo simulation package for macromolecular receptors
Journal of Computational Chemistry 33, 887-900 (2012)
Donnini, S.; Tegeler, F.; Groenhof, G.; Grubmüller, H.: Constant pH molecular dynamics in explicit solvent with lambda-dynamics. Journal of Chemical Theory and Computation 7 (6), pp. 1962 - 1978 (2011)
Go to Editor View