4.5. Notes for Developers and Code Maintainers
This section documents how a few large sections of code with LAMMPS work at a conceptual level. Comments on code in source files typically document what a variable stores, what a small section of code does, or what a function does or its input/outputs. The topics on this page are intended to document code at a higher level.
4.5.1. KSpace PPPM FFT grids
The various KSpace PPPM styles in LAMMPS use FFTs to solve Poisson’s equation. This subsection describes:
how FFT grids are defined
how they are decomposed across processors
how they are indexed by each processor
how particle charge and electric field values are mapped to/from the grid
An FFT grid cell is a 3d volume; grid points are corners of a grid cell and the code stores values assigned to grid points in vectors or 3d arrays. A global 3d FFT grid has points indexed 0 to N-1 inclusive in each dimension.
Each processor owns two subsets of the grid, each subset is brick-shaped. Depending on how it is used, these subsets are allocated as a 1d vector or 3d array. Either way, the ordering of values within contiguous memory x fastest, then y, z slowest.
For the 3d decomposition
of the grid, the global grid is
partitioned into bricks that correspond to the sub-domains of the
simulation box that each processor owns. Often, this is a regular 3d
array (Px by Py by Pz) of bricks, where P = number of processors =
Px * Py * Pz. More generally it can be a tiled decomposition, where
each processor owns a brick and the union of all the bricks is the
global grid. Tiled decompositions are produced by load balancing with
the RCB algorithm; see the balance rcb command.
For the FFT decompostion
of the grid, each processor owns a brick
that spans the entire x dimension of the grid while the y and z
dimensions are partitioned as a regular 2d array (P1 by P2), where P =
P1 * P2.
The following indices store the inclusive bounds of the brick a processor owns, within the global grid:
nxlo_in,nxhi_in,nylo_in,nyhi_in,nzlo_in,nzhi_in = 3d decomposition brick
nxlo_fft,nxhi_fft,nylo_fft,nyhi_fft,nzlo_fft,nzhi_fft = FFT decomposition brick
nxlo_out,nxhi_out,nylo_out,nyhi_out,nzlo_out,nzhi_out = 3d decomposition brick + ghost cells
The in
and fft
indices are from 0 to N-1 inclusive in each
dimension, where N is the grid size.
The out
indices index an array which stores the in
subset of
the grid plus ghost cells that surround it. These indices can thus be
< 0 or >= N.
The number of ghost cells a processor owns in each of the 6 directions is a function of:
neighbor skin distance (since atoms can move outside a proc subdomain)
qdist = offset or charge from atom due to TIP4P fictitious charge
order = mapping stencil size
shift = factor used when order is an even number (see below)
Here is an explanation of how the PPPM variables order
,
nlower
/ nupper
, shift
, and OFFSET
work. They are the
relevant variables that determine how atom charge is mapped to grid
points and how field values are mapped from grid points to atoms:
order = # of nearby grid points in each dim that atom charge/field are mapped to/from
nlower,nupper = extent of stencil around the grid point an atom is assigned to
OFFSET = large integer added/subtracted when mapping to avoid int(-0.75) = 0 when -1 is the desired result
The particle_map() method assigns each atom to a grid point.
If order is even, say 4:
atom is assigned to grid point to its left (in each dim)
shift = OFFSET
nlower = -1, nupper = 2, which are offsets from assigned grid point
window of mapping grid pts is thus 2 grid points to left of atom, 2 to right
If order is odd, say 5:
atom is assigned to left/right grid pt it is closest to (in each dim)
shift = OFFSET + 0.5
nlower = 2, nupper = 2
if point is in left half of cell, then window of affected grid pts is 3 grid points to left of atom, 2 to right
if point is in right half of cell, then window of affected grid pts is 2 grid points to left of atom, 3 to right
These settings apply to each dimension, so that if order = 5, an atom’s charge is mapped to 125 grid points that surround the atom.