LAMMPS Benchmarks

This page lists LAMMPS performance on several benchmark problems, run on various machines, both in serial and parallel.

• Lennard Jones = atomic fluid with Lennard-Jones potential
• Polymer = bead-spring polymer melt of 100-mer chains
• Metal = metallic solid
• Granular = granular chute flow
• Protein = rhodopsin protein in solvated lipid bilayer

Additional benchmark data is given here:

• One processor = relative CPU cost of the 5 benchmarks
• Billion-atom = billion-atom Lennard-Jones benchmarks
• Potentials = relative CPU cost of different interatomic potentials

These are the parallel machines for which benchmark data is given. The "Processors" column is the most number of processors on that machine that LAMMPS was run on. Message passing bandwidth and latency is in units of Mb/sec and microsecs at the MPI level, i.e. what a program like LAMMPS sees. More information on machine characteristics, including their "birth" year, is given below.

One-processor timings are also listed for some older machines whose characteristics are also given below.

A billion-atom LJ timing is also given for a GPU cluster, with characteristics given below.

For each of the 5 benchmarks, fixed- and scaled-size timings are shown in tables and in comparative plots. Fixed-size means that the same problem with 32,000 atoms was run on varying numbers of processors. Scaled-size means that when run on P processors, the number of atoms in the simulation was P times larger than the one-processor run. Thus a scaled-size 64-processor run is for 2,048,000 atoms; a 32K proc run is for ~1 billion atoms.

All listed CPU times are in seconds for 100 timesteps. Parallel efficiencies refer to the ratio of ideal to actual run time. For example, if perfect speed-up would have given a run-time of 10 seconds, and the actual run time was 12 seconds, then the efficiency is 10/12 or 83.3%. In most cases parallel runs were made on production machines while other jobs were running, which can sometimes degrade performance.

The files needed to run these benchmarks are part of the LAMMPS distribution. If your platform is sufficiently different from the machines listed, you can send your timing results and machine info and we'll add them to this page. Note that the CPU time (in seconds) for a run is what appears in the "Loop time" line of the output log file, e.g.

Loop time of 3.89418 on 8 procs for 100 steps with 32000 atoms 

These benchmarks are meant to span a range of simulation styles and computational expense for interaction forces. Since LAMMPS run time scales roughly linearly in the number of atoms simulated, you can use the timing and parallel efficiency data to estimate the CPU cost for problems you want to run on a given number of processors. As the data below illustrates, fixed-size problems generally have parallel efficiencies of 50% or better so long as the atoms/processor is a few hundred or more. Scaled-size problems generally have parallel efficiencies of 80% or more across a wide range of processor counts.

Thanks to the following individuals for running the various benchmarks:

• Christian Trott (U of Technology Ilmenau), Lincoln results
• Paul Crozier (Sandia), IBM BG/L results
• Fiona Reid (Edinburgh Parallel Computing Centre), IBM p690+ results
• Courtenay Vaughan (Sandia), Cray XT3 results

This is a summary of single-processor LAMMPS performance in CPU secs per atom per timestep for the 5 benchmark problems. This is on a Dell 690 desktop Red Hat linux box with dual quad-core 2.66 GHz Intel Xeon processor using the Intel icc compiler. The ratios indicate that if the atomic LJ system has a normalized cost of 1.0, the bead-spring chains and granular systems run 2x faster, while the EAM metal and solvated protein models run 2.7x and 18x slower respectively. These differences are primarily due to the expense of computing a particular pairwise force field for a given number of neighbors per atom.

Input script for this problem.

Atomic fluid:

• 32,000 atoms for 100 timesteps
• reduced density = 0.8442 (liquid)
• force cutoff = 2.5 sigma
• neighbor skin = 0.3 sigma
• neighbors/atom = 55 (within force cutoff)
• NVE time integration

Performance data:

• One-processor performance

• Dual quad-core Xeon desktop performance
• Xeon/Myrinet cluster performance
• IBM p690+ performance
• IBM BG/L performance
• Cray XT3 performance
• Cray XT5 performance

• ASCI Red performance
• Ross performance
• Liberty performance
• Cheetah performance

These plots show fixed-size parallel efficiency for the same 32K atom problem run on different numbers of processors and scaled-size efficiency for runs with 32K atoms/proc. Thus a scaled-size 64-processor run is for 2,048,000 atoms; a 32K proc run is for ~1 billion atoms. Each curve is normalized to be 100% on one processor for the respective machine; one-processor timings are shown in parenthesis. Click on the plot for a larger version.

Input script for this problem.

Bead-spring polymer melt with 100-mer chains and FENE bonds:

• 32,000 atoms for 100 timesteps
• reduced density 0.8442 (liquid)
• force cutoff of 2^(1/6) sigma
• neighbor skin = 0.4 sigma
• neighbors/atom = 5 (within force cutoff)
• NVE time integration

Performance data:

• One-processor performance

• Dual quad-core Xeon desktop performance
• Xeon/Myrinet cluster performance
• IBM p690+ performance
• IBM BG/L performance
• Cray XT3 performance
• Cray XT5 performance

• ASCI Red performance
• Ross performance
• Liberty performance
• Cheetah performance

These plots show fixed-size parallel efficiency for the same 32K atom problem run on different numbers of processors and scaled-size efficiency for runs with 32K atoms/proc. Thus a scaled-size 64-processor run is for 2,048,000 atoms; a 32K proc run is for ~1 billion atoms. Each curve is normalized to be 100% on one processor for the respective machine; one-processor timings are shown in parenthesis. Click on the plot for a larger version.

Input script for this problem.

Cu metallic solid with embedded atom method (EAM) potential:

• 32,000 atoms for 100 timesteps
• force cutoff of 4.95 Angstroms
• neighbor skin = 1.0 Angstrom
• neighbors/atom = 45 (within force cutoff)
• NVE time integration

Performance data:

• One-processor performance

• Dual quad-core Xeon desktop performance
• Xeon/Myrinet cluster performance
• IBM p690+ performance
• IBM BG/L performance
• Cray XT3 performance
• Cray XT5 performance

• ASCI Red performance
• Ross performance
• Liberty performance
• Cheetah performance

These plots show fixed-size parallel efficiency for the same 32K atom problem run on different numbers of processors and scaled-size efficiency for runs with 32K atoms/proc. Thus a scaled-size 64-processor run is for 2,048,000 atoms; a 32K proc run is for ~1 billion atoms. Each curve is normalized to be 100% on one processor for the respective machine; one-processor timings are shown in parenthesis. Click on the plot for a larger version.

Input script for this problem.

Chute flow of packed granular particles with frictional history potential:

• 32,000 atoms for 100 timesteps
• force cutoff of 1.0 sigma
• neighbor skin of 0.1 sigma
• neighbors/atom = 7
• NVE time integration

Performance data:

• One-processor performance

• Dual quad-core Xeon desktop performance
• Xeon/Myrinet cluster performance
• IBM p690+ performance
• IBM BG/L performance
• Cray XT3 performance
• Cray XT5 performance

• ASCI Red performance
• Ross performance
• Liberty performance
• Cheetah performance

These plots show fixed-size parallel efficiency for the same 32K atom problem run on different numbers of processors and scaled-size efficiency for runs with 32K atoms/proc. Thus a scaled-size 64-processor run is for 2,048,000 atoms; a 32K proc run is for ~1 billion atoms. Each curve is normalized to be 100% on one processor for the respective machine; one-processor timings are shown in parenthesis. Click on the plot for a larger version.

Input script for this problem.

All-atom rhodopsin protein in solvated lipid bilayer with CHARMM force field, long-range Coulombics via PPPM (particle-particle particle mesh), SHAKE constraints. This model contains counter-ions and a reduced amount of water to make a 32K atom system:

• 32,000 atoms for 100 timesteps
• LJ force cutoff of 10.0 Angstroms
• neighbor skin of 1.0 sigma
• neighbors/atom = 440 (within force cutoff)
• NPT time integration

Performance data:

• One-processor performance

• Dual quad-core Xeon desktop performance
• Xeon/Myrinet cluster performance
• IBM p690+ performance
• IBM BG/L performance
• Cray XT3 performance
• Cray XT5 performance

• ASCI Red performance
• Ross performance
• Liberty performance
• Cheetah performance

These plots show fixed-size parallel efficiency for the same 32K atom problem run on different numbers of processors and scaled-size efficiency for runs with 32K atoms/proc. Thus a scaled-size 64-processor run is for 2,048,000 atoms; a 32K proc run is for ~1 billion atoms. Each curve is normalized to be 100% on one processor for the respective machine; one-processor timings are shown in parenthesis. Click on the plot for a larger version.

The Lennard-Jones benchmark problem described above (100 timesteps, reduced density of 0.8442, 2.5 sigma cutoff, etc) has been run on different machines for billion-atom tests. For the LJ benchmark LAMMPS requires a little less than 1/2 Terabyte of memory per billion atoms, which is used mostly for neighbor lists.

The parallel efficiencies are estimated from the per-atom CPU or GPU time for a large single processor run on each machine:

• 4.24e-8 sec/atom/timestep on Lincoln
• 1.49e-6 sec/atom/timestep on the Cray XT5
• 2.35e-6 sec/atom/timestep on the Cray XT3
• 8.98e-6 sec/atom/timestep on the IBM BG/L
• 1.96e-5 sec/atom/timestep on ASCI Red

The aggregate flop rate is estimated using the following values for the pairwise interactions, which dominate the run time:

• 27.6 pairwise interactions per atom (with Newton's 3rd law for this density and cutoff)
• 23 flops per LJ interaction

This is a conservative estimate in the sense that flops computed for atom pairs outside the force cutoff, building neighbor lists, and time integration are not counted. For the USER-CUDA package running on GPUs, Newton's 3rd law is not used (because it's faster not to), which doubles the pairwise interaction count, but that is not included in the flop rate either.

The following table summarizes the CPU cost of various potentials, as implemented in LAMMPS, each for a system commonly modeled by that potential. Except that the last 3 entries are for VASP timings, to give a comparison with DFT calculations. The details for the VASP runs are described more fully below.

The listed timing is CPU seconds per timestep per atom for a one processor run. Note that this is per timestep, as is the ratio to LJ; the timestep size is listed in the table. In each case a short 100-step run of a roughly 32000 atom system was performed. The speed-up is for a 4-processor run of the same 32000-atom system. Speed-ups greater than 4x are due to cache effects.

To first order, the CPU and memory cost for simulations with all these potentials scales linearly with the number of atoms N, and inversely with the number of processors P when running in parallel. This assumes the density doesn't change so that the neighbors per atom stays constant as you change N. This holds for N/P ratios larger than some threshhold, say 1000 atoms per processor. Thus you can use this data to estimate the run-time of different size problems on varying numbers of processors.

Notes:

• The 1-processor runs were made on a single core (processor) of a Dell 690 desktop Red Hat linux box, with dual quad-core 2.66 GHz Intel Xeon nodes, using the Intel icc compiler, version 10.0. The 4-processor runs were made on 4 cores of the same system.
• The number of neighbors per atom is based on what LAMMPS tallied at the end of the short run within the neighbor cutoff = force cutoff + neighbor skin distance. There is a double counting of atom I as a neighbor of atom J as well as J as a neighbor of I.
• The tarballs contain the input and output files for each benchmark.
• Potentials with an asterisk can also be run on a GPU. Logfiles for these runs are included in the tarball. They were run on the following hardware. 1-proc: GPU = NVIDIA Tesla C2050; CPU = AMD Opteron 2435 at 2.6 GHz. 4-proc: same with 2 GPUs per node and an Infiniband interconnect. For LJ, the 1 and 4 processor GPU timings were 9.9x and 7.4x times faster than the CPU timings in the table. For GB, the 1 and 4 processor timings were 31.2x and 27.3x faster.
• The VASP quantum DFT calculations and timings are from Thomas Mattsson at Sandia for production runs he performs. The 3 VASP runs (small, medium, large) were performed on large parallel machines: a Cray XT3 for the small, and a dual quad-core Nehalem-based Linux cluster for the medium and large. The single core speed of these machines is slightly slower (maybe 25% slower) of the Dell desktop single core speed for LAMMPS calculations. The "# of Atoms" entry is listed as N/M where N is the number of atoms and M is the number of electrons, since the latter is more relevant for the cost of the DFT calculation. The listed timestep is for ab initio MD calculations. The total memory footprint is not given, but can be inferred from the processor count (320 or 384 procs) since each processor has one to a few Gb of memory. The CPU time per timestep per atom for one processor was calculated from the parallel run-time in the following manner. For the small problem, VASP ran at 15.7 seconds per timestep on 320 procs. The listed CPU time = 320*15.7/192 = 26.2 seconds per atom per timestep per processor. Similarly for the medium and large problems which ran at 126 and 1512 seconds per timestep in parallel. Note that this assumes the parallel VASP run-time is 100% efficient, which thus over-estimates the 1-processor time (assuming these problems could be run on a single processor, which they can't). Also note that the per-atom VASP timings increase with problem size, since DFT scales roughly as O(N^2.5) in the number of electrons. This means the per-atom per-timestep cost of a quantum calculation would increase dramatically if extrapolated to the 32,000 atom problem sizes listed for the MD calculations.

Details for different systems:

• Granular: chute flow, dense packing of sigma = 1 diameter particles, contact forces, neighbor skin = 0.1 sigma, NVE with rotational DOF.
• FENE bead/spring: melt of 320 100-mer chains, rho* = 0.844, FENE bond potential, LJ cutoff = 2^(1/6) sigma, neighbor skin = 0.4 sigma, NVE + Langevin thermoastat at T* = 1.0.
• Lennard-Jones: liquid, rho* = 0.844, T* = 0.75, cutoff = 2.5 sigma, neighbor skin = 0.3 sigma, NVE.
• DPD: pure solvent, rho* = 3.0, T* = 1.0, cutoff = 1.0 sigma, neighbor skin = 0.5 sigma, NVE.
• EAM: bulk fcc Cu at 800K, cutoff = 4.95 Angs, neighbor skin = 1.0 Angs, NVE.
• Tersoff: bulk diamond Si at 500K, cutoff = 3.2 Angs, neighbor skin = 1.0 Angs, NVE.
• Stillinger-Weber: bulk diamond Si at 500K, cutoff = 3.77 Angs, neighbor skin = 1.0 Angs, NVE.
• EIM: NaCl crystal in CsCl structure (would anneal to NaCl polycrystals), cutoff = 7.54 Angs, neighbor skin = 0.3 Angs, NPT.
• SPC/E: liquid water at 300K, SPC/E water model with LJ/Coulombic pair potential, cutoff = 9.8 Angs, neighbor skin = 2.0 Angs, long-range Coulombics via PPPM to 1.0e-4 tolerance every step, SHAKE on bonds and angle, NVT.
• CHARMM + PPPM: rhodopsin protein in solvated lipid bilayer, CHARMM force field with LJ/Coulombic pair potential, cutoff = 10.0 Angs, neighbor skin = 2.0 Angs, long-range Coulombics via PPPM to 1.0e-4 tolerance every step, SHAKE on hydrogens and water, NPT.
• MEAM: bulk fcc Ni at 800K, cutoff = 4.0 Angs, neighbor skin = 1.0 Angs, NVE.
• Peridynamics: brittle fracture of notched sodalime glass, bond-based prototype microelastic brittle (PMB) potential, cutoff = 1.5 mm, neighbor skin = 1.0 mm, NVE.
• Gay-Berne: biaxial ellipsoids at volume fraction = 0.48 and T* = 2.4, ellipsoid shape (in sigma) = 2 by 1.5 by 1, cutoff = 4.0 sigma, neighbor skin = 0.8 sigma, NPT.
• AIREBO: polyethylene chains at 300K, cutoff = 10.2 Angs for C-C interactions, neighbor skin = 0.5 Angs, NVE.
• COMB: bulk alpha-cristobalite SiO2 at 300K, short-range cutoff = 3.1 Angs, long-range Coulombics with cutoff = 12.0 Angs, neighbor skin = 0.5 Angs, QEq via EE every 10 steps to 1.0e-3 tolerance (QEq took 55% of time), NVT.
• eFF: H plasma at 40,000K, short-range cutoff = 12 Bohr = 6.35 Angs, neighbor skin = 2.0 Bohr = 1.06 Angs, NVE.
• ReaxFF: PETN crystal, cutoff = 10.0 Angs, neighbor skin = 2.0 Angs, QEq every step to 1.0e-6 tolerance, NVE. To fit 16240 atoms in one processor's memory, use provided reax_defs.h in lib/reax when compiling Fortran ReaxFF library.
• ReaxFF/C: same system as ReaxFF, run with the newer C-library version of ReaxFF in USER-REAX.
• VASP/small: 64 water molecules with 512 electrons at 300K, using a deuterium mass for H.
• VASP/medium: 64 CO2 molecules with 1024 electrons at 1000K and liquid density.
• VASP/large: 432 Xe atoms with 3456 electrons at 6000K and 10 g/cm3, a high-pressure melting calculation.

This section lists characteristics of machines used in the benchmarking along with options used in compiling LAMMPS. The communication parameters are for bandwidth and latency at the MPI level, i.e. what a program like LAMMPS sees.

Desktop = Dell 690 desktop workstation running Red Hat linux

• processor: dual quad-core 2.66 GHz Intel Xeons
• sited at Sandia National Labs (~2008)
• communication: on chip and between procs, ?? MB/sec bandwidth, ?? usec latency
• Intel compiler: icc -O

Mac laptop = PowerBook G4 running OS X 10.3

• processor: 1 GHz G4 PowerPC
• sited at Sandia National Labs (~2004)
• Mac compiler (g++): c++ -O

ASCI Red = ASCI Intel Tflops MPP

• ~9000 processors
• sited at Sandia National Labs (~2002 for upgrade, original machine ~1998)
• processor: 333 MHz Pentium III
• communication: Intel custom interconnect, 310 MB/sec bandwidth, 18 usec latency
• PGI compiler: CiCC -O4 -Knoieee

Ross = CPlant DEC Alpha/Myrinet cluster

• 1000+ processors
• sited at Sandia National labs (~2004)
• processor: 500 MHz EV6 DEC Alpha
• communication: Myrinet, 100 MB/sec bandwidth, 65 usec latency
• DEC compiler: c++ -O

Liberty = Intel/Myrinet cluster packaged by HP

• 472 procs = 236 dual-processors nodes
• sited at Sandia National Labs (~2004)
• processor: 3.06 GHz Xeon
• communication: Myrinet, 230 MB/sec bandwidth, 9 usec latency
• Intel compiler: mpiCC -O

Cheetah = IBM p690 cluster

• 864 procs = 27 32-processor nodes
• sited at Oak Ridge National Labs (~2004)
• processor: 1.3 GHz Power4
• communication: IBM Federation interconnect, 1.49 GB/sec bandwidth, 7 usec latency
• IBM compiler: mpCC_r -O4 -qnoipa

Xeon/Myrinet cluster = Spirit

• 1024 procs = 512 dual-processors nodes
• sited at Sandia National Labs (~2005)
• processor: 3.4 GHz EM64T Xeon
• communication: Myrinet, 230 MB/sec bandwidth, 9 usec latency
• Intel compiler: mpiCC -O

IBM p690+ cluster = HPCx

• 1600 procs = 50 32-processor nodes
• sited at CCLRC Daresbury Laboratory (~2005)
• processor: 1.7 GHz Power4+
• communication: IBM Federation interconnect, 1.45 GB/sec bandwidth for one link, 4.5 GB/sec bandwidth for all 4 links, 6 usec latency
• IBM compiler: mpCC_r -q64 -O3 -qarch=pwr4 -qtune=pwr4

IBM BG/L = Blue Gene Light

• 65536 (64K) procs
• sited at Lawrence Livermore National Labs (~2005)
• processor: 700 MHz PowerPC 440
• communication: IBM custom interconnect, 150 MB/sec bandwidth, 3 usec latency
• IBM compiler: blrts_xlC -O3

Cray XT3 = Red Storm

• 10368 processors
• sited at Sandia National Labs (~2005)
• processor: 2.0 GHz AMD Opteron
• communication: Cray Cstar custom interconnect, 1100 MB/sec bandwidth, 7 usec latency
• PGI 6.0 compiler: CC -fastsse

Cray XT5 = xtp

• 160 nodes, 1920 cores
• sited at Sandia National Labs (~2010)
• processor: dual hex-core 2.4 GHz AMD Opteron
• communication: Cray Cstar custom interconnect, 1100 MB/sec bandwidth, 7 usec latency
• PGI 9.0 compiler: CC -fastsse

Lincoln = GPU cluster

• 192 nodes, each with 2 CPUs and 2 GPUs
• sited at NCSA (~2010)
• CPU = 2.33 GHz Intel X5300
• GPU = NVIDIA Tesla C1060
• communication: SDR Infiniband
• NVIDIA Cuda compiler