# Scalability

The simulation of a complete three-dimensional seismic wavefield with realistic assumptions of the material properties of the medium the waves are propagating through requires the solution of a system of partial differential equations. Depending on the desired frequency range of the simulated waves and the material properties their spatial wavelengths vary. In particular, high-frequency wave propagation in low-velocity material can reduce the wavelengths dramatically and the necessary resolution of the numerical mesh to discretize the physical model can lead to an enormous amount of mesh elements.

Picture of SuperMUC. Source: Leibniz Rechenzentrum

In our ADER-DG scheme, each element may in principle have its own time step, which is also called local time stepping. This leads, however, to a highly irregular scheme with respect to communication and computation. In order to circumvent the irregularity and reduce the overhead from local time stepping we partition the whole interval of time steps in disjoint intervals, where the boundaries of those intervals are multiples of the minimum timestep. We associate a time cluster with each interval and assign each element to a time cluster depending on the element's timestep. As such we have a regular structure in each cluster and may communicate data in large chunks. Note, that the assignment is solely based on the timestep and has no geometric constraints. As such, the elements of a time cluster may be at arbitrary locations in the mesh.

The images below show our most recent strong scaling results. The code still scales on 6144 nodes even with local time-stepping and only about 550 elements per core.

Mesh with 221 million elements. We show performance per node on 3 recent supercomputers: SuperMUC (M), Shaheen II (S), and Cori (C)

Mesh with 221 million elements. Global time-stepping (GTS) would require over 3 million time-steps. The use of local time-stepping (LTS) leads to a speed-up of 6.8 on SuperMUC.