Running on HPC

The hpc/ folder holds SLURM scripts for the examples. The 2-D cases are serial jobs; the 3-D plume-planform case is a two-rank MPI job that scales through threads. Everything a site has to change (account, partition, how Julia is put on the path) is marked SITE in the scripts.

file	purpose
hpc/plan_3d.jl	DOF count and per-rank memory estimate for each 3-D case
hpc/yinyang_convection_3d.slurm	array job over Ra = 10⁴, 10⁵, 10⁶ for examples/yinyang_convection_3d.jl
hpc/submit_3d_chain.sh	submits that array as a chain of segments, each resuming the last
hpc/annulus_convection_2d.slurm	array job over Ra for examples/annulus_convection_2d.jl

Setting up

The source code is available from the author on request until it is released with a citable publication. In your copy of the repository, install the dependencies:

cd MILEVA
julia --project=. -e 'using Pkg; Pkg.instantiate()'
julia --project=. -e 'using MPI; MPI.install_mpiexecjl()'    # puts mpiexecjl in ~/.julia/bin

By default MPI.jl uses the MPICH binary it ships. That is enough for these jobs, because the 3-D case runs two ranks on one node; the scripts launch it with mpiexecjl. To use the site MPI instead, switch MPI.jl to it once with MPIPreferences.use_system_binary() and submit with MILEVA_LAUNCH=srun, which starts the two ranks with srun.

Sizing the 3-D cases

The rank count is fixed. solve_stokes_distributed! (solvers/solve_stokes_distributed.jl) has one multigrid mode, Phase A, which puts the Yin patch on rank 0 and the Yang patch on rank 1; its multi-rank modes are MUMPS-direct or block-Jacobi Krylov, and MILEVA's rule is that 3-D Stokes is multigrid only. So each job is 2 MPI ranks, and the knobs are the memory per rank and the threads per rank (--cpus-per-task, passed on as JULIA_NUM_THREADS; the SCGS smoother in solvers/multigrid_stokes.jl is threaded).

hpc/plan_3d.jl prints the estimate for the default grids:

julia --project=. hpc/plan_3d.jl 32

Ra     Nr    Nth   Nph      DOF/patch    DOF total    LU line    MG/rank    fits?
1e+04  64    32    96          795648      1591296      6.5 GB      1.3 GB      yes
1e+05  64    48    144        1785600      3571200     14.7 GB      2.9 GB      yes
1e+06  96    72    216        6008256     12016512     49.3 GB      9.6 GB      yes

The DOF counts come from estimate_dofs_raw in utils/dof_estimator.jl. Its own Phase-A memory figure ("LU line") assumes a direct LU factor of ten times the matrix, which the multigrid path never builds; it is shown for reference. The "MG/rank" column is the multigrid estimate: the fine operator, the Galerkin hierarchy (about one seventh of the fine operator in total), the FGMRES basis (restart 50) and the field arrays, with the estimator's constants of 50 nonzeros per row and 16 bytes per entry. Treat it as a lower bound. One data point from the laptop: during the first Stokes solve at the Ra = 10⁵ grid each rank held about 1.8 GB. The array script requests --mem=48G per job (two ranks), sized for the largest case with headroom, and a heap hint of 20 GB per rank.

Estimate the wall time with a short probe

Run a five-step probe before a campaign and work out the wall time from the seconds per step it reports and the step count the CFL condition implies (the 3-D example page has the details):

MILEVA_MAXSTEPS=5 MILEVA_LOGEVERY=1 sbatch --array=1 --time=02:00:00 hpc/yinyang_convection_3d.slurm

Unknowns and estimated memory per rank for the three default 3-D grids, from estimate_dofs_raw as printed by hpc/plan_3d.jl. The direct-LU line is shown for scale only; the multigrid path the driver uses fits a 32 GB node at every Rayleigh number.

Submitting the 3-D sweep

sbatch hpc/yinyang_convection_3d.slurm            # one 48 h segment, all three Ra
sbatch --array=2 hpc/yinyang_convection_3d.slurm  # Ra = 1e6 only
hpc/submit_3d_chain.sh 6                          # six chained segments
hpc/submit_3d_chain.sh 6 2 --partition=long       # Ra = 1e6, extra sbatch options

What one array task does:

1. picks MILEVA_RA from the array index (0, 1, 2 for 10⁴, 10⁵, 10⁶) and an output directory output/hpc/yinyang_Ra<Ra>/ (base overridable with MILEVA_OUTDIR_BASE);

2. exits at once if that directory holds FINISHED;

3. sets MILEVA_RESUME=latest, so it continues from the newest ckpt_rank<r>_step<n>.jld2 if there is one;

4. reads the job's remaining time with squeue and sets MILEVA_WALLTIME_H to 30 minutes less, so the driver checkpoints and stops before SLURM kills it;

5. launches two ranks with JULIA_NUM_THREADS = SLURM_CPUS_PER_TASK and OPENBLAS_NUM_THREADS = 1.

submit_3d_chain.sh N queues N such jobs, each depending on the previous one with afterany, so a long run proceeds through the queue without attention. SLURM output goes to logs/ in the submission directory.

The 2-D sweep

sbatch hpc/annulus_convection_2d.slurm             # Ra = 1e4, 1e5 at Nr = 64; 1e6 at Nr = 128; m = 6
NR=128 M=4 sbatch hpc/annulus_convection_2d.slurm  # other grid (all tasks) or planform

Ra = 10⁶ defaults to Nr = 128 because at Nr = 64 its boundary layer is resolved by only a few cells: a Ra = 10⁶ run at Nr = 64 is a flow visualisation only, not a source of numbers.

Each task is one process with four threads, 16 GB (sized for the Nr = 128 task) and a 72 h limit. On the laptop, 40 steps at 64 × 512 took 0.41 s per step and 2.3 GB. The run length is set by the 0.3 diffusion-time cap and the CFL condition, so Ra = 10⁶ takes the most steps. GKSwstype=nul is set so figures render without a display. The plume and stagnant-lid examples need no script of their own: submit them the same way with the driver and arguments changed.

Checks done on these scripts

All three shell scripts pass bash -n. submit_3d_chain.sh was dry-run with a stand-in sbatch and produced the expected dependency chain. yinyang_convection_3d.slurm was run outside SLURM on the laptop with a tiny grid (MILEVA_NR=8 MILEVA_NTH=8 MILEVA_MGLEVELS=2 MILEVA_MAXSTEPS=2), through mpiexecjl: it launched both ranks, finished, wrote FINISHED, and a second invocation skipped the case. None of them has yet run on a cluster.