Profiling the GPU build

Note

This applies to a GPU (nvc) build running in mode 2 (-mpm 2 / multiprocessor_mode=2 / set_gpu_offload(True)). See Installing for GPU (NVIDIA HPC SDK / nvc) and Compute modes: legacy vs unified (and GPU offload).

The NVIDIA HPC SDK ships the two NVIDIA profilers, plus a memory checker:

Nsight Systems (nsys) — a whole-application timeline: kernel launches, host↔device transfers, CPU gaps, MPI. Start here to see where time goes and which kernels dominate.
Nsight Compute (ncu) — a per-kernel deep dive: occupancy, memory throughput, roofline, warp stalls. Use it on the one or two kernels nsys identifies as hot.
compute-sanitizer — races / out-of-bounds / uninitialised device memory.

Setup

Put the SDK tools on PATH (define NVC_BASE first):

export NVC_BASE=/opt/nvidia/hpc_sdk/Linux_x86_64/26.3
export PATH=$NVC_BASE/compilers/bin:$PATH

Then ncu, nsys, ncu-ui, nsys-ui and compute-sanitizer are on the path. (Adjust the 26.3 version to your installed SDK.)

GPU performance counters. ncu reads restricted GPU counters, which by default require elevated privileges — run it with sudo (as below), or have an admin allow non-root profiling once via the driver option NVreg_RestrictProfilingToAdminUsers=0. nsys timeline profiling does not need sudo.

Whole-run timeline (Nsight Systems)

nsys profile -o anuga_timeline -f true \
  python run_small_towradgi.py -mpm 2 -ft 200

This writes anuga_timeline.nsys-rep. List the hottest kernels from it with:

nsys stats --report cuda_gpu_kern_sum anuga_timeline.nsys-rep

The kernel-summary table gives the exact kernel names (see below) to hand to ncu -k.

Per-kernel deep dive (Nsight Compute)

Profile a single launch of the flux kernel (the main compute kernel), skipping warmup, collecting the full metric set:

sudo $NVC_BASE/compilers/bin/ncu \
  -k "nvkernel_core_compute_fluxes_central_F1L936_36" \
  -c 1 -s 5 --set=full \
  -o my_profile_anuga_fluxes -f \
  $(which python) run_small_towradgi.py -mpm 2 -ft 200

Flags:

Flag	Meaning
`-k <regex>`	Only profile kernels whose name matches (here, the flux kernel).
`-s 5`	Skip the first 5 matching launches (past JIT/warmup, into steady state).
`-c 1`	Collect 1 launch after the skip (per-kernel profiling replays the launch many times, so keep the count small).
`--set=full`	The full metric set (roofline, memory workload, occupancy, …). Thorough but slow; use `--set=basic` or `--set=roofline` for a quick pass.
`-o <name>`	Report file `<name>.ncu-rep`.
`-f`	Overwrite an existing report.

Because ncu replays each profiled kernel many times, always narrow it with -k and a small -c — never profile a whole ANUGA run unfiltered.

Kernel names

nvc names each OpenMP target (offload) region nvkernel_<function>_F<file>L<line>_<index>. For example nvkernel_core_compute_fluxes_central_F1L936_36 is the omp target region at line 936 of anuga/shallow_water/gpu/core_kernels.c, inside core_compute_fluxes_central (the F1 is the compiler’s file index and _36 its kernel index). The build prints these function: line regions as it compiles; nsys stats (above) or an unfiltered ncu also lists them.

Viewing the reports

Open interactively (GUI needed):

ncu-ui  my_profile_anuga_fluxes.ncu-rep     # Nsight Compute report
nsys-ui anuga_timeline.nsys-rep             # Nsight Systems timeline

Or copy the .ncu-rep / .nsys-rep file to a workstation and open it in the Nsight GUIs there. For a quick terminal summary of an ncu report:

ncu --import my_profile_anuga_fluxes.ncu-rep --page details | less