HPDDM / PETSc Framework Webinar

The session aimed to position HPDDM within the Exa-MA framework activities, based on recent work presented at the PETSc Annual Meeting 2026.

The main objectives were to:

HPDDM is a library for the preconditioning of linear systems. It was initially developed as a standalone library, but significant effort has recently been devoted to integrating it more tightly into PETSc.

The recent work consists in progressively replacing HPDDM native internal operations with PETSc data structures and PETSc operations. This makes the code more generic, easier to maintain, and better aligned with PETSc backends, in particular for GPU execution.

Within PETSc, the main component relevant to Exa-MA is PCHPDDM, i.e. HPDDM used as a preconditioner. The KSPHPDDM layer exists, but was not considered central for the current Exa-MA applications, since it is mostly useful for specific Krylov methods such as block methods.

The PETSc interface allows the user or the calling framework to:

A key point is that HPDDM is not used as a standalone solver here. It acts as a preconditioner inside a PETSc linear solver workflow. The Krylov method remains selected through PETSc, and the local subdomain solvers can be direct or iterative. By default, exact local factorizations are often used on subdomains, but this can be changed through PETSc options.

GENEO is the central robust two-level method in HPDDM for SPD problems. It constructs a coarse space from local generalized eigenvalue problems defined on overlapping subdomains. Its main strength is numerical robustness, with theoretical guarantees on the conditioning of the preconditioned operator.

The counterpart is that GENEO is not fully algebraic. It requires Neumann matrices on the overlapping subdomains. These matrices must use natural boundary conditions on the artificial interfaces induced by the domain decomposition, while preserving the essential boundary conditions of the original problem.

This means that the calling framework plays an important role. If the framework has ghost elements or a suitable representation of overlapping subdomains, assembling the Neumann matrices can be relatively natural. This is the case, or could be the case, for frameworks such as FreeFEM, Code_Aster, Gridap.jl, MFEM, and potentially Feel++ once the proper interface is available.

When Neumann matrices are not available, HPDDM can still be used through algebraic variants. These are easier to activate, often directly through PETSc options, but they do not provide the same theoretical guarantees as GENEO.

The discussion clarified the practical positioning of HPDDM.

When algebraic multigrid works well, it is often faster than two-level domain decomposition methods. However, AMG may be sensitive to geometry, mesh quality, coefficient contrasts, discretization choices, and parameter tuning.

HPDDM/GENEO is usually more expensive in setup and sometimes per iteration, but it is more robust in difficult cases. It is especially relevant when the solver must remain reliable under changes in geometry, mesh, discretization, or physical regime.

Two examples were discussed.

A major development direction is the decoupling of the number of MPI processes from the number of subdomains.

Two configurations are targeted:

Similar ideas are already available in PETSc for some preconditioners, such as PCASM, PCGASM, and PCBJACOBI, for example through parameters such as nblocks.

The objective is to bring the same flexibility to PCHPDDM, so that the restricted Schwarz part and the GENEO coarse-space part support the same granularity. This is also relevant for exascale architectures. Several subdomains per MPI process may enable efficient use of batched linear algebra, for example through Kokkos.

The current HPDDM mini-app is located in the apps-petsc repository, in:

It is not an HPDDM-only application, but rather a PETSc application using HPDDM as a linear solver/preconditioner component. The mini-app comes from a discrete fracture network application developed by the SERENA team.

The mini-app demonstrates the full PETSc/HPDDM workflow:

Only a small part of the code is strictly HPDDM-specific. Most of the code is PETSc boilerplate for loading data and reconstructing the correct parallel distribution.

The mini-app is therefore useful as a reference workflow, but it would be useful to produce a simplified version, stripped of the many regression-test-specific branches inherited from PETSc examples.

Larger datasets are being prepared by the SERENA team. These would make it possible to move from a small demonstrator to significantly larger benchmark cases, potentially up to several hundred million degrees of freedom.

Recent work has moved a significant part of the preconditioner application to the GPU when the matrices are already available on the device.

The ideal workflow is:

Several limitations remain:

The DFN mini-app is considered GPU-ready in the sense that the matrix can be loaded on the CPU and then converted to a PETSc GPU matrix type, for example MPIAIJCUSPARSE, before setup and solve. This is not optimal from an I/O point of view because PETSc still loads matrices from disk through the CPU, but it is sufficient to benchmark the solver and preconditioner phases.

This limitation should be explicitly stated in the benchmark report to avoid mixing I/O costs with solver/preconditioner performance.

Regarding AMD, PETSc has a HIP backend, largely derived from the CUDA backend. However, some pieces are still missing in PCHPDDM. Access to AMD hardware is therefore necessary to test and improve this path.

It was agreed that the HPDDM team should be added to the AMD/CINES call in order to test the HIP backend. Kokkos is also seen as an important path toward CUDA/AMD portability, especially for batched linear algebra.

The discussion confirmed that HPDDM is not limited to SPD problems. Saddle point systems are important for Exa-MA and can be handled through PETSc flexibility, in particular through PCFIELDSPLIT.

For Stokes systems and nearly incompressible elasticity, the work of Frederic Nataf and Pierre-Henri Tournier can be used in the PETSc/HPDDM context. The idea is to apply HPDDM to relevant blocks, for instance the A00 block, and possibly to Schur-complement approximations.

For non-symmetric systems, such as Oseen or R13-type equations, several strategies are available:

The theory is less strong than for SPD/GENEO cases, but PETSc/HPDDM already provides practical options to test.

For Oseen, GENEO may work in low-convection regimes, but may fail when the problem becomes convection-dominated. In such cases, algebraic HPDDM variants are more appropriate.

Several Exa-MA use cases were identified where HPDDM should be activated or evaluated.

The meeting confirms that HPDDM is an important component of the Exa-MA software and algorithmic ecosystem, especially for cases where solver robustness is more important than minimizing cost per iteration.

The PETSc integration makes HPDDM accessible to several Exa-MA frameworks with a relatively transparent user workflow, provided that the framework exposes the required matrices and PETSc options.

In the short term, the priority is to produce a clean benchmark of the DFN mini-app, first on CPU and then on GPU if possible. HPDDM should also be tested on at least one application outside apps-petsc, with the MFEM thermo-mechanical benchmark as the most immediate candidate.

In the medium term, the focus should be on consolidating GPU support, testing AMD/HIP, evaluating cuDSS, and integrating HPDDM as a solver/preconditioner option in Feel++, TRUST, and other PETSc-based Exa-MA mini-apps.