M.A.R.S

Mesh Adaptive Refinement for Supercomputing

Read the Full Documentation

MARS is an open-source, GPU-native mesh management library for N-dimensional elements (N <= 4). It is developed in C++20 and makes heavy use of template meta-programming so element dimensions, floating-point precision, and SFC key types are compile-time parameters — giving both compile-time performance optimizations and concise, reusable code.

The main features of MARS consist of:

1. GPU-native unstructured meshes — the mesh is built and stored entirely on the device (CUDA / HIP), with no host round-trips after load. Built on the cornerstone-octree library.

2. Space-filling-curve (SFC) domain decomposition — elements are identified by their lowest SFC corner key and load-balanced across ranks via cornerstone.

3. GPU-native finite-element and CVFEM assembly — element → DOF map → CSR sparsity → assembled matrix, all on the device, with multiple optimized assembly kernels (tensor-product, shared-memory, tensor-core variants).

4. Distributed multi-rank execution via MPI, including a per-node halo for solver communication (CUDA-aware MPI) on top of the cornerstone element halo.

5. GPU-native adaptive mesh refinement (mark → refine → rebuild → solution transfer), with multi-rank support.

6. Lazy composition — adjacency, halo, and coordinate caches are built on first access to minimize VRAM and startup time.

MARS targets multi-core CPUs and GPUs (NVIDIA via CUDA, AMD via HIP); a Kokkos backend covers the older structured-mesh path. Because the mesh stays on the device, libraries built on MARS can run further operations directly on the GPU without going through the host.

Releases & status

Current release: v0.1.0 — see CHANGELOG.md. For the honest stable / experimental / unsupported breakdown (what to rely on and what is still under development), read KNOWN_LIMITATIONS.md. The major version is 0, so the public API may change between minor releases.

Downloading MARS and its dependencies

Clone the repository. MARS has no git submodules — its dependencies (cornerstone-octree, googletest, google/benchmark) are fetched automatically by CMake at configure time, so a plain clone is all you need:

git clone https://github.com/dganellari/mars.git

A network connection is required at configure time for the dependency fetch.

Compiling M.A.R.S for serial usage:

- cd mars/
- mkdir build
- cd build
- cmake ..
- make

MARS Kokkos requirements

Mars depends on both Kokkos and Kokkos Kernels libraries.

It will automatically find Kokkos if installed into your system. It can work with kokkos standalone or with kokkos from the Trilinos library.

Mars looks for KOKKOS_DIR or TRILINOS_DIR into the environment variables. When using Trilinos it will find them from Trilinos in $TRILINOS_DIR otherwise it will look for kokkos and kokkos kernels installations at $KOKKOS_DIR.

Use -DMARS_ENABLE_KOKKOS=ON to use the feature. For more details check CMakeLists.txt.

The default when compiling MARS with Kokkos without specifing any other CMAKE flag is the Kokkos/OpenMP execution space. Kokkos should also be compiled with OpenMP support. Otherwise the default is the serial execution space.

To compile for CUDA the Cmake flag needs to be set: MARS_ENABLE_CUDA=ON. An example would be:

cmake -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_BUILD_TYPE=Release -DMARS_ENABLE_KOKKOS=ON -DMARS_ENABLE_CUDA=ON ..

If compiled for CUDA then Kokkos should also be compiled with CUDA (Kokkos_ENABLE_CUDA=ON) and CUDA_LAMBDA (Kokkos_ENABLE_CUDA_LAMBDA=ON) support.

Unstructured Mesh Support

MARS supports GPU-native unstructured meshes through integration with the Cornerstone library, enabling space-filling curve (SFC) based mesh management for complex geometries and distributed simulations.

Key Features

• GPU-Native Architecture: All data structures live in device memory (DeviceVector via Cornerstone)
• SFC-Based Partitioning: Elements identified by space-filling curve keys for optimal load balancing
• Lazy Composition: Components (adjacency, halo, coordinates) allocated on-demand to minimize VRAM usage
• Thrust Algorithms: CSR building, sorting, and reductions use GPU-optimized Thrust primitives
• MPI Integration: Multi-rank support via Cornerstone domain decomposition
• Element Support: Tetrahedra, hexahedra, triangles, and quadrilaterals

Quick Start

#include "domain.hpp"

// Create GPU-native unstructured domain (read + partition + cstone sync).
// Template params: <ElementTag, RealType, KeyType, AcceleratorTag>.
ElementDomain<HexTag, double, uint64_t, cstone::GpuTag> domain("mesh_dir", rank, numRanks);

// Components built lazily on first access (all device-side):
const auto& offsets = domain.getNodeToElementOffsets();   // builds adjacency (CSR)
domain.cacheNodeCoordinates();                            // caches decoded coords
const auto& d_x = domain.getNodeX();                      // SoA node coordinates
const auto& d_conn = domain.getElementToNodeConnectivity(); // local node IDs per element
const auto& d_owner = domain.getNodeOwnershipMap();       // 0=ghost, 1=owned, 2=shared

Build Configuration

To enable unstructured support:

• Set -DMARS_ENABLE_UNSTRUCTURED=ON during CMake configuration
• Cornerstone is fetched automatically if not found on the system
• GPU support requires -DMARS_ENABLE_CUDA=ON or -DMARS_ENABLE_HIP=ON

Example CMake command for unstructured with CUDA:

cmake .. \
  -DMARS_ENABLE_KOKKOS=OFF \
  -DMARS_ENABLE_CUDA=ON \
  -DMARS_ENABLE_TESTS=ON \
  -DMARS_ENABLE_UNSTRUCTURED=ON \
  -DMARS_ENABLE_FEM_EXAMPLES=ON \
  -DCMAKE_CUDA_ARCHITECTURES=90
make -j

CMAKE_CUDA_ARCHITECTURES is device-specific: 90 for GH200/H100, 80 for A100.

The block above already enables MARS_ENABLE_FEM_EXAMPLES, needed for the CVFEM / FEM example drivers (Poisson, CVFEM assembly, the high-order matrix-free gates). Other optional add-ons:

• -DMARS_ENABLE_HYPRE=ON — BoomerAMG preconditioner. Needed by the AMG-preconditioned solvers and the AMR / Navier-Stokes drivers; not needed for the matrix-free operator gates.
• -DMARS_ENABLE_ADIOS2=ON, -DMARS_ENABLE_VTK=ON — extra I/O backends.

See cmake/MarsOptions.cmake and cmake/MarsDependencies.cmake for the full option list.

Documentation

For comprehensive guides and API references, see:

• Unstructured Meshes Documentation

Getting started & tutorials

• Quickstart - Clone, build, generate a mesh, run your first GPU assembly
• Poiseuille Channel Flow — a From-Scratch CFD Tutorial - Incompressible Navier–Stokes, from the mesh to reading the output
• Taylor–Green Vortex (periodic) - Canonical periodic validation case

FEM

• FEM Assembly - Mesh → DOF map → sparsity → assembled CSR (GPU-native)
• CVFEM Kernels (GPU) - Assembly kernel optimization variants

Mesh & infrastructure

• ElementDomain Overview - Core mesh management class
• Mesh Reading & Partitioning - Binary mesh format and loading
• SFC Mapping - Space-filling curve based load balancing
• Adjacency Structures - CSR-based neighbor finding
• Halo Management - Ghost element handling
• Coordinate Caching - GPU SoA coordinate storage
• Characteristic Sizes - Mesh quality metrics
• GPU Acceleration - CUDA kernel implementation
• Multi-Rank Support - MPI distributed computing

Implementation Details

The unstructured backend uses:

• Lazy initialization for memory efficiency (adjacency, halo, coordinates built on-demand)
• SFC keys as connectivity for sparse global element identification
• Thrust-based CSR building via sort_by_key, reduce_by_key, exclusive_scan
• Lowest SFC corner representation (not centroids) for element identification
• Friend access patterns for zero-copy GPU operations between components

For more details, see the backend/distributed/unstructured directory and its testsuite.

Contributors

Ganellari Daniel, Zulian Patrick and Ramelli Dylan.

License

The software is realized with NO WARRANTY and it is licenzed under BSD 3-Clause license

Copyright

Copyright (c) 2015 Institute of Computational Science - USI Università della Svizzera Italiana, ETH-Z Eidgenössische Technische Hochschule Zürich

Cite MARS

If you use the MARS Serial backend please use the following bibliographic entry

#!bibtex

@misc{mars_serial,
    author = {Zulian, Patrick and Ganellari, Daniel and Rovi, Gabriele and Ramelli, Dylan},
    title = {{MARS} - {M}esh {A}daptive {R}efinement for {S}upercomputing. {G}it repository},
    url = {https://github.com/dganellari/mars},
    year = {2018}
}

If you use the MARS Distributed backends (Kokkos, AMR and Unstructured) please use the following bibliographic entry

#!bibtex

@misc{mars_distributed,
    author = {Ganellari, Daniel and Zulian, Patrick and Rovi, Gabriele and Ramelli, Dylan},
    title = {{MARS} - {M}esh {A}daptive {R}efinement for {S}upercomputing. {G}it repository},
    url = {https://github.com/dganellari/mars},
    year = {2018}
}