Shubham Goswami, PhD

Shubham Goswami, PhD

Postdoctoral Associate at University of Calgary

Setting up PETSc in OpenFOAM

6 minute read

Published:

Guide for coupling PETSc into OpenFOAM.

PETSc, also known as Portable, Extensible Toolkit for Scientific Computation, stands out as a powerful tool designed for the scalable (parallel) resolution of scientific applications governed by partial differential equations (PDEs). Recognized as the world’s most widely used parallel numerical software library for partial differential equations and sparse matrix computations, PETSc plays a pivotal role in tackling linear and nonlinear systems of equations stemming from discretizations of Partial Differential Equations. At the core of PETSc’s versatility is its organizational structure around mathematical concepts tailored for solving complex systems. This numerical library has found extensive use across various domains, and OpenFOAM is no exception. Within the realm of OpenFOAM, a dedicated library named petsc4Foam comes into play. This library introduces a solver that seamlessly integrates PETSc and its external dependencies, such as HYPRE, into diverse OpenFOAM simulations. petsc4Foam opens up a repertoire of solvers and preconditioners available not only in PETSc but also in its external libraries, like HYPRE.

Compilation steps for PETSc

Let’s walk through the steps to build or compile PETSc in OpenFOAM, a process applicable to both Ubuntu and Windows Subsystem for Linux (WSL):

  1. Start by sourcing your OpenFOAM environment. For instance:
    source ~/OpenFOAM/OpenFOAM-v2312/etc/bashrc
  2. Navigate to the ThirdParty folder:
    foam
cd $WM_THIRD_PARTY_DIR
  3. Check if PETSc dependencies are present:
    ./makePETSC

    This should prompt a warning indicating missing sources for PETSc. Note the version, e.g., petsc-x.xx.x.

       Missing sources: 'petsc-x.xx.x'
   Possible download locations for PETSC :
   https://www.mcs.anl.gov/petsc/
   https://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-lite-x.xx.x.tar.gz

    Here, petsc-x.xx.x is basically the version of PETSc, like for example petsc-3.19.2. In my case I will use the latest version, which is petsc-3.19.2.

  4. Download the PETSc source files using the provided link in the warning of the last step, just swap in the version of PETSc you require:
    wget https://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-lite-3.19.2.tar.gz

    This will download a file named petsc-lite-3.19.2.tar.gz. You will need to first un-tar or un-zip the file by using the command

    tar -xzvf petsc-lite-3.19.2.tar.gz

    This step should generate a folder named petsc-3.19.2

  5. Install essential packages - BLAS and LAPACK:
    sudo apt-get install libblas-dev liblapack-dev
  6. Build PETSc:
    ./makePETSC

    This step will take about 5 minutes, depending on your system configuration, and show up as follows

    + ./configure --prefix=/home/ubuntu-students/OpenFOAM/ThirdParty-v2312/platforms/linux64GccDPInt32/petsc-3.19.2 --PETSC_DIR=/home/ubuntu-students/OpenFOAM/ThirdParty-v2312/petsc-3.19.2 --with-petsc-arch=DPInt32 --with-clanguage=C --with-fc=0 --with-x=0 --with-cc=/usr/bin/mpicc --with-cxx=/usr/bin/mpicxx --with-debugging=0 --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 --with-shared-libraries --with-64-bit-indices=0 --with-precision=double --download-hypre
 Using python3 for Python
 ===============================================================================
                   Configuring PETSc to compile on your system
 ===============================================================================
 ===============================================================================
       Trying to download git://https://github.com/hypre-space/hypre for HYPRE
 ===============================================================================
 ===============================================================================
       Running configure on HYPRE; this may take several minutes
 ===============================================================================
 ===============================================================================
       Running make on HYPRE; this may take several minutes
 ===============================================================================
  7. If not loaded automatically, set up PETSc environment:
      export PETSC_DIR=~/OpenFOAM/ThirdParty-v2312/petsc-3.19.2/
  export PETSC_ARCH=DPInt32
  export LD_LIBRARY_PATH=$PETSC_DIR/$PETSC_ARCH/lib:$LD_LIBRARY_PATH
  8. Navigate to the OpenFOAM working directory and build petsc4Foam:
     foam # navigate to working directory
 cd modules/external-solver  # navigate to the external_solver directory
 ./Allwmake -prefix=openfoam # Install everything in FOAM_LIBBIN
  9. Verify PETSc libraries can be found and loaded:
         foamHasLibrary -verbose petscFoam

    The output of this command should be Can load "petscFoam".

  10. Before running any simulations using PETSc, it’s essential to set up the PETSc environment. This can be done by executing the commands provided in Step-7 or by using the following command:
      eval $(foamEtcFile -sh -config petsc -- -force)

First Steps

Once the PETSc library seamlessly integrates into OpenFOAM, you can validate the installation by running a tutorial case found in the external-solver directory. Here are the steps to execute the case:

  # Navigate to the tutorial directory
  cd ~/OpenFOAM/OpenFOAM-v2312/modules/external-solver/tutorials/incompressible/simpleFoam/pitzDaily
  # run the case using given run files
  ./Allrun

The output of this process should resemble the following:

  Copying incompressible/simpleFoam/pitzDaily -> run/
  Add libs (petscFoam) to controlDict
  Rename fvSolution and relink to fvSolution-petsc
  Running blockMesh on /home/goswami/OpenFOAM/OpenFOAM-v2312/modules/external-solver/tutorials/incompressible/simpleFoam/pitzDaily/run
  Running simpleFoam on /home/goswami/OpenFOAM/OpenFOAM-v2312/modules/external-solver/tutorials/incompressible/simpleFoam/pitzDaily/run

To ensure that the tutorial runs the pitzDaily case using petsc solvers, perform the following check:

  cd run/
  tail -n 20 log.simpleFoam

The output should display:

  Time = 281

  PETSc-bicg:  Solving for Ux, Initial residual = 0.000144417, Final residual = 7.09155e-06, No Iterations 2
  PETSc-bicg:  Solving for Uy, Initial residual = 0.00099692, Final residual = 6.16609e-05, No Iterations 2
  PETSc-cg:  Solving for p, Initial residual = 0.000780479, Final residual = 7.3097e-05, No Iterations 68
  time step continuity errors : sum local = 0.0034787, global = -9.27705e-05, cumulative = 0.676227
  PETSc-bicg:  Solving for epsilon, Initial residual = 0.000114612, Final residual = 4.76639e-06, No Iterations 2
  PETSc-bicg:  Solving for k, Initial residual = 0.000203968, Final residual = 1.19536e-05, No Iterations 2
  ExecutionTime = 9.61 s  ClockTime = 10 s


  SIMPLE solution converged in 281 iterations

  streamLine streamlines write:
      seeded 10 particles
      Tracks:10
      Total samples:10885
  End

  Finalizing PETSc

Here, you can observe that velocity, k, and epsilon are solved using the PETSc-bicg solver, employing the Bi-conjugate Gradient method. The pressure is solved using PETSc-cg, utilizing the Preconditioned Conjugate Gradient (PCG) iterative method.

How to use PETSc in your simulations ?

To use PETSc solvers in your simulations, make two essential modifications:

  1. Add the PETSc library to controlDict. Open your controlDict file and insert the following lines above the ‘application’ keyword:
     libs
 (
 "petscFoam"
 );
  2. Edit your fvSolution file and set the solver and preconditioner names in any solver to petsc. For instance, the solvers in the fvSolution file for the tutorial case executed in the last step should be configured as follows:”
       p
   {
       solver petsc;

       petsc
       {
           options
           {
               ksp_type  cg;
               ksp_cg_single_reduction  true;
               ksp_norm_type none;

       // With or without hypre
       #if 0
                       pc_type  hypre;
                       pc_hypre_type boomeramg;
                       pc_hypre_boomeramg_max_iter "1";
                       pc_hypre_boomeramg_strong_threshold "0.25";
                       pc_hypre_boomeramg_grid_sweeps_up "1";
                       pc_hypre_boomeramg_grid_sweeps_down "1";
                       pc_hypre_boomeramg_agg_nl "1";
                       pc_hypre_boomeramg_agg_num_paths "1";
                       pc_hypre_boomeramg_max_levels "25";
                       pc_hypre_boomeramg_coarsen_type HMIS;
                       pc_hypre_boomeramg_interp_type ext+i;
                       pc_hypre_boomeramg_P_max "1";
                       pc_hypre_boomeramg_truncfactor "0.3";
       #else
                       pc_type bjacobi;
                       sub_pc_type ilu;
       #endif
           }

           caching
           {
               matrix
               {
                   update always;
               }

               preconditioner
               {
                   update always;
               }
           }
       }

       tolerance       1e-06;
       relTol          0.1;
   }

 "(U|k|epsilon|omega|f|v2)"
 {
     solver petsc;

     petsc
     {
         options
         {
             ksp_type bicg;
             pc_type bjacobi;
             sub_pc_type ilu;
         }

         caching
         {
             matrix
             {
                 update always;
             }

             preconditioner
             {
                 update always;
             }
         }
     }

     tolerance       1e-05;
     relTol          0.1;
 }

Why use PETSc in your simulations ?

Ideally, when running OpenFOAM simulations, I’ll utilize a High-Performance Computing (HPC) cluster or, in simpler terms, a supercomputer. Most certainly, I’ll run my simulations in parallel, utilizing multiple cores. In OpenFOAM, parallel computation involves mesh and fields decomposition, parallel application execution, and post-processing of the decomposed case. The parallel execution utilizes the openMPI implementation of the Message Passing Interface (MPI). While OpenFOAM’s parallel performance is already commendable, there’s room for improvement. This is where PETSc steps in. PETSc offers exceptional scalability, allowing simulations to scale up to the order of millions of cores. Additionally, PETSc solvers outperform standard OpenFOAM solvers when simulations involve a large number of parallel cores.

For in-depth insights into PETSc, including performance, scalability, and metrics, refer to this informative Article..

You May Also Enjoy

Simulating Pure Pitching and Heaving Motions in OpenFOAM

9 minute read

Published:

Dynamic mesh techniques in Computational Fluid Dynamics (CFD) are pivotal for capturing fluid-structure interactions and other phenomena involving motion. In OpenFOAM, the prescribed motion method allows users to simulate complex scenarios using deformable meshes while maintaining computational efficiency. This article further explores the prescribed mesh motion technique, with a specific focus on simulating pure pitching and pure heaving motions. By the end of this guide, you’ll gain insights into implementing these motions in OpenFOAM and understanding their impact on flow dynamics.

Dynamic Meshes in OpenFOAM: A Deep Dive into Prescribed Mesh Motion

12 minute read

Published:

Simulating real-world fluid dynamics problems often requires the ability to handle dynamic changes in the computational domain. OpenFOAM’s dynamic mesh capabilities provide the necessary tools for this purpose. This blog explores the fascinating concept of dynamic mesh motion in OpenFOAM, specifically the mesh deformation method using prescribed motion technique. I will showcase the case setup and provide a general overview of simulating mesh motion in OpenFOAM.

Meshes in Motion: Making OpenFOAM Go with the Flow

9 minute read

Published:

Simulating flows with moving boundaries and deforming geometries requires specialized computational techniques, and OpenFOAM offers robust tools to tackle such challenges. Dynamic meshes enable mesh motion and deformation, allowing simulations to accurately capture phenomena like valve operations, moving pistons, or deforming structures. In this blog, we will explore the foundational concepts of dynamic meshes in OpenFOAM, covering the principles of mesh motion and their application in CFD. Whether you are a beginner or looking to deepen your understanding, this article sets the stage for harnessing OpenFOAM’s dynamic meshing capabilities.

3D DMD and Visualization with OpenFOAM and Python

9 minute read

Published:

Understanding the complex, dynamic behavior of fluid flows often requires more than just time-averaged statistics. Dynamic Mode Decomposition (DMD) offers a powerful, data-driven approach to uncover the temporal evolution of coherent structures within CFD datasets. In this blog, we’ll dive into the process of applying DMD to three-dimensional flow fields simulated using OpenFOAM and visualize the resulting modes with Python and ParaView. By combining these tools, we can unravel the intricate interactions that define unsteady flows, providing insights into both dominant patterns and transient dynamics.