Add Docker user guide to the README file of the repository

Docker/README.md

@@ -8,19 +8,61 @@ A container is an isolated environment based on a specific operating system (OS)
 In this work the containers are created using software [Docker](https://www.docker.com). A Docker container is built from a dockerfile. After it is created, it is saved as an image that can be copied anywhere is needed or uploaded on the cloud [DockerHub](https://hub.docker.com).
 
 ## Dockerfile
-In the folder Docker/ there are two subfolders ubuntu20/ and ubuntu22/ containing a dockerfile each. A dockerfile usually begins with an image import of the environment OS, which in this case is Ubuntu-20.04 and Ubuntu-22.04. 
-For more details about the dockerfiles created in this work, refer to the DockerHub page [personalDockerHub](https://hub.docker.com/repository/docker/dcodoni/lib/general).
+In the folder Docker/ there are three subfolders solver/, ubuntu20/, ubuntu22/ containing a dockerfile each. A dockerfile usually begins with an image import of the OS, which in this case is Ubuntu-20.04 and Ubuntu-22.04. It could also start by importing an image that contain the desired environment such as in the dockerfile in the solver/ folder.
+For more details about the dockerfiles created in this work, refer to the [simvascular DockerHub page](https://registry.hub.docker.com/u/simvascular).
 
 ## Build a container
-Once the dockerfile is created, it must be built to oget the final image of the container. This can be done by moving to the directory where dockerfile is and run the command:
-
-docker buildx build -t RepositoryName:tagImage .
-
-where -t allows the user to set the name for the image created. 
-If your file is not named dockerfile, then in order to build a particular file just use:
-
-docker buildx build -f filename -t RepositoryName:tagImage .
-
+In this section the steps to build the image with a pre-compiled svFSIplus solver are briefly described. To create the image from the dockerfile provided in Docker/solver, follow the steps below:
+1) build an Ubuntu-based image containing the whole environment in which svFSIplus program can be compiled. The provided dockerfiles are based on Ubuntu-20.04 and Ubuntu-22.04, but they can be easily adapted to use the latest version of Ubuntu, by changing the following line in Docker/ubuntu20/dockerfile or Docker/ubuntu22/dockerfile: 
+```
+FROM ubuntu:20.04 AS base / FROM ubuntu:22.04 AS base
+```
+to 
+```
+FROM ubuntu:latest AS base
+```
+Build the environmnet Docker image: 
+```
+cd Docker/ubuntu20 or cd Docker/ubuntu22
+```
+```
+docker build -t RepositoryName:tagImage .
+```
+where -t allows the user to set the name for the image created. For example:
+```
+docker build -t libraries:latest .
+```
+2) build the image containing the compiled svFSIplus program. This image will be based on the environment created in the previous step (libraries:latest). In order to do this, open the Docker/solver/dockerfile and modify the following lines:
+```
+FROM simvascular/libraries:ubuntu22 AS builder 
+```
+to
+```
+FROM libraries:latest AS builder
+```
+and 
+```
+FROM simvascular/libraries:ubuntu22 AS final 
+```
+to
+```
+FROM libraries:latest AS final
+```
+Build the solver image:
+```
+cd Docker/solver
+```
+```
+docker build -t solver:latest .
+```
+The image include the PETSc-based svFSIplus executable in:
+```
+/build-petsc/svFSIplus-build/bin/svfsiplus
+```
+and the Trilinos-based svFSIplus executable in:
+```
+/build-trilinos/svFSIplus-build/bin/svfsiplus
+```
 ## Run a container
 Once the image is created, it can be run interactively by running the following command:
 

README.md

@@ -29,6 +29,120 @@ The [SimVascular svFSIplus Documentation](https://simvascular.github.io/document
 
 The [svFSIplus Internal Code Documentation](https://simvascular.github.io/svfsiplus/index.html) provides documentation of the svFSIplus source code. It is automatically generated using [Doxygen](https://www.doxygen.nl).
 
+<!--- =================================================================================== -->
+<!---                             Docker container                                        -->
+<!--- =================================================================================== -->
+
+<h1 id="docker_container">  Docker </h1>
+The preferred way to use svFSIplus, is to take advantage of the provided Docker container, which include the latest version of svFSIplus pre-compiled. To use this option, Docker must be installed first. Please refer to [Docker webpage](https://www.docker.com/products/docker-desktop/) to know more about Docker and how to install it on your machine. The following steps describe how to build a Docker image or pull an existent one from DockerHub, and how to run a Docker container. The last section is a brief guide to perform the same steps but in Singularity, since HPC systems usually use Singularity to handle containers.
+
+## Docker image
+A Docker image is a read-only template that may contain dependencies, libraries, and everything needed to run a program. It is like a snapshot of a particular environment. 
+A Docker image can be created directly from a [dockerfile](https://docs.docker.com/reference/dockerfile/#:~:text=A%20Dockerfile%20is%20a%20text,can%20use%20in%20a%20Dockerfile.) or an existent image can be pulled from [DockerHub](https://hub.docker.com). For this repository, both options are available.
+The latest version of svFSIplus program is pre-compiled in a Docker image, built from a dockerfile provided in Docker/solver. The Docker image includes two different type of builds, one where the solver is compiled with Trilinos and the other one where the solver is compiled with PETSc. 
+This Docker image can be downloaded (pulled) from the dockerhub simvascular repository [simvascular/solver](https://registry.hub.docker.com/u/simvascular). To pull an image, run the command:
+```
+docker pull simvascular/solver:latest
+```
+Note that this image was built for AMD64 (x86) architecture, and it will not work on other architectures such as ARM64 (AArch64, also note that the Apple M-series processors are based on ARM-type architectures). In this case, the image has to be built from the provided dockerfile. Please refer to the README inside Docker/ folder for more information on how to build images from the provided dockerfiles. 
+
+## Docker container
+A Docker container is a running instance of a Docker image. It is a lightweight, isolated, and executable unit. 
+Once the image is created, it can be run interactively by running the following command:
+```
+docker run -it -v FolderToUpload:/NameOfFolder solver:latest
+```
+In this command:
+- -it: means run interactively Docker image
+- -v: mounts a directory 'FolderToUpload' from the host machine in the container where the directory has the name '/NameOfFolder'. For example the folder containing the mesh and the input file necessary to run a simulation should be mounted. Once inside the container we can move into the folder jsut mounted and run the simulation, for example with the following command:
+```
+mpirun -n 4 /build-trilinos/svFSIplus-build/bin/svfsiplus svFSIplus.xml
+```
+The previous command will run the solver on 4 processors using the input file svFSIplus.xml and the mesh in the folder 'FolderToUpload' mounted inside the container.
+As an example if we want to run the test case in tests/cases/fluid/pipe_RCR_3d we can proceed as follows:
+```
+docker run -it -v ~/full_path_to/tests/cases/fluid/pipe_RCR_3d:/case solver:latest 
+```
+Now we are inside the container and we run the simulation:
+```
+cd /case
+mpirun -n 4 /build-trilinos/svFSIplus-build/bin/svfsiplus svFSIplus.xml
+```
+once it finishes we can exit the container and the results will be saved inside the tests/cases/fluid/pipe_RCR_3d folder. 
+If you encounter permission problems while running mpirun, try this:
+```
+mpirun --allow-run-as-root -n 4 /build-trilinos/svFSIplus-build/bin/svfsiplus svFSIplus.xml
+```
+
+## Containers on HPC: Singularity
+Most of the HPC systems (if not all) are based on AMD64 architecture and the solver image can be directly pulled from [simvascular/solver](https://hub.docker.com/r/simvascular/solver). First of all, make sure the singularity module is loaded on the HPC system. Then, pull the solver image (it is recommended to run the following command on the compute node for example through an interactive job):
+```
+singularity pull docker://simvascular/solver:latest
+``` 
+After the pull is complete, you should have a file with extension .sif (solver image). This image contains the two executables of the svFSIplus program build with PETSc and Trilinos support, respectively. 
+In the following, we provide two example of job submission's scripts that can be used as a reference to run a simulation using the svFSIplus solver on an HPC cluster. 
+1) single-node job script:
+```
+#!/bin/bash
+#SBATCH --job-name
+#SBATCH --output
+#SBATCH --partition
+#SBATCH --nodes=1
+#SBATCH --ntasks-per-node=
+#SBATCH --mem=0 
+#SBATCH -t 48:00:00
+
+NTASKS = # number of tasks
+FOLDER_TO_BIND1 = # path to folder to bind to the container (it will be accessible to the container)
+FOLDER_TO_BIND2 = # path to folder to bind to the container (it will be accessible to the container)
+PATH_TO_IMAGE = # full path to image, including the image name (*.sif file)
+
+# For single node, no modules should be loaded to avoid incongruences between HPC and containers environments
+module purge
+
+singularity run --bind $FOLDER_TO_BIND1, $FOLDER_TO_BIND2, # and so on \
+$PATH_TO_IMAGE \
+mpirun -n $NTASKS /build-trilinos/svFSIplus-build/bin/svfsiplus svFSIplus.xml
+```
+
+2) multi-node job script
+```
+#!/bin/bash
+#SBATCH --job-name
+#SBATCH --output
+#SBATCH --partition
+#SBATCH --nodes
+#SBATCH --ntasks-per-node
+#SBATCH --mem
+#SBATCH -t 00:00:00
+
+# The following 'export' may not work on all the HPC systems
+export UCX_TLS=ib
+export PMIX_MCA_gds=hash
+export OMPI_MCA_btl_tcp_if_include=ib0
+
+NTASKS = # number of tasks
+FOLDER_TO_BIND1 = # path to folder to bind to the container (it will be accessible to the container)
+FOLDER_TO_BIND2 = # path to folder to bind to the container (it will be accessible to the container)
+PATH_TO_IMAGE = # full path to image, including the image name (*.sif file)
+
+module purge
+# Load here all the modules necessary to use the HPC MPI, for example: module load openmpi
+
+mpirun -n $NTASKS singularity run --bind $FOLDER_TO_BIND1, $FOLDER_TO_BIND2, # and so on \
+$PATH_TO_IMAGE \
+/build-trilinos/svFSIplus-build/bin/svfsiplus svFSIplus.xml
+```
+Since the multi-node relies on both MPI, the one on the HPC and the one inside the container, there may be some problems. In the following, we give a solution (workaround) for two common problems:
+- if the HPC OpenMPI was built with cuda support, then it may happen that it is expecting that OpenMPI inside the container to be built with cuda support too, which is not the case. Possible solution is to add --mca mpi_cuda_support 0:
+```
+mpirun --mca mpi_cuda_support 0 -n #TotalNumberOfTasks ... 
+```
+- if for some reason, it is complaining about not finding 'munge' then add --mca psec ^munge:
+```
+mpirun --mca psec ^munge -n #TotalNumberOfTasks ... 
+```
+
 <!--- =================================================================================== -->
 <!---                       Building the svFSIplus                                        -->
 <!--- =================================================================================== -->
@@ -273,15 +387,4 @@ A simulation can be run in parallel on four processors using
 ```
 mpiexec -np 4 svfsiplus fluid3.xml
 ```
-In this case a directory named `4-procs` containing the simulation results output will be created. Results from different processors will be combined into a single file for a given time step.
-
-<!--- =================================================================================== -->
-<!---                             Docker container                                        -->
-<!--- =================================================================================== -->
-
-<h1 id="docker_container">  Docker Container </h1>
-
-
-
-
-
+In this case a directory named `4-procs` containing the simulation results output will be created. Results from different processors will be combined into a single file for a given time step.