-
Notifications
You must be signed in to change notification settings - Fork 66
Cokurt GenTen Pele
Cyrus Harrison edited this page Mar 18, 2022
·
9 revisions
Public install of ascent with genten support on perlmutter:
/global/cfs/cdirs/alpine/software/ascent/0.8.0_genten
this was built using:
module load PrgEnv-gnu
cudatoolkit/11.4
cpe-cuda/22.02
Example full compile and link of ascent_first_light_example (non cmake case):
# load proper modules
. /global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/ascent_permutter_setup_env_gcc_cuda.sh
# copy the basic example
cp /global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/ascent_permutter_setup_env_gcc_cuda.sh /global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/ascent-install/examples/ascent/tutorial/ascent_intro/cpp/ascent_first_light_example.cpp .
# compile
g++ -std=c++11 -I /global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/conduit-0.8.2-22fkox3s7rcncoyyt4otbg6wwtulkv5a/include/conduit -I /global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/conduit-0.8.2-22fkox3s7rcncoyyt4otbg6wwtulkv5a/include -I/global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/hdf5-1.8.22-tuxzhm5l4ex3ygllox4a43ijmp4aw2mc/include -I /global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/ascent-develop-7meohvriquuxivky3mo2aborxdh6k2fr/include/ascent ascent_first_light_example.cpp -Wl,-rpath,/global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/ascent-develop-7meohvriquuxivky3mo2aborxdh6k2fr/lib -Wl,-rpath,/global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/conduit-0.8.2-22fkox3s7rcncoyyt4otbg6wwtulkv5a/lib -Wl,-rpath,/global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/hdf5-1.8.22-tuxzhm5l4ex3ygllox4a43ijmp4aw2mc/lib -Wl,-rpath,/global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/metis-5.1.0-h33tywgfjm3nhitrv5c7wdadnhjsrvno/lib -Wl,-rpath,/global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/parmetis-4.0.3-ffjm7a4jry674zmxvl67eidlkxiiqggd/lib -Wl,-rpath,/global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/umpire-6.0.0-att6v7ljfdwhy7imyk7ek3nhoavx2afp/lib -Wl,-rpath,/global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/vtk-h-0.8.1-qbabdxuwnpifi5o5fahjzwktsn3b43lo/lib -Wl,-rpath,/global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/vtk-m-1.7.1-kduhciajruu3xghtazntkdb6rqzfoorr/lib -Wl,-rpath,/global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/genten-master-a5kz7hi3hzh3syhozkzcqabe7eaqrw5r/lib /global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/ascent-develop-7meohvriquuxivky3mo2aborxdh6k2fr/lib/bcal_vtkm.o -L /global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/ascent-develop-7meohvriquuxivky3mo2aborxdh6k2fr/lib -lascent -lascent_flow -lascent_lodepng -lrover -L /global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/vtk-h-0.8.1-qbabdxuwnpifi5o5fahjzwktsn3b43lo/lib -lvtkh_rendering -lvtkh_compositing -lvtkh_filters -lvtkh_core -lvtkh_utils -lvtkh_lodepng -L /global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/vtk-m-1.7.1-kduhciajruu3xghtazntkdb6rqzfoorr/lib -lvtkm_rendering-1.7 -lvtkm_filter_contour-1.7 -lvtkm_filter_gradient-1.7 -lvtkm_filter_extra-1.7 -lvtkm_filter_common-1.7 -lvtkm_worklet-1.7 -lvtkm_source-1.7 -lvtkm_io-1.7 -lvtkm_lodepng-1.7 -lvtkm_cont-1.7 -lvtkmdiympi_nompi -L /global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/conduit-0.8.2-22fkox3s7rcncoyyt4otbg6wwtulkv5a/lib -lconduit_blueprint -lconduit_relay -lconduit -L /global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/hdf5-1.8.22-tuxzhm5l4ex3ygllox4a43ijmp4aw2mc/lib -lhdf5 -lm -ldl /global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/zlib-1.2.11-g733nra4ip5s6y7pt3lz4egiidasjthu/lib/libz.a -ldl -lrt -L /global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/genten-master-a5kz7hi3hzh3syhozkzcqabe7eaqrw5r/lib64 -lgt_higher_moments -lgentenlib -lgenten_mathlibs_c -L /global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/kokkos-3.4.00-k243ywep3feenciclbb4gtup3p6mpoia/lib64/ -lkokkoscore -lkokkoscontainers -L /opt/nvidia/hpc_sdk/Linux_x86_64/21.9/math_libs/11.4/lib64/ -lcusolver -lcublas -L /global/cfs/cdirs/alpine/software/ascent/0.8.0_genten/perlmutter/cuda/gnu/spack/opt/spack/cray-sles15-zen3/gcc-10.3.0/umpire-6.0.0-att6v7ljfdwhy7imyk7ek3nhoavx2afp/lib -lumpire /opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4/lib64/libcudadevrt.a /opt/nvidia/hpc_sdk/Linux_x86_64/21.9/cuda/11.4/lib64/libcudart_static.a -ldl -lrt -lblas -llapack -pthread -o ascent_first_light_example
Genten support has now been merged into ascent@develop.
Here is how to build ascent + genten:
git clone --recursive https://github.com/Alpine-DAV/ascent.git
cd ascent
./scripts/spack_install/nersc_perlmutter_gcc_10.3.0_cuda_11.4_genten_install.sh
# then to use, make sure you load
ml PrgEnv-gnu
ml cpe-cuda
ml cudatoolkit
This will create a build under:
$PSCRATCH/ASCENT_INSTALL
Currently, we are building the GenTen libs and VTK-m, and building VTK-m cannot be build on the login node (you will run out of memory). Thus, you need to build the dependencies (via the python command below) using a batch job or interactive node. To get an interactive node use this command:
bsub -XF -W 120 -nnodes 1 -P CSC340 -Is /bin/bash
When the allocation is granted, you will have 120 minute to complete the build, and as of the time of this writing (9/28/2021), it will be enough time to build vtkm. You will only have to build the dependencies once, and any changes to ascent can be compiled on the login node (with the correct modules loaded).
git clone -b genten --recursive https://github.com/Alpine-DAV/ascent.git
cd ascent
python3 scripts/uberenv/uberenv.py --spack-config-dir=./scripts/uberenv_configs/spack_configs/configs/olcf/summit --spec="+genten ^conduit@develop ^vtk-h@develop"
mkdir build
mkdir install
cd build
ml cuda/11.0.2
ml gcc/9.1.0
ml cmake/3.20.2
cmake -C ../uberenv_libs/[email protected] ../src/ -DCMAKE_INSTALL_PREFIX=path_to_install
make install
cd ../..
Note: the cmake file in uberenv_libs will be named differently.
Neither GenTen nor Kokkos natively integrate with a Make build system. While Ascent provides a path for this, we are not currently maintaining a path for this in the file:
ascent_install_path/share/ascent/ascent_config.mk
To add the correct lines to link against Kokkos and GenTen (and its cublas deps) the several libraries need to be added to the ASCENT_VTKM_LIB_FLAGS variable.
ASCENT_VTKM_LIB_FLAGS = -L $(ASCENT_VTKM_DIR)/lib \
-lvtkm_rendering-$(ASCENT_VTKM_VERSION) \
-lvtkm_filter_contour-$(ASCENT_VTKM_VERSION) \
-lvtkm_filter_gradient-$(ASCENT_VTKM_VERSION) \
-lvtkm_filter_extra-$(ASCENT_VTKM_VERSION) \
-lvtkm_filter_common-$(ASCENT_VTKM_VERSION) \
-lvtkm_worklet-$(ASCENT_VTKM_VERSION) \
-lvtkm_source-$(ASCENT_VTKM_VERSION) \
-lvtkm_io-$(ASCENT_VTKM_VERSION) \
-lvtkm_lodepng-$(ASCENT_VTKM_VERSION) \
-lvtkm_cont-$(ASCENT_VTKM_VERSION) \
-lvtkmdiympi_nompi \
/gpfs/alpine/world-shared/csc340/cokurt/ascent_genten/ascent/uberenv_libs/spack/opt/spack/linux-rhel8-power9le/gcc-9.1.0/genten-master-wwhycryv3yextg625h5sr2ybzgvtzzr2/lib64/libgt_higher_moments.a /gpfs/alpine/world-shared/csc340/cokurt/ascent_genten/ascent/uberenv_libs/spack/opt/spack/linux-rhel8-power9le/gcc-9.1.0/genten-master-wwhycryv3yextg625h5sr2ybzgvtzzr2/lib64/libgentenlib.a /gpfs/alpine/world-shared/csc340/cokurt/ascent_genten/ascent/uberenv_libs/spack/opt/spack/linux-rhel8-power9le/gcc-9.1.0/genten-master-wwhycryv3yextg625h5sr2ybzgvtzzr2/lib64/libgenten_mathlibs_c.a /gpfs/alpine/world-shared/csc340/cokurt/ascent_genten/ascent/uberenv_libs/spack/opt/spack/linux-rhel8-power9le/gcc-9.1.0/kokkos-3.4.00-b3gns6simasdq7qtmp5fnnv6le7dzwpn/lib64/libkokkoscontainers.a /gpfs/alpine/world-shared/csc340/cokurt/ascent_genten/ascent/uberenv_libs/spack/opt/spack/linux-rhel8-power9le/gcc-9.1.0/kokkos-3.4.00-b3gns6simasdq7qtmp5fnnv6le7dzwpn/lib64/libkokkoscore.a
Above you have to scroll to the right to see the extra libs but I'll list them here:
/gpfs/alpine/world-shared/csc340/cokurt/ascent_genten/ascent/uberenv_libs/spack/opt/spack/linux-rhel8-power9le/gcc-9.1.0/genten-master-wwhycryv3yextg625h5sr2ybzgvtzzr2/lib64/libgt_higher_moments.a
/gpfs/alpine/world-shared/csc340/cokurt/ascent_genten/ascent/uberenv_libs/spack/opt/spack/linux-rhel8-power9le/gcc-9.1.0/genten-master-wwhycryv3yextg625h5sr2ybzgvtzzr2/lib64/libgentenlib.a
/gpfs/alpine/world-shared/csc340/cokurt/ascent_genten/ascent/uberenv_libs/spack/opt/spack/linux-rhel8-power9le/gcc-9.1.0/genten-master-wwhycryv3yextg625h5sr2ybzgvtzzr2/lib64/libgenten_mathlibs_c.a
/gpfs/alpine/world-shared/csc340/cokurt/ascent_genten/ascent/uberenv_libs/spack/opt/spack/linux-rhel8-power9le/gcc-9.1.0/kokkos-3.4.00-b3gns6simasdq7qtmp5fnnv6le7dzwpn/lib64/libkokkoscontainers.a
/gpfs/alpine/world-shared/csc340/cokurt/ascent_genten/ascent/uberenv_libs/spack/opt/spack/linux-rhel8-power9le/gcc-9.1.0/kokkos-3.4.00-b3gns6simasdq7qtmp5fnnv6le7dzwpn/lib64/libkokkoscore.a
More concisely, these are the libs in linking order:
libgt_higher_moments.a
libgentenlib.a
libgenten_mathlibs_c.a
libkokkoscontainers.a
libkokkoscore.a
Also replace the ASCENT_CUDA_LIBS with this:
ASCENT_CUDA_LIBS = /usr/lib64/libcuda.so /sw/summit/cuda/11.0.3/lib64/libcublas.so /sw/summit/cuda/11.0.3/lib64/libcusolver.so /sw/summit/cuda/11.0.2/lib64/libcudadevrt.a /sw/summit/cuda/11.0.2/lib64/libcudart_static.a -ldl -lrt -lpthread
Your paths will be different, but if you followed the instructions above, then they will be located inside the uberenv_libs directory in a path similar to uberenv_libs/spack/opt/spack/linux-rhel8-power9le/gcc-9.1.0/
IMPORTANT NOTE: each time you make install ascent this config file will be overwritten as part of the installation process, but its contents will not change unless you rebuilt the dependencies. Once you make the changes to the file, keep a copy around so you don't have to grep for the libs again.