Issue 309 cl data vector

[marina-ricci]

This PR is dedicated to put together the different part of the Cluster pipeline.

[marina-ricci]

Trying to work on feeding cluster ensemble with cluster tomography :

[marina-ricci]

This is what I obtain for the ensemble profile on the 20deg2 :

The error bars are from the sample covariance.

[marina-ricci]

This is the stacked profiles in bin of redshift/richness for 20deg2. There is a clear problem with the profile, but the pipeline is running !

[marina-ricci]

There is also a problem to fix maybe on the CLMM side : when there is no background galaxy in a radial bin we want to fill the profile with None to preserve the size of the profile otherwise we can't stack.

[marina-ricci]

What was done :
we now have preliminary pipeline to compute stacked delta_sigma profiles for each richness/redshift bins. This works on the 20deg2 extracted from cosmoDC2.
Problems, limitations and to-dos so far :

• The cluster_esemble stage output pickle files. All Cluster related stage are not parallelized. This may not scale.
• There is a clear problem with a profiles that are wrong. I suspect a problem in the cluster_shear stage. We need to investigate.
• There is a problem with the radius : if a given cluster does not have background source in a given radial bin, this prevent the stack to be compute. This likely has to be changed in CLMM directly.
• We miss the last stage to read outputs and only save needed information in SACC format.

[combet]

• There is a problem with the radius : if a given cluster does not have background source in a given radial bin, this prevent the stack to be compute. This likely has to be changed in CLMM directly.

In CLMM, make_radial_profile has a keyword include_empty_bins that is False by default. Turning it to True when calling that function in TXPipe (here) should solve the problem (I think).

[marina-ricci]

Ah great! I'll update that then (Or anyone before me)

[combet]

@marina-ricci - I started completing the CLClusterShearCatalogs demo notebook, in the spirit of what you did for the counts one. Not finished and still needs work though.

[marina-ricci]

I need someone looking that Run_CL_pipeline.ipynb makes sense and runs at CCin2P3 and nersc.

[marina-ricci]

OK, we noticed some bugs at nersc but this is most probably due to the installation, as the CLMM version is too old. This PR works with CLMM v1.8.0 (not the latest version). I would like to proceed with the merge as it works at CCin2p3 and marginally at NERSC (only the clusterensemble part fails). We can (and should) update it later but this will be too big for this PR. @combet and @cavestruz I would be happy if you can accept (no need for thorough review). There are some test failing but that does not come from this PR.

[cavestruz]

I approve the PR, but a quick note that we (likely Tomomi) will need to explicitly run the notebooks on NERSC still.

[cavestruz]

Is this still cori-batch? Or perlmutter-batch?

[joezuntz]

It should be nersc-batch now.

[marina-ricci]

Yes, and there will likely be needed updates with the new CLMM version.