benchmark on c3potato

Notes on how to run the benchmarks on c3potato.

Set up

Ideally, I would like to run asv from outside the source tree. However, that ran into some issues, possibly related to the non-standard set up of the MDAnalysis/mdanalysis repo (we need to use the new feature in the JSON file "repo_subdir":"package" from merged asv PR 611 so that we can use package/setup.py). At the moment I run the benchmarks from inside the checked out repo (in the benchmarks directory) and store the results and environments elsewhere. This allows making the results a separate repo without fear of interference or having to use git subrepositories.

benchmarking/
   benchmarks/       # the MDAnalysis/benchmarks repo
       asv.conf.json
       results/      # all benchmark results
   env/              # cache with asv environments 
   html/             # html output, becomes the MDAnalysis/benchmarks gh-pages branch
repositories/
   asv/              # offical asv 
   mdanalysis/       # MDAnalysis/mdanalysis
       benchmarks/   # asv benchmark files; benchmarks are run in this directory
       package/      # source code including setup.py
       testsuite/    # unit tests
miniconda3/          # anaconda environment

I added the function

function add_miniconda () {
   echo ">> adding miniconda3 to PATH"
   # added by Miniconda3 installer
   export PATH="${HOME}/MDA/miniconda3/bin:$PATH"
}

to .bashrc so that I can add the miniconda environment on demand.

Python environment

I installed miniconda3. It is not enabled by default so to get started do

add_miniconda
source activate benchmark

and work in the benchmark environment.

Run benchmark

Python 2.7

Benchmarks take a long time so run in byobu.

cd ~/MDA/repositories/mdanalysis/benchmarks
asv run --config asv_c3potato.conf.json -e -j 4 "release-0.11.0..HEAD --merges" 2>&1 | tee asv_log.txt

We are running the benchmarks since release 0.11.0 because the transition from 0.10 to 0.11 broke so many things in the API that it is too painful to write the performance tests to also cater to the pre-0.11.0 code. (At least for now.)

Python 3.6

Starting with the development after 1.0, Python 2.7 support was dropped (June 2020). Benchmarks should run under Python 3 (currently, Python 3.6 – see MDAnalysis/benchmarks#9 and MDAnalysis/mdanalysis#2747).

(Also run with newer version of ASV, 0.4.2 instead of a 0.3.0-dev+patch.)

• change build matrix from 2.7 to 3.6
• only build history from release 0.17.0 onwards (first full Py 3 support)

Build initial history:

asv run --config asv_c3potato.conf.json -e -j 4 "release-0.17.0..HEAD --merges" 2>&1 | tee asv_36_log_0.txt

TODO: currently failing as numpy is missing from the build environment

Only code changes

Re-run benchmarks from the last release (instead of release 0.11.0), e.g.

asv run --config asv_c3potato.conf.json -e -j 4 "release-0.18.0..HEAD --merges"

New benchmarks

Just run the new benchmark NAME from the beginning

asv run --config asv_c3potato.conf.json -e -j 4 "release-0.11.0..HEAD --merges" --bench NAME

Store results

The results are stored in the git repository https://github.com/MDAnalysis/benchmarks/ in the master branch.

cd benchmarking/benchmarks
asv publish  
git add results/
git commit -m 'updated benchmarks'
git push

Process and publish results

We want to create html pages of the results and push it to the gh-pages branch of the MDAnalysis/benchmarks repo. In principle asv gh-pages should do this automatically. In practice it fails for this set up with a cryptic failure at the push step ("asv.util.ProcessError: Command '/usr/bin/git push origin gh-pages' returned non-zero exit status 1").

For right now, force-push manually:

cd benchmarking/benchmarks
asv publish     # might be superfluous
asv gh-pages --no-push
git push origin +gh-pages

Check the output on https://www.mdanalysis.org/benchmarks/.