LHCb Open Data Curation scripts #154
Conversation
lhcb-YYYY-open-data-curation/LICENCE
Outdated
| @@ -0,0 +1,674 @@ | |||
| GNU GENERAL PUBLIC LICENSE | |||
There was a problem hiding this comment.
We can remove LINCENSE from this directory, since we have the top-level one already.
| @@ -0,0 +1,36 @@ | |||
| import os | |||
There was a problem hiding this comment.
BTW the file lhcb-YYYY-open-data-curation/scripts/staging/ExistingFilesOnEosPublic.txt should be probably excluded from git, since it is a moving object...
What we use elsewhere is:
- we put it into
inputsfolder (since it's not strictly speaking a "script" - we add it to
.gitignore
so, when you run the scripts, the file will live "locally" but won't be committed to the upstream data-curation repository
Would this be OK?
There was a problem hiding this comment.
It would be nice to host this somewhere from the ODP (maybe on eospublic itself?). If a new person starts working on the release or something changes and we don't commit the file it breaks the workflow because metadata_writer expects the file to be there. I can add some functionality to prepare files for staging even without this file but we should point to it as well.
| @@ -0,0 +1,39 @@ | |||
| # How to prepare the metadata for an LHCb OD release | |||
There was a problem hiding this comment.
Currently the PR has:
- lhcb-YYYY-open-data-curation/README.md
- lhcb-YYYY-open-data-curation/scripts/README.md
What about creating only one README file living in lhcb-YYYY-open-data-curation directory that would contain everything people need to know when working with scripts, and leave the scripts directory without any README?
(That's the technique we use for other data curation campaigns.)
| @@ -0,0 +1,39 @@ | |||
| #!/usr/bin/env python | |||
There was a problem hiding this comment.
Cosmetics: we try to standardise on using the black code formatter. If OK with you, we could add pass these scripts via black too...
There was a problem hiding this comment.
There are two approaches to this, one was more manual, the other one should work a bit more automatically but needs to be checked.
| @@ -0,0 +1,51 @@ | |||
| #!/bin/bash | |||
There was a problem hiding this comment.
Cosmetics: we could pass the shell scripts via shellcheck , there are several observations to improve some quote expansions etc.
There was a problem hiding this comment.
Sure, I will look into that.
|
Pushed the most up to date code for preparing OD release |
Adding LHCb metadata writer and stripping pages creation scripts. As well as the codebase with helper scripts / directories of data to write out.