Command Line Interface Background / HPC Certification USE1.1-B and 1.3B

[dbarbi]

Thank you for your interest in developing and sharing lesson materials! To submit lesson materials or suggest a topic for future curricular development, please answer the questions below. Our Curriculum Development Team will follow up to suggest next steps in your lesson's trajectory. Questions? Please email [email protected].

1. What is the topic of your lesson or lesson proposal?

I am working on a series of courses preparing students and scientist for working on HPC Clusters. The "shell-novice" course is great, but does not cover all necessary knowledge defined by the HPC Certification Forum in code USE1.1-B and 1.3B (see https://www.hpc-certification.org/wiki/skill-tree/use/1/1/b ). As I see the importance of the commands listed there, but also liking the style of SW carpentries courses, I would like to develop this in Carpentries Incubator.
This is by no means meant to replace "shell-novice" or "shell-extras". In an ideal setting I can imagine going through "shell-novice" first, then "shell-extras" and finally this one, so that all HPC-relevant topics are covered.
A good name would be "hpc-shell" maybe.

2. Do you already have a draft of your lesson? You're welcome to share materials at any stage of development. If you already have drafted materials, please include a link.

No, but about to.

Thank you for sharing your lesson with The Carpentries community!

[tobyhodges]

Hi @dbarbi thank you for proposing this lesson. The topics you identify sound like a good fit for the HPC Carpentry project. One of their lessons, hpc-intro, is in The Carpentries Incubator already. Rather than starting a new lesson, I wonder if you might prefer to review which of these skills are already being covered in that lesson, and/or propose that any that are missing be added there?

Let me know what you think. If the focus of your lesson would be rather distinct from what is being covered in that hpc-intro lesson, I'll be happy to open a new repository for you to work in, but I always prefer to see the community collaborate on a single project where possible.

[ocaisa]

@dbarbi We indeed have an hpc-shell lesson under HPC Carpentry but we are actually about to archive that as we felt that it wasn't giving enough value beyond the shell-novice lesson. Instead we are moving relevant content (that isn't already covered in shell-novice) into our hpc-intro lesson.

I'm familiar with the HPC Certification Forum but, personally, I would take the knowledge items listed there as guides rather than requirements. For example, I would say that understanding the use case for /etc and /opt as mentioned in 1.3B is not a necessary skill for using an HPC system (particularly since even as a developer you usually only operate in user space).

[reid-a]

Hi @dbarbi -- this looks interesting, and in particular I think I was not aware of that skill tree you've linked to.

As @ocaisa mentioned, we recently changed our approach to the shell lesson -- we found that there was significant overlap between what a stand-alone HPC Shell lesson needs to do and what the Software Carpentry Unix Shell Intro lesson already does, and the latter has a larger maintainer community, and generally maintains a high level of quality.

Our scheme was to require Unix Shell Intro as a pre-requisite, and include the necessary HPC-specific supplementary shell material "in-line" in the first part of our HPC Intro lesson. The idea is to be able to benefit from the high quality of the Unix Shell Intro and allowing our community to focus our labor on the HPC-specific material that is our focus.

There's no guarantee that this is the right approach, of course, but it's informed by our experience, and it seems like a pretty good answer.

[dbarbi]

Dear all (@ocaisa, @tobyhodges and @reid-a) , thanks for the quick replies.

First of all, I know the HPC Carpentries site, but wasn't sure how much that integrates into the "official" Carpentries. As I really like those I didn't want to develop around / against SW Carpentries in any way. Could you please bring some light into that relation between HPC Carpentries and SW Carpentries?

Second of all, yes, there is a course called hpc-shell, that from my point of view (model supporter for more than 10 years) does not go far enough. I want to teach students as well as scientists in a way so that they end up with a basic understanding (in this case of Linux/UNIX), that is still profound enough to get them through at least 90% of what they might see in their day-to-day work.

Lastly, I find the HPC Certification codes quite helpful, and even though I also agree that some items are more helpful than others, I decided to use these codes to organize my courses. They are also used for discussion of courses on a national level (NHR in Germany). So there is some "political" reason why I want to use those as well. That is just my explanation for why I will use these codes as a guideline, that isn't really the question. The question is if I can develop this course material in SW Carpentries, HPC Carpentries, or if I should just do it for our own needs without sharing it with the community.

[tobyhodges]

First of all, I know the HPC Carpentries site, but wasn't sure how much that integrates into the "official" Carpentries. As I really like those I didn't want to develop around / against SW Carpentries in any way. Could you please bring some light into that relation between HPC Carpentries and SW Carpentries?

I think I'm best placed to answer this part. HPC Carpentry is indeed not an official part of The Carpentries organisation (Software, Library, and Data Carpentry are what we refer to as Lesson Programs: collections of lessons that we support e.g. via central administration of workshops etc). HPC Carpentry is not a Lesson Program, but has some ambitions to become one at some point in the future.

In recognition of the considerable overlap in community membership and the shared values and approach between the two organisations, you can consider HPC Carpentry right now as something like a Friend of The Carpentries: they aren't part of our family but we like them, we talk to them fairly frequently, and we want them to succeed.

The question is if I can develop this course material in SW Carpentries, HPC Carpentries, or if I should just do it for our own needs without sharing it with the community.

I'm glad I prompted this conversation, and the next step is up to you, @dbarbi. If collaborating with HPC Carpentry is not the best way to meet your objectives, and you prefer to work on a separate lesson, I am happy to open a repository in the Incubator and help you get started. Let me know how you would like to proceed.

[dbarbi]

Thanks @tobyhodges for clarifying. I am really open to both HPC Carpentries and SW Carpentries. This first course isn't really HPC-related (apart from using the codes of the HPC Certification), but I am planning on many more to come if everything works as planned.
So, if that is what is best, I would be happy to contribute my lesson to HPC Carpentry, provided that doesn't annoy the SW Carpentry people.
If I understand the HPC Carpentry page correctly, which states:

The Carpentries Incubator is for:

Collaborative lesson development (from conceptual to stable lessons).
Providing wider visibility for lessons that are being worked on.

HPC Carpentry is for:

Repository of peer-reviewed, short-format, lessons that use the teaching approach and lesson design from The Carpentries.
Getting peer-review on the content of the lesson in the way traditional journal peer-review wouldn’t be able to provide.

That means the place to start and to develop my lessons is the carpentries incubator anyway, right?

[tobyhodges]

If I understand the HPC Carpentry page correctly, which states:

The Carpentries Incubator is for:

Collaborative lesson development (from conceptual to stable lessons).
Providing wider visibility for lessons that are being worked on.

HPC Carpentry is for:

Repository of peer-reviewed, short-format, lessons that use the teaching approach and lesson design from The Carpentries.
Getting peer-review on the content of the lesson in the way traditional journal peer-review wouldn’t be able to provide.

That means the place to start and to develop my lessons is the carpentries incubator anyway, right?

I am not sure I agree completely with that description of HPC Carpentry, but that is a discussion for another day! 😆

Regardless, yes I think the Incubator is the place to be if you want to start a separate lesson. I will invite you to the organisation and create a new repository for you. Are you ok with the repository name carpentries-incubator/shell-hpc (consistent with the structure of shell-novice and shell-extras)?

[dbarbi]

Yup, shell-hpc is fine (for now). At least that is the current aim.
If you have any suggestion of how to develop for HPC Carpentry directly, let me hear it.

[reid-a]

To develop for HPC Carpentry directly, you should contribute issues or PRs to our repos. We have a coordination repo, it's mostly about policy and goals, as well as a number of lessons. The most mature and active has been mentioned here, it's HPC Intro, in the incubator. The others are listed in our HPC Carpentry Github organization. We have regular monthly co-working and coordination meetings, typically these are announced on the coordination repo, on the #hpc-carpentry channel of the Carpentries slack, and on the "discuss-hpc" mailing list on the Carpentries Topicbox.

The next coordination meeting is tomorrow, a draft agenda is up now. There are two sessions to help cover various time zones.

[dbarbi]

Ok, thanks for the info. I will try to attend tomorrow at 11 UTC. Not 100% sure if I will be able to make it though as I might be sitting in a train. I will ask about my planned lessons tomorrow and then get back to you here, ok?

[wirawan0]

Hi Dirk @dbarbi , do you have any follow-on thoughts on how you may want to proceed building your lesson after discussing with the HPC-carpentry folks in their coordination meeting in 2022-05-19?