Engagement with global HPC communities

[wirawan0]

G'day everyone... I've been thinking of two things these days:

1. Do we have a strategic planning / roadmap for our HPC community?

2. Shall we up the effort to engage with global HPC communities everywhere (where we can have access to them)?

If we have the strategic planning, could we make this more visible / centrally accessible (+linked to the various brainstormings). That should probably a separate issue compared to this one. In this issue, I want to bring up some high-level questions (some of which have been nagging us for awhile) which we can use to engage meaningful conversations with our peer communities:

• Who are the new people today that can benefit from (that is, in a great need of) hands-on onboarding like that provided by HPC Carpentry? What research / computing needs drive them to seek HPC-based solutions?

• Are these people mostly using command-line, MPI, etc? I.e. the traditional bread-and-butter HPC? I think the demography has shifted significantly in the last few years, that now a completely different breed of people seem to outnumber the traditionally more competent HPC users. What are their challenges?

• What are the basic knowledge & competencies needed by them to use HPC effectively? What are the steps to learn them? Could we define stepping stones, or learning paths for them?

• Is a certain technology ("X", where X could be MPI, Singularity container, Python, etc...) essential for learners to know? We have quite a bit discussions among ourselves on this. Having a community's input is going to help solidifying the direction we'd put more effort.

• How can HPC Carpentry engage with the broader cyberinfrastructure communities (I’m thinking all over the world, such as: XSEDE, CaRCC; in Europe: PRACE; in Australia: NCI & Pawsey, NESI in New Zealand, and I don’t know whom else to reach in Japan, China, South Asia, Africa) to make our effort even more relevant, applicable, and adopted by the global HPC communities?

[reid-a]

Thanks for this, these are good framing questions for a lot of the issues we have been working around, and having this conversation is very important.

To the degree there is a strategic plan, it surrounds the goal of getting our lesson material into a format and level of maturity that it can be comprehended and set up as workshops by the Carpentries organization at large, on the hoped-for future date when HPC Carpentry joins the Carpentries. At the same time, we have a lot of very valuable material that does not seem like it will fit (or fit easily or soon) into this framework, so a secondary part of the plan is to identify, and where appropriate, develop, additional material to capture that value for learners.

All of the secondary issues you cite are excellent questions. I think we don't have solid answers, and in some cases, will just need to make some decisions. I think we have done this to some extent, where we've decided that HPC Intro will not have a coding component, but will use the Amdahl's Law "stunt" code to demonstrate how to run in parallel on a cluster, and the benefits of doing so. In this sense, we've rejected "coding" as a necessary piece of HPC usage, although of course it remains a part of "advanced" HPC usage.

As for the issue of "where learners are coming from", this is also something we have some control over, and need to make a decision, I think -- presumably learners are out there coming from all directions, so part of the issue we face is which subset of learners do we want to address most effectively.

At the moment, and in the near future, HPC operations require some command-line knowledge, so IMO getting a broad spectrum of users up to speed on what you call "traditional bread-and-butter HPC" is high-value, and is something our existing material and existing expertise is well-positioned to execute on, so that seems like a logical place to focus, it's something we can get done, and serves our strategic goals.

But, as in all things Carpentries, empiricism rules -- if something we like isn't working, we need to change it.

[reid-a]

A very brief strategic plan, consisting of a few bullets and sub-bullets, was added as PR 57 against the main website repo.

Outreach ideas are part of the on-going coordination meeting agenda.

[tkphd]

@bkmgit did a great job reaching out to the owners of various forks: https://gitlab.com/hpc-carpentry/incubation-request/-/issues/3. @sebranchett even attended a Coordination meeting and contributed to our Incubator application! Let's try to keep some of the more willing respondents engaged, and explore how we can incorporate improvements on the forks.