Command Line Envar Parsing

Processing of both environmental variables and command line options is done in a collaborative manner between PMIx and PRRTE. Some of the responsibility is executed by PMIx in order to provide general support for OMPI (and other programming libraries) across direct-launch systems (e.g., Slurm and PBSPro) in a scalable manner. For example, a large application launched via "srun" shouldn't cause every process to open a common default MCA parameter file.

Therefore, the following description contains steps that may be executed by either PMIx or PRRTE. I have tried to identify which is actually performing each operation. However, in the following discussion, I will only be dealing with information pertaining to OMPI applications. PRRTE separately handles its own envar/cmd line parsing for its daemons to ensure that none of the PRRTE-related values are seen by OMPI applications.

Note that PRRTE (and its daemons) will see directives included in the PMIx default system and user-level parameter files. In general, PMIx clearly distinguishes between its server and client parameters (e.g., pmix_server_get_verbose vs pmix_client_get_verbose). However, there are overlaps in areas such as component selection within a given framework. If you truly need a clean separation between PRRTE and the OMPI application, you may need to manually start the PRRTE DVM (providing it with the desired directives) and use prun directly to launch your job.

There are now two separate MCA parameter files to be parsed for OMPI applications - one for PMIx and the other for OMPI itself. Each set consists of both a system-level default file and a user-level file. Neither of these are required to exist, nor are they required to contain anything more than comments should they exist.

Regardless of whether the application is started via prun or a host RM launcher (e.g., srun), the launcher is required to call the PMIx_server_setup_application API prior to actually initiating launch of the application. PMIx uses this opportunity to prepare the environment to be given to application processes at time of execution. For example, PMIx will parse both the PMIx system default parameter file and any user-level parameter file to include its contents. User-level directives overwrite any system-level values. In addition, PMIx will include all envars prefixed with PMIX_MCA_ - these values will overwrite any corresponding value provided by the system and user-level files.

Thus, the PMIx library will add PMIx-related envars to the application using the "system file" --> "user file" --> "environment" precedence ordering, with priority given in ascending order.

In addition, the call to PMIx_server_setup_application gives PMIx's plugins an opportunity to parse any relevant default configuration files for content. For example, the "ompi" plugin parses the OMPI default and user-level MCA parameter files and includes their values, along with any envars prefixed with "OMPI_MCA_". This requires that the corresponding programming model be identified in the call to PMIx_server_setup_application. In the case of launch via PRRTE, the "ompi" programming model is included by default when executed using the mpirun symlink, or can be explicitly identified by the --personality ompi command line option when using prun. Other launch environments will hopefully include this support, but may well use a different cmd line option to indicate the personality.

A variety of plugins may contribute to the returned data along with the "ompi" plugin. This includes the "opa" plugin which forwards the user's envars relating to that (PSM-1 and PSM-2) fabric provider. Availability of plugins is left to the relevant vendors as well as the usual component build/selection criteria.

Precedence processing of the plugin-related envars is left to the plugin authors. In the case of OMPI, precedence is processed using the "system file" --> "user file" --> "environment" ordering, with priority given in ascending order.

In each case, a flag is added to the application's environment indicating that the system/user-level default parameter files have been processed. This is done to help the OMPI application determine whether or not the host environment executed the PMIx-based support as some environments may not have implemented adequate PMIx integration to support this scale-enabling feature. OMPI v5 has been updated to look for the flag prior to opening/parsing the default files, thus making it scale better when the support is available.

Once the base environment has been established, prun begins processing of its command line in the following order:

• any --tune file1,file2 options. The --amca option has been deprecated in favor of --tune as the two heavily overlapped. The files can contain any directives that would normally be included on a cmd line. The list is parsed left-to-right, with values in the files overwriting any corresponding value obtained by a prior step.

  However, any conflicting directives contained within each file or across the files in the list will generate an error message and cause prun to exit. For example, if file1 contains a directive mca_btl_tcp_foo = 5 and file2 contains a directive mca_btl_tcp_foo = 6, prun will output an error message including the conflict (and their sources) and exit due to the ambiguous result. Collisions that specify the same value (e.g., A=5 in file1 as well as file2) will be ignored as no ambiguity exists.

• all other command line options. This includes -x directives for forwarding/setting envars as well as the usual --mca foo bar option. Any directives provided explicitly on the cmd line will override a corresponding directive provided by a prior step. However, conflicting command line directives will generate an error - e.g., including both -x A=5 and -x A=6 on a cmd line will result in an error.

  Note that this is also true for indirectly conflicting options. Thus, a directive of -x A=5 and --mca mca_base_env_list A=6;B=3 will also generate an error as prun has been given conflicting directives for the A envar. Ordering of conflicting directives on the cmd line is ignored - the existence of the conflict is all that is required to generate the error. Again, note that overwriting on the cmd line of a directive provided in a prior step is permitted.

• processing of the "override" file. OMPI allows a system administrator to specify an "override" file containing directives that supersede anything provided by the user or its environment.

The end result of the above procedure is provided to each application process in its environment at launch.