Update FARM ver 2023 Sep

[wanghy-anl]

---

Pull Request Description

What issue does this change request address?

In original release of FARM Delta, the linear constraints in MOAS cannot be efficiently removed, which leads to sub-optimal solutions in multi-unit power dispatch
#1114

What are the significant changes in functionality due to this change request?

Updated FARM Delta, with a new function that could remove redundant linear constraints more efficiently.

---

For Change Control Board: Change Request Review

The following review must be completed by an authorized member of the Change Control Board.

• 1. Review all computer code.
• 2. If any changes occur to the input syntax, there must be an accompanying change to the user manual and xsd schema. If the input syntax change deprecates existing input files, a conversion script needs to be added (see Conversion Scripts).
• 3. Make sure the Python code and commenting standards are respected (camelBack, etc.) - See on the wiki for details.
• 4. Automated Tests should pass, including run_tests, pylint, manual building and xsd tests. If there are changes to Simulation.py or JobHandler.py the qsub tests must pass.
• 5. If significant functionality is added, there must be tests added to check this. Tests should cover all possible options. Multiple short tests are preferred over one large test. If new development on the internal JobHandler parallel system is performed, a cluster test must be added setting, in XML block, the node <internalParallel> to True.
• 6. If the change modifies or adds a requirement or a requirement based test case, the Change Control Board's Chair or designee also needs to approve the change. The requirements and the requirements test shall be in sync.
• 7. The merge request must reference an issue. If the issue is closed, the issue close checklist shall be done.
• 8. If an analytic test is changed/added is the the analytic documentation updated/added?
• 9. If any test used as a basis for documentation examples (currently found in raven/tests/framework/user_guide and raven/docs/workshop) have been changed, the associated documentation must be reviewed and assured the text matches the example.

[joshua-cogliati-inl]

Hm, we got some diffs, can this be fixed?

Output of'/home/civet/civet/build_0/raven/plugins/FARM/tests/heron_validator/FARM_Beta/HERONvalidatorFARMbeta':
Starting HERON ...
 ... RAVEN located at "/home/civet/civet/build_0/raven"
 ... Activating CONDA environment ...
/home/civet/.conda/envs/raven_libraries/bin/mamba
CONDA
raven_libraries       *  /home/civet/.conda/envs/raven_libraries
raven_libraries_alt      /home/civet/.conda/envs/raven_libraries_alt
Running HERON ...
Loading plugin "TEAL" at /home/civet/civet/build_0/raven/plugins/TEAL
 ... successfully imported "TEAL" ...
Loading plugin "HERON" at /home/civet/civet/build_0/raven/plugins/HERON
 ... successfully imported "HERON" ...
Loading plugin "SR2ML" at /home/civet/civet/build_0/raven/plugins/SR2ML
 ... successfully imported "SR2ML" ...
Loading plugin "LOGOS" at /home/civet/civet/build_0/raven/plugins/LOGOS
 ... successfully imported "LOGOS" ...
Loading plugin "SRAW" at /home/civet/civet/build_0/raven/plugins/SRAW
 ... successfully imported "SRAW" ...
Loading plugin "FARM" at /home/civet/civet/build_0/raven/plugins/FARM
 ... successfully imported "FARM" ...
Loading plugin "ExamplePlugin" at /home/civet/civet/build_0/raven/plugins/ExamplePlugin
 ... successfully imported "ExamplePlugin" ...
DISPATCHES has not been found in current conda environment. This is only needed when running the DISPATCHES workflow through HERD.
(            ) UTILS                    : Message         -> importing module /home/civet/civet/build_0/raven/plugins/FARM/tests/heron_validator/FARM_Beta/transfers
(    1.46 sec) <HERON ARMA>      : Message -> Checking ROM at "/home/civet/civet/build_0/raven/plugins/FARM/tests/heron_validator/ARMA/ARMA_Sine_12h/arma_Sine_12h.pk"
(    1.48 sec) <HERON ARMA>      : Message -> For DataGenerator <ARMA> "flex", detected:  interpolated,  clustered.
(    1.48 sec) <Template Driver> : Message -> ========================
(    1.48 sec) <Template Driver> : Message -> HERON: writing files ...
(    1.48 sec) <Template Driver> : Message -> ========================
(    1.48 sec) <Template Driver> : Message -> Wrote "outer.xml" to "/home/civet/civet/build_0/raven/plugins/FARM/tests/heron_validator/FARM_Beta/"
(    1.48 sec) <Template Driver> : Message -> Wrote "inner.xml" to "/home/civet/civet/build_0/raven/plugins/FARM/tests/heron_validator/FARM_Beta/"
(    1.49 sec) <Template Driver> : Message -> Wrote "heron.lib" to "/home/civet/civet/build_0/raven/plugins/FARM/tests/heron_validator/FARM_Beta/"
(    1.49 sec) <Template Driver> : Message -> Wrote "write_inner.py" to "/home/civet/civet/build_0/raven/plugins/FARM/tests/heron_validator/FARM_Beta/"
 ... HERON completed successfully.

Copyright 2017 Battelle Energy Alliance, LLC

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
  

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

      .---.        .------######       #####     ###   ###  ########  ###    ###
     /     \  __  /    --###  ###    ###  ###   ###   ###  ###       #####  ###
    / /     \(  )/    --###  ###    ###   ###  ###   ###  ######    ### ######
   //////   ' \/ `   --#######     #########  ###   ###  ###       ###  #####
  //// / // :    :   -###   ###   ###   ###    ######   ####      ###   ####
 // /   /  /`    '---###    ###  ###   ###      ###    ########  ###    ###
//          //..\\
===========UU====UU=============================================================
           '//||\\`
             ''``
    
RAVEN Python dependencies located and checked.
Loading plugin "TEAL" at /home/civet/civet/build_0/raven/plugins/TEAL
 ... successfully imported "TEAL" ...
Loading plugin "HERON" at /home/civet/civet/build_0/raven/plugins/HERON
 ... successfully imported "HERON" ...
Loading plugin "SR2ML" at /home/civet/civet/build_0/raven/plugins/SR2ML
 ... successfully imported "SR2ML" ...
Loading plugin "LOGOS" at /home/civet/civet/build_0/raven/plugins/LOGOS
 ... successfully imported "LOGOS" ...
Loading plugin "SRAW" at /home/civet/civet/build_0/raven/plugins/SRAW
 ... successfully imported "SRAW" ...
Loading plugin "FARM" at /home/civet/civet/build_0/raven/plugins/FARM
 ... successfully imported "FARM" ...
Loading plugin "ExamplePlugin" at /home/civet/civet/build_0/raven/plugins/ExamplePlugin
 ... successfully imported "ExamplePlugin" ...
/home/civet/civet/build_0/raven/ravenframework/CodeInterfaceClasses/Prescient/PrescientCodeInterface.py:25: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources
/home/civet/.conda/envs/raven_libraries/lib/python3.10/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/home/civet/.conda/envs/raven_libraries/lib/python3.10/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.logging')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/home/civet/.conda/envs/raven_libraries/lib/python3.10/site-packages/pkg_resources/__init__.py:2350: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(parent)
/home/civet/.conda/envs/raven_libraries/lib/python3.10/site-packages/pkg_resources/__init__.py:2871: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('mpl_toolkits')`.
Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
  declare_namespace(pkg)
/home/civet/.conda/envs/raven_libraries/lib/python3.10/site-packages/pyDOE2/doe_factorial.py:16: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
  import imp
(    0.00 sec) SIMULATION               : Message         -> Global verbosity level is "all"
( XML  UTILS ) Replaced text in <Input> with variable group "GRO_capacities"
( XML  UTILS ) Replaced text in <Output> with variable group "GRO_outer_results"
InputData: Using param spec "DataSet" to read XML node "PointSet.
(            ) UTILS                    : Message         -> importing module /home/civet/civet/build_0/raven/plugins/FARM/tests/heron_validator/FARM_Beta/write_inner.py
INPUT SPEC ERROR adding checked and unchecked to grid  in  variable len 1
( XML  UTILS ) Replaced text in <vars> with variable group "GRO_capacities"
(    0.00 sec) SIMULATION               : Message         -> Simulation started at 2023-09-15 12:20:39
(    0.00 sec) SIMULATION               : Message         -> -- Beginning MultiRun step "sweep" ... --
(    0.00 sec) STEP MULTIRUN            : Message         -> ***  Beginning initialization ***
(    0.01 sec) SAMPLER GRID             : Message         -> No restart for SAMPLER GRID
(    0.01 sec) GRID ENTITY              : Message         -> Starting initialization of grid 
(    0.01 sec) GRID ENTITY              : Message         -> Grid initialized...
(    0.01 sec) CODE MODEL               : Message         -> job "1" submitted!
(    0.01 sec) STEP MULTIRUN            : Message         -> Submitted input 1
(    0.01 sec) STEP MULTIRUN            : Message         -> ***    Initialization done    ***
(    0.01 sec) STEP MULTIRUN            : Message         -> ***       Beginning run       ***
(            ) UTILS                    : Message         -> importing module /home/civet/civet/build_0/raven/plugins/FARM/tests/heron_validator/FARM_Beta/write_inner.py
current working dir /home/civet/civet/build_0/raven/plugins/FARM/tests/heron_validator/FARM_Beta/Sweep_Runs_o/sweep/1/Sweep_Runs_i
already exists, this might imply deletion of present files
(    0.03 sec) CODE MODEL               : Message         -> Execution command submitted: python /home/civet/civet/build_0/raven/raven_framework.py inner.xml 
(    6.99 sec) STEP MULTIRUN            : Message         -> Just collected job 1 and sent to output "grid"
(    7.03 sec) STEP MULTIRUN            : Message         -> Just collected job 1 and sent to output "sweep"
(    7.03 sec) CODE MODEL               : Message         -> job "2" submitted!
(            ) UTILS                    : Message         -> importing module /home/civet/civet/build_0/raven/plugins/FARM/tests/heron_validator/FARM_Beta/write_inner.py
current working dir /home/civet/civet/build_0/raven/plugins/FARM/tests/heron_validator/FARM_Beta/Sweep_Runs_o/sweep/2/Sweep_Runs_i
already exists, this might imply deletion of present files
(    7.05 sec) CODE MODEL               : Message         -> Execution command submitted: python /home/civet/civet/build_0/raven/raven_framework.py inner.xml 
(   13.74 sec) STEP MULTIRUN            : Message         -> Just collected job 2 and sent to output "grid"
(   13.78 sec) STEP MULTIRUN            : Message         -> Just collected job 2 and sent to output "sweep"
(   13.78 sec) STEP MULTIRUN            : Message         -> ***       Run finished        ***
(   13.78 sec) STEP MULTIRUN            : Message         -> ***     Closing the step      ***
(   13.78 sec) STEP MULTIRUN            : Message         -> ***        Step closed        ***
(   13.78 sec) SIMULATION               : Message         -> -- End step sweep of type: MultiRun --

(   13.78 sec) SIMULATION               : Message         -> Run complete!




DIFF in /home/civet/civet/build_0/raven/plugins/FARM/tests/heron_validator/FARM_Beta/Sweep_Runs_o/sweep.csv: 
  Columns are not the same! Different: med_TotalActivity__electr_market__production__electricity, var_TotalActivity__SES__production__electricity, perc_5_TotalActivity__BOP__production__electricity, perc_95_TotalActivity__SES__production__electricity, samp_TotalActivity__electr_market__production__electricity, perc_95_TotalActivity__electr_market__production__electricity, std_TotalActivity__electr_market__production__electricity, var_TotalActivity__electr_market__production__electricity, min_TotalActivity__BOP__production__electricity, var_TotalActivity__BOP__production__electricity, mean_TotalActivity__BOP__production__electricity, mean_TotalActivity__SES__production__electricity, perc_95_TotalActivity__BOP__production__electricity, samp_TotalActivity__BOP__production__electricity, min_TotalActivity__SES__production__electricity, med_TotalActivity__BOP__production__electricity, perc_5_TotalActivity__electr_market__production__electricity, std_TotalActivity__BOP__production__electricity, std_TotalActivity__SES__production__electricity, samp_TotalActivity__SES__production__electricity, max_TotalActivity__electr_market__production__electricity, med_TotalActivity__SES__production__electricity, perc_5_TotalActivity__SES__production__electricity, min_TotalActivity__electr_market__production__electricity, max_TotalActivity__SES__production__electricity, mean_TotalActivity__electr_market__production__electricity, max_TotalActivity__BOP__production__electricity
(12F4/12) Diff   ( 19.37sec)/home/civet/civet/build_0/raven/plugins/FARM/tests/heron_validator/FARM_Beta/HERONvalidatorFARMbeta
FAILED:
Diff /home/civet/civet/build_0/raven/plugins/FARM/tests/heron_validator/FARM_None/HERONvalidatorFARMnone
Diff /home/civet/civet/build_0/raven/plugins/FARM/tests/heron_validator/FARM_Alpha/HERONvalidatorFARMalpha
Diff /home/civet/civet/build_0/raven/plugins/FARM/tests/heron_validator/FARM_Gamma_LTI/HERONvalidatorFARMgammaLTI
Diff /home/civet/civet/build_0/raven/plugins/FARM/tests/heron_validator/FARM_Beta/HERONvalidatorFARMbeta

PASSED: 2
SKIPPED: 6
FAILED: 4
... there were failed tests for plugin FARM!

[wanghy-anl]

Hm, we got some diffs, can this be fixed?

Joshua, I updated the gold files under HERON 731720c3. Let's see if it passes all the tests now. Thanks, Haoyu

[moosebuild]

Job Test Ubuntu 18-2 Python 3 on e143cff : invalidated by @joshua-cogliati-inl

Diff tests/framework/PostProcessors/LimitSurface/testLimitSurfacePostProcessor

[joshua-cogliati-inl]

Updates to FARM devel commit 64c5254