New Parser: cloudformation template.yaml

[georgettica]

Hello!

I wanted to use cat template.yaml | jc --yaml for processing, but the yaml formatter didn't know how to cope with the schema

an example yaml snipper

# yaml-language-server: $schema=https://github.com/aws/serverless-application-model/raw/develop/samtranslator/validator/sam_schema/schema.json
# also read https://github.com/redhat-developer/vscode-yaml/issues/669 in order to make most of the warning disappear
Parameters:
  age:
    Type: Number
    Default: 10
Conditions:
  IsOld: !Equals [!Ref age, 10]

the error I get is:

$ cat template.yaml | LC_ALL=C jc -d --yaml
Traceback (most recent call last):
  File "/opt/homebrew/bin/jc", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/jc/cli.py", line 965, in main
    JcCli().run()
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/jc/cli.py", line 947, in run
    self._run()
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/jc/cli.py", line 911, in _run
    self.standard_parse_and_print()
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/jc/cli.py", line 803, in standard_parse_and_print
    self.create_normal_output()
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/jc/cli.py", line 755, in create_normal_output
    self.data_out = self.parser_module.parse(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/jc/parsers/yaml.py", line 158, in parse
    for document in yaml.load_all(data):
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/ruamel/yaml/main.py", line 476, in load_all
    yield constructor.get_data()
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 110, in get_data
    return self.construct_document(self.composer.get_node())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 125, in construct_document
    for _dummy in generator:
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 633, in construct_yaml_map
    value = self.construct_mapping(node)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 429, in construct_mapping
    return BaseConstructor.construct_mapping(self, node, deep=deep)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 244, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 147, in construct_object
    data = self.construct_non_recursive_object(node)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 181, in construct_non_recursive_object
    data = constructor(self, node)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 647, in construct_undefined
    raise ConstructorError(
ruamel.yaml.constructor.ConstructorError: could not determine a constructor for the tag '!Equals'
  in "<unicode string>", line 6, column 10

I searched online and found:

• https://github.com/gristlabs/yaml-cfn | in js but parses the cloudformation as a yaml and has more stars
• https://github.com/puresec/aws-parsecf | in python (has a parse_yaml function)

if you want me to help let me know.
if I were to do it I could add a jc --cloudformation and parse the yaml with the special parser or write one myself

the comments in the beginning of the template are there for further reading and how other tools partially implement supporting cloudformation

[kellyjonbrazil]

Cool - yeah I think that sounds like a good new parser!

[georgettica]

as seen in this issue I linked to here, I found a very nice and documented repo, that makes the code work perfectely.

I don't want to just copy it and maintain myself / push the code here and I am seeing if they are ok with extracting the code to a generic section which will be in python so I can work with it in the PR.

let me know if that seems like a good idea / if you have a different approach to this / what I wrote is not reflected in the issue I created and I will correct and reconsolidate what I wrote to fit into what I want to happen

[kellyjonbrazil]

Could work, but not sure. Since the project is coded in Golang I think you'd still need to install the Go library/dependencies before using the Python bindings, so it couldn't simply be "vendored" in like we have done with other dependencies. (see the x.509 parsers) The license seems pretty open, so that's good. This seems more like a custom parser route because even if we included a compiled version of the command and used subprocess, this would be architecture-dependent.

[georgettica]

That is great insight and I didn't even think this was an option to pull in the binary.

If the license permits, I'll try to incorporate the golang code to python (like I tried before you said but now that the license is good I can also feel better just taking this code snippet and pushing here)

I am concerned that the internal yaml implementation they have is way more stable and easy to extend than in python.

[kellyjonbrazil]

I recently made a change in the the YAML parser and cli.py where the JSON serialization happens. (in dev)

In the YAML parser I made the YAML loading less strict so it will automatically pull in the custom tags.

In cli.py I made a default encoder that converts any non-JSON-serializable objects into strings.

I think this might actually the problem for cloudformation templates and other YAML documents with custom tags.

[kellyjonbrazil]

I believe this may be covered in jc v1.25.4. Closing this issue. Let me know if it doesn't work as expected.