docs: Generate manual from markdown using mkdocs

[landam]

Goal

Build user manual from Markdown using mkdocs

How to test

1. Install requirements, see man/mkdocs/requirements.txt
2. Create file /tmp/code.lua:

function CodeBlock (cb)
    return pandoc.RawBlock('markdown', '```bash\n' .. cb.text .. '\n```\n')
end

3. Convert HTML files to Markdown: -> see docs: script to convert HTML manual pages to markdown #4620

for f in `find . -name *.html`; do echo $f; cat $f | sed 's#<div class="code"><pre>#<pre><code>#g' | sed 's#</pre></div>#</code></pre>#g' | pandoc --from=html --to=markdown -t gfm --lua-filter /tmp/code.lua > ${f%%.html}.md;done

4. Uncomment parts commented in 6743f1a and ab67825
5. Recompile GRASS (make distclean; ./configure ...; make)
6. Build documentation from markdown files: (cd man; make build-mkdocs)
7. Open dist.x86_64-pc-linux-gnu/docs/mkdocs/site/index.html in web browser

Tasks

Critical:

• fix CI
• clean-up mkmarkdown.py
• man : integrate changes from main
• separate makefile rules (Markdown.make, MarkdownRules.make) ?
• make install
• fix gui/script/ compilation
• fix compilation issue in raster
• fix compilation issue in man
• fix segfault in --md-description
• fix code blocks
• fix commit link

Important:

• implement build_md.py and indices
• solve warnings generated by mkdocs
• add missing meta page comments to MD files (as YAML metadata block)
• consider to keep supporting HTML in g.extension for the addons
• parser_standard_options: choose formatting than table
• index: choose better formatting than table
• replace html with md in links
• remove toc-related code from mkmarkdown.py and change mkdocs style to show page-related TOC
• fix "## DESCRIPTION" appearance from keyword list
• logo points to index.html
• site_name to "GRASS GIS 8.5.0dev Reference Manual"
• fix menu items
• implement parser_standard_options
• implement manual_gallery
• use theme footer

Nice to have:

• implement class_graphical (https://grass.osgeo.org/grass84/manuals/class_graphical.html is empty ???)
• manual_gallery: different number of images (md vs html), fix layout (run in man dir: GISBASE="/home/martin/src/grass/dist.x86_64-pc-linux-gnu" ARCH="x86_64-pc-linux-gnu" ARCH_DISTDIR="/home/martin/src/grass/dist.x86_64-pc-linux-gnu" VERSION_NUMBER=8.5.0dev VERSION_DATE=2024 python3 ./build_manual_gallery.py) and see dist.x86_64-pc-linux-gnu/docs/markdown/site/manual_gallery.html and https://grass.osgeo.org/grass84/manuals/manual_gallery.html

Tasks after merge(?)

• remove rest-related scripts
• generate man from markdown
• remove HTML-based documentation and clean-up scripts

[echoix]

I was about to approve, but did a quick other check. There's two things that stood out.

[wenzeslaus]

I get an error with !ENV:

... > make build-mkdocs

MkDocs encountered as error parsing the configuration file: could not determine a constructor for the tag '!ENV'
  in ".../dist.x86_64-pc-linux-gnu/docs/mkdocs/mkdocs.yml", line 2, column 12
make: *** [Makefile:247: build-mkdocs] Error 1

After deleting !ENV from the referenced file:

... > make build-mkdocs
ERROR   -  Config value: 'theme'. Error: Unrecognised theme name: 'material'. The available installed themes are: mkdocs, readthedocs 
ERROR   -  Config value: 'markdown_extensions'. Error: Failed loading extension "pymdownx.details". 
ERROR   -  Config value: 'plugins'. Error: The "glightbox" plugin is not installed 

Aborted with 3 Configuration Errors!
make: *** [Makefile:247: build-mkdocs] Error 1

[landam]

I get an error with !ENV:

Environmental variables are supported by mkdocs: https://www.mkdocs.org/user-guide/configuration/#environment-variables

@wenzeslaus Which version of mkdocs do you have?

[landam]

... > make build-mkdocs
ERROR - Config value: 'theme'. Error: Unrecognised theme name: 'material'. The available installed themes are: mkdocs, readthedocs
ERROR - Config value: 'markdown_extensions'. Error: Failed loading extension "pymdownx.details".
ERROR - Config value: 'plugins'. Error: The "glightbox" plugin is not installed

please install requirements: man/mkdocs/requirements.txt

See How To, step 1.

[landam]

@wenzeslaus Which version of mkdocs do you have?

Probably you have to install https://github.com/waylan/pyyaml-env-tag

[landam]

@wenzeslaus Which version of mkdocs do you have?

Probably you have to install https://github.com/waylan/pyyaml-env-tag

If so, we have to update requirements file accordingly. Let us now.

[wenzeslaus]

I used the requirements file. I have mkdocs 1.1.2. I also added pyyaml-env-tag and reinstalled. I will try to trash everything and start over, but I think I'm still the three mkdocs plugins.

[wenzeslaus]

Thanks. Starting from scratch with the pyyaml-env-tag package helped. The result (for core repo) looks great!

[wenzeslaus]

This creates a great minimal working product (assuming a conversion is done on the side). It does not break the current HTML doc build. while there is a lot of places to improve, we can work on them as PRs against main.

I have only the 2-3 comments about dependencies.

I would be also curious to see a list of special cases you encountered, i.e., what special pages we have. It is sort of hard to keep track. We have the standard options table, intro pages, family, topic and keyword lists, anything else?

[landam]

Thanks. Starting from scratch with the pyyaml-env-tag package helped. The result (for core repo) looks great!

Ok, requirements file updated in d45c0b6

[neteler]

@landam Overall it looks good to me, thanks for your hard work!

One thing I don't understand yet: if we keep both HTML and MD in parallel for a short period, how to deal with the comment parts as seen in 6743f1a? Should this now be activated in the PR?

[landam]

One thing I don't understand yet: if we keep both HTML and MD in parallel for a short period, how to deal with the comment parts as seen in 6743f1a? Should this now be activated in the PR?

@neteler No, otherwise the GRASS compilation fails (and breaks CI) since there are no MD files in repo. It can be activated in #4620 or in follow-up PR which adds generated MD files to git.

[neteler]

Tested on Fedora 40, the result, i.e. the generated manual pages based on markdown look great!

[wenzeslaus]

Thanks for addressing the dependencies. I tested the previous version as discussed in my previous review.

[neteler]

It seems that this change was missing in this PR:

--- a/lib/gis/parser_rest_md.c
+++ b/lib/gis/parser_rest_md.c
@@ -517,7 +517,7 @@ void print_escaped_for_md_keywords(FILE *f, const char *str)
                 fputc(*s, f);
             }
         }
-        fprintf(f, ".html)");
+        fprintf(f, ".md)");
     }
     else { /* first and other than second keyword */
         if (st->n_keys > 0 && strcmp(st->module_info.keywords[0], str) == 0) {
@@ -532,13 +532,13 @@ void print_escaped_for_md_keywords(FILE *f, const char *str)
                     fputc(*s, f);
                 }
             }
-            fprintf(f, ".html)");
+            fprintf(f, ".md)");
         }
         else {
             /* keyword index */
             char *str_link;
             str_link = G_str_replace(str_s, " ", "%20");
-            fprintf(f, "[%s](keywords.html#%s)", str_s, str_link);
+            fprintf(f, "[%s](keywords.md#%s)", str_s, str_link);
             G_free(str_link);
         }
     }

This fixes

[...]
WARNING -  Doc file 'd.what.rast.md' contains a link 'display.html', but the target is not found among documentation files. Did you mean 'display.md'?
WARNING -  Doc file 'd.what.rast.md' contains a link 'topic_vector.html', but the target is not found among documentation files. Did you mean 'topic_vector.md'?
WARNING -  Doc file 'd.what.vect.md' contains a link 'display.html', but the target is not found among documentation files. Did you mean 'display.md'?
WARNING -  Doc file 'd.what.vect.md' contains a link 'topic_vector.html', but the target is not found among documentation files. Did you mean 'topic_vector.md'?
WARNING -  Doc file 'd.where.md' contains a link 'display.html', but the target is not found among documentation files. Did you mean 'display.md'?
WARNING -  Doc file 'd.where.md' contains a link 'topic_sampling.html', but the target is not found among documentation files. Did you mean 'topic_sampling.md'?

[...]
WARNING -  Doc file 'g.copy.md' contains a link 'general.html', but the target is not found among documentation files. Did you mean 'general.md'?
WARNING -  Doc file 'g.copy.md' contains a link 'topic_map_management.html', but the target is not found among documentation files.
WARNING -  Doc file 'g.copy.md' contains a link 'keywords.html#copy', but the target 'keywords.html' is not found among documentation files. Did you mean
           'keywords.md#copy'?
WARNING -  Doc file 'g.dirseps.md' contains a link 'general.html', but the target is not found among documentation files. Did you mean 'general.md'?
[...]

I am not fully sure if this change is correct - it helps to get rid of above error messages but still dist.x86_64-pc-linux-gnu//docs/mkdocs/site/sql.html contains .md suffixes (e.g. dist.x86_64-pc-linux-gnu/docs/mkdocs/site/grass-sqlite.md) which should be .html.

So I kindly ask @landam and others for support.

[landam]

So I kindly ask @landam and others for support.

For record: #4740