Shell-escape

[dbrgn]

Does tectonic support the --shell-escape flag? It is used to do things like syntax highlighting with pygments (using the minted.sty package).

[pkgw]

Tectonic very explicitly does not, because the engine is intended to be fully standalone, so that it can have fully reproducible document compiles. Calling out to arbitrary programs that may or may not be available on arbitrary machines breaks that. I really want to discourage the use of shell-escape techniques as much as possible.

However, I believe that biblatex requires a shell-escape to biber (#35), and that seems to be a very popular feature. Tectonic is all about improving the user experience, so I'm willing to be practical.

[dbrgn]

The question is if there are alternatives for things like syntax highlighting :)

Might it be possible to write LaTeX packages in Rust? (One of the downsides of that would be that it's not compatible with other implementations anymore though.)

[pkgw]

I just recently had to deal with this — my solution was the listings package, which can do basic syntax highlighting without shell-escape.

I believe the correct solution for this sort of thing is not to attempt to integrate Rust (or, say, Lua) into the TeX engine, but to instead write programs that can generate (La)TeX code that's then fed into the engine. TeX is impossible to parse, but not hard to generate.

[khaledhosny]

Which is what these programs that are called via shell-escape do :)

[pkgw]

OK yes but they do it at the wrong time :-) You can't encapsulate/embed the TeX engine if it needs to run arbitrary executables at runtime — it should be a handed a complete set of inputs that it can process without needing to talk to the outside world. I don't feel like I've converged on the most persuasive way to make this case but I have an extremely strong engineering/design sense that shell-escape is a bad path to go down.

[dbrgn]

It definitely is a bad path, and documentation always warns about that flag, but I don't see a good alternative.

Maybe it could be allowed in an "impure" mode, with proper warnings?

[jtojnar]

Maybe it could be made to only access an isolated environment like chroot, or even containers with only programs specified in Tectonic.toml available.

[tecosaur]

@pkgw just my 2c since this seems to just be sitting here.

I really want --shell-escape. It may be bad practice (encapsulation etc.), and I completely get why it feels like a bad idea, but unfortunately a lot of fairly popular, very useful packages depend on this (minted, biber, to name a few).

Would it be possible to add it so that users aren't blocked from using these packages, just strongly discourage it? e.g.

$ tectonic --shell-escape file.tex
*Warning!* --shell-escape is _strongly_ discouraged,
and should be avoided unless absolutely necessary
...

[ratmice]

If it would be possible to implement minted commands as a rust implementation which doesn't require shell-escape. I have written code to syntax highlight with syntect and convert to tex which does not require shell-escape.

I just don't know yet the right way to export this code to the tex engine, or how to package themes, and language definitions for the bundle, nor whether this should be an external command, feature, or sandboxed webassembly 😬.

All these questions really belong in a separate issue, but i'd much rather put in the effort than see us cave on shell-escape.

[tecosaur]

Are you proposing creating a fork of minted that uses syntext instead of pygments? If this approach is used, I must admit I have quite a few concerns about the maintanability involved with your own rust-ed forks of minted, biber, TikZ externalise, pythontex, etc.

[ratmice]

yes, here is an example that is already using tectonic, built into a preprocessor that writes tex like pkgw's comment, recommends.

Reading though reading again he advocates against hooking into rust code to implement tex commands as I advocate above. I understand why, it's a slippery slope as you remind me but given that it can be done without sacrificing reproducibility (and it can) its at least worth considering for only the most popular packages. If only to quell the stalemate.

[tecosaur]

I see. At the moment I'm tempted by tectonic (hence my commenting here), but the main thing keeping me away is not being able to use my the packages that have become essential to my daily workflow (minted, biber,...). It sounds like at the moment I'll have to stick with the standard engines for --shell-escape, but this may become a viable option in the future.

[pkgw]

At this point, it seems to me that it would help a lot of people to add a -Z shell-escape option, so I'm coming around to the prospect of making it available. I certainly don't have the bandwidth to prepare a PR for it, but if someone else does, we'll give it careful consideration.

(I would like to implement any command-line flag with something like a -Z option, cf. #475 which I just created.)

[cyruseuros]

New to using tectonic, but with --shell-escape I really don't see the point. I already see the Makefiles being written. Might as well stick to latexmk. If something like that is necessary, install them like you do latex classes. When tectonic sees [backend=biber] it should know what to do etc.

[durka]

Do you know where to go in xetex to enable shellescape? I started poking around, but it seems the code to actually run commands is missing. In particular, xetex-shipout.c always prints "runsystem(...)... disabled." Am I looking in the wrong place?

[pkgw]

@durka The relevant code was stripped out a long time ago since it got in the way of compilation. Here are some of the relevant commits:

• 9f23984
• 0adaa8d
• 63fa2de
• 139ce1f

[rugk]

My use case to use --shell-escape is indeed minted. Why minted instead of listings? Because it is moderner, and supports more languages. (also upstream support for pygments is very good)

[breuerfelix]

I loved tectonic. As a new Latex user it was just beautiful. Installing = easy, compiling = easy, output = nice, auto downloading packages = awesome, not able to use minted = quit tectonic.

pretty sad but the lstlisting package is garbage in terms of linting. not even yaml works.

why do you wanna make it so hard with the -Z flag. what about --unsafe "--shell-escape", just appending to pdflatex everything which is in --unsafe. That would be a workaround for future problems aswell.

[rugk]

Yeah, I mean if you find another way to make minted work with tectonic and without --shell-escape that's fine too.
The use case, however, is very popular (see also the upvotes given above) and it also fits the idea of tectonic: modernizing and simplifying LaTeX.

Minted is a modern and (in contrast to listings) "proper" syntax highlighting method.

[DanySK]

No -shell-escape option is a blocker for me as well.
I totally agree with @pkgw on the fact that allowing arbitrary shell programs to run is a bad design choice and most definitely a leaky abstraction. Nevertheless, it is there, LaTeX is not going to abolish it, and consequently I believe it should be somehow supported by Tectonic as well.

[breuerfelix]

Since i cannot wait until the PR is merged (got some deadlines over here) and tectonic is by far the smoothest LaTeX compiler, i created a small helper script to use minted with tectonic with a tutorial until it is built in.

https://breuer.dev/tutorial/tectonic-minted

Just comment on the post if something is not working properly and i will try to fix it :)

[tecosaur]

For any *cough Emacs users here, I've made a minted replacement (for Org exports) that:

• produces higher-quality syntax highlighting
• compiles around 8x faster than un-cached minted
• doesn't require shell-escape

See here for a sample, and the implementation at https://tecosaur.github.io/emacs-config/config.html#pretty-code-blocks.

[DanySK]

@breuerfelix I had a similar need, and ended up switching to Rubber. Which however does not pull packages. Sic. 😞

[breuerfelix]

@DanySK also heard of it and tried alternatives. nothing is as easy-to-use as tectonic :)
if you still have to do your work, try out the script. requires 5 min to setup and once this is done, it is dead simple to add new code highlighting!

[DanySK]

@breuerfelix that's a very nice effort, but it hardly applies to documents that already have minted used across the board with inline code. I will consider it for future papers though...

[douglasrizzo]

Hi there. I have a thesis that uses minted, biber and standalone and all three packages need -shell-escape, so unfortunately tectonic is a no-no for me. But, as a LaTeX developer myself, I appreciate the effort of trying to make a reproducible compiler which automatically downloads dependencies. Linux users thank you!

[rugk]

Wohaaa! Niice! Out of nowhere. Awesome. 🎉

[charmoniumQ]

How does one enable --shell-escape with tectonic -X build?

[dbrgn]

@charmoniumQ -Z shell-escape

[charmoniumQ]

I tried that:

$ tectonic -X build -Z shell-escape
error: Found argument '-Z' which wasn't expected, or isn't valid in this context

USAGE:
    build [FLAGS]

For more information try --help

$ tectonic -Z shell-escape -X build
error: Found argument '-X' which wasn't expected, or isn't valid in this context

USAGE:
    tectonic <input> -Z <option>...

For more information try --help

$ tectonic --version               
Tectonic 0.8.0

[VojtechStep]

From browsing through the codebase, I think unstable options are not implemented for the V2 CLI (there is no call to .unstables() in the session builder), so it seems like you have to use the V1 interface if you want -Z shell-escape

[ratmice]

It is supported through Tectonic.toml for the V2 interface.

Edit: my understanding is that it is intentionally only enabled through Tectonic.toml as it furthers reproducibility for the arguments passed in not to affect the processing.

[cogumbreiro]

I wanted to clarify #38 (comment):

Edit: my understanding is that it is intentionally only enabled through Tectonic.toml as it furthers reproducibility for the arguments passed in not to affect the processing.

As #38 (comment) correctly points out, that's not the case:

$ tectonic tutorial.tex 
Running TeX ...
warning: tutorial.tex:4: 
runsystem(mkdir -p _minted-tutorial)...disabled.
error: tutorial.tex:4: Package minted Error: You must invoke LaTeX with the -shell-escape flag.

See the minted package documentation for explanation.
Type  H <return>  for immediate help
error: halted on potentially-recoverable error as specified

Now, if we use tectonic -Z shell-escape the file compiles:

$ tectonic -Z shell-escape tutorial.tex 
Running TeX ...
[ ... ]
note: running shell command: `which pygmentize && touch tutorial.aex`
/usr/bin/pygmentize
note: running shell command: `rm tutorial.aex`
[...]
Writing `tutorial.pdf` (65.32 KiB)
Skipped writing 2 intermediate files (use --keep-intermediates to keep them)

[ratmice]

@cogumbreiro Yes, my comment referred strictly to the V2 CLI e.g. tectonic -X ..., where -Z shell-escape is not a valid option.
When not using the V1 CLI -Z shell-ecape still works fine. It is only when using V2 CLI that it can only be controlled via Tectonic.toml.

[Caellian]

I wonder whether minted is the only thing people need the shell-escape for. Anyway, it's pretty straightforward to generate code blocks in bulk externally with pygments, and then just \input those.

If you're using a makefile or latexmk it's as straightforward as adding some build steps.

[ratmice]

@Caellian it's also pretty easy using tectonic as a library to just read (code blocks from) markdown files directly and parsing with something like pulldown-cmark and using syntect for syntax highlighting (or just use syntect directly I suppose), the tex generated by the following is probably fairly specific to the project, but here is some code that works like that https://github.com/ratmice/lumpy-leandoc/blob/master/src/tex_gen.rs

One of the benefits of that is it keeps the install phase down to a single cargo install command, and the build phase down to a single command too. I'd be happy to help make that code more generally useful if someone has a need for it.

Edit: I really only mention markdown here, because that is what the linked code doing, to just use syntect itself would be much simpler... The extent of the actual syntax highlighting code is the fn stylish_tex, in the previous link, and the theme loading here: https://github.com/ratmice/lumpy-leandoc/blob/master/src/syntax_hilight.rs

[Caellian]

@ratmice Right, but if you're simply working on a document it might be unpractical to roll your own version of tectonic to support that as it could take well over a week to just get familiar with it and write the logic required to simply get started writing the actual document. It makes sense only if you're trying to automate the process for a lot of documents.

[ratmice]

Yeah, I think it would take some amount of work to make it usable for everybody as a general solution, where we use something like what was discussed in #1120 to add additional fields to Tectonic.toml like syntax_highlight = ["some_src_file.rs"] and have some tectonic-highlight binary which links to tectonic, but my point is basically that it is entirely possible to get it working without any shell execution, by augmenting the main tectonic binary with some additional pre-parsers.

[pkgw]

@ratmice I do envision that sort of arrangement becoming common, and is something I expect to need for projects like tectonopedia.

That being said, compiling the full Tectonic crate is relatively hard and expensive. Approaches that don't require a full (re)build will be a lot easier on people. I have two main ideas about how one could go about this.

One direction is "outside-in": a stock tectonic executable executes other programs — so, in other words, pretty much traditional shell-escape. To achieve reproducibility, it's all about controlling and constraining the environment used by those other programs. Tectonic could potentially integrate with something like Pixi, which is based on Conda with a per-project orientation that would map really well to the Tectonic document model (and is implemented in Rust!). This idea feels quite promising because it seems to me that this approach could once-and-for-all solve the perennial biblatex version mismatching issue. And there are other cases, like generating SVG output, where it would be nice to add on functionality without having to compile everything into one ever-growing executable.

Or, you could take an "inside-out" approach, where some external program drives a stock tectonic executable. In this approach, Tectonic is more like rustc and the external driver is more like cargo. You'd need to add some kind of bidirectional communication layer for Tectonic to "ask" its parent to do some processing, but that would be doable. This seems a bit more awkward, but my engineering intuition does suggest that a sort of cargo/rustc division of responsibilities could work nicely. You could imagine having a new Rust crate, tectonic_driver or something, that allows you to build documents by invoking a separate tectonic executable, which means that you can incorporate it in various projects a lot more easily than something that needs to build and link with the full-on tectonic crate.

[ratmice]

Yeah, build times were a bit long for my project. The main advantage was that it had a single cargo install. I'm not terribly familiar with the linkme crate and if that works with shared objects, and dll's. But one other thing that could be done is building a tectonic-community binary which has an extra feature for each pre-processor which then loads into a DistributedSlice.

But I would need to play with it to understand if that would actually be a viable option.