Allow secure lua in restricted mode in pdflatex

[tobiasBora]

I'd prefer to code libraries in lua while maintaining compatibility with pdflatex, without shell-escape. While it may be possible to implement a lua interpreter in pure latex, it would be significantly simpler and more efficient if a restricted lua environment (with os.execute restricted to allowed binaries, write access allowed only in the current folder etc) would be available in restricted mode.

So I have a few questions/requests:

1. In lualatex, how do you restrict lua when it is run in restricted mode (no shell escape)? Is there a way to start a lua interpreter in the same mode as the one used in lualatex easily?
2. Would you agree to move this lua interpreter to the allowed list of packages?

[gucci-on-fleek]

(Also discussed in the comments on this TeX.SE question)

I'd prefer to code libraries in lua while maintaining compatibility with pdflatex, without shell-escape.

I don't really see much practical use for this. For this to be useful, you need to be unable to use LuaLaTeX directly, and you would only want to use this for fairly substantial calculations, because shell escape is slow enough that you wouldn't want to use it hundreds of times per document. And the vast majority of people prefer other languages (Python, Perl, Ruby, etc.) to Lua, so I doubt that this would be a very popular option.

Regarding the specific case mentioned in the TeX.SE question (journals/arXiv that don't support LuaTeX), I don't think that this solution would actually work—any journal that only supports pdfTeX is likely using a very old version of TeX Live, so adding this to TL26 won't help very much. And if you're only using this for larger calculations, you would probably want to precompute them before submitting the article anyways, which means that you wouldn't require the program to be available with restricted shell-escape.

While it may be possible to implement a lua interpreter in pure latex

Possible in the sense of “TeX is Turing-complete, so it can compute anything”, not possible in the sense of “someone could practically make this”.

In lualatex, how do you restrict lua when it is run in restricted mode (no shell escape)?

This is implemented in the engine itself, mostly in luatex-core.lua (but also in a half dozen other files).

Is there a way to start a lua interpreter in the same mode as the one used in lualatex easily?

No, but it would be fairly easy to patch the LuaTeX engine to allow this. I've been considering this for awhile now, so I'll try and submit a patch before TL26.

Would you agree to move this lua interpreter to the allowed list of packages?

Provided that it meets all the requirements for restricted shell-escape, we're theoretically willing to add essentially any program to the default allowlist. Whether or not we're willing to add a specific program depends heavily on how it's implemented, so we can't guarantee anything either way until we can review a complete implementation.

[tobiasBora]

Thanks for your answer!

I don't really see much practical use for this. For this to be useful, you need to be unable to use LuaLaTeX directly, and you would only want to use this for fairly substantial calculations

Well this is not so uncommon in journals/conferences. E.g., quoting the famous Nature journal:

If you are submitting to Snapp your files must be able to compile using pdflatex

Also, if you believe that lualatex can fully and always replace pdflatex, then why do you even bother continuing to package in texlive pdflatex, and why so much effort on latex3? (while lua seems to be a superior alternative) I will likely write my packages only in lua either when lua is included in pdflatex, or if pdflatex is officially deprecated and removed from texlive.

Regarding the compilation time, a few remarks:

• I'd better be slow that unable to compile my document at all when trying to publish it
• if lua is only used to compile once the code to an equivalent latex translation, then it might not be much slower than a native latex code
• it is also possible to cache results to avoid running shell-escape too ofter.
• I'm curious to do benchmarks: given how slow latex can be, it might be worth losing some time with shell-escape and gaining time with lua
• Is there any reason a lua interpreter is not packed directly inside pdflatex? It would give us a real and efficient programming language with the wide adoption of pdflatex.

Any journal that only supports pdfTeX is likely using a very old version of TeX Live

Hum, I don't know: journals may keep an up-to-date texlive, but don't want to bother updating their template & workflow to lualatex + provide a way to ask the user which software they want to use. Moreover, when the editor combines multiple articles into one, it is certainly simpler for them to use a single software, if some articles are compatible with pdflatex while others with lualatex, it might be a mess to combine.

not possible in the sense of “someone could practically make this”.

Ahah, curious to give it a try if I find some time.

This is implemented in the engine itself, mostly in luatex-core.lua (but also in a half dozen other files).

Yeah, I saw this file, but it seems that it is not enough alone, for instance it never checks the value 2 of shellescape, so everything related to restricted mode must live somewhere else.

No, but it would be fairly easy to patch the LuaTeX engine to allow this. I've been considering this for awhile now, so I'll try and submit a patch before TL26.

Awesome, please let me know if you do that! But I'm curious, wasn't this the point of texlua? I was quite surprised to see that texlua --shell-restricted --safer test.lua was allowing arbitrary code execution.

Provided that it meets all the requirements for restricted shell-escape, we're theoretically willing to add essentially any program to the default allowlist.

Great. Is there an official list of requirements beside i) no write in other folders than current one ii) no arbitrary program execution beside the authorized list?

Also, adding lua to this allowlist may reduce your own work: you just need to review your own lua wrapper, not any single package that may want to get into this authorized list that is not even written by you.

[gucci-on-fleek]

if you believe that lualatex can fully and always replace pdflatex,

I personally believe that LuaTeX can replace pdfTeX in the vast majority of circumstances, but (1) I'm pretty sure that I'm the only one on the TeX Live team who feels this way, and (2) there are some fairly obscure cases where LuaTeX isn't a perfect drop-in replacement for pdfTeX.

then why do you even bother continuing to package in texlive pdflatex,

Because TeX Live packages anything TeX-related that is (1) free software, (2) useful to more than 1 person, and (3) not technically incompatible with our build system. This is why we package things like “some random university with 30 students thesis class”, AmSTeX (the defunct format, not at all related to the amsmath package), mflua, etc.

and why so much effort on latex3? (while lua seems to be a superior alternative)

No one on the TeX Live team is involved with LaTeX (and expl3) development at all. Why does the LaTeX team avoid adding deep integrations with Lua like ConTeXt and OpTeX do? I myself have complained about this too, so I'm really not the right person to ask :).

if pdflatex is officially deprecated and removed from texlive.

I don't want to say that this will never happen, but TeX Live itself going defunct seems more likely than it removing pdfTeX. This won't happen for at least 30 years, if not longer.

Regarding the compilation time, a few remarks:

• I'd better be slow that unable to compile my document at all when trying to publish it

• if lua is only used to compile once the code to an equivalent latex translation, then it might not be much slower than a native latex code

• it is also possible to cache results to avoid running shell-escape too ofter.

Sure, this is all possible, but it also adds complexity, and all of this would need to be programmed in TeX, which would somewhat defeat the point of this whole exercise.

• I'm curious to do benchmarks: given how slow latex can be, it might be worth losing some time with shell-escape and gaining time with lua

pgfplots does this, and it can make a fairly big difference for sufficiently complicated plots. But unless you're doing lots of numerical calculations in a loop, using shell-escape probably won't be faster.

• Is there any reason a lua interpreter is not packed directly inside pdflatex? It would give us a real and efficient programming language with the wide adoption of pdflatex.

Sure, the version of pdfTeX with Lua integrated is known as “LuaTeX” :). LuaTeX is a direct descendent of pdfTeX, and shares most of its code, so just using LuaTeX is usually the best solution. (Yes, some publishers still require pdfTeX, but that's outside the TeX Live team's control)

Any journal that only supports pdfTeX is likely using a very old version of TeX Live

Hum, I don't know: journals may keep an up-to-date texlive, but don't want to bother updating their template & workflow to lualatex + provide a way to ask the user which software they want to use.

Yes, I'm sure that that applies to some journals, but as far as I'm aware, the vast majority use ancient versions of TeX.

This is implemented in the engine itself, mostly in luatex-core.lua (but also in a half dozen other files).

Yeah, I saw this file, but it seems that it is not enough alone, for instance it never checks the value 2 of shellescape, so everything related to restricted mode must live somewhere else.

There's also a little bit in luainit.c, which passes some options to kpathsea, which then determines if kpse_out_name_ok_silent_extended and such allow certain files or not. loslibext.c also has some shell-escape handling code.

But I'm curious, wasn't this the point of texlua?

No, the point of texlua was to have a cross-platform scripting language that TeX Live packages can assume to be present in any TeX Live installation. This is its main advantage over Python and Ruby and similar, since users will need to manually install those themselves.

I was quite surprised to see that texlua --shell-restricted --safer test.lua was allowing arbitrary code execution.

When in texlua-mode, LuaTeX ignores all the shell-escape flags (you can see the comment in luainit.c above)

Provided that it meets all the requirements for restricted shell-escape, we're theoretically willing to add essentially any program to the default allowlist.

Great. Is there an official list of requirements beside i) no write in other folders than current one ii) no arbitrary program execution beside the authorized list?

That's essentially it, but it's quite a bit harder than it sounds. For example, here's the patch that removed a shell-escape bypass that I found, but it's really really hard to figure out how to exploit that, even when you can see what the fix is.

Also, adding lua to this allowlist may reduce your own work: you just need to review your own lua wrapper, not any single package that may want to get into this authorized list that is not even written by you.

Yes-ish? It's really easy to review a program that does hardly anything, but it's much harder to review a program that can run arbitrary scripts. And the next Lua-based restricted-shell-escape program that we'll probably add is memoize-extract.lua, but that won't be able to use the “safe Lua” wrapper since it depends on shelling out to GhostScript.

Still, as I've mentioned earlier, I do like the idea of a safe-by-default version of texlua (rtexlua?), so I'll probably implement this at some point.

[tobiasBora]

Thanks for this interesting conversation, I guess we agree on most points.

Sure, the version of pdfTeX with Lua integrated is known as “LuaTeX” :)

Ahah, LuaTex is a bit more than that, otherwise I guess we wouldn't need to have two binaries ^^

but it's much harder to review a program that can run arbitrary scripts

Makes some sense… but you anyway need to do basically the same work for lualatex (that can already run arbitrary scripts), so hopefully it shouldn't be much more difficult to create this extra safe texlua ;-)

Still, as I've mentioned earlier, I do like the idea of a safe-by-default version of texlua (rtexlua?), so I'll probably implement this at some point.

Perfect, can't wait for this to happen!

[kberry]

For the record, regarding requirements for shell_escape programs, I attempted to write this up (a while ago) at https://tug.org/texlive/pkgcontrib.html#shellesc. The previous section, on executables in general, is also relevant.

[cfr42]

@gucci-on-fleek

Yes-ish? It's really easy to review a program that does hardly anything, but it's much harder to review a program that can run arbitrary scripts. And the next Lua-based restricted-shell-escape program that we'll probably add is memoize-extract.lua, but that won't be able to use the “safe Lua” wrapper since it depends on shelling out to GhostScript.

That was the plan, but the current plan is to drop gs and use the new Lua-based pdf writer.

[But maybe you know this already.]

[gucci-on-fleek]

@cfr42

And the next Lua-based restricted-shell-escape program that we'll probably add is memoize-extract.lua, but that won't be able to use the “safe Lua” wrapper since it depends on shelling out to GhostScript.

That was the plan, but the current plan is to drop gs and use the new Lua-based pdf writer.

[But maybe you know this already.]

Ah, I did not know that; thanks for letting me know! That appears to be done in the luapdfrw.lua file, which seems to be a pretty clever solution.