[discussion] Static Perl compilation without PAR + make optional / drop more dependencies #338
Comments
|
I would be interested in helping to look into this. |
|
Cool :) I'll post here my building attempts |
|
As part of prep work, I'm working on compiling cperl (a small Perl compiler that supports perlcc in a more stable way) statically with all modules in order to run tlmgr from C code (a smaller Perl codebase than biber). If that works, that could be an alternative way to WebPerl (WebPerl does same for regular Perl), and it would also run natively. If you're interested to also work on this, feel free to slide into perl11/cperl#423 (comment) |
|
An update: I wanted to see if one can get a fully static build (Emscripten has a dynamic linking support, but it's new; and a static build is valuable on its own for embedding in other programs).
|
|
@plk What is the minimum set of biber's dependencies for a minimial local-only working setup? (without downloading stuff from internet, without running external programs) Full list from Build.pl: |
|
That is the minimum dependencies you have listed there ... |
|
E.g. Mozilla::CA or LWP should not be needed in theory if consulting internet is disabled, right? Same for IPC. Does biber require running external programs for minimal functioning? |
|
Well, I'd have to see whether things would work without them as their integration isn't really that modular as it was never designed to be used without them. |
|
Okay! In the meanwhile I'll try to build it with all dependencies. but for sure it'd be more robust for these cross-compilation scenarios (and have smaller binary size!) if there is a mode "no-calling-external-programs + no-downloading-from-internet" that enables running even if LWP/IPC are not installed/present |
|
Do you know how to skip tests during It may be that |
Why does the https://github.com/plk/biber/blob/dev/dist/linux_x86_64/build.sh PAR run command contain much fewer dependencies? namely, only --module=Pod::Simple::TranscodeSmart \
--module=Pod::Simple::TranscodeDumb \
--module=List::MoreUtils::XS \
--module=List::SomeUtils::XS \
--module=List::MoreUtils::PP \
--module=HTTP::Status \
--module=HTTP::Date \
--module=Encode:: \
--module=File::Find::Rule \
--module=IO::Socket::SSL \
--module=IO::String \
--module=PerlIO::utf8_strict \
--module=Text::CSV_XS \
--module=DateTime \ |
|
Also, I'm trying to set up a Github Actions build script in https://github.com/vadimkantorov/buildbiber/blob/master/.github/workflows/build.yml Maybe it would be good for biber to have it for some testing as well in the main repo |
Did you try |
Not yet. Will try! |
It does depend on the platform - it's somewhat empirical, you have to tweak the build script until the executable works in terms of module includes. This is because the dependency checking modules are quite sensitive to platforms and the idiosyncracies of particular perl builds. If in doubt, use a |
|
Regarding testing, I haven't looked this in github as it required setting up a perl environment etc. and that's probably not trivial ... |
|
They have an Ubuntu installation, but without a perl distribution pre-installed. I opted for compiling one from scratch (since going forward I'd like to replace it by a custom statically-built WebAssembly one): |
|
Running https://github.com/vadimkantorov/buildbiber/blob/master/.github/workflows/build.yml, essentially: # perl and cpan are built from sources
cpan -T Test::More Test::Differences File::Which Module::Build Config::AutoConf ExtUtils::LibBuilder autovivification Class::Accessor Data::Dump Data::Compare Data::Uniqid DateTime::Format::Builder DateTime::Calendar::Julian File::Slurper IPC::Cmd IPC::Run3 List::AllUtils List::MoreUtils List::MoreUtils::XS Mozilla::CA Regexp::Common Log::Log4perl Unicode::Collate Unicode::Normalize Unicode::LineBreak Unicode::GCString Encode::Locale Encode::EUCJPASCII Encode::JIS2K Encode::HanExtra Parse::RecDescent PerlIO::utf8_strict XML::LibXML XML::LibXML::Simple XML::LibXSLT XML::Writer Sort::Key Storable Text::CSV Text::CSV_XS Text::Roman IO::String URI Text::BibTeX LWP::UserAgent LWP::Protocol::https Business::ISBN Business::ISSN Business::ISMN Lingua::Translit
URLBIBER=https://github.com/plk/biber/archive/v2.15.tar.gz
URLTESTFILES=https://master.dl.sourceforge.net/project/biblatex-biber/biblatex-biber/testfiles
BIBERTESTFILES="test.bib test.bcf test-dev.bcf unifont.ttf"
mkdir biber
wget $URLBIBER
tar -xf $(basename $URLBIBER) --strip-components=1 --directory biber
pushd biber
perl ./Build.PL
perl ./Build install
wget $(printf "$URLTESTFILES/%s " $BIBERTESTFILES)
perl ./Build test
perl ./bin/biber --validate-control --convert-control testproduced a lot of I attach the full test log log.txt |
|
What version of perl and U::C is on the test box? |
|
I install there perl 5.32.0 from sources in https://github.com/Perl/perl5/tree/v5.32.0 I could easily upgrade the perl version there if needed by just specifying a different version |
|
I don't know what U::C version it installs from CPAN. If a particular U::C version is required, I can check if possible to install a specific version from CPAN |
|
Do you know what version of Unicode::Collate is there? |
|
I see - I think that my install is not using the latest U::C - let me update and update the test results and then let's see. It would be best to currently use perl 5.30 for the tests until I migrate all the builds/tests to 5.32. Ah, I see the sortinithash issues with U::C 1.29 will fix in the tests in DEV branch and update here. |
|
In all likelihood it's: |
|
If it matters, the latest release at https://github.com/Perl/perl5/releases is 5.33.4 |
|
I never use odd numbers as they tend to be considered experimental. 5.32 would be the latest stable version. |
|
My github workflow file doesn't do PAR packing, since I didn't need it, but if it's useful for a single beginner/simple/free_CI build script, please feel free to build upon it |
|
DEV branch is updated with requirement for U::C 1.29 and all tests should have the correct |
|
Can PAR output the final list of resolved module dependencies? I'm advancing with compiling WebPerl, so a full list of modules to try bundling with Perl at build time would be very useful. |
|
Have a look at the docs for |
|
Instead of a binary, I produced a zipball with PAR and dumped its file listing: log.txt Here is a listing of archive's |
|
I will try to have a static perl build with just these modules (for native and for wasm). If you can make optional LWP/internet access/process launching (and corresponding module imports), it would greatly increase the chance of success. |
|
Here're the shared library that PAR packs: I'll see if perl can build these modules statically at compile time |
|
i.e. packages with shared libraries in the PAR-file are below. So probably it makes sense to start with them for a static_ext Perl build. @plk Are there any that strike you as unnecessary? Since I'm going to manually find all dependencies, we could get rid of some unneeded ones. |
vadimkantorov
changed the title
|
Hi both I wanted to make you aware that changes made in support of WebAssembly could also be of great help for Biber+Tectonic integration, which right now is difficult due to interop complexity. Most people are using this upstream by invoking Biber themselves in their IDE of choice in addition to Tectonic. Integration is being discussed in the following issue. tectonic-typesetting/tectonic#35 Being able to build without internet functionality could well be appreciated for this use case, in line with the compilation reproducibility goals of Tectonic. Thanks again for your work here! |
|
@aterenin I made some attempts at compiling Perl statically with all the dependencies. It almost worked, but I don't have time to continue this fight at the moment. If you'are interested to hear about my advancements, ping me at @vadimkantorov on Telegram or on [email protected] :) On the side note, I also lazily develop https://busytex.github.io - completely client-side wasm TexLive compiler and basic integration with github (~3k lines for Makefiles and JavaScript). Let know if you wish to collaborate. |
vadimkantorov
changed the title
|
Will revisit in future is there is sufficient interest. |
|
I've retried again. Here is the list of PAR-discovered dependencies: |
|
@plk Btw I managed to build statically all these So there might be path towards building statically all of biber with musl - without any packer utils. Maybe a remaining question for more portability is: does biber use backticks/system shell calls? If so, for portability it would be better to replace them with native Perl function calls or at least localize all such system shell calls in one perl source file to ease inspection and replacing them with more portable Perl code in the future. This can also open path to providing biber as a library (if needed) |
|
No, no backticks. Any such thing is done with OS neutral perl modules by design. |
|
This is great news, as I found that tlmgr.pl / install-tl.pl are using backticks all over the place... |
|
SInce |
|
@plk I succeded in compiling a fully static variant of Perl (without any I'll at some point try to use it instead of PAR to compile/pack a biber - without any temporary file extraction and so on, everything is embedded in a read-only, virtual FS (including |
|
@plk for building libbtparse from source, is it https://metacpan.org/pod/LaTeX::BibTeX and https://metacpan.org/release/AMBS/Text-BibTeX-0.89/source/btparse and https://www.ctan.org/tex-archive/biblio/bibtex/utils/btOOL/? |
|
I only use the AMBS/Text-BibTeX package which includes a modified |
|
https://github.com/ambs/Text-BibTeX right? or some other place from CTAN/CPAN? |
|
Yes, that's it. That's just the dev site for the package than ends up in CPAN which is where I pull it from for the build. |
|
Exactly, that's |
|
@plk Where I used to do wget https://master.dl.sourceforge.net/project/biblatex-biber/biblatex-biber/testfiles/test.bib https://master.dl.sourceforge.net/project/biblatex-biber/biblatex-biber/testfiles/test.bcf https://master.dl.sourceforge.net/project/biblatex-biber/biblatex-biber/testfiles/test-dev.bcf https://master.dl.sourceforge.net/project/biblatex-biber/biblatex-biber/testfiles/unifont.ttf
./biber-linux_x86_64 --validate-control --convert-control testbut how to run tests now? |
|
Ah, sorry, they were moved into the git repo so that they are more up to date. For the DEV branch: |
|
this is nice! curious, why |
|
I forgot to delete it ... |
|
I somehow made it work with a fully static Perl build (instead of using PAR): https://github.com/busytex/busybiber/blob/main/.github/workflows/busybiber.yml <- in this file you can see how many dependencies currently biber has... It would be nice to be able to know which ones are actually not needed and remove them. I'm embedding all Perl and data modules inside the executable. And somehow getting an error How can I check that |
|
Well, that error will tell you if you can find it or not. Sometimes you have to explicitly package files in the |
Hi! I'm building yet-another-attempt-of-latex-in-the-browser: https://vadimkantorov.github.io/busytext/busytex.html. So far I managed to build xetex+bibtex8.
I was thinking to somehow try to "compile" biber into WebAssembly. I was considering two options:
What would be your thoughts on feasibility of these two variants? Have you tried using perlcc on biber? Are there any "nasty" required dependencies?
Does biber require popen / signals / web requests for normal functioning? If not, it probably can be made working in WASM.
The text was updated successfully, but these errors were encountered: