-
Notifications
You must be signed in to change notification settings - Fork 36
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Build script for demo fails on macOS/Apple M1 as some .mjs files aren't found #30
Comments
How long did the "./build-with-docker.sh" script take to run? It's looking like these errors are caused because the JS & Wasm failed to generate in the first place. Can we get a view of "$BUILD/llvm/bin/"?, There is some copying going on towards the end of the build process, we want to look at the directory its actually being built in. Just to add aswell, BoxProcess.mjs, FileSystem.mjs, Python3Process.mjs etc all ship from within the repo, this doesn't mean we are having any success with the build. |
After running $ tree ../build/llvm/
../build/llvm/
├── CMakeCache.txt
├── CMakeFiles
│ └── cmake.check_cache
└── build.ninja I see what you mean with copies, and noticed there were several errors that didn't outright terminate the build: /Users/s1g/Workspaces/emception
configure: cmake -G Ninja -S /Users/s1g/Workspaces/emception/upstream/brotli/ -B /Users/s1g/Workspaces/emception/build/brotli/ -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE=/emsdk/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -DCMAKE_CROSSCOMPILING_EMULATOR=/emsdk/node/14.18.2_64bit/bin/node;--experimental-wasm-threads
-- Build type is 'Release'
-- Performing Test BROTLI_EMSCRIPTEN
-- Performing Test BROTLI_EMSCRIPTEN - Success
-- Compiler is EMSCRIPTEN
-- Looking for log2
-- Looking for log2 - found
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/s1g/Workspaces/emception/build/brotli
sed: preserving permissions for '/Users/s1g/Workspaces/emception/build/brotli/sedWV8fJN': Permission denied
sed: preserving permissions for '/Users/s1g/Workspaces/emception/build/brotli/sedzTYldi': Permission denied
[1/36] Building C object CMakeFiles/brotlienc-static.dir/c/enc/dictionary_hash.c.o
[2/36] Building C object CMakeFiles/brotlienc-static.dir/c/enc/command.c.o
[3/36] Building C object CMakeFiles/brotlienc-static.dir/c/enc/compound_dictionary.c.o
[4/36] Building C object CMakeFiles/brotlienc-static.dir/c/enc/brotli_bit_stream.c.o Some checking host system type... wasm32-unknown-emscripten
checking for --with-build-python... configure: error: invalid or missing build python binary "/Users/s1g/Workspaces/emception/build/cpython-native/python"
emconfigure: error: '/Users/s1g/Workspaces/emception/upstream/cpython/configure -C --host=wasm32-unknown-emscripten --build=x86_64-pc-linux-gnu --with-emscripten-target=browser --disable-wasm-dynamic-linking --with-suffix=.mjs --disable-wasm-preload --enable-wasm-js-module --with-build-python=/Users/s1g/Workspaces/emception/build/cpython-native/python' failed (returned 1)
make: make -j10
make: *** No targets specified and no makefile found. Stop.
emmake: error: 'make -j10' failed (returned 2)
/Users/s1g/Workspaces/emception
/Users/s1g/Workspaces/emception/build/cpython /Users/s1g/Workspaces/emception
make: make -j10
make: *** No targets specified and no makefile found. Stop.
emmake: error: 'make -j10' failed (returned 2) Even some JS errors: file:///Users/s1g/Workspaces/emception/packs/emscripten/split_packages.js:1
const cp = require("child_process");
^
ReferenceError: require is not defined in ES module scope, you can use import instead
This file is being treated as an ES module because it has a '.js' file extension and '/Users/s1g/Workspaces/emception/package.json' contains "type": "module". To treat it as a CommonJS script, rename it to use the '.cjs' file extension.
at file:///Users/s1g/Workspaces/emception/packs/emscripten/split_packages.js:1:12
at ModuleJob.run (internal/modules/esm/module_job.js:183:25)
at async Loader.import (internal/modules/esm/loader.js:178:24)
at async Object.loadESM (internal/process/esm_loader.js:68:5) But I think the most pressing issue is the fact that the directories you mentioned don't exist, presumably because of a permissions issue: /Users/s1g/Workspaces/emception/build/packs/emscripten/emscripten /Users/s1g/Workspaces/emception/build/packs/emscripten /Users/s1g/Workspaces/emception
/Users/s1g/Workspaces/emception/build/packs/emscripten /Users/s1g/Workspaces/emception
/Users/s1g/Workspaces/emception
cp: cannot stat '/Users/s1g/Workspaces/emception/build/cpython/usr/local/lib/python3.*': No such file or directory
unzip: cannot find or open /Users/s1g/Workspaces/emception/build/cpython/usr/local/lib/python3*.zip, /Users/s1g/Workspaces/emception/build/cpython/usr/local/lib/python3*.zip.zip or /Users/s1g/Workspaces/emception/build/cpython/usr/local/lib/python3*.zip.ZIP.
No zipfiles found.
/Users/s1g/Workspaces/emception/build/packs/cpython /Users/s1g/Workspaces/emception
/Users/s1g/Workspaces/emception
/Users/s1g/Workspaces/emception/build/packs/wasm /Users/s1g/Workspaces/emception
cp: cannot stat '../../llvm/bin/llvm-box.wasm': No such file or directory
cp: cannot stat '../../cpython/python.wasm': No such file or directory But I have a really hard time imagining why that'd be, because I've never run into permission issues with Docker before and the directory belongs to me. I tried to log the entire build process but |
Hmm, I think its best we try and diagonse any issues with the LLVM build first, as its the first thing happening. Looks like some configuration is happening, which is good, but when it comes to running CMake something must be off. Do you have a console log for that? If permissions seem to be an issue, it could be an issue with your docker setup... I assume you have ran "$ sudo usermod -aG docker $USER" at some point? |
Its very possible mac works like that, apologise for sending you on the goose chase but its worth checking. For future tests to minimise time its probably worth going into the "build.sh" script and commenting out all but "build-tooling.sh" and "build-llvm.sh", we can add the rest of the build back in later. This should also make the errors clearer. We already have a major issue early on: "fatal: the remote end hung up unexpectedly" whilst trying to clone the LLVM Project, was this done in a fresh repo with all dangling docker images cleared? This is definitely the first error you want to approach, we need the source files to build anything. Did the same issue appear on the Linux machine? Strangely binaryen and cpython seems to have cloned fine... Just an extra thing to add, how much RAM does each machine have? The LLVM build can be pretty greedy, >16 GB if running in parallel. |
I gave it a go, and there are permission problems when building with Docker Desktop. I tried building with Docker Desktop and had a rather quick failure. I guesss that's related to how files are shared with the VM. |
To fix the issue with
|
@Johnn333 right this is diff --git a/build.sh b/build.sh
index eaa7f29..f3e8244 100755
--- a/build.sh
+++ b/build.sh
@@ -13,9 +13,9 @@ BUILD=$(realpath "$BUILD")
$SRC/build-tooling.sh "$BUILD"
$SRC/build-llvm.sh "$BUILD" "$LLVM_SRC"
-$SRC/build-binaryen.sh "$BUILD" "$BINARYEN_SRC"
-$SRC/build-cpython.sh "$BUILD" "$CPYTHON_SRC"
-$SRC/build-quicknode.sh "$BUILD" "$QUICKNODE_SRC"
-$SRC/build-brotli.sh "$BUILD" "$BROTLI_SRC"
+# $SRC/build-binaryen.sh "$BUILD" "$BINARYEN_SRC"
+# $SRC/build-cpython.sh "$BUILD" "$CPYTHON_SRC"
+# $SRC/build-quicknode.sh "$BUILD" "$QUICKNODE_SRC"
+# $SRC/build-brotli.sh "$BUILD" "$BROTLI_SRC"
-$SRC/build-emception.sh "$BUILD"
\ No newline at end of file
+# $SRC/build-emception.sh "$BUILD" and after $ time ./build-with-docker.sh
~/Workspaces/emception/docker ~/Workspaces/emception
[+] Building 1.4s (8/8) FINISHED docker:desktop-linux
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 804B 0.0s
=> [internal] load metadata for docker.io/emscripten/emsdk:3.1.24 1.3s
=> [auth] emscripten/emsdk:pull token for registry-1.docker.io 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [1/3] FROM docker.io/emscripten/emsdk:3.1.24@sha256:41a295159bab31f36261e6f3616dc44efc1207bc396e488a30530e40085d8cfe 0.0s
=> CACHED [2/3] RUN DEBIAN_FRONTEND=noninteractive apt --no-install-recommends -qy update && DEBIAN_FRONTEND=noninteractive apt --no-install-recommends -qy install docker.io pkg-config ninja 0.0s
=> CACHED [3/3] RUN mkdir -p /.npm && chmod a+rwx /.npm 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:0782e7e9768828a827992878f261401c9d832a1f19b17cbf6d47c3072b370d73 0.0s
=> => naming to docker.io/library/emception_build 0.0s
View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/zo7eeatuy3k3eflr5xdo4ujhv
What's Next?
View a summary of image vulnerabilities and recommendations → docker scout quickview
~/Workspaces/emception
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
shared:INFO: (Emscripten: Running sanity checks)
cache:INFO: generating system headers: sysroot_install.stamp... (this will be cached in "/emsdk/upstream/emscripten/cache/sysroot_install.stamp" for subsequent builds)
cache:INFO: - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/lto/libGL.a... (this will be cached in "/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libGL.a" for subsequent builds)
system_libs:INFO: compiled 3 inputs
cache:INFO: - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/lto/libal.a... (this will be cached in "/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libal.a" for subsequent builds)
system_libs:INFO: compiled 1 inputs
cache:INFO: - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/lto/libhtml5.a... (this will be cached in "/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libhtml5.a" for subsequent builds)
system_libs:INFO: compiled 2 inputs
cache:INFO: - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/lto/libstubs.a... (this will be cached in "/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libstubs.a" for subsequent builds)
system_libs:INFO: compiled 2 inputs
cache:INFO: - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/lto/libnoexit.a... (this will be cached in "/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libnoexit.a" for subsequent builds)
system_libs:INFO: compiled 1 inputs
cache:INFO: - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/lto/libc.a... (this will be cached in "/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libc.a" for subsequent builds)
system_libs:INFO: compiled 998 inputs
cache:INFO: - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/lto/libdlmalloc.a... (this will be cached in "/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libdlmalloc.a" for subsequent builds)
system_libs:INFO: compiled 2 inputs
cache:INFO: - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/lto/libcompiler_rt.a... (this will be cached in "/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libcompiler_rt.a" for subsequent builds)
system_libs:INFO: compiled 175 inputs
cache:INFO: - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/lto/libc++-noexcept.a... (this will be cached in "/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libc++-noexcept.a" for subsequent builds)
system_libs:INFO: compiled 46 inputs
cache:INFO: - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/lto/libc++abi-noexcept.a... (this will be cached in "/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libc++abi-noexcept.a" for subsequent builds)
system_libs:INFO: compiled 16 inputs
cache:INFO: - ok
cache:INFO: generating system library: sysroot/lib/wasm32-emscripten/lto/libsockets.a... (this will be cached in "/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/lto/libsockets.a" for subsequent builds)
system_libs:INFO: compiled 21 inputs
cache:INFO: - ok
cache:INFO: generating system asset: sysroot/lib/wasm32-emscripten/struct_info.json... (this will be cached in "/emsdk/upstream/emscripten/cache/sysroot/lib/wasm32-emscripten/struct_info.json" for subsequent builds)
cache:INFO: - ok
Cloning into '/Users/s1g/Workspaces/emception/upstream/llvm-project'...
Updating files: 100% (142111/142111), done.
/Users/s1g/Workspaces/emception/upstream/llvm-project /Users/s1g/Workspaces/emception
fatal: detected dubious ownership in repository at '/Users/s1g/Workspaces/emception/upstream/llvm-project'
To add an exception for this directory, call:
git config --global --add safe.directory /Users/s1g/Workspaces/emception/upstream/llvm-project
error: https://github.com/llvm/llvm-project.git did not send all necessary objects
fatal: detected dubious ownership in repository at '/Users/s1g/Workspaces/emception/upstream/llvm-project'
To add an exception for this directory, call:
git config --global --add safe.directory /Users/s1g/Workspaces/emception/upstream/llvm-project
fatal: detected dubious ownership in repository at '/Users/s1g/Workspaces/emception/upstream/llvm-project'
To add an exception for this directory, call:
git config --global --add safe.directory /Users/s1g/Workspaces/emception/upstream/llvm-project
/Users/s1g/Workspaces/emception
CMake Error at CMakeLists.txt:3 (cmake_minimum_required):
CMake 3.20.0 or higher is required. You are running version 3.16.3
-- Configuring incomplete, errors occurred!
Error: could not find CMAKE_PROJECT_NAME in Cache
configure: cmake -G Ninja -S /Users/s1g/Workspaces/emception/upstream/llvm-project/llvm/ -B /Users/s1g/Workspaces/emception/build/llvm/ -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD=WebAssembly -DLLVM_ENABLE_PROJECTS=clang;lld;clang-tools-extra -DLLVM_ENABLE_DUMP=OFF -DLLVM_ENABLE_ASSERTIONS=OFF -DLLVM_ENABLE_EXPENSIVE_CHECKS=OFF -DLLVM_ENABLE_BACKTRACES=OFF -DLLVM_BUILD_TOOLS=OFF -DLLVM_ENABLE_THREADS=OFF -DLLVM_BUILD_LLVM_DYLIB=OFF -DLLVM_INCLUDE_TESTS=OFF -DLLVM_TABLEGEN=/Users/s1g/Workspaces/emception/build/llvm-native/bin/llvm-tblgen -DCLANG_TABLEGEN=/Users/s1g/Workspaces/emception/build/llvm-native/bin/clang-tblgen -DCMAKE_TOOLCHAIN_FILE=/emsdk/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -DCMAKE_CROSSCOMPILING_EMULATOR=/emsdk/node/14.18.2_64bit/bin/node;--experimental-wasm-threads
CMake Error at CMakeLists.txt:3 (cmake_minimum_required):
CMake 3.20.0 or higher is required. You are running version 3.16.3
-- Configuring incomplete, errors occurred!
emcmake: error: 'cmake -G Ninja -S /Users/s1g/Workspaces/emception/upstream/llvm-project/llvm/ -B /Users/s1g/Workspaces/emception/build/llvm/ -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD=WebAssembly -DLLVM_ENABLE_PROJECTS=clang;lld;clang-tools-extra -DLLVM_ENABLE_DUMP=OFF -DLLVM_ENABLE_ASSERTIONS=OFF -DLLVM_ENABLE_EXPENSIVE_CHECKS=OFF -DLLVM_ENABLE_BACKTRACES=OFF -DLLVM_BUILD_TOOLS=OFF -DLLVM_ENABLE_THREADS=OFF -DLLVM_BUILD_LLVM_DYLIB=OFF -DLLVM_INCLUDE_TESTS=OFF -DLLVM_TABLEGEN=/Users/s1g/Workspaces/emception/build/llvm-native/bin/llvm-tblgen -DCLANG_TABLEGEN=/Users/s1g/Workspaces/emception/build/llvm-native/bin/clang-tblgen -DCMAKE_TOOLCHAIN_FILE=/emsdk/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -DCMAKE_CROSSCOMPILING_EMULATOR=/emsdk/node/14.18.2_64bit/bin/node;--experimental-wasm-threads' failed (returned 1)
sed: can't read /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
sed: can't read /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
/Users/s1g/Workspaces/emception /Users/s1g/Workspaces/emception
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
cat: /Users/s1g/Workspaces/emception/build/llvm/build.ninja: No such file or directory
/Users/s1g/Workspaces/emception
Error: could not find CMAKE_PROJECT_NAME in Cache
./build-with-docker.sh 0.24s user 0.27s system 0% cpu 10:47.06 total It certainly reduced the execution time immensely! I would paste the (successful) build of LLVM on Linux but selecting all of the terminal output after 24 minutes of compilation causes it to stop responding and spam the '"Console" is not responding" dialogue every 3 seconds forcing me to either "Force Quit" or "Wait", the latter of which just reopens the dialogue.. yea I'm really not having much success showing the output of the build script.
Yea I learned that the hard way, had to quadruple my swap on Linux but don't think that's the problem, as it freezes up my PC if it runs out of memory and the macOS swap partition is dynamically sized AFAIK. @jprendes I am indeed a Docker Desktop user, and figured that was the way to go. But I tried removing the diff --git a/build-with-docker.sh b/build-with-docker.sh
index a047b8e..a227c2c 100755
--- a/build-with-docker.sh
+++ b/build-with-docker.sh
@@ -16,7 +16,6 @@ docker run \
-v /var/run/docker.sock:/var/run/docker.sock \
-v $(pwd):$(pwd) \
-v $(pwd)/build/emsdk_cache:/emsdk/upstream/emscripten/cache \
- -u $(id -u):$(id -g) \
$(id -G | tr ' ' '\n' | xargs -I{} echo --group-add {}) \
emception_build:latest \
- bash -c "cd $(pwd) && ./build.sh"
\ No newline at end of file
+ bash -c "cd $(pwd) && ./build.sh" and that on a fresh clone in addition to commenting $ time ./build-with-docker.sh
...
[1920/2623] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaCast.cpp.o
[1921/2623] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaCoroutine.cpp.o
[1922/2623] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaCUDA.cpp.o
[1923/2623] Building CXX object tools/clang/lib/AST/CMakeFiles/obj.clangAST.dir/Interp/Disasm.cpp.o
FAILED: tools/clang/lib/AST/CMakeFiles/obj.clangAST.dir/Interp/Disasm.cpp.o
/emsdk/upstream/emscripten/em++ -DGTEST_HAS_RTTI=0 -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/clang/lib/AST -I/Users/s1g/Workspaces/emception/upstream/llvm-project/clang/lib/AST -I/Users/s1g/Workspaces/emception/upstream/llvm-project/clang/include -Itools/clang/include -Iinclude -I/Users/s1g/Workspaces/emception/upstream/llvm-project/llvm/include -Dwait4=__syscall_wait4 -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O3 -DNDEBUG -fno-exceptions -fno-rtti -std=c++17 -MD -MT tools/clang/lib/AST/CMakeFiles/obj.clangAST.dir/Interp/Disasm.cpp.o -MF tools/clang/lib/AST/CMakeFiles/obj.clangAST.dir/Interp/Disasm.cpp.o.d -o tools/clang/lib/AST/CMakeFiles/obj.clangAST.dir/Interp/Disasm.cpp.o -c /Users/s1g/Workspaces/emception/upstream/llvm-project/clang/lib/AST/Interp/Disasm.cpp
em++: error: '/emsdk/upstream/bin/clang++ -target wasm32-unknown-emscripten -fignore-exceptions -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -DEMSCRIPTEN -I/emsdk/upstream/emscripten/cache/sysroot/include/SDL --sysroot=/emsdk/upstream/emscripten/cache/sysroot -Xclang -iwithsysroot/include/compat -DGTEST_HAS_RTTI=0 -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Itools/clang/lib/AST -I/Users/s1g/Workspaces/emception/upstream/llvm-project/clang/lib/AST -I/Users/s1g/Workspaces/emception/upstream/llvm-project/clang/include -Itools/clang/include -Iinclude -I/Users/s1g/Workspaces/emception/upstream/llvm-project/llvm/include -Dwait4=__syscall_wait4 -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -fno-common -Woverloaded-virtual -Wno-nested-anon-types -O3 -DNDEBUG -fno-exceptions -fno-rtti -std=c++17 -MD -MT tools/clang/lib/AST/CMakeFiles/obj.clangAST.dir/Interp/Disasm.cpp.o -MF tools/clang/lib/AST/CMakeFiles/obj.clangAST.dir/Interp/Disasm.cpp.o.d -c /Users/s1g/Workspaces/emception/upstream/llvm-project/clang/lib/AST/Interp/Disasm.cpp -o tools/clang/lib/AST/CMakeFiles/obj.clangAST.dir/Interp/Disasm.cpp.o' failed (received SIGKILL (-9))
[1924/2623] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaChecking.cpp.o
[1925/2623] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaConcept.cpp.o
[1926/2623] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaExceptionSpec.cpp.o
[1927/2623] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaCodeComplete.cpp.o
[1928/2623] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaAvailability.cpp.o
[1929/2623] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaDeclObjC.cpp.o
[1930/2623] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaDeclAttr.cpp.o
[1931/2623] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaDecl.cpp.o
[1932/2623] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaDeclCXX.cpp.o
[1933/2623] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaExprCXX.cpp.o
[1934/2623] Building CXX object tools/clang/lib/Sema/CMakeFiles/obj.clangSema.dir/SemaExpr.cpp.o
ninja: build stopped: subcommand failed.
./build-with-docker.sh 0.48s user 0.51s system 0% cpu 49:35.17 total So it has definitely been a step in the right direction, looks as if the UID/GID line were the cause of permission issues on macOS, and it actually starts compiling LLVM now. That said, I don't know what caused |
Could it be a full disk or out of memory? |
Hmm, just tried again after freeing up more than enough space but it still failed. How would you go about applying those @Johnn333? I don't see them anywhere in the codebase and I'm not sure which of the build steps are involved. |
Here: https://github.com/jprendes/emception/blob/master/build-llvm.sh#L60 Im not sure of the default value, it could already be set to 1 but worth a try. Add -DLLVM_PARALLEL_COMPILE_JOBS=1 and -DLLVM_PARALLEL_LINK_JOBS=1. |
After applying both it's still failing surprisingly fast for the same reason, despite having >200 GiB storage available 12 GiB of free memory available to a recently restarted Docker. Is it a memory limit imposed on Docker? Because watching Activity Monitor, the memory used never exceeds 12 GiB but the Docker VM instance never crosses 8 GiB. Working in a dirty directory it's |
Activity monitors can be a bit misleading due to the update intervals not fully tracking memory usage, alot can happen in just a few ms. I'd clear any old repos out, clear any docker images. Clone a new emception project, reboot the machine. Close anything unecessary, and then give it a run. If this fails the only other thing I could suggest is finding if the "-j {N-Cores}" flag works here: https://github.com/jprendes/emception/blob/master/build-llvm.sh#L98. It may be that you have to invoke ninja directy to get this to work however. |
That does explain alot if the VM only has 8gb allocated to it. Sounds like there should be an advanced options tab if you poke around in the GUI, no idea where though. In reply to this: "cmake --build $LLVM_BUILD/ -j 1 -- llvm-box", yes although maybe I'd change the order to "cmake --build $LLVM_BUILD/ -- llvm-box -j 1". CMake might not do anything with this though. |
Yes I raised it to what appears to be the max of 16 GiB memory and 4 GiB swap. In my experience that wasn't enough to compile on Linux, so I really hope the swap limit of 4 GiB can be citcumvented in some way, but we'll see.. If this fails I'll try with |
Okay wow so that definitely took a while, but it does seem to be compiling llvm successfully now! 🎉 After applying the changes to There are still several permission issues, but I'm not sure if they're actually breaking anything. The breaking issue seems to be stemming from npm run build
> build
> webpack --mode=production --stats-children
assets by status 85.2 MiB [cached] 311 assets
orphan modules 3.05 MiB [orphan] 362 modules
runtime modules 7.38 KiB 12 modules
built modules 6.52 MiB (javascript) 66.3 KiB (asset) [built]
modules by path ./node_modules/monaco-editor/esm/vs/ 6.12 MiB (javascript) 66.3 KiB (asset) 276 modules
modules by path ./node_modules/style-loader/dist/runtime/*.js 5.75 KiB 6 modules
modules by path data:image/ 3.99 KiB 5 modules
modules by path ./node_modules/css-loader/dist/runtime/*.js 2.91 KiB 3 modules
modules by path ./node_modules/xterm/ 332 KiB
./node_modules/xterm/lib/xterm.js 327 KiB [built] [code generated]
./node_modules/css-loader/dist/cjs.js!./node_modules/xterm/css/xterm.css 4.56 KiB [built] [code generated]
./index.js + 11 modules 62.6 KiB [not cacheable] [built] [code generated]
./node_modules/xterm-addon-fit/lib/xterm-addon-fit.js 1.68 KiB [built] [code generated]
./node_modules/css-loader/dist/cjs.js!./style.css 4.09 KiB [built] [code generated]
Entrypoint HtmlWebpackPlugin_0-0 =
runtime modules 440 bytes 3 modules
cacheable modules 514 bytes
data:text/javascript,__webpack_public_path__ = __webpack_base_uri__ = htmlWebpackPluginPublicPath; 77 bytes [built] [code generated]
./node_modules/html-webpack-plugin/lib/loader.js!./node_modules/html-webpack-plugin/default_index.ejs 437 bytes [built] [code generated]
Child HtmlWebpackCompiler compiled successfully
assets by status 115 KiB [cached] 1 asset
orphan modules 391 KiB [orphan] 34 modules
runtime modules 221 bytes 1 module
./node_modules/monaco-editor/esm/vs/editor/editor.worker.js + 34 modules 392 KiB [built] [code generated]
Child vs/editor/editor compiled successfully
assets by status 81.2 MiB [cached] 304 assets
orphan modules 117 KiB [orphan] 15 modules
runtime modules 1.67 KiB 5 modules
modules by path ../build/emception/packages/*.br 12.3 KiB (javascript) 79.9 MiB (asset) 301 modules
javascript modules 567 KiB
modules by path ../build/emception/ 450 KiB
../build/emception/wasm-package/wasm-package.mjs 88.7 KiB [built] [code generated]
../build/emception/llvm/llvm-box.mjs 92.9 KiB [built] [code generated]
../build/emception/binaryen/binaryen-box.mjs 99 KiB [built] [code generated]
../build/emception/quicknode/quicknode.mjs 95.4 KiB [built] [code generated]
../build/emception/brotli/brotli.mjs 73.6 KiB [built] [code generated]
./emception.worker.js + 15 modules 117 KiB [built] [code generated]
(ignored asset) 26 bytes [built] [code generated]
../build/emception/wasm-package/wasm-package.wasm 42 bytes (javascript) 143 KiB (asset) [built] [code generated]
../build/emception/brotli/brotli.wasm 42 bytes (javascript) 786 KiB (asset) [built] [code generated]
ERROR in ../build/emception/Python3Process.mjs 2:0-48
Module not found: Error: Can't resolve './cpython/python.mjs' in '/Users/s1g/Workspaces/emception/build/emception'
resolve './cpython/python.mjs' in '/Users/s1g/Workspaces/emception/build/emception'
using description file: /Users/s1g/Workspaces/emception/package.json (relative path: ./build/emception)
Field 'browser' doesn't contain a valid alias configuration
using description file: /Users/s1g/Workspaces/emception/package.json (relative path: ./build/emception/cpython/python.mjs)
Field 'browser' doesn't contain a valid alias configuration
/Users/s1g/Workspaces/emception/build/emception/cpython/python.mjs doesn't exist
@ ./emception.js 5:0-58 91:20-34 92:20-34 93:20-34
@ ./emception.worker.js 2:0-39 4:22-31
Child worker-loader /Users/s1g/Workspaces/emception/demo/emception.worker.js compiled with 1 error
webpack 5.61.0 compiled with 1 error in 15839 ms Looking for ## line 4043:
Cloning into '/Users/s1g/Workspaces/emception/upstream/cpython'...
Updating files: 100% (4875/4875), done.
/Users/s1g/Workspaces/emception/upstream/cpython /Users/s1g/Workspaces/emception
error: RPC failed; curl 56 GnuTLS recv error (-9): Error decoding the received TLS packet.
fatal: the remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed
fatal: Could not parse object 'b8a9f13abb61bd91a368e2d3f339de736863050f'.
/Users/s1g/Workspaces/emception
/Users/s1g/Workspaces/emception/upstream/cpython /Users/s1g/Workspaces/emception
error: patch failed: Modules/Setup.stdlib.in:118
error: Modules/Setup.stdlib.in: patch does not apply
error: Modules/_emception.c: No such file or directory
error: patch failed: configure.ac:1253
error: configure.ac: patch does not apply
## line 5400:
cp: cannot stat '/Users/s1g/Workspaces/emception/build/cpython/python.mjs': No such file or directory
## line 6024:
cp: cannot stat '/Users/s1g/Workspaces/emception/build/cpython/usr/local/lib/python3.*': No such file or directory
unzip: cannot find or open /Users/s1g/Workspaces/emception/build/cpython/usr/local/lib/python3*.zip, /Users/s1g/Workspaces/emception/build/cpython/usr/local/lib/python3*.zip.zip or /Users/s1g/Workspaces/emception/build/cpython/usr/local/lib/python3*.zip.ZIP.
No zipfiles found.
/Users/s1g/Workspaces/emception/build/packs/cpython /Users/s1g/Workspaces/emception
/Users/s1g/Workspaces/emception
/Users/s1g/Workspaces/emception/build/packs/wasm /Users/s1g/Workspaces/emception
cp: cannot stat '../../cpython/python.wasm': No such file or directory |
It looks like a network issue downloading cpython's source |
Guess that was a poor example, but after running it two more times it failed once more (seemingly for the same reason) and it succeeded to clone I tried commenting out the other build targets to cut down on time: diff --git a/build.sh b/build.sh
index eaa7f29..885de3e 100755
--- a/build.sh
+++ b/build.sh
@@ -12,10 +12,11 @@ BUILD=$(realpath "$BUILD")
$SRC/build-tooling.sh "$BUILD"
-$SRC/build-llvm.sh "$BUILD" "$LLVM_SRC"
-$SRC/build-binaryen.sh "$BUILD" "$BINARYEN_SRC"
+# $SRC/build-llvm.sh "$BUILD" "$LLVM_SRC"
+# $SRC/build-binaryen.sh "$BUILD" "$BINARYEN_SRC"
$SRC/build-cpython.sh "$BUILD" "$CPYTHON_SRC"
-$SRC/build-quicknode.sh "$BUILD" "$QUICKNODE_SRC"
-$SRC/build-brotli.sh "$BUILD" "$BROTLI_SRC"
+# $SRC/build-quicknode.sh "$BUILD" "$QUICKNODE_SRC"
+# $SRC/build-brotli.sh "$BUILD" "$BROTLI_SRC"
+
+# $SRC/build-emception.sh "$BUILD"
-$SRC/build-emception.sh "$BUILD"
\ No newline at end of file And upon building it, it still complains about the patch: https://gist.github.com/s1gtrap/1854ebc742b95ba8c273ca6e327f4cd4#file-gistfile1-txt-L71-L75 and proceeds to build # line 1411:
The following modules are *disabled* in configure script:
_sqlite3
The necessary bits to build these optional modules were not found:
_bz2 _ctypes _ctypes_test
_curses _curses_panel _dbm
_gdbm _hashlib _lzma
_ssl _tkinter _uuid
readline
To find the necessary bits, look in configure.ac and config.log.
Could not build the ssl module!
Python requires a OpenSSL 1.1.1 or newer
# line 1437:
checking for --with-build-python... configure: error: invalid or missing build python binary "/Users/s1g/Workspaces/cpython/build/cpython-native/python"
emconfigure: error: '/Users/s1g/Workspaces/cpython/upstream/cpython/configure -C --host=wasm32-unknown-emscripten --build=x86_64-pc-linux-gnu --with-emscripten-target=browser --disable-wasm-dynamic-linking --with-suffix=.mjs --disable-wasm-preload --enable-wasm-js-module --with-build-python=/Users/s1g/Workspaces/cpython/build/cpython-native/python' failed (returned 1)
make: make -j10
make: *** No targets specified and no makefile found. Stop.
emmake: error: 'make -j10' failed (returned 2)
/Users/s1g/Workspaces/cpython
/Users/s1g/Workspaces/cpython/build/cpython /Users/s1g/Workspaces/cpython
make: make -j10
make: *** No targets specified and no makefile found. Stop.
emmake: error: 'make -j10' failed (returned 2)
/Users/s1g/Workspaces/cpython
./build-with-docker.sh 0.28s user 0.33s system 0% cpu 11:13.21 total For the record |
Is python native actually building here? configure: error: invalid or missing build python binary "/Users/s1g/Workspaces/cpython/build/cpython-native/python" Is suggesting not, can we get a view of this directory, does it even exist? |
$ ls /Users/s1g/Workspaces/cpython/build/cpython-native/python
Python-ast.o bltinmodule.o dtoa.o frozen_modules getversion.o mysnprintf.o pyhash.o pytime.o
Python-tokenize.o bootstrap_hash.o dynamic_annotations.o frozenmain.o hamt.o mystrtoul.o pylifecycle.o specialize.o
_warnings.o ceval.o dynload_shlib.o future.o hashtable.o pathconfig.o pymath.o structmember.o
asdl.o ceval_gil.o errors.o getargs.o import.o perf_trampoline.o pystate.o suggestions.o
asm_trampoline.o codecs.o fileutils.o getcompiler.o importdl.o preconfig.o pystrcmp.o symtable.o
ast.o compile.o formatter_unicode.o getcopyright.o initconfig.o pyarena.o pystrhex.o sysmodule.o
ast_opt.o context.o frame.o getopt.o marshal.o pyctype.o pystrtod.o thread.o
ast_unparse.o deepfreeze frozen.o getplatform.o modsupport.o pyfpe.o pythonrun.o traceback.o |
Interesting, The native build doesn't seem to be completing, and as such the Wasm build is immediately failing. This guy seems to have had the same problem: #24 Here is the entry point for the patch you mentioned: https://github.com/jprendes/emception/blob/master/build-cpython.sh#L56 but I'm not sure if this is a problem. CPython seems to have cloned fine and reverted to a known commit, assuming the project can still be built on the current commit its possible the patch is failing anyway, but still building fine @jprendes do you know if this is the case? |
Hmm so I did a little more digging, and after successfully building cpython alone on Linux and diffing the generated output on macOS, the point where they really start diverging is line 1433/1437 respectively: diff --git a/linux b/macos
index ad06c14..ad665fd 100644
--- a/linux
+++ b/macos
@@ -1,16 +1,14 @@
checking for git... found
checking build system type... x86_64-pc-linux-gnu
checking host system type... wasm32-unknown-emscripten
-checking for --with-build-python... /home/s1g/Workspaces/cpython/build/cpython-native/python
-checking for Python interpreter freezing... /home/s1g/Workspaces/cpython/build/cpython-native/python
-checking for python3.12... (cached) /home/s1g/Workspaces/cpython/build/cpython-native/python
-checking Python for regen version... Python 3.12.0a1+
-checking for wasm32-unknown-emscripten-pkg-config... no
-checking for pkg-config... /usr/bin/pkg-config
-configure: WARNING: using cross tools not prefixed with host triplet
-checking pkg-config is at least version 0.9.0... yes
-checking for --enable-universalsdk... no
-checking for --with-universal-archs... no
-checking MACHDEP... "emscripten"
-checking for wasm32-unknown-emscripten-gcc... /emsdk/upstream/emscripten/emcc
-
+checking for --with-build-python... configure: error: invalid or missing build python binary "/Users/s1g/Workspaces/cpython/build/cpython-native/python"
+emconfigure: error: '/Users/s1g/Workspaces/cpython/upstream/cpython/configure -C --host=wasm32-unknown-emscripten --build=x86_64-pc-linux-gnu --with-emscripten-target=browser --disable-wasm-dynamic-linking --with-suffix=.mjs --disable-wasm-preload --enable-wasm-js-module --with-build-python=/Users/s1g/Workspaces/cpython/build/cpython-native/python' failed (returned 1)
+make: make -j10
+make: *** No targets specified and no makefile found. Stop.
+emmake: error: 'make -j10' failed (returned 2)
+/Users/s1g/Workspaces/cpython
+/Users/s1g/Workspaces/cpython/build/cpython /Users/s1g/Workspaces/cpython
+make: make -j10
+make: *** No targets specified and no makefile found. Stop.
+emmake: error: 'make -j10' failed (returned 2)
+/Users/s1g/Workspaces/cpython which got me to examine the $ file /home/s1g/Workspaces/cpython/build/cpython-native/python
/home/s1g/Workspaces/cpython/build/cpython-native/python: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=4401836f5165f232fa9e2a3d68eb4e277da406fa, for GNU/Linux 3.2.0, with debug_info, not stripped $ file "/Users/s1g/Workspaces/cpython/build/cpython-native/python"
/Users/s1g/Workspaces/cpython/build/cpython-native/python: directory
$ ls "/Users/s1g/Workspaces/cpython/build/cpython-native/python"
Python-ast.o bltinmodule.o dtoa.o frozen_modules getversion.o mysnprintf.o pyhash.o pytime.o
Python-tokenize.o bootstrap_hash.o dynamic_annotations.o frozenmain.o hamt.o mystrtoul.o pylifecycle.o specialize.o
_warnings.o ceval.o dynload_shlib.o future.o hashtable.o pathconfig.o pymath.o structmember.o
asdl.o ceval_gil.o errors.o getargs.o import.o perf_trampoline.o pystate.o suggestions.o
asm_trampoline.o codecs.o fileutils.o getcompiler.o importdl.o preconfig.o pystrcmp.o symtable.o
ast.o compile.o formatter_unicode.o getcopyright.o initconfig.o pyarena.o pystrhex.o sysmodule.o
ast_opt.o context.o frame.o getopt.o marshal.o pyctype.o pystrtod.o thread.o
ast_unparse.o deepfreeze frozen.o getplatform.o modsupport.o pyfpe.o pythonrun.o traceback.o So no wonder the following build step fails? I mean if expecting an executable but given a directory I'm not even sure how it ought to react but I guess this is the source of the issue? I linked both outputs but here is the |
It's worth having a dig through to see if you can find the Python executable elsewhere on the mac system, maybe there is some line deep in the build changing the output location. Nothing is suggesting it isn't being built from what I've seen of the logs. If you can find it, change the line here: https://github.com/jprendes/emception/blob/master/build-cpython.sh#L83 to point to it. |
Sorry for the delay, I did find the python executable, turns out it was named I don't know where or when it decided to generate a Would it make sense to toss together a PR? As stated I'm not sure where it opted to append |
I tried to get the demo to compile from the instructions in the README, but it seems to be failing regardless of what I do.
Following the instructions, running:
all executes fine, but running the build script right after doesn't:
Running
ls ../build/emception
gives:so it looks like the primary modules are generated correctly, but their immediate successors aren't:
In fact it looks like they're not being generated at all:
For the record I also tried checking out
783fa10
(the same commit where the instructions were added), and it still fails (despite being run on a completely fresh clone):Am I missing a step in compilation or am I missing a dependency of sorts?
The text was updated successfully, but these errors were encountered: