Implemented support for known gems that use system calls for direct file access

.github/workflows/test-and-release.yml

@@ -91,7 +91,8 @@ jobs:
     - name: Install Ruby
       run: |
         apk --no-cache --upgrade add build-base cmake git bash   \
-        autoconf make binutils-dev pkgconfig tar ruby-dev
+        autoconf make binutils-dev pkgconfig tar ruby-dev        \
+        openjdk10-jre-headless
 
     - name: Install bundler
       shell: bash

.rubocop.yml

@@ -15,6 +15,9 @@ Style/StringLiteralsInInterpolation:
   Enabled: true
   EnforcedStyle: double_quotes
 
+Style/Documentation:
+  Enabled: false
+
 Gemspec/RequireMFA:
   Enabled: false
 

Gemfile

@@ -30,3 +30,6 @@ source "https://rubygems.org"
 gemspec
 
 gem "rake", "~> 12.0"
+
+# https://github.com/fontist/seven_zip_ruby/issues/13
+gem "seven-zip", git: "https://github.com/fontist/seven_zip_ruby.git"

README.adoc

@@ -4,24 +4,27 @@
 
 image:https://github.com/maxirmx/tebako-runtime/actions/workflows/test-and-release.yml/badge.svg["Test and release", link="https://github.com/maxirmx/tebako-runtime/actions/workflows/test-and-release.yml"]
 
-Tebako is an executable packager. It packages a set of files into a DwarFS file
-system for read-only purposes.
+Tebako is an executable packager. It packages a set of files into a DwarFS file system for read-only purposes.
 
-After packaging the file system into an image, Tebako produces a single
-executable binary that allows the user to execute a selected file from the
-packaged software from a point in the file system.
+After packaging the file system into an image, Tebako produces a single executable binary that allows the user to run a selected file from the packaged filesystem.
 
 Tebako image is essentially a patched Ruby with embedded filesystem as shown in the diagram.
 
 image:https://user-images.githubusercontent.com/2081498/150532110-75b60f61-0dc0-4697-abe9-59133878ae8c.jpg["Tebako architecture", link="https://user-images.githubusercontent.com/2081498/150532110-75b60f61-0dc0-4697-abe9-59133878ae8c.jpg"]
 
-Inside tebako image there are Ruby gems that acess native extensions and/or external libraries and/or data files.
+Inside tebako image there are Ruby gems that can access native extensions. If a gem loads native extension using rubygem features this call is intercepted, a copy extension shared object is placed to
+host temporary folder, all further calls to extension are routed to the copy of extension (Item 2 on the diagram).
 
-If a gem loads native extension using rubygems features this call is intercepted, a copy extension shared object is placed to host temp folder,
-all further calls to extension are routed to the copy of extension (Item 2 on the diagram).
+Gems and extensions can reference other libraries, executable and data files using native system calls (Items 5, 6). Tebako cannot intercept such calls and route them correctly to
+memory filesystem.  Tebako shall offload required file from memfs to temporary folder and reroute system calls as needed.
 
-Gems and extensions can reference other libraries, executable and data files using native system calls (Item 6). Tebako cannot intercept such calls and route them correctly to
-memory filesystem.  Such Gem shall be aware that they are running in tebako environment and offload required file from memfs to temporary folder.
+tebako-runtime (this Gem) provides support for known Gems that use system calls to access executable or data files.
 
-tebako-runtime (this Gem) provides a set of tools to help Gems to work in tebako environment.
-It is intalled automatically by the core tebako packager. There is no need to install it manually.
+== Supported Gems
+
+* https://rubygems.org/gems/ffi[ffi]
+* https://rubygems.org/gems/mn2pdf[mn2pdf]
+* https://rubygems.org/gems/mnconvert[mnconvert]
+* https://rubygems.org/gems/ruby-jing[ruby-jing]
+* https://rubygems.org/gems/sassc[sassc]
+* https://rubygems.org/gems/seven-zip[seven-zip]

bin/console

@@ -2,7 +2,7 @@
 # frozen_string_literal: true
 
 require "bundler/setup"
-require "tebako/runtime"
+require "tebako-runtime"
 
 # You can add fixtures and/or initialization code here to make experimenting
 # with your gem easier. You can also use a different console, if you like.

lib/tebako-runtime.rb

@@ -0,0 +1,81 @@
+# frozen_string_literal: true
+
+# Copyright (c) 2023 [Ribose Inc](https://www.ribose.com).
+# All rights reserved.
+# This file is a part of tebako
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS
+# BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+
+require "fileutils"
+require "pathname"
+
+require_relative "tebako-runtime/version"
+require_relative "tebako-runtime/memfs"
+
+# Module TenakoRuntime
+# Adds two additional steps for original require
+# - an option to run some pre-processing 'BEFORE'
+# - an option to implement adapter 'AFTER'
+module TebakoRuntime
+  PRE_REQUIRE_MAP = {
+    "seven_zip_ruby" => "tebako-runtime/pre/seven-zip"
+  }.freeze
+
+  POST_REQUIRE_MAP = {
+    "ffi" => "tebako-runtime/adapters/ffi",
+    "jing" => "tebako-runtime/adapters/jing",
+    "mn2pdf" => "tebako-runtime/adapters/mn2pdf",
+    "mnconvert" => "tebako-runtime/adapters/mnconvert",
+    "sassc" => "tebako-runtime/adapters/sassc"
+  }.freeze
+
+  def self.full_gem_path(gem)
+    Gem::Specification.find_by_name(gem).full_gem_path
+  end
+
+  def self.log_enabled
+    @log_enabled ||= false
+  end
+
+  def self.process(name, map, title)
+    return !log_enabled unless map.key?(name)
+
+    puts "Tebako runtime: req/#{title} [#{name} => #{map[name]}]" if log_enabled
+    res_inner = require_relative map[name]
+    puts "Tebako runtime: skipped [#{name}]" if log_enabled && !res_inner
+    log_enabled
+  end
+end
+
+# Some would call it 'monkey patching' but in reality we are adding
+# adapters to gems that shall be aware that they are running in tebako environment
+module Kernel
+  alias original_require require
+  def require(name)
+    f1 = TebakoRuntime.process(name, TebakoRuntime::PRE_REQUIRE_MAP, "pre")
+    res = original_require name
+    f2 = TebakoRuntime.process(name, TebakoRuntime::POST_REQUIRE_MAP, "post")
+
+    puts "Tebako runtime: req [#{name}]" unless f1 || f2
+    res
+  end
+end

lib/tebako/runtime/adaptors/ffi.rb → lib/tebako-runtime/adapters/ffi.rb

@@ -25,6 +25,8 @@
 # ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 
+require_relative "../memfs"
+
 # Wrapper for FFI.map_library_name method
 # If the library file to be mapped to is within memfs it is extracted to tmp folder
 module FFI

lib/tebako-runtime/adapters/jing.rb

@@ -0,0 +1,44 @@
+# frozen_string_literal: true
+
+# Copyright (c) 2023 [Ribose Inc](https://www.ribose.com).
+# All rights reserved.
+# This file is a part of tebako
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS
+# BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+
+# Unpack jing.jar
+class Jing
+  tmp = DEFAULT_JAR
+  remove_const("DEFAULT_JAR")
+  DEFAULT_JAR = TebakoRuntime.extract_memfs(tmp)
+
+  alias original_initialize initialize
+  alias original_validate validate
+
+  def initialize(schema, options = nil)
+    original_initialize(TebakoRuntime.extract_memfs(schema, wild: true), options)
+  end
+
+  def validate(xml)
+    original_validate(TebakoRuntime.extract_memfs(xml))
+  end
+end

lib/tebako-runtime/adapters/mn2pdf.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+# Copyright (c) 2023 [Ribose Inc](https://www.ribose.com).
+# All rights reserved.
+# This file is a part of tebako
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS
+# BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+
+# Unpack mn2pdf.jar
+module Mn2pdf
+  remove_const("MN2PDF_JAR_PATH")
+  MN2PDF_JAR_PATH = TebakoRuntime.extract_memfs(File.join(TebakoRuntime.full_gem_path("mn2pdf"), "bin", "mn2pdf.jar"))
+
+  singleton_class.send(:alias_method, :convert_orig, :convert)
+  singleton_class.send(:remove_method, :convert)
+
+  def self.convert(url_path, output_path, xsl_stylesheet, options)
+    convert_orig(TebakoRuntime.extract_memfs(url_path), output_path, TebakoRuntime.extract_memfs(xsl_stylesheet),
+                 options)
+  end
+end

lib/tebako-runtime/adapters/mnconvert.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+# Copyright (c) 2023 [Ribose Inc](https://www.ribose.com).
+# All rights reserved.
+# This file is a part of tebako
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS
+# BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+
+# Unpack mn2pdf.jar
+module MnConvert
+  remove_const("MNCONVERT_JAR_PATH")
+  MNCONVERT_JAR_PATH = TebakoRuntime.extract_memfs(File.join(TebakoRuntime.full_gem_path("mnconvert"), "bin",
+                                                             "mnconvert.jar"))
+
+  singleton_class.send(:alias_method, :convert_orig, :convert)
+  singleton_class.send(:remove_method, :convert)
+
+  def self.convert(input_file, opts = {})
+    opts[:xsl_file] = TebakoRuntime.extract_memfs(opts[:xsl_file]) if opts[:xsl_file]
+    convert_orig(TebakoRuntime.extract_memfs(input_file), opts)
+  end
+end

lib/tebako-runtime/adapters/sassc.rb

@@ -0,0 +1,65 @@
+# frozen_string_literal: true
+
+# Copyright (c) 2023 [Ribose Inc](https://www.ribose.com).
+# All rights reserved.
+# This file is a part of tebako
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+# ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
+# TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS
+# BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+# POSSIBILITY OF SUCH DAMAGE.
+
+require "fileutils"
+require_relative "../memfs"
+
+module SassC
+  # Load style files for sassc
+  class Engine
+    # rubocop:disable Style/ClassVars
+    @@loaded_pathes = []
+    @@loaded_pathes_semaphore = Mutex.new
+    # rubocop:enable Style/ClassVars
+
+    def load_files(path, m_path)
+      FileUtils.mkdir_p(m_path)
+      FileUtils.cp_r(File.join(path, "."), m_path) if File.exist?(path)
+      @@loaded_pathes << m_path
+    end
+
+    def load_path(path, new_paths)
+      if path.start_with?(TebakoRuntime::COMPILER_MEMFS)
+        m_path = path.sub(TebakoRuntime::COMPILER_MEMFS, TebakoRuntime::COMPILER_MEMFS_LIB_CACHE.to_s)
+        @@loaded_pathes_semaphore.synchronize do
+          load_files(path, m_path) unless @@loaded_pathes.include?(m_path)
+        end
+        new_paths << m_path
+      else
+        new_paths << path
+      end
+    end
+
+    def load_paths
+      paths = (@options[:load_paths] || []) + SassC.load_paths
+      new_paths = []
+      paths.each { |path| load_path path, new_paths }
+      pp = new_paths.join(File::PATH_SEPARATOR) unless new_paths.empty?
+      pp
+    end
+  end
+end

lib/tebako/runtime/memfs.rb → lib/tebako-runtime/memfs.rb

@@ -48,13 +48,14 @@ def extract(file, wild, extract_path)
       FileUtils.cp_r files, extract_path
     end
 
-    def extract_memfs(file, cache_path: COMPILER_MEMFS_LIB_CACHE)
+    # wild == true means "also extract other files with the same extension"
+    def extract_memfs(file, wild: false, cache_path: COMPILER_MEMFS_LIB_CACHE)
       is_quoted = file.quoted?
       file = file.unquote if is_quoted
       return is_quoted ? file.quote : file unless File.exist?(file) && file.start_with?(COMPILER_MEMFS)
 
       memfs_extracted_file = cache_path + File.basename(file)
-      extract(file, false, cache_path) unless memfs_extracted_file.exist?
+      extract(file, wild, cache_path) unless memfs_extracted_file.exist?
 
       is_quoted ? memfs_extracted_file.to_path.quote : memfs_extracted_file.to_path
     end