Rust Workspace support for multiple submodule for python package

[evan0greenup]

Description

For some project (like database or some other stuff which include both high-level stuff and low-level stuff). There are several low-level part, each low-level part is little coupling with each other. And the the developer don't want to expose the low level part to user, instead, only high level part is exposed to user.

For performance consideration, those low level part want to be implemented in Rust.

As a result, single rust module for python package is not enough.

Example

There is an Python specific Object-Oriented database called ZODB.

If the developers want to refactor this software. (not for implementation, but for descriptive purpose), transaction, picklezodb, and all variant of storage back-ends can be rewritten via Rust.

Currently, ZODB package several important component as separate Python packages. This is not smart, because even if those component is less-coupling but they are highly relevant. packing in single package can make development and deployment much easier.

Proposal

Here is an sample project directory:

hello_py_workspace/
├── Cargo.toml
├── hello_py_workspace
│   ├── high1
│   │   └── __init__.py
│   ├── __init__.py
│   ├── low1
│   │   └── __init__.py
│   └── low2
│       └── __init__.py
├── pyproject.toml
└── rust_submodule_root
    ├── low1
    │   ├── Cargo.toml
    │   └── src
    │       └── lib.rs
    └── low2
        ├── Cargo.toml
        └── src
            └── lib.rs

Just for demonstration purpose, if there is details not strict, please tolerant them.

The content of hello_py_workspace/pyproject.toml is:

[build-system]
requires = ["maturin>=0.14,<0.15"]
build-backend = "maturin"

[project]
name = "hello_py_workspace"
requires-python = ">=3.7"
classifiers = [
    "Programming Language :: Rust",
    "Programming Language :: Python :: Implementation :: CPython",
    "Programming Language :: Python :: Implementation :: PyPy",
]

[tool.maturin]
rust_submodule_root = "rust_submodule_root/"

The content of hello_py_workspace/Cargo.toml is:

[workspace]
members = ["rust_submodules/*"]

[dependencies]
pyo3 = { version = "0.18.0", features = ["extension-module"] }

hello_py_workspace/rust_submodule_root/low1/Cargo.toml is:

[package]
name = "low1"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[lib]
name = "low1"

crate-type = ["cdylib"]

[dependencies]
pyo3 = { version = "0.18.0", features = ["extension-module"] }

The other details can be found in hello_py_workspace.tar.gz.

rust_submodule_root can be some other name else (by change the value in tool.maturin section in pyproject.toml).

maturin will traverse all the pyo3-ready crates under rust_submodule_root. And make one-to-one mapping according to relative path and copy <submodule_name>.so or <submodule_name>.pyd to relevant location in python modules.

For example hello_py_workspace/rust_submodule_root/low1/Cargo.toml ==> hello_py_workspace/hello_py_workspace/low1/low1.<platform-triple>.so

For non-pyo3 ready crates (for dependency purpose). It can be anywhere else (just include it in workspace members).

[evan0greenup]

@messense How do you think about it?

[messense]

Thanks for the proposal, I'd like to explore existing solutions first.

pyo3 can add submodules, so I think for maintainability you can split a big extension module into multiple submodules each in its own crate and create a single facade crate that depends on these submodules and expose them to Python.

For you example, you can define functions, classes, modules and etc in low1 and low2 but only have them exposed to Python in a hello_py pyo3 ready crate.

use low1;
use low2;

#[pymodule]
fn hello_py(py: pyo3::Python<'_>, m: &pyo3::types::PyModule) -> pyo3::PyResult<()> {
    m.add_submodule(low1::create_submodule(py)?)?;
    m.add_submodule(low2::create_submodule(py)?)?;
    Ok(())
}

// low1::create_submodule example

pub fn create_submodule(py: Python<'_>) -> PyResult<&PyModule> {
    let submod = PyModule::new(py, "low1")?;
    submod.add_wrapped(pyo3::wrap_pyfunction!(function1))?;
    Ok(submod)
}

This will only build one dylib and you can re-export the submodules in Python code to better reflect the intended API namespace if you want.