Build blocks using Reth 1.1.0 #115

hashcashier · 2024-10-29T08:34:17Z

This PR migrates zeth from directly using revm to using reth for building blocks. This allows zeth to support all forks supported by reth. The changes are an overhaul of many of zeth's internal components and project structure. Notably, because of Cargo's feature unification, and reth's dependence on feature flags for enabling/disabling OP functionality, two separate zeth binaries are now necessary.

The project has been restructured into the following crates:

zeth-core Library for defining and using the sparse MPT to statelessly build a block using reth.
zeth-core-ethereum Specialization of the core library for Ethereum blocks.
zeth-core-optimism Specialization of the core library for Optimism blocks.
zeth-preflight Library for populating the data required to statelessly build a block using zeth-core.
zeth-preflight-ethereum Specialization of the preflight library for Ethereum RPCs.
zeth-preflight-optimism Specialization of the preflight library for Optimism RPCs.
zeth-guests-reth-ethereum A binary for running the stateless block building process for Ethereum blocks.
zeth-guests-reth-optimism A binary for running the stateless block building process for Optimism blocks.
zeth-guests Library for exporting the compiled rv32im bytecode of zeth-guests-reth.
zeth A library for running a CLI to use the RISC Zero zkVM to generate/verify proofs for the stateless block building process.
zeth-ethereum A binary using the zeth crate for Ethereum blocks.
zeth-optimism A binary using the zeth crate for Optimism blocks.
zeth-benchmark A binary for benchmarking proving performance for randomly sampled blocks.

Functionality changes:

All ethereum types have been migrated to alloy types.
All Bonsai and profiling logic has been delegated to r0vm's environment variables.
Optimism consensus verification (block derivation) logic has been removed.
Ethereum foundation testing is unfinished.

Note: v1.81 of the toolchain is required:

rzup update

Note: Separate binaries must be built individually

cargo build -p zeth-ethereum --bin zeth-ethereum

cargo build -p zeth-optimism --bin zeth-optimism

Issues affected:

pdg744 · 2024-11-07T22:52:42Z

@hashcashier @SchmErik iiuc v1.81 toolchain should now be ready to use!

Wollac · 2024-11-20T15:43:10Z

crates/core/src/db.rs

@@ -95,6 +95,7 @@ impl<DB: DatabaseRef + Recoverable> DatabaseRef for Wrapper<DB> {

 impl<DB: DatabaseCommit + Recoverable> DatabaseCommit for Wrapper<DB> {
    fn commit(&mut self, changes: HashMap<Address, Account>) {
+        dbg!("COMMIT!");


I believe that should be removed.

Wollac · 2024-11-20T15:50:08Z

crates/preflight/src/client.rs

+                // resolve storage orphans
+                if let Some((trie, _)) = storage_tries.get_mut(&account_proof.address) {
+                    for storage_proof in account_proof.storage_proof {
+                        // let proof_nodes = parse_proof(&storage_proof.proof)?;


There is still quite a bit of commented code, this should be removed

Wollac · 2024-11-20T15:51:49Z

crates/preflight/src/db.rs

@@ -151,6 +153,48 @@ impl<N: Network, R: CoreDriver, P: PreflightDriver<R, N>> PreflightDB<N, R, P> {
        apply_changeset(&mut self.inner, state_changeset)
    }

+    pub fn sanity_check(&mut self, state_changeset: StateChangeset) -> anyhow::Result<()> {


Unused function

yeah this is vestigial but it's also public and comes in handy when debugging

Wollac · 2024-11-20T16:02:21Z

crates/preflight/src/trie.rs

-            mpt_from_proof(&proof_nodes).context("invalid storage_proof")?;
+            let proof_nodes = parse_proof(&storage_proof.proof)
+                .context("extend_proof_tries/parse storage proof")?;
+            mpt_from_proof(&proof_nodes).context(format!(


should be a with_context

Wollac · 2024-11-20T16:10:10Z

crates/preflight/src/provider/mod.rs

        Ok(vec.into_iter().collect())
    }
 }
+
+pub mod opt_ordered_map {


This is never used, ordered_map + default is used instead

Wollac · 2024-11-20T19:10:59Z

crates/core-ethereum/src/lib.rs

+
+    fn chain_spec(chain: &NamedChain) -> Option<Arc<Self::ChainSpec>> {
+        match chain {
+            NamedChain::Mainnet => Some(MAINNET.clone()),


We do not support all possible forks. I know for a fact that the pre Byzantine validation is not correct in Reth: Back then, the TxReceipt still contained intermediate state hashes which was left out in Reth for performance reasons https://github.com/paradigmxyz/reth/blob/e3702cfc87449294da061f02a571a23061b8ab50/crates/ethereum/consensus/src/validation.rs#L31
So we must exclude at least everything before Byzantine.

What about possible future forks that are (or will be with newer reth versions) only partially supported? Do they also need to be explicitly excluded?

yeah this seems underconstrained as well like with the PoW checks

Wollac · 2024-11-20T19:13:46Z

testing/ef-tests/testdata

I assume the submodule was removed to prevent Cargo from always checking out the huge tests.
However, then the ef-tests should be adapted to include a script or documentation that checks out the EF tests in the correct version

I actually didn't migrate the ef tests because reth takes care of much of the logic we had previously put together ourselves. But I agree this would be great to bring back.

Wollac · 2024-11-20T19:14:57Z

Cargo.toml

@@ -1,6 +1,10 @@
 [workspace]
 resolver = "2"
-members = ["guests", "host", "lib", "primitives", "testing/ef-tests"]
+members = [


The ef-tests have been removed here. Thus, they have not been adapted to the new reth version and will not compile.

Wollac · 2024-11-20T19:17:12Z

.github/workflows/ci.yml

    - uses: risc0/cargo-install@v1
      with:
        crate: cargo-binstall
-    - run: cargo binstall -y --force cargo-risczero@$RISC0_VERSION
-    - run: cargo risczero install --version $RISC0_TOOLCHAIN_VERSION
-    - run: cargo test --workspace --all-targets -F ef-tests,debug-guest-build


We should absolutely not disable the ef-tests. Even when we are now using reth they contain crucial testcases that we must also pass.

Wollac · 2024-11-21T00:32:11Z

crates/core/src/rescue.rs

+}
+
+#[derive(Default)]
+pub struct Wrapper<T: Recoverable> {


An alternative to "abusing" the drop to get the Database back is to move the DB in and out similar to https://github.com/risc0/risc0-ethereum/blob/73eebd4efb283c7f99748d45fc730a53ee2edf5c/steel/src/contract.rs#L211
However, this should probably not also be in this PR.

The code to do this would have be on the reth side so to keep the changes there minimal I went with rescuing the dropped value

If you're constrained by reth, then I'd say this is a pretty neat work around. It's "non-local" and thus maybe less intuitive than the move-in-move-out approach -- but I do appreciate that drop() only gets called once, which simplifies the process of reasoning about the Mutex.

Nice trick.

intoverflow

Approved pending concurrence from Wolf

~~ Some additional comments

This is a really exciting refactor. It makes good on our long-held goal of deleting code and leveraging the growing ecosystem of libraries, and incorporates some architectural lessons learned. This was obviously a ton of work and I deeply appreciate you taking it so seriously.

It's also a lot of code so I can't be 100% sure I didn't miss some important things. So, while I've read the PR and left some general comments, I still have a few higher level questions:

(Checklist item) Are there any portions of code that need additional attributions?
(Checklist item) Have you recently walked the security-critical code paths to convince yourself you didn't accidentally leave any parts commented out/disabled? Does the logic still look good to you?
What's your least favorite part of the code?
Which parts do you think will require the most ongoing maintenance? Chainspec seems like an obvious answer. Any others? Are there any parts that are "temporary" that I may have missed?
It would be awesome if 1 or 2 of the core modules had some top-level cargo doc that explained the role of the key traits/types. Stateless? Drivers? Strategies? I think I was able to piece it together from context but it'd be very nice to have them laid out somewhere, even if only tersely.

Thank you again brother

intoverflow · 2024-12-03T09:52:58Z

crates/core-ethereum/src/lib.rs

+        // let block_with_senders = BlockWithSenders {
+        //     block,
+        //     senders: vec![], // todo: recover signers with non-det hints
+        // };


intoverflow · 2024-12-03T09:56:16Z

crates/core/src/lib.rs

@@ -1,4 +1,4 @@
-// Copyright 2023 RISC Zero, Inc.
+// Copyright 2024 RISC Zero, Inc.


I believe the usual convention is to make a list of years, including the year of the most recent edit. Example:

"Copyright 2023, 2024 RISC Zero, Inc."

intoverflow · 2024-12-03T10:01:38Z

crates/core/src/rescue.rs

+}
+
+#[derive(Default)]
+pub struct Wrapper<T: Recoverable> {


If you're constrained by reth, then I'd say this is a pretty neat work around. It's "non-local" and thus maybe less intuitive than the move-in-move-out approach -- but I do appreciate that drop() only gets called once, which simplifies the process of reasoning about the Mutex.

Nice trick.

intoverflow · 2024-12-03T10:06:32Z

crates/core/src/db.rs

+
+impl Recoverable for MemoryDB {
+    fn rescue(&mut self) -> Option<Self> {
+        Some(core::mem::take(self))


intoverflow · 2024-12-03T10:07:00Z

crates/core/src/db.rs

+
+impl<DB: DatabaseCommit + Recoverable> DatabaseCommit for Wrapper<DB> {
+    fn commit(&mut self, changes: HashMap<Address, Account>) {
+        // dbg!("COMMIT!");


intoverflow · 2024-12-03T10:07:32Z

crates/core/src/db.rs

+    Ok(())
+}
+
+/// Copied from [BundleState::into_plane_state]. Modified to retain account code.


[BundleState::into_plane_state]

Do we need to add additional attribution here? Is this from reth?

intoverflow · 2024-12-03T10:10:51Z

crates/core/src/stateless/client.rs

+use std::io::Read;
+use std::sync::{Arc, Mutex};
+
+pub type RescueDestination<D> = Arc<Mutex<Option<D>>>;


What's this for? Does it get used?

hashcashier added 30 commits October 16, 2024 07:10

update zkvm and add basic reth guest

2c60f0e

remove OP derivation code

e70f1fa

switch to tiny-keccak

f6ab9e9

remove more op derive code

1e632ce

consensus checks

56fc8c8

(unfinished) refactor into core crate

6c3bfa1

post-exec split

9067612

finalization

ff8a7ea

StatelessClientStrategy

b9381f4

reth guest

90fa5e4

preflight providers

f695193

preflight db

d7a77b7

preflight client

eba328b

preflight client lib

ac54f34

uncle blocks

5688b05

reth preflight client

ac25ef5

basic bin lib

313404c

rescue db post preflight

b2de53c

provider exec fix

17a693d

refactor & fix

811dd96

block building

225b76d

zkvm exec

8bf41f4

time & ordering fix + rpc data

9464cef

proof generation

390fcf5

add prover opts to receipt file name derivation

4bbf416

receipt verification

401a061

cleanup: delete old code

0b3af8b

downgrade reth msrv + cleanup

bd2e934

block batch journal derivation

da2a99a

GAT in strategies

b80ef64

hashcashier added 18 commits October 31, 2024 13:08

refactor

93276d8

feature unification workaround

d1d4711

op support

337e971

rename eth guest

0c9587f

fix code drop from account

a361d89

multichain support

705217c

readme

5ee963d

preflight speedup

3e5014e

clipp

8b2e87c

cache optimizations

8b42557

fix alloy primitives

2f02b37

remove ef tests submodule

db833a0

fix ci

7a38b7d

versioning nit

00c716a

ci test commands

3caf21a

fix cli short arg clash

736909a

profile flag

cb05478

benchmark tool + refactor

0583389

hashcashier added 6 commits November 11, 2024 10:48

risc0 1.1.3 w/ release builds

67a461f

fmt benchmark

2bcf20f

ci nit

496e671

ci nit

ff833d0

refactor

5bd18b7

preflight using debug namespace

0836298

Wollac reviewed Nov 21, 2024

View reviewed changes

austinabell mentioned this pull request Nov 25, 2024

switch to 1.2 rc and new k256 patch #119

Draft

delete unused code & commends

6037b33

hashcashier requested a review from Wollac December 2, 2024 17:39

intoverflow approved these changes Dec 3, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build blocks using Reth 1.1.0 #115

Build blocks using Reth 1.1.0 #115

hashcashier commented Oct 29, 2024 •

edited

Loading

pdg744 commented Nov 7, 2024

Wollac Nov 20, 2024

Wollac Nov 20, 2024

Wollac Nov 20, 2024

hashcashier Dec 2, 2024

Wollac Nov 20, 2024

Wollac Nov 20, 2024

Wollac Nov 20, 2024

Wollac Nov 20, 2024

hashcashier Dec 2, 2024

Wollac Nov 20, 2024

hashcashier Dec 2, 2024

Wollac Nov 20, 2024

Wollac Nov 20, 2024

Wollac Nov 21, 2024

hashcashier Dec 2, 2024

intoverflow Dec 3, 2024

intoverflow left a comment

intoverflow Dec 3, 2024

intoverflow Dec 3, 2024

intoverflow Dec 3, 2024

intoverflow Dec 3, 2024

intoverflow Dec 3, 2024

intoverflow Dec 3, 2024

intoverflow Dec 3, 2024

intoverflow Dec 3, 2024

		@@ -1,4 +1,4 @@
		// Copyright 2023 RISC Zero, Inc.
		// Copyright 2024 RISC Zero, Inc.

Build blocks using Reth 1.1.0 #115

Are you sure you want to change the base?

Build blocks using Reth 1.1.0 #115

Conversation

hashcashier commented Oct 29, 2024 • edited Loading

pdg744 commented Nov 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

intoverflow left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hashcashier commented Oct 29, 2024 •

edited

Loading