[DON'T MERGE] Migrating PeerDAS to Pectra + soft-forking #163

Partially addresses Consensys#7762 Signed-off-by: Paul Harris <[email protected]>

…#8485) Partially addresses Consensys#7762 Signed-off-by: Paul Harris <[email protected]>

partially addresses Consensys#7762 Signed-off-by: Paul Harris <[email protected]>

* wip array type definition * tests added, fixed error Signed-off-by: Paul Harris <[email protected]> * Add post attestation v2 interface * add missing final * rename json files * refactoring * add integration tests * refactoring --------- Signed-off-by: Paul Harris <[email protected]> Co-authored-by: Paul Harris <[email protected]>

Signed-off-by: Paul Harris <[email protected]>

) The concept of `? extends Attestation` got a little messy... wondering if we need to make it closer to the way blocks are, where Attestation is a class, but also not sure how much 'better' this would look without spiking it. partially addresses Consensys#7762 Signed-off-by: Paul Harris <[email protected]>

Signed-off-by: Gabriel Fukushima <[email protected]>

* refactor create aggregate attestation request * refactor

* Remove -X prefix Signed-off-by: Gabriel Fukushima <[email protected]> * add changelog Signed-off-by: Gabriel Fukushima <[email protected]> --------- Signed-off-by: Gabriel Fukushima <[email protected]>

* remove deprecated GetNewBlindedBlock Signed-off-by: Gabriel Fukushima <[email protected]> * add changelog Signed-off-by: Gabriel Fukushima <[email protected]> * removed json schema and response Signed-off-by: Gabriel Fukushima <[email protected]> --------- Signed-off-by: Gabriel Fukushima <[email protected]>

* Cleaned up interfaces for remote request Also added some tests for minimal typedef functions which were missing. I'd like to ultimately move everything to this model where we're testing per class, but it's a larger cleanup, so here's a good sanity check point. partially addresses Consensys#7762 Signed-off-by: Paul Harris <[email protected]>

This meant handling our failure list responses also, which will be needed for a couple of other interfaces also, so i've put the handling into a spot we can get to. partially addresses Consensys#7762 Signed-off-by: Paul Harris <[email protected]>

* Remove BlocksV2 Signed-off-by: Gabriel Fukushima <[email protected]> * remove API resources and tests from GetBlockV2 Signed-off-by: Gabriel Fukushima <[email protected]> * add changelog Signed-off-by: Gabriel Fukushima <[email protected]> --------- Signed-off-by: Gabriel Fukushima <[email protected]>

- moved the last 2 endpoints to typedef - cleaned up interface ValidatorRestApiClient interface - Renamed SendSyncCommitteeRequest This removes the last usages of the old schema based interfaces in the VC. fixes Consensys#7762 Signed-off-by: Paul Harris <[email protected]>

…ys#8487) * support schema for oneOf electra and phase0 Signed-off-by: Gabriel Fukushima <[email protected]> * fix tests Signed-off-by: Gabriel Fukushima <[email protected]> --------- Signed-off-by: Gabriel Fukushima <[email protected]>

* Moved slashing protection interchange format away from using JsonProvider. Signed-off-by: Paul Harris <[email protected]>

…ader (Consensys#8516) * add Eth-Consensus-Version header to getBlobSidecars response Signed-off-by: Gabriel Fukushima <[email protected]> * spotless Signed-off-by: Gabriel Fukushima <[email protected]> --------- Signed-off-by: Gabriel Fukushima <[email protected]>

This reverts commit 89dd8fa.

* remove experimental flag and useBlindedBlock variable now that V3 can handle blind and unblinded flows. Signed-off-by: Gabriel Fukushima <[email protected]> --------- Signed-off-by: Gabriel Fukushima <[email protected]>

also restored some deserializers for JsonProvider that are likely used by external signer... Signed-off-by: Paul Harris <[email protected]>

Signed-off-by: Paul Harris <[email protected]>

- migrated TekuBeaconNode towards using mostly ObjectMapper where possible, though i did use a couple of typedefs. - Added ObjectMapper to Node, but haven't yet removed JsonProvider because its still heavily used in AT. - Several functions now return more specific things, which simplifies using ObjectMapper. Signed-off-by: Paul Harris <[email protected]>

* add get aggregate attestation request * fix integration test * refactoring * rename test * spotless * refactor aggregattion api * refactoring * refactoring * rename aggregate attestation request * rename request * apply spotless * use schema definition cache * refactoring * use spec when submit * refactor create aggregate request

* add post aggregate and proofs V2 API * add changelog entry * update error reponse description * refactoring

* use header based selector with slot fallback

Signed-off-by: Paul Harris <[email protected]>

* better support for enum type in open api doc

* Add post attestation v2 request

* add post aggregate and proofs v2 request

fixes Consensys#8538 Signed-off-by: Paul Harris <[email protected]>

Added test for validator exit triggered by the EL

* handle missing committee index

…onsensys#8550)

* add --Xattestations-v2-apis-enabled option

- our version of web3signer in AT was very dated. - this is only single node Signed-off-by: Paul Harris <[email protected]>

Signed-off-by: Paul Harris <[email protected]>

another batch of changes. Partially addresses Consensys#8526 Signed-off-by: Paul Harris <[email protected]>

…onsensys#8576) This reverts commit 22e2350.

Signed-off-by: gconnect <[email protected]> Co-authored-by: Paul Harris <[email protected]>

* Revert "Revert "attestation duties generation optimization (Consensys#8553)" (Consensys#8576)" This reverts commit 44b811c. * reintroduce optimization with fixed benchmark

AbstractDataBackedRestAPIIntegrationTest no longer requires JsonProvider. fixes Consensys#8526 Signed-off-by: Paul Harris <[email protected]>

This used to be needed for v2 but now that we're completely using v3 there's no real need for the message to exist... fixes Consensys#8526 Signed-off-by: Paul Harris <[email protected]>

…ys#8582) * Add explicit suffix to metrics automatically changed by prometheus Signed-off-by: Gabriel Fukushima <[email protected]> * changelog entry Signed-off-by: Gabriel Fukushima <[email protected]> --------- Signed-off-by: Gabriel Fukushima <[email protected]>

* separate attestation-related schemas --------- Co-authored-by: Enrico Del Fante <[email protected]>

* add EnumTypeHeaderDefinition class to deal with EnumType for response header that serialize into openAPI differently of content * add Eth-Consensus-Version header to endpoints response definition Signed-off-by: Gabriel Fukushima <[email protected]> --------- Signed-off-by: Gabriel Fukushima <[email protected]>

Signed-off-by: asli <[email protected]> Co-authored-by: Paul Harris <[email protected]>

…Consensys#8531) Signed-off-by: Paul Harris <[email protected]> Co-authored-by: bomanaps <[email protected]> Co-authored-by: Mercy Boma Naps Nkari <[email protected]> Co-authored-by: Paul Harris <[email protected]>

) Signed-off-by: gconnect <[email protected]>

adding header type definition to deal with OpenApiSerialization for response headers Signed-off-by: Gabriel Fukushima <[email protected]> --------- Signed-off-by: Gabriel Fukushima <[email protected]>

* removing URL constructor deprecated Signed-off-by: Gabriel Fukushima <[email protected]> * replace getID() deprecated Signed-off-by: Gabriel Fukushima <[email protected]> --------- Signed-off-by: Gabriel Fukushima <[email protected]>

Signed-off-by: Gabriel Fukushima <[email protected]>

* upgrade rocksdb to latest * removed deprecated statistics (removed in 8.0.0 release) * tweak some defaults from RocksDb * turn compression on * add jemalloc detection Signed-off-by: Gabriel Fukushima <[email protected]> --------- Signed-off-by: Gabriel Fukushima <[email protected]>

…clusion (Consensys#8613) * add log to debug level to track possible incorrect report of block inclusion * add state root and move log to debug Signed-off-by: Gabriel Fukushima <[email protected]> --------- Signed-off-by: Gabriel Fukushima <[email protected]>

* update holesky bootnodes ENRs Signed-off-by: Gabriel Fukushima <[email protected]> --------- Signed-off-by: Gabriel Fukushima <[email protected]>

Signed-off-by: Gabriel Fukushima <[email protected]>

…#8611) Co-authored-by: Paul Harris <[email protected]>

* start EL and CL pair together to avoid delay * give and extra 30 secs to genesis Signed-off-by: Gabriel Fukushima <[email protected]> --------- Signed-off-by: Gabriel Fukushima <[email protected]>

This is no longer required, cleaning up. fixes Consensys#8636 Signed-off-by: Paul Harris <[email protected]>

Signed-off-by: gconnect <[email protected]> Co-authored-by: Paul Harris <[email protected]> Co-authored-by: Gabriel Fukushima <[email protected]>

…#8633) Co-authored-by: Gabriel Fukushima <[email protected]>

…onsensys#8640) * Implement a DatabaseArchiveWriter and add to pruneOldestBlobSidecars. Currently not configurable and only implements a NoopWriter.

Update Electra penalty computation.

* add get-earliest-available-block-slot to debug tool Signed-off-by: Gabriel Fukushima <[email protected]> --------- Signed-off-by: Gabriel Fukushima <[email protected]>

* Gossip_V_1_2 * changelog --------- Co-authored-by: Stefan Bratanov <[email protected]>

* Expose discovery NodeId to DiscoveryPeer and LibP2PPeer (Consensys#38) * Add DiscoveryPeer.getNodeId() * Add Eth2Peer.getDiscoveryNodeId() method. * Refactoring, discovery updated to 24.9.1 --------- Co-authored-by: Anton Nashatyrev <[email protected]>

* add engine get blobs

* initial implementation * improve * better debug logging * improve tests * better error handling * better error msg * add changelog line * missed toString change

Signed-off-by: Gabriel Fukushima <[email protected]>

…onsensys#8675)

* add rpc method return type checks in unit tests

…ng (Consensys#8681)

…ys#8646)

- created a new suite of tests to show a number of different signing bodies. - had to do manual testing for external signer becuase no AT i think - I was able to get from phase0 -> bellatrix so this should cover most scenarios. - I believe most of the wrappers should be available in the api partially addresses Consensys#8527 Signed-off-by: Paul Harris <[email protected]>

Signed-off-by: gconnect <[email protected]> Co-authored-by: Paul Harris <[email protected]>

…#8668)

* fix proposer config loader

* changing max pending partial withdrawals per sweep according to spec for minimal and swift * fix partial pending withdrawals count * bump reference test to alpha.8 Signed-off-by: Gabriel Fukushima <[email protected]> --------- Signed-off-by: Gabriel Fukushima <[email protected]>

* migrate Attestation and AttnetsENR schemas to registry * allow `addMilestoneMapping` with same `milestone` and `untilMilestone`

Signed-off-by: Paul Harris <[email protected]>

…tion flow (Consensys#8700)

Co-authored-by: Lucas Saldanha <[email protected]>

* reduce wait period to avoid flakiness Signed-off-by: Gabriel Fukushima <[email protected]> * Reduce parallelism for ATs Signed-off-by: Gabriel Fukushima <[email protected]> --------- Signed-off-by: Gabriel Fukushima <[email protected]>

Co-authored-by: Paul Harris <[email protected]>

…ost-Electra (Consensys#8717)

Co-authored-by: Enrico Del Fante <[email protected]>

* remove test fixed from the list of ignored tests * move processWithdrawalRequests to BlockProcessorElectra * expose supplier so it can be accessed from ElectraBLockProcessor --------- Signed-off-by: Gabriel Fukushima <[email protected]>

…8727) * Add debug command to retrieve peerId from a node key file Signed-off-by: Gabriel Fukushima <[email protected]> * spotless Signed-off-by: Gabriel Fukushima <[email protected]> * add default flags to command Signed-off-by: Gabriel Fukushima <[email protected]> --------- Signed-off-by: Gabriel Fukushima <[email protected]>

Add the option --Xdata-storage-blobs-archive-path to allow pruned BlobSidecars to be written to disk. Each block/slot has all associated BlobSidecars written to disk in a filename of the block root hash and written out in JSON format.

* set up state pruner when switching from archive to minimal or prune mode

* update EL get blobs logs

…Consensys#8712) Co-authored-by: Enrico Del Fante <[email protected]>

…ky with TODO)

* make state pruner faster

Co-authored-by: Lucas Saldanha <[email protected]>

Co-authored-by: Lucas Saldanha <[email protected]> Co-authored-by: Enrico Del Fante <[email protected]>

…ure module)

…onsensys#8744)

…onsensys#8736) * migrate-attester-slashing-and-indexed-att * fix tests

…m VC (Consensys#8754) When producing a block, JSON payloads require the Eth-Consensus-version header. Generally using SSZ this was correct, but there was no test case for JSON block production, and it wasn't providing the header correctly. Because probably 99% of our block production is via SSZ, this was not really visible. This would not be a problem for local BN+VC, but if the teku VC ever falls back to json (or is instructed to produce JSON), block production could potentially fail if the BN is following spec. Changed an AT to check both SSZ and JSON block production. fixes Consensys#8753 Signed-off-by: Paul Harris <[email protected]>

fixes Consensys#8756 Added test cases for VC block requests, and added a validation to the Attestation request test. These were the 2 VC cases where we now require that header to be specified or we will fail. There was another case in DSL for AT, but we'll see those quickly if we use it. Signed-off-by: Paul Harris <[email protected]>

In the case of JSON specifically, we do have the slot number, and we don't actually need the header with a slot present. Signed-off-by: Paul Harris <[email protected]>

* add http client unit tests

Co-authored-by: Lucas Saldanha <[email protected]>

Adding a variable in the database to store the earliest available block slot which is used when peers request blocks by range. This is a query that can vary in some case and I've seen it causing disconnection in some cases due to multiple requests getting timed out. Changing it to be stored in a variable brings the query to 1 - 2 ms response time. It does add a bit of logic to handle the sync of historical batches and prune which are currently the workflow in the code base that affect that column family. Signed-off-by: Gabriel Fukushima <[email protected]> --------- Signed-off-by: Gabriel Fukushima <[email protected]>

…8765)

…Impl (Consensys#8766) Change the builderResultCache of ExecutionLayerBlockProductionManagerImpl keys from slot to SlotAndBlockRoot. This allows the cache to return proposals based on slot and block root, which is required if there's multiple operators requesting different beacon block proposals. fixes Consensys#8625

* refactor post block integration tests

--------- Signed-off-by: Paul Harris <[email protected]> Co-authored-by: Enrico Del Fante <[email protected]>

fixes Consensys#8777 Signed-off-by: Paul Harris <[email protected]>

)

…8772) Co-authored-by: Enrico Del Fante <[email protected]>

)

Co-authored-by: Enrico Del Fante <[email protected]>

Signed-off-by: gconnect <[email protected]> Co-authored-by: Paul Harris <[email protected]>

* add start scripts to detect jemalloc presence Signed-off-by: Gabriel Fukushima <[email protected]> --------- Signed-off-by: Gabriel Fukushima <[email protected]>

Signed-off-by: Gabriel Fukushima <[email protected]>

* use the correct schema definitions

Signed-off-by: Gabriel Fukushima <[email protected]>

…onsensys#8780) Co-authored-by: Stefan Bratanov <[email protected]>

…nsys#8806) * use correct schema definitions when creation full beacon block

use forkChoiceUpdatedV4 with PayloadAttributesV4 Co-authored-by: Mehdi AOUADI <[email protected]> Co-authored-by: Enrico Del Fante <[email protected]>

fixes Consensys#8777 Signed-off-by: Paul Harris <[email protected]>

* update blob sidecars gossip topics * use misc helpers to get blob sidecars subnet count * remove unused method in gossip filter test

* update blobs rpc methods

Consensys#8830)

Co-authored-by: Enrico Del Fante <[email protected]>

* migrate beacon state * fix historical summary schema * fix ref test compilation * fix ref tests

* migrate-block-content-schemas * deleted too much * lookup cache first

Fix incorrect variable syntax in state download request Co-authored-by: Paul Harris <[email protected]>

…ys#8855)

* use attestation bits aggregator in performance tracker * add electra tests * fix spotless * fix unit test * always clear objects * refactor bits aggregator or operation * revert bits aggregator or operation return type

Commits on Aug 7, 2024

update changelog (Consensys#8499 )

lucassaldanha authored Aug 7, 2024

Configuration menu

View commit details

Copy full SHA for 7defe58

Browse repository at this point

Copy the full SHA

7defe58 View commit details

Browse the repository at this point in the history

Commits on Nov 20, 2024

update netty (Consensys#8838 )

tbenr authored Nov 20, 2024

Configuration menu

View commit details

Copy full SHA for af1320e

Browse repository at this point

Copy the full SHA

af1320e View commit details

Browse the repository at this point in the history

Commits on Nov 25, 2024

Merge branch 'master' into das-electra

zilm13 committed Nov 25, 2024

Configuration menu

View commit details

Copy full SHA for 463d203

Browse repository at this point

Copy the full SHA

463d203 View commit details

Browse the repository at this point in the history

[DON'T MERGE] Migrating PeerDAS to Pectra + soft-forking #163

Are you sure you want to change the base?

[DON'T MERGE] Migrating PeerDAS to Pectra + soft-forking #163

Commits on Jul 31, 2024

Commits on Aug 1, 2024

Commits on Aug 2, 2024

Commits on Aug 5, 2024

Commits on Aug 6, 2024

Commits on Aug 7, 2024

Commits on Aug 11, 2024

Commits on Aug 12, 2024

Commits on Aug 13, 2024

Commits on Aug 19, 2024

Commits on Aug 20, 2024

Commits on Aug 21, 2024

Commits on Aug 22, 2024

Commits on Aug 26, 2024

Commits on Aug 27, 2024

Commits on Aug 28, 2024

Commits on Aug 29, 2024

Commits on Aug 30, 2024

Commits on Sep 1, 2024

Commits on Sep 2, 2024

Commits on Sep 4, 2024

Commits on Sep 5, 2024

Commits on Sep 6, 2024

Commits on Sep 9, 2024

Commits on Sep 11, 2024

Commits on Sep 12, 2024

Commits on Sep 13, 2024

Commits on Sep 16, 2024

Commits on Sep 17, 2024

Commits on Sep 19, 2024

Commits on Sep 23, 2024

Commits on Sep 24, 2024

Commits on Sep 25, 2024

Commits on Sep 26, 2024

Commits on Sep 27, 2024

Commits on Sep 28, 2024

Commits on Sep 29, 2024

Commits on Sep 30, 2024

Commits on Oct 2, 2024

Commits on Oct 3, 2024

Commits on Oct 4, 2024

Commits on Oct 6, 2024

Commits on Oct 7, 2024

Commits on Oct 8, 2024

Commits on Oct 9, 2024

Commits on Oct 10, 2024

Commits on Oct 11, 2024

Commits on Oct 14, 2024

Commits on Oct 15, 2024

Commits on Oct 16, 2024

Commits on Oct 17, 2024

Commits on Oct 18, 2024

Commits on Oct 19, 2024

Commits on Oct 20, 2024

Commits on Oct 21, 2024

Commits on Oct 22, 2024

Commits on Oct 23, 2024

Commits on Oct 24, 2024

Commits on Oct 25, 2024

Commits on Oct 28, 2024

Commits on Oct 31, 2024

Commits on Nov 4, 2024

Commits on Nov 5, 2024

Commits on Nov 6, 2024

Commits on Nov 7, 2024

Commits on Nov 11, 2024

Commits on Nov 14, 2024

Commits on Nov 15, 2024

Commits on Nov 19, 2024

Commits on Nov 20, 2024

Commits on Nov 21, 2024

Commits on Nov 22, 2024

Commits on Nov 24, 2024

Commits on Nov 25, 2024

Commits on Nov 27, 2024

Commits on Nov 28, 2024

Commits on Nov 29, 2024