Failed to request signed beacon block

[Jor-Tech]

Hi,

Trying out this awesome exporter with Prometheus and Grafana and got it working, but still see an error in the logs.
Using Geth (latest) for EL and Prysm (latest) for CL.

The error in the logs show this:

ethereum-metrics-exporter[25186]: {"component":"exporter","error":"failed to request signed beacon block: GET failed with status 500: {\"message\":\"GetBlockV2: rpc error: code = NotFound desc = Could not find requested block: signed beac on block can't be nil\",\"code\":500}","level":"error","module":"consensus/beacon","msg":"Subscriber error","time":"2022-08-01T11:21:00+02:00","topic":"block"}

Is this a misconfiguration from my side or could it be something else?

Flags used: --consensus-url=http://localhost:3500 --execution-url=http://localhost:8545

Prysm public API: https://docs.prylabs.network/docs/how-prysm-works/prysm-public-api

[samcm]

Hey! I think this is just an incompatibility with Prysm - nothing on your side. I'll have a look in to it and see what the best option is. Thanks for the issue!

[ybstaked]

I also still see this with prysm 3.1.1

[samcm]

Should be fixed when prysmaticlabs/prysm#11666 is released 👌

[Jor-Tech]

That would be great, will keep an eye out when next Prysm release is out, thanks!

[Jor-Tech]

Just updated Prysm to latest release v3.2.0 which includes:
"Use the correct block root for block events api prysmaticlabs/prysm#11666"

And updated ethereum-metrics-exporter to latest release v0.19.0

Getting these messages ending with some errors at the end of the log;

dec 16 18:13:22 ethereum-desktop systemd[1]: Started A Prometheus metrics exporter for Ethereum execution and consensus nodes. 

dec 16 18:13:22 ethereum-desktop ethereum-metrics-exporter[195132]: {"cfgFile":"","level":"info","msg":"Loading config","time":"2022-12-16T18:13:22+01:00"} 

dec 16 18:13:22 ethereum-desktop ethereum-metrics-exporter[195132]: {"component":"exporter","level":"info","msg":"Initializing...","time":"2022-12-16T18:13:22+01:00"} 
dec 16 18:13:22 ethereum-desktop ethereum-metrics-exporter[195132]: {"component":"exporter","level":"info","modules":"eth, net, web3","msg":"Initializing execution...","time":"2022-12-16T18:13:22+01:00"} 

dec 16 18:13:22 ethereum-desktop ethereum-metrics-exporter[195132]: {"component":"exporter","exporter":"execution","level":"info","msg":"Enabling sync status metrics","time":"2022-12-16T18:13:22+01:00"} 

dec 16 18:13:22 ethereum-desktop ethereum-metrics-exporter[195132]: {"component":"exporter","exporter":"execution","level":"info","msg":"Enabling general metrics","time":"2022-12-16T18:13:22+01:00"} 

dec 16 18:13:22 ethereum-desktop ethereum-metrics-exporter[195132]: {"component":"exporter","exporter":"execution","level":"info","msg":"Enabling block metrics","time":"2022-12-16T18:13:22+01:00"} 
dec 16 18:13:22 ethereum-desktop ethereum-metrics-exporter[195132]: {"component":"exporter","exporter":"execution","level":"info","msg":"Enabling web3 metrics","time":"2022-12-16T18:13:22+01:00"} 

dec 16 18:13:22 ethereum-desktop ethereum-metrics-exporter[195132]: {"component":"exporter","exporter":"execution","level":"info","msg":"Enabling net metrics","time":"2022-12-16T18:13:22+01:00"} 

dec 16 18:13:22 ethereum-desktop ethereum-metrics-exporter[195132]: {"component":"exporter","consensus_url":"http://localhost:3500","execution_url":"http://localhost:8545","level":"info","msg":"Starting metrics
server on :9092","time":"2022-12-16T18:13:22+01:00"} 

dec 16 18:13:22 ethereum-desktop ethereum-metrics-exporter[195132]: {"component":"exporter","execution_url":"http://localhost:8545","level":"info","msg":"Starting execution metrics...","time":"2022-12-16T18:13:2
2+01:00"} 

dec 16 18:13:22 ethereum-desktop ethereum-metrics-exporter[195132]: {"component":"exporter","consensus_url":"http://localhost:3500","level":"info","msg":"Starting consensus metrics...","time":"2022-12-16T18:13:2
2+01:00"} 

dec 16 18:13:22 ethereum-desktop ethereum-metrics-exporter[195132]: {"component":"exporter","exporter":"execution","level":"info","msg":"Started metrics exporter jobs","time":"2022-12-16T18:13:22+01:00"} 

dec 16 18:13:22 ethereum-desktop ethereum-metrics-exporter[195132]: {"component":"exporter","level":"info","module":"consensus/beacon","msg":"Initializing beacon state","time":"2022-12-16T18:13:22+01:00"} 

dec 16 18:13:22 ethereum-desktop ethereum-metrics-exporter[195132]: {"component":"exporter","level":"info","module":"consensus/beacon","msg":"Beacon state initialized! Ready to serve requests...","time":"2022-12
-16T18:13:22+01:00"} 

dec 16 18:13:27 ethereum-desktop ethereum-metrics-exporter[195132]: {"component":"exporter","level":"info","module":"consensus/beacon","msg":"Subscribing to events upstream","time":"2022-12-16T18:13:27+01:00","t
opics":["head","voluntary_exit","contribution_and_proof","attestation","block","chain_reorg","finalized_checkpoint"]} 

dec 16 18:13:37 ethereum-desktop ethereum-metrics-exporter[195132]: {"component":"exporter","error":"block is nil","level":"error","module":"consensus/beacon","msg":"Subscriber error","time":"2022-12-16T18:13:37
+01:00","topic":"block"} 

dec 16 18:13:48 ethereum-desktop ethereum-metrics-exporter[195132]: {"component":"exporter","error":"block is nil","level":"error","module":"consensus/beacon","msg":"Subscriber error","time":"2022-12-16T18:13:48
+01:00","topic":"block"} 

dec 16 18:14:00 ethereum-desktop ethereum-metrics-exporter[195132]: {"component":"exporter","error":"block is nil","level":"error","module":"consensus/beacon","msg":"Subscriber error","time":"2022-12-16T18:14:00
+01:00","topic":"block"} 

dec 16 18:14:11 ethereum-desktop ethereum-metrics-exporter[195132]: {"component":"exporter","error":"block is nil","level":"error","module":"consensus/beacon","msg":"Subscriber error","time":"2022-12-16T18:14:11
+01:00","topic":"block"} 

and then keeps repeating this message:

dec 16 18:21:48 ethereum-desktop ethereum-metrics-exporter[195132]: {"component":"exporter","error":"block is nil","level":"error","module":"consensus/beacon","msg":"Subscriber error","time":"2022-12-16T18:21:48
+01:00","topic":"block"} 

[dturner34]

Same issue with the same version of Prysm. Even if this is an issue with prysm doing something non-standard, is it possible for ethereum-metrics-exporter to gracefully handle this and still export the rest of the data?

[FloatingUpstream]

Hi,

so this was merged in https://github.com/prysmaticlabs/prysm/releases/tag/v3.2.0
I am still seeing the error on Prysm/v3.2.1/0cf1ca8b3cf0c4124fa3183412975721bd31d183.

If I try, I an sucessfully subscribe

curl "http://consensus:5052/eth/v1/events?topics=block" -H "accept: text/event-stream"
event: block
data: {"slot":"5926306","block":"0x16d72f09518687f178d3ee14e10aecf1b77761cfd03bab669ba6cd0e809290e2","execution_optimistic":false}

event: block
data: {"slot":"5926307","block":"0x8291b1a3c584275efba4a4c4ac8652b4f9057e4d1f3e1d66392a069a2d25e672","execution_optimistic":false}

event: block
data: {"slot":"5926308","block":"0x72b6a97b1ea5b3903b9704c99f4d5e7df6d0a5dd3cdb4ee550c5829f26afdb77","execution_optimistic":false}

I figure this is where the log output is produced:
https://github.com/ethpandaops/beacon/blob/master/pkg/beacon/metrics_beacon.go#L287
https://github.com/ethpandaops/beacon/blob/master/pkg/beacon/metrics_beacon.go#L406

But I cannot quite figure out why FetchBlock fails in that case.