Why the node cannot keep up with head of the Binance chain?

[Aracki]

We've followed the guide from #41, but apparently several times a day one of our nodes is lagging ~1-2k blocks behind the tip of the chain.

We've tried to tweak --bodies.cache & --batchSize several times but without help. It always sync in batches which mean constant lag behind the geth binance nodes. We haven't even noticed any difference between syncing when using smaller or larger values for these two flags.

Anyone managed to run node-real erigon nodes which are always in sync with the Binance chain?

[blxdyx]

Actually my local dev is aways with head of bsc. It's the problem about p2p network. Please show you start command.

[Aracki]

Currently it is like this @blxdyx

      --datadir=/erigon/.ethereum
      --chain=bsc
      --networkid=56
      --db.pagesize=8kb
      --bodies.cache=268435456
      --batchSize=256M
      --http.addr=0.0.0.0
      --http.port=8545
      --http.vhosts=*
      --http.corsdomain=*
      --http.api=eth,erigon,web3,net,debug,trace,txpool
      --private.api.addr=0.0.0.0:9090
      --txpool.api.addr=0.0.0.0:9090
      --ws
      --ws.port=8546
      --metrics
      --metrics.addr=0.0.0.0
      --metrics.port=6060
      --log.dir.disable=true
      --healthcheck
      --torrent.download.rate=512mb
      --bootnodes=enode://e17c2ec7ea588c46c0ceb03a2d621a020a49def8564276e3b82bb90f1d5aa9f929e33a3e894a69783a7a87d92d3f2c28af581171ce29f22e12434541c6c0c53a@18.195.40.224:33303,enode://f6313a784ec90da86ddae53fb76830c3eefc69e6e796f3f5e6e4ce4cc45fbb1dc8684e34ab5951a5047a5d376584d3a668ff88eae4dd4f0f0ced6d96e942ee89@18.195.40.224:33313,enode://1cc4534b14cfe351ab740a1418ab944a234ca2f702915eadb7e558a02010cb7c5a8c295a3b56bcefa7701c07752acd5539cb13df2aab8ae2d98934d712611443@52.71.43.172:30311,enode://28b1d16562dac280dacaaf45d54516b85bc6c994252a9825c5cc4e080d3e53446d05f63ba495ea7d44d6c316b54cd92b245c5c328c37da24605c4a93a0d099c4@34.246.65.14:30311,enode://5a7b996048d1b0a07683a949662c87c09b55247ce774aeee10bb886892e586e3c604564393292e38ef43c023ee9981e1f8b335766ec4f0f256e57f8640b079d5@35.73.137.11:30311
      --staticpeers=enode://551c8009f1d5bbfb1d64983eeb4591e51ad488565b96cdde7e40a207cfd6c8efa5b5a7fa88ed4e71229c988979e4c720891287ddd7d00ba114408a3ceb972ccb@34.245.70.169:30311,enode://c637c90d6b9d1d0038788b163a749a7a86fed2e7d0d13e5dc920ab144bb432ed1e3e00b54c1a93cecba479037601ba9a5937a88fe0be949c651043473c0d1e5b@52.17.25.179:30311,enode://bac6a548c7884270d53c3694c93ea43fa87ac1c7219f9f25c9d57f6a2fec9d75441bc4bad1e81d78c049a1c4daf3b1404e2bbb5cd9bf60c0f3a723bbaea110bc@54.171.201.213:30311,enode://a232f92d1e76447b93306ece2f6a55ac70ca4633fae0938d71a100757eaf8526e6bbf720aa70cba1e6d186be17291ad1ee851a35596ec6caa2fdf135ce4b6b68@107.20.124.16:30311,enode://accbc0a5af0af03e1ec3b5e80544bdceea48011a6928cd82d2c1a9c38b65fd48ec970ba17bd8c0b0ec21a28faec9efe1d1ce55134784b9207146e2f62d8932ba@54.162.32.1:30311,enode://16c7e98f78017dafeaa4129647d1ec66b32ee9be5ec753708820b7363091ceb310f575e7abd9603005e0e34d7b3316c1a4b6c8c42d7f074ed2eb4d073f800a03@3.85.216.212:30311
      --maxpeers=200

[blxdyx]

remove those flag:

--bootnodes=enode://e17c2ec7ea588c46c0ceb03a2d621a020a49def8564276e3b82bb90f1d5aa9f929e33a3e894a69783a7a87d92d3f2c28af581171ce29f22e12434541c6c0c53a@18.195.40.224:33303,enode://f6313a784ec90da86ddae53fb76830c3eefc69e6e796f3f5e6e4ce4cc45fbb1dc8684e34ab5951a5047a5d376584d3a668ff88eae4dd4f0f0ced6d96e942ee89@18.195.40.224:33313,enode://1cc4534b14cfe351ab740a1418ab944a234ca2f702915eadb7e558a02010cb7c5a8c295a3b56bcefa7701c07752acd5539cb13df2aab8ae2d98934d712611443@52.71.43.172:30311,enode://28b1d16562dac280dacaaf45d54516b85bc6c994252a9825c5cc4e080d3e53446d05f63ba495ea7d44d6c316b54cd92b245c5c328c37da24605c4a93a0d099c4@34.246.65.14:30311,enode://5a7b996048d1b0a07683a949662c87c09b55247ce774aeee10bb886892e586e3c604564393292e38ef43c023ee9981e1f8b335766ec4f0f256e57f8640b079d5@35.73.137.11:30311
    --staticpeers=enode://551c8009f1d5bbfb1d64983eeb4591e51ad488565b96cdde7e40a207cfd6c8efa5b5a7fa88ed4e71229c988979e4c720891287ddd7d00ba114408a3ceb972ccb@34.245.70.169:30311,enode://c637c90d6b9d1d0038788b163a749a7a86fed2e7d0d13e5dc920ab144bb432ed1e3e00b54c1a93cecba479037601ba9a5937a88fe0be949c651043473c0d1e5b@52.17.25.179:30311,enode://bac6a548c7884270d53c3694c93ea43fa87ac1c7219f9f25c9d57f6a2fec9d75441bc4bad1e81d78c049a1c4daf3b1404e2bbb5cd9bf60c0f3a723bbaea110bc@54.171.201.213:30311,enode://a232f92d1e76447b93306ece2f6a55ac70ca4633fae0938d71a100757eaf8526e6bbf720aa70cba1e6d186be17291ad1ee851a35596ec6caa2fdf135ce4b6b68@107.20.124.16:30311,enode://accbc0a5af0af03e1ec3b5e80544bdceea48011a6928cd82d2c1a9c38b65fd48ec970ba17bd8c0b0ec21a28faec9efe1d1ce55134784b9207146e2f62d8932ba@54.162.32.1:30311,enode://16c7e98f78017dafeaa4129647d1ec66b32ee9be5ec753708820b7363091ceb310f575e7abd9603005e0e34d7b3316c1a4b6c8c42d7f074ed2eb4d073f800a03@3.85.216.212:30311
    --maxpeers=200

[wtdcode]

@blxdyx My nodes are also lagging 5-10 blocks and the logs indicates another story:

erigon-1  | [INFO] [06-14|10:04:15.022] Timings (slower than 50ms)               Bodies=80ms Execution=1.187s HashState=862ms IntermediateHashes=1.011s CallTraces=299ms AccountHistoryIndex=174ms StorageHistoryIndex=470ms LogIndex=1.236s TxLookup=262ms

The disk I'm using is raid0 of TLC ssds and going well with mainnet clients (erigon, reth etc). It also works well with previous bsc-erigon versions. The IntermediateHashes and LogIndex stages take too long and I will investigate it a bit.

[blxdyx]

What's the log before?

[Aracki]

@blxdyx I've tried that as you suggested but without any help. I still got these "staircases" (syncing in batches).

It's also interesting (second screenshot) that lag is really lower during the weekends. Probably it has something to do with "transaction per block" and network activity.

Can you share your local dev setup, or do you monitor your peer number over time?

[blxdyx]

It's my start command. Use a nvme device.
erigon --bodies.cache=214748364800 --batchSize=4096M --txpool.disable --metrics.addr=0.0.0.0 --log.console.verbosity=eror --log.dir.verbosity=dbug --http --ws --http.api=web3,net,eth,debug,trace,txpool,admin --http.addr=0.0.0.0 --db.pagesize=16k --datadir ./data --private.api.addr=localhost:9090 --chain=bsc --metrics

[Aracki]

Ok, it definitely comes down that the reason is not using NVME.

Unfortunately when using GCP persistent disks, it's not possible.

Thanks for the help anyway!

[blxdyx]

Ok, it definitely comes down that the reason is not using NVME.

Unfortunately when using GCP persistent disks, it's not possible.

Thanks for the help anyway!

@Aracki Working on integrate erigon3, it can solve this problem maybe.