Large calldata in eth_sendRawTransaction causes a timeout exceeded error

[thusaras-vc]

Description

Making calls to eth_sendRawTransaction with large calldata (i.e. 12+ kB) return the following error:

• Error invoking RPC: timeout exceeded

1. There are several internal calls to /transactions/... leading up to this error that result in code 404
2. Followed by this error message:

• Failed to successfully submit sendRawTransaction for transaction .....

However, the same calls to /transactions/... succeed when tried again several minutes later

Steps to reproduce

make a call to the sendRawTransaction endpoint with a large transaction

curl my-rpc-endpoint -X POST -H "Content-Type: application/json" \
  --data '{"jsonrpc":"2.0","method":"eth_sendRawTransaction","params":["0x[LARGE_TRANSACTION]"],"id":1}'

Additional context

Logs - newest to oldest

--------

[2024-10-24 10:12:14.069] ERROR (rpc-server/89 on mainnet-json-rpc-relay-797979dc48-8m8sq): 
[Request ID] Error invoking RPC: timeout exceeded
}
"validNetworkError": false
},
"_code": 21
"status": {
at rejected (/home/node/app/packages/relay/dist/lib/clients/sdkClient.js:25:65)
at Generator.throw (<anonymous>)
at SDKClient.<anonymous> (/home/node/app/packages/relay/dist/lib/clients/sdkClient.js:447:40)

--------


Error: timeout exceeded
"stack":
"message": "timeout exceeded",
"type": "SDKClientError",
err: {
[2024-10-24 10:12:13.069] ERROR (relay-eth/89 on mainnet-json-rpc-relay-797979dc48-8m8sq): 
[Request ID] Failed to successfully submit sendRawTransaction for transaction XXX...YYY...ZZZ
}
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at rejected (/home/node/app/packages/relay/dist/lib/clients/mirrorNodeClient.js:25:65)
at Generator.throw (<anonymous>)
at MirrorNodeClient.<anonymous> (/home/node/app/packages/relay/dist/lib/clients/mirrorNodeClient.js:213:22)
at MirrorNodeClient.handleError (/home/node/app/packages/relay/dist/lib/clients/mirrorNodeClient.js:246:31)

--------


Error: Request failed with status code 404
"stack":
"message": "Request failed with status code 404",
"type": "Error",
err: {
[2024-10-24 10:12:12.069] ERROR (mirror-node/89 on mainnet-json-rpc-relay-797979dc48-8m8sq): 
[Request ID] [GET] transactions/0.0.XXX-YYY-ZZZ?nonce=0 404 status
}
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at rejected (/home/node/app/packages/relay/dist/lib/clients/mirrorNodeClient.js:25:65)
at Generator.throw (<anonymous>)
at MirrorNodeClient.<anonymous> (/home/node/app/packages/relay/dist/lib/clients/mirrorNodeClient.js:213:22)
at MirrorNodeClient.handleError (/home/node/app/packages/relay/dist/lib/clients/mirrorNodeClient.js:246:31)

--------


Error: Request failed with status code 404
"stack":
"message": "Request failed with status code 404",
"type": "Error",
err: {
[2024-10-24 10:12:11.069] ERROR (mirror-node/89 on mainnet-json-rpc-relay-797979dc48-8m8sq): 
[Request ID] [GET] transactions/0.0.XXX-YYY-ZZZ?nonce=0 404 status
}
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at rejected (/home/node/app/packages/relay/dist/lib/clients/mirrorNodeClient.js:25:65)
at Generator.throw (<anonymous>)
at MirrorNodeClient.<anonymous> (/home/node/app/packages/relay/dist/lib/clients/mirrorNodeClient.js:213:22)
at MirrorNodeClient.handleError (/home/node/app/packages/relay/dist/lib/clients/mirrorNodeClient.js:246:31)

--------


Error: Request failed with status code 404
"stack":
"message": "Request failed with status code 404",
"type": "Error",
err: {

[2024-10-24 10:12:10.069] ERROR (mirror-node/89 on mainnet-json-rpc-relay-797979dc48-8m8sq): 
[Request ID] [GET] transactions/0.0.XXX-YYY-ZZZ?nonce=0 404 status

Hedera network

mainnet

Version

v0.56.0

Operating system

Linux

[Nana-EC]

Thanks @thusaras-vc , we're actively looking into this and will update on a direction to improve the experience here

[quiet-node]

Heyy @thusaras-vc trhank you for taking the time to report this! This appears to be a known issue, for which we recently pushed a fix (PR#3161). Occasionally, transactions encounter "SDK timeout exceeded" (and sometimes "Connection Dropped") errors from the SDK but still potentially reach the consensus level. The fix in PR#3161 implemented filtering for these specific errors and continuing to poll the mirror node to confirm whether the transaction indeed failed or still made it to the EVM.

Just for a POC, I submitted a transaction with a calldata length of 12.3 KB on the Hedera testnet and things were working as expected. The transaction can be viewed on Hashscan here.

A few questions for clarification:

1. Did you find this issue occurs consistently or intermittently?
2. Could you please provide an example of large calldata to help with testing?
3. Which Relay are you using for submission? Was it your own Relay or the Hashio Relay?
4. Finally, could you please kindly run this test with the large calldata again with the latest Relay and confirm if the issue still happens?
   Appreciate the time and effort!

Additionally, regarding the 404 errors, these are expected as the mirror node sometimes requires extra time for records to fully populate. These errors may result from polling requests made by the Relay to the MN to check for records.

[quiet-node]

Reported in and tracked by hashgraph/hedera-sdk-js#2614