Support for yielded execution #519

akhi3030 · 2023-11-17T12:49:29Z

No description provided.

render · 2023-11-17T12:49:33Z

Your Render PR Server URL is https://nomicon-pr-519.onrender.com.

Follow its progress at https://dashboard.render.com/static/srv-clbm2n54lnec73ashvhg.

DavidM-D

Looks like a really good start and I'm broadly happy with it in it's current form. The biggest thing I'd change is the changing promise ID each time resume is called, everything else is nits.

DavidM-D · 2023-12-11T01:04:46Z

neps/nep-519-yield-execution.md

+/// either respond to the caller or create another promise.
+///
+/// If the contract fails to call `yield_resume()` within `yield_num_blocks`,
+/// then the protocol will call the method on the smart contract that is


Suggested change

/// then the protocol will call the method on the smart contract that is

/// then the protocol may call the method on the smart contract that is

I don't think this needs to be required in the protocol, rather it's an option the protocol has.

Hmm... I would've thought that this must happen at some point otherwise the smart contract might leak memory. If the contract created some state and then called yield_create which it then wanted to free in the callback, that might never happen.

Also I see this as analogous to the case where contract A calls B and B fails to respond to A. Then the protocol generates a response for A.

OK I agree with this, it makes life easier for contract developers if there is always a response eventually.

DavidM-D · 2023-12-11T01:09:16Z

neps/nep-519-yield-execution.md

+/// `method_name_len` and `method_name_ptr` and this method will be expected to
+/// either respond to the caller or create another promise.
+///
+/// If the contract fails to call `yield_resume()` within `yield_num_blocks`,


Suggested change

/// If the contract fails to call `yield_resume()` within `yield_num_blocks`,

/// If the contract fails to, in a `yield_resume()`, call `value_return`, `panic` or `abort` within `yield_num_blocks`,

This makes the API much more flexible, but does it make it harder to implement?

If I understood the idea properly, then I think we need to think about the semantics some more. Let's say the following has happened:

contract A called the signer to get a signature and the signer has called yield create.

contract B called the signer to get a signature and the signer has called yield create.

now some other contract C calls the signer and the signer panics, which one of the above two outstanding executions should we terminate? I don't think we can really pick between them so the only sensible choice seems to be to terminate both. It doesn't seem like that that is a good choice either.

So if contract C calls the signer and that call panics before it calls yield_create then the call from contract C should panic and none of the others.

This refers to a situation where contract D calls yield_create, the signer calls yield_resume and then the resumed call panics, ending contract D's suspended call. It shouldn't effect any other outstanding calls.

neps/nep-519-yield-execution.md

bowenwang1996

@akhi3030 @saketh-are I'd like to see an end-to-end example of how this is going to be used in a smart contract. While there is some high-level description on the execution flow, I think some important details are missing. For example, I don't think the API works well as is because the caller of the yield_resume is expected to pass in a receipt id, not a promise index, whereas for runtime, it needs the index instead of receipt id. Also, it is not clear from the existing API how an argument would be passed to the callback, which is crucial for chain signatures

akhi3030

Thanks for the feedback @DavidM-D. I've added some comments. I am going to make some changes to the NEP to remove the concept of yield_num_blocks. Unfortunately, given the crummy github UI, it might become challenging to continue the comment threads after the changes. We will have to stumble along somehow.

akhi3030 · 2023-12-12T09:28:09Z

neps/nep-519-yield-execution.md

+/// either respond to the caller or create another promise.
+///
+/// If the contract fails to call `yield_resume()` within `yield_num_blocks`,
+/// then the protocol will call the method on the smart contract that is


Hmm... I would've thought that this must happen at some point otherwise the smart contract might leak memory. If the contract created some state and then called yield_create which it then wanted to free in the callback, that might never happen.

Also I see this as analogous to the case where contract A calls B and B fails to respond to A. Then the protocol generates a response for A.

neps/nep-519-yield-execution.md

akhi3030 · 2023-12-12T09:33:50Z

neps/nep-519-yield-execution.md

+/// `method_name_len` and `method_name_ptr` and this method will be expected to
+/// either respond to the caller or create another promise.
+///
+/// If the contract fails to call `yield_resume()` within `yield_num_blocks`,


If I understood the idea properly, then I think we need to think about the semantics some more. Let's say the following has happened:

contract A called the signer to get a signature and the signer has called yield create.

contract B called the signer to get a signature and the signer has called yield create.

now some other contract C calls the signer and the signer panics, which one of the above two outstanding executions should we terminate? I don't think we can really pick between them so the only sensible choice seems to be to terminate both. It doesn't seem like that that is a good choice either.

neps/nep-519-yield-execution.md

Co-authored-by: DavidM-D <[email protected]>

bowenwang1996 · 2024-01-06T17:26:24Z

At the protocol working group meeting yesterday, different members expressed concerns on these new runtime APIs. I'll attempt the summarize them below:

There is nothing that stops someone from emulating the API in the sdk by saving the promise in the state and triggering it again once it is ready to be resumed. The call stack is discontiguous but that could potentially be fixed on the tooling level and it seems that Aurora already does this.
Whether or not the delayed promise has an expiration time, the introduction of this API could lead to a new type of failure that does not previously exist where the contract execution fails with no way to clean up the state. It could be fine for the chain signature use case, but if some (potentially malicious) developer uses this in a defi contract for oracles (for example), then it is possible that the contract call stalls/fails in the middle, which results in users losing access to their fund.
The API does not specify who can call "resume". This means that a malicious attacker can just call resume with invalid data to make a transaction fail. While the attacker does need to spend gas themselves, it is quite cheap. This allows for a relatively easy DoS attack, especially given that the gas that a user needs to spend may be larger than what an attacker needs to spend to cancel the transaction. In addition, failure in a chain of calls A -> B -> C may also lead to inconsistent state in one of the smart contract involved.

cc @mm-near @birchmd @mfornet

akhi3030 · 2024-01-08T13:20:28Z

* There is nothing that stops someone from emulating the API in the sdk by saving the promise in the state and triggering it again once it is ready to be resumed. The call stack is discontiguous but that could potentially be fixed on the tooling level and it seems that Aurora already does this.

This main motivation for this work is the MPC project. If we can satisfy the requirements without having to change the protocol, that much better.

Can you provide some links to how aurora does this? Maybe then @DavidM-D and @itegulov can review them.

* Whether or not the delayed promise has an expiration time, the introduction of this API could lead to a new type of failure that does not previously exist where the contract execution fails with no way to clean up the state. It could be fine for the chain signature use case, but if some (potentially malicious) developer uses this in a defi contract for oracles (for example), then it is possible that the contract call stalls/fails in the middle, which results in users losing access to their fund.

I am not sure I completely understand this point. Especially about not having a way to clean up the state. If things expire, the protocol will call the contract. So is the contract may not handle that case properly? Isn't that similar to how the current Promise API works?

* The API does not specify who can call "resume". This means that a malicious attacker can just call resume with invalid data to make a transaction fail. While the attacker does need to spend gas themselves, it is quite cheap. This allows for a relatively easy DoS attack, especially given that the gas that a user needs to spend may be larger than what an attacker needs to spend to cancel the transaction.

If contract A called yield_create, only contract A can call resume on that promise. No other contract can.

In addition, failure in a chain of calls A -> B -> C may also lead to inconsistent state in one of the smart contract involved.

Can you please expand on this? If A is calling B, then B can fail for any number of reasons today, (it can trap, run out of gas, have invalid state). Is the problem that we are introducing a new failure mode now that B has to handle?

neps/nep-519-yield-execution.md

bowenwang1996 · 2024-01-10T04:00:36Z

I am not sure I completely understand this point. Especially about not having a way to clean up the state. If things expire, the protocol will call the contract. So is the contract may not handle that case properly? Isn't that similar to how the current Promise API works?

I misread the proposal and therefore thought incorrectly that the execution would just fail at that point.

If contract A called yield_create, only contract A can call resume on that promise. No other contract can.

Yes, I guess the concern here is that a contract developer can shoot themselves in the foot if they are not careful with how they valid the computation result that is passed to yield_resume because while only the contract itself can resume the computation. Anyone can call the method (signature_available in this case) that provides the computation result. If there is no proper validation of the result, then a malicious attacker can easily submit something that causes the original transaction to fail. I don't think this is a concern for MPC signature per se since that contract will be carefully written and audited, but still worth noting for the general use case.

akhi3030 · 2024-01-10T14:33:10Z

Yes, I guess the concern here is that a contract developer can shoot themselves in the foot if they are not careful with how they valid the computation result that is passed to yield_resume because while only the contract itself can resume the computation. Anyone can call the method (signature_available in this case) that provides the computation result. If there is no proper validation of the result, then a malicious attacker can easily submit something that causes the original transaction to fail. I don't think this is a concern for MPC signature per se since that contract will be carefully written and audited, but still worth noting for the general use case.

Yes, this is always possible. Luckily, we have some nice tools (https://github.com/aurora-is-near/near-plugins/blob/master/near-plugins/src/access_controllable.rs) available that could be used.

saketh-are · 2024-02-14T22:14:19Z

neps/nep-519-yield-execution.md

+/// should be resumed.
+///
+/// `payload_len` and `payload_ptr`: the smart contract can provide an optional
+/// list of arguments that should be passed to the method that will be resumed.


nit: This is not really a list of arguments from the yield/resume perspective, just an arbitrary byte sequence. The rust sdk makes it easy to jsonify a tuple of objects and parse them back out, but that is not relevant at the level of the host function.

Hopefully improved now.

saketh-are · 2024-02-14T22:18:49Z

neps/nep-519-yield-execution.md

+
+## Summary
+
+Today, when a smart contract is called by a user or another contract, it has no sensible way to delay responding to the caller.  There exist use cases where contracts would benefit from being able to delay responding till some arbitrary time in the future.  This proposal introduces such possibility into the NEAR protocol.


till some arbitrary time

This feels potentially misleading to me given that we enforce a timeout. Could it be clearer to say something like "await a future transaction" rather than "delay responding"?

Hopefully improved now.

saketh-are · 2024-02-14T22:20:27Z

neps/nep-519-yield-execution.md

+
+Examples include when a smart contract (`S`) provides MPC signing capabilities parties external to the NEAR protocol are computing the signature.  The rough steps are:
+1. Signer contract provides a function `fn sign_payload(Payload, ...)`.
+2. When called, the contract updates some contract state which is being monitored by external indexers to indicate that a new signing request has been received.  It also defers replying to the caller.


In practice I think the signers will directly index the transactions on the account and observe the receipts generated by yield (no need for any storage in contract state). But perhaps it is easier to explain things with this example.

Hopefully improved now. I'll add the example in a subsequent commit.

akhi3030 · 2024-02-28T14:28:12Z

One thing I have a hard time wrapping my head around it the necessity for the "Reference Implementation" section

I agree, I also found it a bit difficult to figure out how much details to present here. Inspired by this NEP I think it is fine to just provide a link to the PR. I will make the change to this PR now. And let the SMEs decide if they are happy with the change or not.

akhi3030 · 2024-02-28T18:30:46Z

@birchmd, @nagisa: thank you very much for the review. I believe I have addressed all the feedback that you provided. There is an ongoing discussion on the edge case around when a timeout has happened but the callback has not executed and the contract calls resume. Once we resolve the discussion, I will update the spec. But this should be a minor change. So I believe that this should be ready for you to look once again.

victorchimakanu · 2024-02-29T02:48:01Z

NEP Status (Updated by NEP Moderators)

Status: Approved

SME reviews:

Protocol SME @nagisa: Support for yielded execution Support for yielded execution #519
Protocol SME @birchmd Support for yielded execution Support for yielded execution #519

Protocol Work Group voting indications (❔ | 👍 | 👎 ):

birchmd

Thank for updating the NEP @akhi3030 ! It looks good to me now.

As both an SME and working group member I approve this NEP and look forward to seeing the feature go live on testnet in the future.

victorchimakanu · 2024-03-01T10:48:59Z

As the moderator, I would like to thank the author @akhi3030 for submitting this NEP extension, and the work group members for your review. Based on the voting indications above, it seems like this proposal is close to a decision. We are therefore scheduling the 8th Protocol Work group meeting next week.

Anyone can discuss the technical details by adding your comments to this discussion. And join the call to learn about the final decision and how to get more involved.

Meeting Info:

📅 Date: Friday, March 8th, 4 PM UTC
✏️ Register here

neps/nep-519-yield-execution.md

bowenwang1996 · 2024-03-06T23:19:04Z

As a working group member, I lean towards accepting this proposal. It is a great step towards enabling smart contracts to interact with offchain computation while still preserving the call stack.

mfornet · 2024-03-07T02:29:11Z

As a working group member, I lean toward approving this NEP.

This NEP solves a problem that can't be solved today, which is having simultaneously:

Delayed responses
Guaranteed execution of callback
Abstraction from the caller about the delayed response.

Delayed responses are particularly useful to seamlessly use off-chain computation on-chain. This could involve for example oracle queries (eg fetch_tweet(tweet_id)) or privacy protocol execution such as signing some data offchain.

mm-near · 2024-03-07T16:49:37Z

As a working group member, I lean toward approving this NEP.

one small comment: in comments you say:

/// `data_id_len` and `data_it_ptr`: Used to pass the unique resumption token
/// that was returned to the smart contract in the `promise_yield_create()`
/// function.

But promise_yield_create seems to be returning u64

akhi3030 · 2024-03-07T17:42:20Z

* one small comment: in comments you say:

/// `data_id_len` and `data_it_ptr`: Used to pass the unique resumption token
/// that was returned to the smart contract in the `promise_yield_create()`
/// function.

But promise_yield_create seems to be returning u64

Yeah, the comment is referring to the resumption token returned in the register. I'll update the comment to address this.

Co-authored-by: Simonas Kazlauskas <[email protected]>

neps/nep-519-yield-execution.md

walnut-the-cat · 2024-06-17T17:09:16Z

can we merge this into master?

akhi3030 · 2024-06-17T17:17:01Z

@near/nep-moderators: please review and approve the PR. The work was just released to the mainnet.

WIP: yield execution

c23b15c

akhi3030 mentioned this pull request Nov 17, 2023

Runtime triggers #516

Open

akhi3030 added 2 commits November 17, 2023 13:59

rename file as per instructions

1f8dbbd

Add a summary and update some metadata

0931acd

walnut-the-cat mentioned this pull request Nov 17, 2023

[Tracking issue] Protocol implementation for Yield execution near/nearcore#10210

Open

frol added WG-protocol Protocol Standards Work Group should be accountable S-draft/needs-author-revision A NEP in the DRAFT stage that needs an author revision. A-NEP A NEAR Enhancement Proposal (NEP). labels Nov 17, 2023

akhi3030 added 3 commits November 20, 2023 14:10

minor cleanup

5b54b47

add motivation

20c993f

add draft of host function calls

b8d882c

DavidM-D reviewed Dec 11, 2023

View reviewed changes

bowenwang1996 reviewed Dec 11, 2023

View reviewed changes

akhi3030 commented Dec 12, 2023

View reviewed changes

Apply suggestions from code review

fcb5230

Co-authored-by: DavidM-D <[email protected]>

mfornet reviewed Jan 8, 2024

View reviewed changes

neps/nep-519-yield-execution.md Outdated Show resolved Hide resolved

itegulov mentioned this pull request Jan 10, 2024

🔷 [Epic] Multichain near/mpc#326

Closed

nagisa mentioned this pull request Jan 22, 2024

Yield Execution (NEP 516 / NEP 519) near/nearcore#10455

Open

akhi3030 added 2 commits February 14, 2024 17:13

update the API

bc6708b

add alternatives

aef122a

saketh-are reviewed Feb 14, 2024

View reviewed changes

akhi3030 added 4 commits February 19, 2024 10:53

address some review comments

5ce8652

add discussion of security implications

4cc7d88

fix lint issue

a57d2a5

add a reference implementation

ce2a28d

cleanup templates that are not needed

09cc2e0

akhi3030 added 6 commits February 28, 2024 15:29

remove ref impl

8facd6b

clarify how the args are available and some formatting

43fa6f3

minor improv english

cd9bc85

apply suggestion from review

812cff3

apply suggestion from review

2728727

Improve specification

577970e

akhi3030 requested review from nagisa and birchmd February 28, 2024 18:30

birchmd approved these changes Feb 29, 2024

View reviewed changes

nagisa reviewed Mar 1, 2024

View reviewed changes

neps/nep-519-yield-execution.md Outdated Show resolved Hide resolved

akhi3030 and others added 2 commits March 7, 2024 18:45

Update neps/nep-519-yield-execution.md

13ebeb6

Co-authored-by: Simonas Kazlauskas <[email protected]>

some minor improvements to the comments

10920a6

saketh-are reviewed Mar 8, 2024

View reviewed changes

neps/nep-519-yield-execution.md Outdated Show resolved Hide resolved

apply minor nit comment

77bb83e

walnut-the-cat approved these changes Jun 18, 2024

View reviewed changes

akhi3030 merged commit 48c9f16 into master Jun 18, 2024
5 of 6 checks passed

akhi3030 deleted the yield-execution branch June 18, 2024 14:29

flmel added S-approved A NEP that was approved by a working group. and removed S-review/needs-wg-to-assign-sme A NEP that needs working group to assign two SMEs. labels Oct 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for yielded execution #519

Support for yielded execution #519

akhi3030 commented Nov 17, 2023

render bot commented Nov 17, 2023

DavidM-D left a comment

DavidM-D Dec 11, 2023

akhi3030 Dec 12, 2023

DavidM-D Dec 14, 2023

DavidM-D Dec 11, 2023

akhi3030 Dec 12, 2023

DavidM-D Dec 14, 2023

bowenwang1996 left a comment

akhi3030 left a comment

akhi3030 Dec 12, 2023

akhi3030 Dec 12, 2023

bowenwang1996 commented Jan 6, 2024

akhi3030 commented Jan 8, 2024

bowenwang1996 commented Jan 10, 2024

akhi3030 commented Jan 10, 2024

saketh-are Feb 14, 2024

akhi3030 Feb 19, 2024

saketh-are Feb 14, 2024

akhi3030 Feb 19, 2024

saketh-are Feb 14, 2024

akhi3030 Feb 19, 2024

akhi3030 commented Feb 28, 2024

akhi3030 commented Feb 28, 2024

victorchimakanu commented Feb 29, 2024 •

edited

Loading

birchmd left a comment •

edited

Loading

victorchimakanu commented Mar 1, 2024

bowenwang1996 commented Mar 6, 2024

mfornet commented Mar 7, 2024

mm-near commented Mar 7, 2024

akhi3030 commented Mar 7, 2024

walnut-the-cat commented Jun 17, 2024

akhi3030 commented Jun 17, 2024

	/// then the protocol will call the method on the smart contract that is
	/// then the protocol may call the method on the smart contract that is

	/// If the contract fails to call `yield_resume()` within `yield_num_blocks`,
	/// If the contract fails to, in a `yield_resume()`, call `value_return`, `panic` or `abort` within `yield_num_blocks`,


		## Summary

		Today, when a smart contract is called by a user or another contract, it has no sensible way to delay responding to the caller. There exist use cases where contracts would benefit from being able to delay responding till some arbitrary time in the future. This proposal introduces such possibility into the NEAR protocol.

Support for yielded execution #519

Support for yielded execution #519

Conversation

akhi3030 commented Nov 17, 2023

render bot commented Nov 17, 2023

DavidM-D left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bowenwang1996 left a comment

Choose a reason for hiding this comment

akhi3030 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bowenwang1996 commented Jan 6, 2024

akhi3030 commented Jan 8, 2024

bowenwang1996 commented Jan 10, 2024

akhi3030 commented Jan 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akhi3030 commented Feb 28, 2024

akhi3030 commented Feb 28, 2024

victorchimakanu commented Feb 29, 2024 • edited Loading

NEP Status (Updated by NEP Moderators)

birchmd left a comment • edited Loading

Choose a reason for hiding this comment

victorchimakanu commented Mar 1, 2024

Meeting Info:

bowenwang1996 commented Mar 6, 2024

mfornet commented Mar 7, 2024

mm-near commented Mar 7, 2024

akhi3030 commented Mar 7, 2024

walnut-the-cat commented Jun 17, 2024

akhi3030 commented Jun 17, 2024

victorchimakanu commented Feb 29, 2024 •

edited

Loading

birchmd left a comment •

edited

Loading