feat: add support for nested transaction rollbacks via savepoints in sql #4375

LucianBuzzo · 2023-10-17T21:59:12Z

This is my first OSS contribution for a Rust project, so I'm sure I've made some stupid mistakes, but I think it should mostly work :)

This change adds a mutable depth counter, that can track how many levels deep a transaction is, and uses savepoints to implement correct rollback behaviour. Previously, once a nested transaction was complete, it would be saved with COMMIT, meaning that even if the outer transaction was rolled back, the operations in the inner transaction would persist. With this change, if the outer transaction gets rolled back, then all inner transactions will also be rolled back.

Different flavours of SQL servers have different syntax for handling savepoints, so I've had to add new methods to the Queryable trait for getting the commit and rollback statements. These are both parameterized by the current depth.

I've additionally had to modify the begin_statement method to accept a depth parameter, as it will need to conditionally create a savepoint.

Client PR: prisma/prisma#21678

CLAassistant · 2023-10-17T22:00:40Z

All committers have signed the CLA.

LucianBuzzo · 2023-10-18T16:01:39Z

This should resolve the issue described here prisma/prisma#15212

LucianBuzzo · 2023-10-20T08:32:05Z

@miguelff @Weakky If you have a moment could you take a look a this? If it works as I think it does ( 🤞 ) it's going to fix a lot of headaches for me and my team! TIA 🙏

janpio · 2023-10-27T00:55:55Z

Would you consider this a breaking change compared to current behavior @LucianBuzzo? It smells a bit like that to me because current functionality would change. Agree?

LucianBuzzo · 2023-10-27T08:25:10Z

@janpio I think that the current behaviour is unexpected, so this would be a bug fix or new "feature". It's possible that people have made applications that rely on the current behaviour, but this is true of any bug IMO.
As an example of how I consider this a bug: the documentation for transactions shows an example where a transfer between two accounts I rolled back, but it's not explained that this will not work if the transaction is nested. If you try to make behaviour like this and couple it with something like the RLS example from the client extensions, it will fail and the user has to do a lot of debugging to find out why.

quaint/src/connector/mssql.rs

janpio · 2023-10-28T22:03:08Z

Thanks for the explanation @LucianBuzzo, I get it now!

I could also connect it to prisma/prisma#19346 then which is about the same problem.

Did you find a bug report about the incorrect behavior? Optimally this issue would close that one so we have a closed bug in our release notes when we add this, that makes it clearer that this is a "breaking change" only in the context of that it fixes a bug. (Maybe you can create the issue if non exists yet!? Thanks.)

(There are also the related prisma/prisma#9710 and prisma/prisma#12898, but I think they want to completely replace transactions with BEGIN/COMMIT with savepoints - which this PR does not do. Do you agree that there is no overlap?)

This should probably be documented in the future with a new "Nested interactive transactions" sub headline under https://www.prisma.io/docs/concepts/components/prisma-client/transactions#interactive-transactions?

What would be good test cases for prisma/prisma? (Optimally those fail right now, but will succeed when this PR here is merged to show that this properly improves things)

janpio · 2023-10-28T22:04:38Z

You can ignore all the failing "Driver Adapters" tests, and also the Vitess and MySQL on Linux ones - those are currently flaky.

But these look relevant and need to be fixed:

Lots of formatting things: https://github.com/prisma/prisma-engines/actions/runs/6553297366/job/18151456208?pr=4375
Some Rust things: https://github.com/prisma/prisma-engines/actions/runs/6553297349/job/18151455513?pr=4375

LucianBuzzo · 2023-10-29T07:40:05Z

Thanks for the review @janpio I'll get the code issues resolved.

The issue prisma/prisma#19346 describes the exact problem, I just haven't referenced it as I wanted to have this PR reviewed first. For prisma/prisma#9710 and prisma/prisma#12898 this PR will resolve those issues, as long as they are using an SQL DB.
To get integration tests in an isolated transaction, you would simply need to run your test cases inside an interactive transaction:

describe('some test', () => {
  it('should work', async () => {
    try {
    await prisma.$transaction(async (tx) => {
      //...
      expect(x).toBe(y)
      
      throw new Error('rollback')
    })
    } catch (e) {
      // ignore e
    }
  })
 })

This is a similar pattern to how I discovered this issue myself - trying to test if a user is able to perform an operation when running Yates RBAC.

I'll add some lines about nested transactions to the docs. As for a test case in https://github.com/prisma/prisma a simple way to do it would be to create a client extension that wraps all queries in a transaction and then test that the interactive transaction example in the docs (transferring money between accounts) works as expected.

janpio · 2023-10-29T10:47:00Z

Wouldn't it then also be possible to just create a standalone test case that wraps multiple transactions, without the need to change how we run tests or use an extension? (I would assume the extension just does that automatically, but it should be possible to also express that explicitly and simpler)

For prisma/prisma#9710 and prisma/prisma#12898 this PR will resolve those issues, as long as they are using an SQL DB.

Don't these suggest to use just savepoints for the actual tests, instead of normal transactions? I am still a bit unclear about the approach.

LucianBuzzo · 2023-10-29T12:01:56Z

From my reading of those issues, it seems that you could achieve the result they want using a nested transaction that utilises savepoints. I would let the original authors correct me if this is not the case though!

LucianBuzzo · 2023-10-29T12:03:16Z

And yes you could not use a client extension in the test case, I simply provided the example above as one possibility 👍

janpio · 2023-10-29T12:11:36Z

CI run is done, some more Clippy stuff to make it able to compile I guess: https://github.com/prisma/prisma-engines/actions/runs/6682077529/job/18158479734?pr=4375 (I also don't know any Rust, so unfortunately can't be of help here)

janpio · 2023-10-29T16:19:21Z

(Can you do a fake PR to README.md adding a newline or some other minimal, non-intrusive change that I can merge easily? Then your commits in this PR will automatically run tests going forward - and you don't have to wait for me to click the button😆)

LucianBuzzo · 2023-10-30T10:18:56Z

@janpio Here's a PR for the Quaint README #4399

LucianBuzzo · 2023-10-30T10:33:40Z

@janpio I've also opened a PR with a failing test case for the prisma client here prisma/prisma#21678

Jolg42 · 2023-11-01T14:19:48Z

Note: I brought the branch into the repo, it will release the engines automatically after a while
https://github.com/prisma/prisma-engines/tree/integration/sql-nested-transactions
https://buildkite.com/prisma/test-prisma-engines/builds/21364

janpio · 2023-11-01T15:54:36Z

Happened, version shared here: prisma/prisma#21678 (comment)

LucianBuzzo · 2023-11-02T08:43:50Z

@janpio @Jolg42 I think that this commit 5d2cc0a should allow us to send an existing transaction ID to the engine in the case of a nested transaction. I'm not sure how to test this behaviour in this repo - any tips on how to do this, or an existing test I could use as an example?

aqrln

First round of comments, didn't review the whole PR yet. I'd also like to chat with you about the overall design and what the API on TS side would look like because I don't have the full context here.

quaint/src/connector/mssql/native/mod.rs

quaint/src/connector/queryable.rs

quaint/src/connector/transaction.rs

aqrln · 2024-09-27T11:46:00Z

quaint/src/connector/transaction.rs

@@ -36,23 +43,26 @@ pub(crate) struct TransactionOptions {
 /// transaction object will panic.
 pub struct DefaultTransaction<'a> {
    pub inner: &'a dyn Queryable,
+    pub depth: Arc<Mutex<i32>>,


I'd still prefer an AtomicU32

So something like this?

Suggested change

pub depth: Arc<Mutex<i32>>,

pub depth: AtomicU32,

aqrln · 2024-09-27T11:49:59Z

quaint/src/connector/transaction.rs

        Ok(this)
    }
 }

 #[async_trait]
 impl<'a> Transaction for DefaultTransaction<'a> {
+    async fn begin(&mut self) -> crate::Result<()> {


Why do these methods take self by mutable reference? You have interior mutability in depth so this seems unnecessarily restrictive.

I think this might be leftover from the initial version of the PR, before I encapsulated depth inside the Transaction. I'll fix this 👍

If I switch from the Mutex to AtomicU32 I think these will need to be a mutable reference, unless I'm misunderstanding

No, atomics have interior mutability

query-engine/connectors/query-connector/src/interface.rs

query-engine/connectors/mongodb-query-connector/src/interface/transaction.rs

query-engine/driver-adapters/executor/src/bench.ts

query-engine/driver-adapters/src/proxy.rs

luisgrases · 2024-11-06T20:53:01Z

Still watching this initiative closely and with great interest

LucianBuzzo · 2024-11-06T21:48:29Z

@luisgrases Now that #5001 is merged I think this PR is very close to completion 🚀

LucianBuzzo · 2024-11-25T14:01:10Z

Update from my end: all tests are passing except on Mssql, where the default transaction methods are used instead of the Mssql implementation. If anyone has any insight into how to debug this issue please let me know!

This is my first OSS contribution for a Rust project, so I'm sure I've made some stupid mistakes, but I think it should mostly work :) This change adds a mutable depth counter, that can track how many levels deep a transaction is, and uses savepoints to implement correct rollback behaviour. Previously, once a nested transaction was complete, it would be saved with `COMMIT`, meaning that even if the outer transaction was rolled back, the operations in the inner transaction would persist. With this change, if the outer transaction gets rolled back, then all inner transactions will also be rolled back. Different flavours of SQL servers have different syntax for handling savepoints, so I've had to add new methods to the `Queryable` trait for getting the commit and rollback statements. These are both parameterized by the current depth. I've additionally had to modify the `begin_statement` method to accept a depth parameter, as it will need to conditionally create a savepoint. When opening a transaction via the transaction server, you can now pass the prior transaction ID to re-use the existing transaction, incrementing the depth. Signed-off-by: Lucian Buzzo <[email protected]>

aqrln · 2024-11-28T09:28:44Z

quaint/src/connector/mssql/native/mod.rs

+    // MSSQL doesn't have a "RELEASE SAVEPOINT" equivalent, so in a nested
+    // transaction we just continue onwards
+    fn release_savepoint_statement(&self, _depth: u32) -> Cow<'static, str> {
+        Cow::Owned("".to_string())


Suggested change

Cow::Owned("".to_string())

Cow::Borrowed("")

LucianBuzzo requested a review from a team as a code owner October 17, 2023 21:59

LucianBuzzo requested review from miguelff and Weakky and removed request for a team October 17, 2023 21:59

LucianBuzzo force-pushed the lucianbuzzo/sql-nested-transactions branch from 6f375b8 to b984ae4 Compare October 17, 2023 22:00

janpio reviewed Oct 28, 2023

View reviewed changes

quaint/src/connector/mssql.rs Outdated Show resolved Hide resolved

janpio self-assigned this Oct 28, 2023

This comment was marked as off-topic.

Sign in to view

janpio removed request for miguelff and Weakky October 29, 2023 16:19

LucianBuzzo force-pushed the lucianbuzzo/sql-nested-transactions branch from da206e7 to a8af640 Compare October 30, 2023 10:16

LucianBuzzo mentioned this pull request Oct 30, 2023

feat: Add test for nested transaction rollback behaviour in SQL databases prisma/prisma#21678

Open

LucianBuzzo force-pushed the lucianbuzzo/sql-nested-transactions branch from b0bab65 to a220cf5 Compare September 26, 2024 15:31

aqrln assigned aqrln and unassigned SevInf Sep 27, 2024

aqrln reviewed Sep 27, 2024

View reviewed changes

query-engine/connectors/query-connector/src/interface.rs Show resolved Hide resolved

aqrln reviewed Sep 27, 2024

View reviewed changes

query-engine/connectors/mongodb-query-connector/src/interface/transaction.rs Show resolved Hide resolved

aqrln reviewed Sep 27, 2024

View reviewed changes

query-engine/driver-adapters/executor/src/bench.ts Outdated Show resolved Hide resolved

aqrln reviewed Sep 27, 2024

View reviewed changes

query-engine/driver-adapters/src/proxy.rs Outdated Show resolved Hide resolved

LucianBuzzo force-pushed the lucianbuzzo/sql-nested-transactions branch from f787095 to b877e71 Compare September 30, 2024 13:22

LucianBuzzo force-pushed the lucianbuzzo/sql-nested-transactions branch 12 times, most recently from b8ca794 to 670e5ad Compare November 12, 2024 15:36

0xdevalias mentioned this pull request Nov 26, 2024

Support Nested Transactions / Savepoints prisma/prisma#19346

Open

LucianBuzzo added 2 commits November 27, 2024 22:29

use separate methods for handling savepoints

8f090ee

LucianBuzzo force-pushed the lucianbuzzo/sql-nested-transactions branch 2 times, most recently from cf219c7 to 8f090ee Compare November 28, 2024 07:00

aqrln reviewed Nov 28, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add support for nested transaction rollbacks via savepoints in sql #4375

feat: add support for nested transaction rollbacks via savepoints in sql #4375

LucianBuzzo commented Oct 17, 2023 •

edited by aqrln

Loading

CLAassistant commented Oct 17, 2023 •

edited

Loading

LucianBuzzo commented Oct 18, 2023

LucianBuzzo commented Oct 20, 2023

janpio commented Oct 27, 2023

LucianBuzzo commented Oct 27, 2023

janpio commented Oct 28, 2023

janpio commented Oct 28, 2023

This comment was marked as off-topic.

LucianBuzzo commented Oct 29, 2023 •

edited

Loading

janpio commented Oct 29, 2023

LucianBuzzo commented Oct 29, 2023

LucianBuzzo commented Oct 29, 2023

janpio commented Oct 29, 2023

janpio commented Oct 29, 2023

LucianBuzzo commented Oct 30, 2023

LucianBuzzo commented Oct 30, 2023

Jolg42 commented Nov 1, 2023

janpio commented Nov 1, 2023

LucianBuzzo commented Nov 2, 2023

aqrln left a comment •

edited

Loading

aqrln Sep 27, 2024

LucianBuzzo Sep 30, 2024

aqrln Nov 28, 2024

aqrln Sep 27, 2024

LucianBuzzo Sep 30, 2024

LucianBuzzo Sep 30, 2024

aqrln Nov 26, 2024

luisgrases commented Nov 6, 2024

LucianBuzzo commented Nov 6, 2024

LucianBuzzo commented Nov 25, 2024

aqrln Nov 28, 2024

feat: add support for nested transaction rollbacks via savepoints in sql #4375

Are you sure you want to change the base?

feat: add support for nested transaction rollbacks via savepoints in sql #4375

Conversation

LucianBuzzo commented Oct 17, 2023 • edited by aqrln Loading

CLAassistant commented Oct 17, 2023 • edited Loading

LucianBuzzo commented Oct 18, 2023

LucianBuzzo commented Oct 20, 2023

janpio commented Oct 27, 2023

LucianBuzzo commented Oct 27, 2023

janpio commented Oct 28, 2023

janpio commented Oct 28, 2023

This comment was marked as off-topic.

LucianBuzzo commented Oct 29, 2023 • edited Loading

janpio commented Oct 29, 2023

LucianBuzzo commented Oct 29, 2023

LucianBuzzo commented Oct 29, 2023

janpio commented Oct 29, 2023

janpio commented Oct 29, 2023

LucianBuzzo commented Oct 30, 2023

LucianBuzzo commented Oct 30, 2023

Jolg42 commented Nov 1, 2023

janpio commented Nov 1, 2023

LucianBuzzo commented Nov 2, 2023

aqrln left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luisgrases commented Nov 6, 2024

LucianBuzzo commented Nov 6, 2024

LucianBuzzo commented Nov 25, 2024

Choose a reason for hiding this comment

LucianBuzzo commented Oct 17, 2023 •

edited by aqrln

Loading

CLAassistant commented Oct 17, 2023 •

edited

Loading

LucianBuzzo commented Oct 29, 2023 •

edited

Loading

aqrln left a comment •

edited

Loading