Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support async methods in stateful testing #4107

Open
Zac-HD opened this issue Sep 18, 2024 · 0 comments
Open

Support async methods in stateful testing #4107

Zac-HD opened this issue Sep 18, 2024 · 0 comments
Labels
interop how to play nicely with other packages new-feature entirely novel capabilities or strategies

Comments

@Zac-HD
Copy link
Member

Zac-HD commented Sep 18, 2024

@given() doesn't natively support async functions, but thanks to #1343 that can be handled downstream e.g. in pytest-trio. However, we don't have a good solution for stateful testing with async methods, and it's harder to add downstream - for example, hypothesis-trio copy-pasted all of stateful.py to add a few awaits, but is now several years of bugfixes and performance improvements behind.

I'd like to allow RuleBasedStateMachines to define a mix of sync and async methods for rules, invariants, and teardown.

Instead of a firm design, some notes on options:

  • We're not aiming to run multiple rules concurrently, just support concurrency within each rule. This might mean we can [asyncio/anyio/trio].run() each async method separately, which greatly simplifies the implementation.

    • Add a new method __execute_fn(self, fn, *args, /, **kwargs); this can default to return fn(*args, **kwargs) with the obvious more-complicated options to support async frameworks (branching on inspect.iscoroutinefunction(fn)). We can ship first-party subclasses {AnyIO,Asyncio,Trio}RuleBasedStateMachine with the appropriate implementations.
  • On the other hand, if we need to maintain some (async) state over the whole run, that gets more complicated.

    • ⭐ we'll want to add a new interface to hold context managers open for the duration of the test - perhaps teach @initialize() to recognize methods wrapped with @contextlib.(async)contextmanager and handle them appropriately?
      • this sounds pretty nice anyway, which hints that we're in this scenario
    • we'll have to make the whole implementation syntactically async, and then have some reasonable logic for the all-synchronous case to keep our current UX (traceback trimming plus a thin asyncio impl?). I still want to avoid taking a position on what async framework users should use, and think I still prefer framework-specific subclasses defining an executor method over a stateful_async_backend setting.
    • I don't want to deal with unstructured concurrency from e.g. asyncio launching a background task - when await whatever_rule() returns, it should be finished. No idea what we could do about that though, beyond documenting that it's not supported.
      • Trio can have lesser issues from e.g. a spanning context with some background task running; the trio.testing.wait_all_* functions might be useful to let things settle down between rules.
  • we should think about the deadline setting: async stateful testing makes it pretty natural to cancel tests that exceed the grace period; and with that in mind we could do this in sync tests that call back into Hypothesis interfaces too - potentially a pretty nice speedup, independent of the rest of this issue.

Thoughts?

@Zac-HD Zac-HD added new-feature entirely novel capabilities or strategies interop how to play nicely with other packages labels Sep 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
interop how to play nicely with other packages new-feature entirely novel capabilities or strategies
Projects
None yet
Development

No branches or pull requests

1 participant