Navigate compatibility

[henrypinkard]

@nornil
@AdvancedImagingUTSW and @annie-xd-wang, very excited you're interested in trying out ExEngine!

Here's a guide for what I think a good roadmap is in order of benefits to Navigate / effort required. For most of these things, you may be the first to look over the documentation I've written, so please ask questions early and often so that I know what needs improvement!

I think (1) will be not so hard and offers immediate obvious benefits to Navigate. (2) will be a bit more involved but provides the threading benefits of ExEngine to Navigate. (3-5) are essentially refactoring Navigate to be as if you were building it on ExEngine from the start. This will offer immediate benefits to the wider community, because it would modularize and standardize your components to make them more easily reusable, and it should allow others to contribute fixes and improvements to components you rely on. In the long run, as people build more things on top of ExEngine, it will also give the benefit that you should be various types of smart microscopy/networked-based control on your hardware with minimal additional work.

1) Try out ExEngine with the Micro-Manager device backend

This will give you a feel for how ExEngine's device abstraction works, and it should save you the effort of having to make compatibility for other MM devices yourself. I think you can add exengine as a dependency to your project with only the micromanager device backend, and this may not even add any additional dependencies to Navigate.

More instructions here

2) Implement your devices as ExEngine devices

This would give you the immediate benefits of the executor/threading for your devices

Based on seeing your code, it seems like the simplest thing to try first would be to have your MicroscopeModel inherit from exengine.base_classes.Device. After testing that, the better approach would be to do this at the level of individual devices within your MicroscopeModel, as that would give you more fine-grained control over threading, and open up the possibility of others using your devices.

(Edit: I a writeup of the threading features we talked about)

3) Refactor your higher level code in terms of ExEngine device APIs

Next, you could try to make your devices inherit from ExEngine's device base classes. You should be able to do this without breaking any higher level code that currently calls your devices by just trying to implement these abstract methods in terms of your existing APIs. Note that ExEngine's device APIs are in early days. There are definitely more devices needed, and if existing APIs don't work for your devices, it would be great to know that so they could possibly be changed.

Next, you could consider adding Navigate devices as a new device backend, which would allow all users of ExEngine to make use of them

4) Investigate making features into events

I think your concept of "Features" is very similar, if not the exact same thing as events. If you were able to implement them as ExecutorEvents and have them take only Exengine device types in constructors they could made to work with all backends.

5) Storage backends

I see that Navigate has a lot of great options for storage backends. Can these be broken out into individual libraries for reading and writing? If so, you could make these into storage backends in ExEngine, which could open them up for use by a much bigger audience.

I would not be surprised if the ExEngine storage API needs expansion to accommodate some of the light sheet-specific features, and I'd be quite curious to know what these are.

Please feel free to ask any questions or share your thoughts as you look into this!

[AdvancedImagingUTSW]

@henrypinkard - I just wanted to let you know that this is on my radar and I will look into it once things settle down on my side. Thanks...