External services via M3O

[asim]

M3O is a cloud platform to explore, discover and consume public APIs. It was built on top of Go Micro and Micro as a way to enable sharing and reuse of services rather than everyone building them themselves. My hopes with the open source efforts over the past 7 years - (Go Micro was open sourced and launched on HN January 15th 2015 https://news.ycombinator.com/item?id=8895794) - was to try level us all up, so that we could focus on higher level problems. What GitHub did for source code, I really wanted to enable for living, breathing services. This was hard as a framework and as open source. M3O is what I believe the end state for such a solution and now my hope is I can offer this back to Go Micro developers too.

If you look at the services directory, I've provided the clients and interfaces for all the services on the M3O platform. You can now basically just import and use them as if they were local. Not only that, you can mock the interfaces or replace them where needed. It's still early days, I think a lot of this will evolve, but I'm hoping this is the next step forward whether integrated with Go Micro or external elsewhere.

Let me know your thoughts.

[refs]

In my opinion having a set of basic building blocks as services makes total sense. One can make the case of complex applications are composed of sets of API calls M3O has to offer. For instance one can combine a Geolocation API with the Weather API, making composability easy, as they will be at the reach of a single client. An example application of this is for building fast functional prototypes. If there are some efforts that need to be done on the consumer side, the interface to adhere to is already provided by go-micro and can be easily swapped, as you mentioned. This has been always a good selling point for me at least, interchangeable interfaces implementations.

There can also be the case of not only having high level APIs but lower level APIs, such as arithmetic operations, or even services that hosts one's own functions, sort of like a lambda, allowing users to run different computations on demand. I pretty much like the idea of composability and problem solving from a vantage point, it elevates problem solving and makes services and APIs less "aggressive", and I can see myself taking a different path when building something complex; rather than thinking, for instance, on protobuf definitions, I could first consult the go-micro list of services, see if something fits my needs, if there is something perfect, I could use it right away and it will sit perfectly with my ecosystem, if not, I could decide to either implement it upstream and make the entire ecosystem more robust or implement it externally.

It is a system that will benefit from a great number of users and increase in quality because of it. The next question is if this is viable, if developers could stick to such a way of developing. I'm thinking a developer survey might not be a bad idea. Get people to share their usage / experience with either micro or go-micro, and what their expectations are.

[asim]

Thanks for the feedback, hopefully devs can come here and share some comments also. One thing I'm just trying to factor in is how to make this stuff available to Go Micro users. Whether it needs to be part of the library or something external. It already exists at https://github.com/m3o/m3o-go but trying to put it front and center is really what I'm interested in and doing it in a way that's really true to Go Micro itself. Go Micro was always about hand crafted interfaces that are native to Go. Right now we're generating client code from protos but I think there's still significant room for improvement. The potential of Go Micro to be the next standard library on top of Go is pretty huge.

[refs]

Go Micro users

I read "users" as developers consuming asim/go-micro, is that right? If so, a client (m3o/m3o-go) is have a very convenient way to do this. There is also the question of how is M3O being served. If it is on-prem or SaaS. I like to imagine it more like a SaaS, so, a way of democratizing such public APIs and offer a free market for competitors. Having a look around at the reference services, they are all currently implemented in asim/go-micro but having a global service registry will open the doors to different implementations of the same interface essentially, both running in parallel, which is the entire purpose of go-micro of sticking to the interfaces. On such ecosystem I can even imagine running payment through go-micro, integrating with Stripe or PayPal would just be a matter of configuration, and it has the potential of keeping everything under the same framework, so maintenance is shifted to the go-micro service provider (perhaps the payment API example is not a good one, as you might want the transaction to happen only between the client and the payment provider... but I think my intent there is clear :D ).

The potential of Go Micro to be the next standard library on top of Go is pretty huge.

This being a standard library for building services?

[asim]

I read "users" as developers consuming asim/go-micro, is that right?

That's right

I like to imagine it more like a SaaS, so, a way of democratizing such public APIs and offer a free market for competitors.

This is how I imagined the long term future of M3O, hence everything being open source

Having a look around at the reference services, they are all currently implemented in asim/go-micro but having a global service registry will open the doors to different implementations of the same interface essentially, both running in parallel, which is the entire purpose of go-micro of sticking to the interfaces.

This is something I've written about here https://github.com/micro/network and tried to implement on a few occasions. I think there's the potential to create a free market for services where we do conform to a specific service interface. Part of that is booting not just the infrastructure to run services but to also globally interconnect.

The other piece is just trying to figure out how to pull Go Micro back into the original vision or whether it needs to continue to evolve standalone for a while. When you have a lot of moving parts; Go Micro / Micro / M3O it becomes very hard to cohesively pull these together. Instead building on top of each other is actually much easier. So I'm very careful not to try force something back into Go Micro that doesn't fit.

This being a standard library for building services?

It's definitely a missing component in the industry. Right now Micro served as that purpose for us and we've built 50+ services in https://github.com/micro/services but the popularity of Go Micro as a framework makes me think we need to find a way to use that.

[lambdaR]

Following to our discussion about Webassembly go-micro and M3O, I found two projects WasmEdge (C++) and Wasmtime (Rust) which both provide an API for many languages in order to integrate Webassembly runtime in your application, these API allow you to pass arguments to a Webassembly binaries and retrieve the result back.

You can try an MVP where you convert the Put, Get and Delete methods of the Cache service, for instance, into a Webassembly binary and run it inside the Wastime/WasmEdge runtime.

This way a Python developer for instance can write an application with an embedded Webassebly runtime which in turn runs Webassembly binaries of methods mentioned above; the Cache service will be running as standalone.

One problem though, TYPES …. The documentation below has mentioned some of the scenarios and solutions that can be used to solve it.

If that MVP went well, you can imagine scenarios where M3O can play a role here.
You could provide a Store service with many flavors Redis, Postgres, cockroach etc. This way developers can pick any flavor and combine Services to establish their backend, then you charge per service call.

Question is, is there a demand for a Service like this? Or do they prefer to host their own backend.

Wasmtime has an API for C, Rust, Python, Go, .Net, Java, Perl, Zig and Ruby
WasmEdge has an API for C, Go, Rust, NodeJs and Python

WasmEdge developers claim that it is the fastest and it has better documentation

[asim]

I've been tracking the intel micro wasm runtime https://github.com/bytecodealliance/wasm-micro-runtime. It supports request/response and pubsub and can be embedded. I think that's probably the most appropriate runtime. I just haven't had the chance to dig deeper into it.

So I can imagine a scenario in which we have something that's able to execute wasm binaries primarily written with Go Micro and a specific set of plugins that let us map the IO (req/rsp, pubsub). But it's not clear to me yet what value that offers beyond running someone's code because the outbound network IO required to access external APIs and things will be limited since wasm still doesn't have proper networking support.

Question is, is there a demand for a Service like this? Or do they prefer to host their own backend.

The web3 ecosystem is showing there is a demand for this because they are all entirely built as wasm based running smart contracts. I think from an old school cloud perspective and when I started building a product around Go Micro, it was clear the existing users were just interested in being helped to run their stuff self hosted on AWS with Kubernetes. Today maybe the focus is shifting to next generation services so there might be an opportunity to do something there.