Prevent data loss in lame duck mode

[ritmas]

What's the best way for the client to handle lame duck mode, when NATS cluster members are gracefully restarting?
I'm using the following code snippet, but it doesn't look reliable much.

servers := "nats://nats.cluster.svc:4222"

nc, err := nats.Connect(servers,
    nats.LameDuckModeHandler(func(conn *nats.Conn) {
        log.Printf("Server gracefully shutdowns. Let's reconnect")

        // using the following functions fails due to error
        // <func> undefined (type *nats.Conn has no field or method selectNextServer)
        // conn.selectNextServer()
        // conn.bindToNewConn()

        conn.Drain()
        conn.Close()

        os.Exit(0)
    }),
)

[derekcollison]

Reconnects happen automatically. You can hook those events, but it is not required.

Towards end of year clients will get intelligent reconnect where they can proactively respond to the lame duck signal from a server.

[ritmas]

Automatic reconnect works as intended, but my concern is to prevent data loss upon publish request to a cluster node which is in Lame duck mode (LDM). Now what happens is that the 1st publish request fails due to EOF, then reconnect logics kick in and the client can continue publishing messages, but the initial failed request's data is lost - there is no default retry logics. I'd expect nats.go could apply some retry logics itself.

2023/08/25 07:57:25 Msg: "Hello World 1" # <-- data loss
2023/08/25 07:57:25 Got disconnected! Reason: "EOF"
2023/08/25 07:57:26 Got reconnected to nats://10.x.x.x:4222!
2023/08/25 07:57:26 Msg: "Hello World 2" # <-- successfully published

So the question remains - what would be the best practice for that? Implementing own retry logics for cluster nodes in LDM? Or use this clumsy "drain, close & exit" way, mentioned above?

[derekcollison]

There are many ways to approach problem based on your ultimate goals. Retries are very common, and you could even architect to have multiple responses and pick the fastest one that is returned. This is a common pattern when latency and getting a result are the most important.

For the responders, you could hook the LDM and do a drain if you wanted. Not sure its worth it and our intelligent reconnect will probably fit the bill once released.

[ritmas]

1. Is there a way of initiating reconnection to other cluster member upon LDM rather than closing client connection? It sounds like selectNextServer() or bindToNewConn() would fit our needs, but they cannot be used upon LameDuckModeHandler event.
2. What is the ETA of releasing this "intelligent reconnect" feature?

[derekcollison]

If latency and avoiding retries is the # priority have multiple responses, that way if one is affected by a LDM server the other one can succeed.

Intelligent reconnect is scheduled for end of year.

[ajax-koval-d]

@derekcollison, I've glanced through release notes and don't see any mentions of Intelligent reconnect feature. Do you still have plans to implement it?

[ripienaar]

We do have plans but not sure when we'll do that. For now its not released or started.