Publishing? #280

Hoverbear · 2018-06-21T22:28:28Z

I'm interested in publishing this crate so that it can be used as part of the upcoming TiKV client in Rust (which will be published on crates.io).

This already builds cleanly on nightly and has no illegal dependencies, so there should be nothing blocking publishing this.

I would recommend renaming this crate to tikv_proto though, so in the future tikv_client and tikv_server will be consistent.

The text was updated successfully, but these errors were encountered:

Hoverbear · 2018-07-05T00:34:43Z

@siddontang PTAL

Hoverbear · 2019-02-11T22:31:36Z

We need to seriously consider this with the tikv-client for Rust. We cannot publish the client without all of its dependencies also being published.

nrc · 2019-05-20T22:32:53Z

Some things we need to do/think about:

Use a crates.io version of Raft (I think this just needs us to finish the Prost migration and other Raft versioning issues.)
Consider the name - 'kvproto' is not very self-explanatory (tikv_proto is suggested above, I prefer tikv-proto (consistent with rust-client, which should probably be tikv-client.rs or something)
Consider and document the process for releases
What should a Rust crate look like? We probably should not include the Go and CPP code in the distributed tarball, for example.
building - are we happy with the current process and requirements for building (see also Installation requires cmake, protoc, golang, perl... #354)
think about the project org maybe? (Some things worth considering: why is Raft the source of truth for it's Protos, but this project is for the other protos? Why would a proto be here vs elsewhere? Should we be committing the generated code? Is having a multi-language project the right approach (especially if we add more languages in the future)?
should the repo live in the TiKV org?

Probably more...

Hoverbear · 2019-05-20T22:41:09Z

@nrc Do you think extracting only the tikv related protos and making them a dependency here would be a good first step to enable this? Maybe a tikv-api crate? Then we could publish that.

nrc · 2019-05-20T23:11:32Z

Do you think extracting only the tikv related protos and making them a dependency here would be a good first step to enable this? Maybe a tikv-api crate?

I think that is probably a good idea. I wonder whether the Raft protos should be part of that?

ice1000 · 2019-05-21T03:31:16Z

These protos were too big for Raft (which is relatively small) to depend on, but the situation may change after prost

BusyJay · 2019-05-21T03:59:24Z

No, raft is a standard library that should keep its own protocols.

nrc · 2019-05-21T04:15:41Z

I don't understand either of those points, sorry.

These protos were too big for Raft (which is relatively small) to depend on

Why is size an issue?

No, raft is a standard library that should keep its own protocols.

What do you mean by "standard" (and by implication why are TikV, etc., not standard?)

BusyJay · 2019-05-21T05:44:08Z

Sorry, it should be standalone. 🤣

ice1000 · 2019-05-21T22:29:20Z

Why is size an issue?

Hmm, compiling kvproto takes a long time.

nrc · 2019-05-22T01:13:18Z

Hmm, compiling kvproto takes a long time.

This is true of a full build (1m 16s for me), but building kvproto itself only takes 7s. Given that any deps of kvproto should be required by Raft too, it is that which matters (in practice, I think we pull in grpc-rs, which we don't really need to, I'll file an issue about that).

nrc · 2019-05-22T01:21:48Z

Sorry, it should be standalone. 🤣

That makes more sense, thanks! However, I'm still a bit unclear why Raft is a standalone library in this sense. If it is meant to be self-contained, why do we need to re-export the proto interface here?

BusyJay · 2019-05-22T03:00:33Z

...why do we need to re-export the proto interface here?

Because kvproto uses the struct of raft protocol directly, and due to the limited of protobuf compiler, we need to reexport raft proto interface to make the generated code compiled.

nrc · 2019-05-22T03:31:46Z

Because kvproto uses the struct of raft protocol directly, and due to the limited of protobuf compiler, we need to reexport raft proto interface to make the generated code compiled.

Cool, thanks!

nrc · 2019-06-04T21:08:39Z

Proposal:

Move the Raft protos into a RaftProto repo
Raft depends on RaftProto
KvProto depends on RaftProto, not Raft
Move the code for generating protos in Go into protobuf-build so that KvProto and RaftProto can share that code rather than duplicate it
Publish RaftProto and KvProto on crates.io

How does that sound? @BusyJay @Hoverbear

BusyJay · 2019-06-05T03:20:26Z

It sounds good to me. But I have also been thinking about removing raft dependency from kvproto, so that raft is free to change its protocols, or even no protocols at all.

/cc @siddontang what do you think?

ice1000 · 2019-06-05T06:26:34Z

so that raft is free to change its protocols, or even no protocols at all.

I suppose raft and kvproto will depend on the same "raftproto" library mentioned in @nrc's comment

Hoverbear · 2019-06-05T18:28:41Z

@nrc I approve of this idea!

nrc · 2019-06-14T02:42:45Z

It sounds good to me. But I have also been thinking about removing raft dependency from kvproto, so that raft is free to change its protocols, or even no protocols at all.

@BusyJay @siddontang do you have any more thoughts on this?

BusyJay · 2019-06-17T05:50:02Z

I just take a look into the protocols, it seems there are many places using eraft protos. It's not practical to remove the dependency now. So your proposal seems the only way to publish.

nrc · 2019-06-17T07:40:50Z

Cool, thanks for looking at the protocols!

We now only depend on crates.io crates, cc pingcap#280 Signed-off-by: Nick Cameron <[email protected]>

* Use the raft-proto crate instead of raft We now only depend on crates.io crates, cc #280 Signed-off-by: Nick Cameron <[email protected]> * Update protobuf-build Fixes a bunch of warnings Signed-off-by: Nick Cameron <[email protected]>

Daniel-B-Smith · 2019-09-25T22:04:50Z

Any update on publishing the crate?

Hoverbear · 2019-09-26T16:03:18Z

@Daniel-B-Smith We're currently not able to publish this due to some ongoing work by @nrc . We're planning to publish this crate (along with our Rust client) in Q4. Sorry for the delays.

kvproto/Cargo.toml

Lines 31 to 32 in 121d118

    
           "protobuf-codegen:2.8.0" = { git = "https://github.com/nrc/rust-protobuf", branch = "v2.8" } 
        
           "protobuf:2.8.0" = { git = "https://github.com/nrc/rust-protobuf", branch = "v2.8" }

Daniel-B-Smith · 2019-09-27T02:04:58Z

Thanks for the update!

* Use the raft-proto crate instead of raft We now only depend on crates.io crates, cc pingcap#280 Signed-off-by: Nick Cameron <[email protected]> * Update protobuf-build Fixes a bunch of warnings Signed-off-by: Nick Cameron <[email protected]>

xxchan · 2021-08-23T07:50:06Z

Any update?

disksing · 2021-08-23T08:19:52Z

anyone know the status of this issue? cc @sticnarf @BusyJay

BusyJay · 2021-08-24T05:48:30Z

kvproto now uses forked rust-protobuf, I believe forked rust-protobuf needs to be published first.

disksing · 2021-08-24T06:06:10Z

kvproto now uses forked rust-protobuf, I believe forked rust-protobuf needs to be published first.

How can we proceed? Do we currently have any plan for the next step?

BusyJay · 2021-08-24T13:15:23Z

As far as I know, no one is working on it.

xxchan · 2021-08-25T05:24:30Z

Maybe related post: Move kvproto to tikv org

Hoverbear added the question label Jun 21, 2018

This was referenced Feb 14, 2019

Resolving blockers for publishing tikv/client-rust#32

Closed

Replace script with build script, use features tikv/raft-rs#175

Closed

nrc added a commit to nrc/kvproto that referenced this issue Jun 18, 2019

Use the raft-proto crate instead of raft

6c1fc56

We now only depend on crates.io crates, cc pingcap#280 Signed-off-by: Nick Cameron <[email protected]>

This was referenced Jun 18, 2019

Move protos to a separate crate tikv/raft-rs#247

Merged

Depend on raft-proto crate instead of raft #411

Merged

nrc added a commit to nrc/kvproto that referenced this issue Jun 19, 2019

Use the raft-proto crate instead of raft

c8efe8a

We now only depend on crates.io crates, cc pingcap#280 Signed-off-by: Nick Cameron <[email protected]>

nrc added a commit to nrc/kvproto that referenced this issue Jun 20, 2019

Use the raft-proto crate instead of raft

948ab86

We now only depend on crates.io crates, cc pingcap#280 Signed-off-by: Nick Cameron <[email protected]>

This comment has been minimized.

Sign in to view

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Publishing? #280

Publishing? #280

Hoverbear commented Jun 21, 2018 •

edited

Loading

Hoverbear commented Jul 5, 2018

Hoverbear commented Feb 11, 2019

nrc commented May 20, 2019 •

edited

Loading

Hoverbear commented May 20, 2019

nrc commented May 20, 2019

ice1000 commented May 21, 2019

BusyJay commented May 21, 2019

nrc commented May 21, 2019

BusyJay commented May 21, 2019

ice1000 commented May 21, 2019

nrc commented May 22, 2019

nrc commented May 22, 2019

BusyJay commented May 22, 2019

nrc commented May 22, 2019

nrc commented Jun 4, 2019

BusyJay commented Jun 5, 2019

ice1000 commented Jun 5, 2019

Hoverbear commented Jun 5, 2019

nrc commented Jun 14, 2019

BusyJay commented Jun 17, 2019

nrc commented Jun 17, 2019

Daniel-B-Smith commented Sep 25, 2019

Hoverbear commented Sep 26, 2019

Daniel-B-Smith commented Sep 27, 2019

xxchan commented Aug 23, 2021

disksing commented Aug 23, 2021 •

edited

Loading

BusyJay commented Aug 24, 2021

disksing commented Aug 24, 2021

BusyJay commented Aug 24, 2021

This comment has been minimized.

xxchan commented Aug 25, 2021

Publishing? #280

Publishing? #280

Comments

Hoverbear commented Jun 21, 2018 • edited Loading

Hoverbear commented Jul 5, 2018

Hoverbear commented Feb 11, 2019

nrc commented May 20, 2019 • edited Loading

Hoverbear commented May 20, 2019

nrc commented May 20, 2019

ice1000 commented May 21, 2019

BusyJay commented May 21, 2019

nrc commented May 21, 2019

BusyJay commented May 21, 2019

ice1000 commented May 21, 2019

nrc commented May 22, 2019

nrc commented May 22, 2019

BusyJay commented May 22, 2019

nrc commented May 22, 2019

nrc commented Jun 4, 2019

BusyJay commented Jun 5, 2019

ice1000 commented Jun 5, 2019

Hoverbear commented Jun 5, 2019

nrc commented Jun 14, 2019

BusyJay commented Jun 17, 2019

nrc commented Jun 17, 2019

Daniel-B-Smith commented Sep 25, 2019

Hoverbear commented Sep 26, 2019

Daniel-B-Smith commented Sep 27, 2019

xxchan commented Aug 23, 2021

disksing commented Aug 23, 2021 • edited Loading

BusyJay commented Aug 24, 2021

disksing commented Aug 24, 2021

BusyJay commented Aug 24, 2021

This comment has been minimized.

xxchan commented Aug 25, 2021

Hoverbear commented Jun 21, 2018 •

edited

Loading

nrc commented May 20, 2019 •

edited

Loading

disksing commented Aug 23, 2021 •

edited

Loading