Ballista reloaded - proposed changes to core ballista #1066
Conversation
... as it was never finished
not quite sure why it is there but i dont see obvious reason to keep it.
... the crate used need some love, and there are is newer crate as well.
... and all other dependant libraries
|
Will have something ready later today or tomorrow @andygrove, I was thinking to put proposal of the next steps on top of this PR, started already but did not have time to finish it yet. Now, this PR has many things removed, but not too many code changed, would you like to take it step by step and re-do this work again or we can just merge this as once off, without squashing commits so we can revert removal if needed @andygrove ? I understand that it is a bit of big PR.
|
|
also, we could remove deps update commits and cherry pick it to new PR |
Honestly, this is too big for me to do a comprehensive review within the limited weekend time that I have available. It seems that some very unrelated changes would be easy to split out, so we can start with those first to at least reduce the size of this PR? For example, these could be separate smaller PRs that would be easy to review:
We could get through these very quickly, I think. |
|
all right, will cherry-pick issue from big branch. I'm doing this at my spare time so bear with me :) created #1067 listing all features for removal, and #1068 as high level road map proposal to kick off discussions and ideas. is it ok just to refer PR to #1067 list or you prefer having issue created for each PR @andygrove |
That's great. Thank you. No need to create separate issues per item. |
relates to: apache#1066 & apache#1067
it looks like not used anywhere. Relates to: apache#1066 & apache#1067
relates to: apache#1066 & apache#1067
As it may be useful I'd argue that it should not be part of core ballista. We see in the community caches build on top of object_store. If functionality like this is needed it should be implemented as scheduler-policy or similar. Relates to: apache#1066 & apache#1067
Which issue does this PR close?
Closes #.
Rationale for this change
First of all, apologies for a gigantic PR, but I did not see better way of making this proposal.
With this proposal I'd like to propose one way forward for ballista, direction is not set in stone so we could discuss alternatives.
I'd like to set the stage with driving idea, it may help understanding PR better.
I believe if we want to revive ballista we need to make it a library rather than full fledge application with all batteries included. If we continue developing it as an application, core code will just grow, putting maintainers on pressure to maintain code which may not be related to core ballista mission.
Making ballista as library, which could bootstrap schedulers and executors with few lines of code, with possibility to plug in additional functionalities will keep use-case specific code out of ballista. We should make it as simple as bootstrapping datafusion application.
Second goal, I believe, should be keeping ballista functionalities on par with datafusion releases. At the moment ballista is missing a lot of functionalities, like writes, sinks, to name a few. The main, short term, goal would be to get writes, inserts supported.
Lastly, client-scheduler, and scheduler-executor interaction should be improved. Client-scheduler at the moment communicate using custom made gprc protocol, which could get support for UDF(s), configuration ... current client-scheduler protocol can be made an optional component, so users can bring their own frontend such as spark connect, or similar. scheduler-executor as well can be extended to support session configuration information, ...
So with those high level goals in mind, I have done a bit of code trimming and maintenance. I have removed some code which I personally believe should/could be an "add-on" to ballista.
What changes are included in this PR?
This PR removes:
This PR adds no functionality, but updates most of the dependencies.
Are there any user-facing changes?