Add ability to select models and edit system/assistant prompts #1051
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
**Goal: ** Users should be able to select the model from the chat interface and receive a response from that model.
Currently: we just send the request and take the response from whatever model the "server" process has loaded. To load a different model, the user would have to stop the server and re-start it with different CLI args.
Solution: This is a bit tricky. OpenAI can just route the request to a system that already has the model loaded. If we wanted to replicate this functionality, the user's system would quickly run out of memory to have even two instances of low-parameter models loaded. To get around this, we'll just assume that the system will only ever have a single model loaded.
Flow:
Warning This PR is still in draft status and does not yet include server side changes.