Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement UpdateModel backend workflow #117

Merged
merged 3 commits into from
Oct 1, 2024

Conversation

petermuller
Copy link
Contributor

@petermuller petermuller commented Sep 27, 2024

Creates UpdateModel state machine implementation with frontend handler validations to force as many synchronous validations as possible before actually hitting the async workflow, which is a much user friendly experience, and easier to debug.

Model only polls for capacity between stopped and in-service states as the model is still functional in other scenarios. When starting a model, the state machine first waits for the desired number of instances to spin up, THEN it waits for the user-defined warmup time before adding the model to LiteLLM, that way customers can't try to make inference requires prior to the models actually spinning up. the model instances pop up healthy before the models are fully initialized, so this wait is necessary to ensure that the models are working before we open them back up to requests.

known issues:

  • UI does not seem to send the correct payload for updating ASG configuration to the backend
    • UI is sending autoScalingConfig instead of autoScalingUpdateConfig
  • UI cannot update from embedding to textgen or from non-streaming to streaming
  • UI does not allow a user to start a stopped model without refreshing the page first if the model was stopped in that browser session

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@petermuller petermuller self-assigned this Sep 27, 2024
@petermuller petermuller force-pushed the feature/updatemodel-statemachine-impl branch from f7de2ff to 93fe870 Compare September 27, 2024 20:20
@petermuller petermuller marked this pull request as ready for review September 27, 2024 20:21
@petermuller petermuller changed the title Feature/updatemodel statemachine impl Implement UpdateModel backend workflow Sep 27, 2024
@petermuller petermuller force-pushed the feature/updatemodel-statemachine-impl branch from 93fe870 to 616bd97 Compare September 30, 2024 21:02
@estohlmann estohlmann merged commit 8d12ca6 into develop Oct 1, 2024
4 checks passed
@estohlmann estohlmann deleted the feature/updatemodel-statemachine-impl branch October 1, 2024 03:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants