Fix KV cache error in wildguard.py #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ValueError: The model's max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache (30448). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.
kavelrao
reviewed
Jul 3, 2024
There was a problem hiding this comment.
Thanks for submitting this fix, I hadn't run it on 3090 before so I didn't encounter this.
But I wonder if there's a more general way to address it by detecting the memory limit of the device instead of special-casing GTX 3090? I bet other consumer GPUs would have the same issue.
kavelrao
reviewed
Jul 5, 2024
| self.model = LLM(model=MODEL_NAME) | ||
| gpu_name = torch.cuda.get_device_name(0) | ||
| if gpu_name == 'NVIDIA GeForce RTX 3090': | ||
| self.model = LLM(model=MODEL_NAME,max_model_len=30448) |
There was a problem hiding this comment.
Suggested change
| self.model = LLM(model=MODEL_NAME,max_model_len=30448) | |
| self.model = LLM(model=MODEL_NAME, max_model_len=30448) |
Comment on lines
+182
to
+183
| gpu_name = torch.cuda.get_device_name(0) | ||
| if gpu_name == 'NVIDIA GeForce RTX 3090': |
There was a problem hiding this comment.
Suggested change
| gpu_name = torch.cuda.get_device_name(0) | |
| if gpu_name == 'NVIDIA GeForce RTX 3090': | |
| if torch.cuda.get_device_properties(0).total_memory < 30e9: |
And to make this work with ephemeral_model=True, you will also need to add a parameter for max_model_len to subprocess_inference_with_vllm and pass it through to create_and_inference_with_vllm.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ValueError: The model's max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache (30448). Try increasing
gpu_memory_utilizationor decreasingmax_model_lenwhen initializing the engine.