litellm batch_completion

[kingennio]

Hi there. I was trying to use instructor with liteLLM. I can make il work for standard chat completions, as shown in the example, but I was trying to exploit batching supported by liteLLM through their batch_completion fanction. But I'm unable to make it work with instructor. I'm I right that this feature cannot be exploited with instructor? Thanks

[jxnl]

not yet, can you share docs and maybe someone can try to support this

[krrishdholakia]

hey @kingennio - litellm maintainer here - how do you want to use batch completions with instructor?

[kingennio]

Hi there, thank you for your interest.
When I sumbitted the request there was not a way to patch the "batch_completion" function with instructor
https://docs.litellm.ai/docs/completion/batching
I'm not sure if there's another way to use instructor with batching and litellm.
BTW, I did implement my own batching function following the code in your blog to use instructor. It kinda works but it's half hacked and not very robust

[threewisemonkeys-as]

Hi there. Currently from_litellm(batch_completion) does not work -

from litellm import batch_completion
import instructor

client = instructor.from_litellm(batch_completion)

from pydantic import BaseModel
class Ans(BaseModel):
    ans: int

results = client.chat.completions.create(model="gpt-4o", messages=[[{"role": "user", "content": "1+1"}], [{"role": "user", "content": "2+2"}]], response_model=Ans)

Results in -

Traceback (most recent call last):
  File "/Users/ays57/miniconda3/envs/rdv/lib/python3.12/site-packages/instructor/retry.py", line 188, in retry_sync
    return process_response(
           ^^^^^^^^^^^^^^^^^
  File "/Users/ays57/miniconda3/envs/rdv/lib/python3.12/site-packages/instructor/process_response.py", line 142, in process_response
    model = response_model.from_response(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ays57/miniconda3/envs/rdv/lib/python3.12/site-packages/instructor/function_calls.py", line 139, in from_response
    if completion.choices[0].finish_reason == "length":
       ^^^^^^^^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'choices'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/ays57/miniconda3/envs/rdv/lib/python3.12/site-packages/instructor/retry.py", line 182, in retry_sync
    for attempt in max_retries:
                   ^^^^^^^^^^^
  File "/Users/ays57/miniconda3/envs/rdv/lib/python3.12/site-packages/tenacity/__init__.py", line 443, in __iter__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ays57/miniconda3/envs/rdv/lib/python3.12/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/Users/ays57/miniconda3/envs/rdv/lib/python3.12/site-packages/tenacity/__init__.py", line 419, in exc_check
    raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x117a928d0 state=finished raised AttributeError>]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ays57/miniconda3/envs/rdv/lib/python3.12/site-packages/instructor/client.py", line 119, in create
    return self.create_fn(
           ^^^^^^^^^^^^^^^
  File "/Users/ays57/miniconda3/envs/rdv/lib/python3.12/site-packages/instructor/patch.py", line 291, in new_create_sync
    response = retry_sync(
               ^^^^^^^^^^^
  File "/Users/ays57/miniconda3/envs/rdv/lib/python3.12/site-packages/instructor/retry.py", line 220, in retry_sync
    raise InstructorRetryException(
instructor.exceptions.InstructorRetryException: 'list' object has no attribute 'choices'

Im assuming this is because handling batch completions (which returns list[list[Response]]) is not supported by the instructor patching. Is there any way around this?

[pablospe]

Hi there. I was trying to use instructor with liteLLM. I can make il work for standard chat completions, as shown in the example, but I was trying to exploit batching supported by liteLLM through their batch_completion fanction. But I'm unable to make it work with instructor. I'm I right that this feature cannot be exploited with instructor? Thanks

Would it work if you call directly first with liteLLM and then use instructor for the parsing in a 2nd call (even perhaps with a different model)? Idea from here:

call1: prompt -> LLM  -> text_response  
call2: prompt (which is only the text_response)  -> LLM (with response_model) -> response_model_Object

#580 (reply in thread)