How to do both text output and FunctionCall in the same request?

[jaromiru]

Hello, I want the model to reason before making a function call in one request, which is possible in other libraries. However, I haven't found a way to do this in magentic. The usual signature of a method is:

def do_something(param: str) -> FunctionCall: ...

which does not allow any other outputs than the function call.

I'd be happy to do something like:

def do_something(param: str) -> tuple[Annotated[str, "reasoning"], FunctionCall]: ...

but this results in a pydantic error. It applies to ParallelFunctionCall also.

[jackmpcollins]

Hi @jaromiru

The OpenAI API returns either a message or tool call(s), while the Anthropic API can return "thoughts" before making a tool call (see related issue #220). Because of this, a return type that allows "thinking" before making a function call would only work with some LLM providers, but it might still be useful to have.

One approach you could try is having separate steps for the thinking and the function call. Something like

@prompt(...)
def think_about_doing_something(param: str) -> str: ...

@prompt("Based on thoughts {thoughts}. Do thing")
def do_something(thoughts: str, param: str) -> str: ...

thoughts = think_about_doing_something(param)
function_call = do_something(thoughts, param)

Another option which would reduce this to a single query would be to set the return type to a pydantic model with an "explanation" field and fields for the function parameters. You would need to do this for all available functions and union them in the return type. Similar to https://magentic.dev/structured-outputs/#chain-of-thought-prompting

class ExplainedFunctionParams(BaseModel):
    explanation: str = Field(description="reasoning for the following choice of parameters")
    param: str

@prompt(...)  # No `functions` provided here
def do_something(param: str) -> ExplainedFunctionParams: ...

func_params = do_something(param)
my_function(func_params.param)

Please let me know if either of these approaches would work for you. Thanks for using magentic.

[jaromiru]

Hi @jackmpcollins, thanks for the answer and the proposed workarounds. Obviously, both of them have their drawbacks.
My main motivation was to reduce costs by having one LLM call (especially when the context is large), and the first method does not help with this.
The second proposal makes it complicated, brittle, hand-engineered, prone to errors and avoids the ease-of-use of Magentic library.

I currently have no solutions on my own, so we can close the ticket, or keep it open for followup discussion.

[jackmpcollins]

@jaromiru Let's leave this open because it would be great for magentic to support this for the Anthropic API. Please let me know if/how you solve this for OpenAI using another library or their API directly because that could inform how to solve it generally for magentic.