Different response behaviors when using OpenAIStream. How do you test your stream output?

[StephenTangCook]

(Discord cross-post)

Depending on what client/tool I use to test my API responses, I get troubling different behaviors when OpenAIStream is used to wrap the response. I can't tell if this is a bug, or the tools are implemented differently, or what the best practice for testing this is?

I've tested with nearly an identical setup as the Vercel AI SDK example. My code: https://github.com/StephenTangCook/vercel-ai-sdk-example/blob/b6d34406f7e7c636924c8504684f0dc8cfc24236/app/api/completion/route.ts#L32

Versions:
"ai": "^2.1.20",
"next": "13.4.9",
"openai-edge": "^1.2.0",
node v18.16.0

1. With OpenAiStream and StreamingTextResponse:

  const stream = OpenAIStream(response);
  return new StreamingTextResponse(stream, {
    headers: {"Content-Type": "text/event-stream"},
  });

• Browser (chrome): streams the text incrementally
• Postman: connection opens, then closes. No data chunks received.
• cURL: the entire text is shown at once (no stream):

$ curl --location 'http://localhost:3000/api/completion'
Hello Bob! I understand your concern about the car's paint. I'm here to help you address this issue. Could you please provide me with more details about the car and the paint?

2. Only StreamingTextResponse (no OpenAiStream):

  return new StreamingTextResponse(response.body, {
    headers: {"Content-Type": "text/event-stream"},
  });

• Browser (chrome): displays incremental data chunks of the stream
• Postman: connection opens, displays incremental data chunks of the stream, then closes
• cURL: receives incremental data chunks of the stream:

$ curl --location 'http://localhost:3000/api/completion-without-OpenAiStream'
data: {"id":"chatcmpl-7c6UCcdYeH22z93RFkXwqWXIvz6sM","object":"chat.completion.chunk","created":1689315752,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-7c6UCcdYeH22z93RFkXwqWXIvz6sM","object":"chat.completion.chunk","created":1689315752,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-7c6UCcdYeH22z93RFkXwqWXIvz6sM","object":"chat.completion.chunk","created":1689315752,"model":"gpt-3.5-turbo-0613","choices":[{"index":0,"delta":{"content":" Bob"},"finish_reason":null}]}

As you can see, only # 2 consistently returns a stream. What does wrapping the response with OpenAIStream do in # 1 that makes these tools unhappy?

[MaxLeiter]

We do some processing in OpenAIStream that could be related. Thanks for the high quality report @StephenTangCook, I’ll try and investigate early next week

[StephenTangCook]

Thanks @MaxLeiter! Looking forward to it.

Here's another data point -- if I don't set any custom headers:

  const stream = OpenAIStream(response);
  return new StreamingTextResponse(stream);

• Browser (chrome): displays entire text at once (no stream)
• Postman: displays entire text at once (no stream)
• cURL: displays entire text at once (no stream)

The default headers get set as:

Connection: close
Content-Encoding: gzip
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked
Vary: Accept-Encoding

Seems a bit odd to me, like it's not streaming at all? Now I'm wondering on the flip side what is useChat() doing when it receives this response? Does it convert the completed string into a stream? (I haven't tested that part yet) That would be inefficient.

[MaxLeiter]

Transfer-Encoding: chunked is correct.

The header being text/plain is expected; if you want to stream JSON or something else you can override the header or not use StreamingTextResponse.

https://github.com/vercel-labs/ai/blob/78477d34a1e838101633e9ed777826c0035e0278/packages/core/streams/streaming-text-response.ts#L28

The fact it's not streaming sounds like a bug, but the headers aren't part of it, AFAICT.

[StephenTangCook]

@MaxLeiter should I log an issue?