Get Streaming Chunks as they arrive in useChat

[alokwhitewolf]

Context - I want to extract individual chunks sent by StreamingTextResponse in the backend, that is used by useChat in the client side.

Existing Implementation
I am currently trying to use onResponse callback to read the data. Current implementation

onResponse: async (response: Response) => {
      const reader = response.body?.getReader();

      if (reader) {
        const decoder = new TextDecoder("utf-8");
        let chunk = await reader.read();
        let chunkValue = '';
        let first_time = true;
        while (!chunk.done) {
          if (first_time){
            first_time=false;
            chunkValue += decoder.decode(chunk.value);
          }          
          chunk = await reader.read();
          chunkValue += decoder.decode(chunk.value);
          console.log("TEST")
          console.log(messages);
          const newMessages = (existingMessages: Message[]) => {
            
            return [
              ...existingMessages,
              {
                id: Date.now().toString(),
                role: 'assistant',
                content: chunkValue,
              } as Message,
            ];
          };
          setMessages(newMessages(messages));
          
        }
      }
    },

I am having to also use setMessages, as it seems like if i use const reader = response.body?.getReader();, the incoming response is not updated in the messages object. However, with this implementation, it seems like the messages array always seems to have one less element ( for eg, the last user sent message is never included ), so the last message that the user sent gets skipped

Any help on how I can - go through each message chunk while still maintaining the same behaviour of messages/useChat?

Environment Setup
SDK Version : 3.0.23
Environment : TS / Next.js

[pretrator]

Try this

onResponse: async (response: Response) => {
      const reader = response.body?.getReader();
     const allMessage = []
      if (reader) {
        const decoder = new TextDecoder("utf-8");
        let chunk = await reader.read();
        let chunkValue = '';
        let first_time = true;
        while (!chunk.done) {
          if (first_time){
            first_time=false;
            chunkValue += decoder.decode(chunk.value);
          }          
          chunk = await reader.read();
          chunkValue += decoder.decode(chunk.value);
          console.log("TEST")
          console.log(messages);
          allMessage.push({
                id: Date.now().toString(),
                role: 'assistant',
                content: chunkValue,
              });
          setMessages(allMessage); 
        }
      }
    },

[mcavaliere]

It would be great to have an onChunk callback for this.

[BrianHung]

TypeError: Failed to execute 'getReader' on 'ReadableStream': ReadableStreamDefaultReader constructor can only accept readable streams that are not yet locked to a reader

Can't do it in onResponse. Will probably need a patch.