New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use redis stream as a message queue for duplicating chat stream. #481

Open

634750802 opened this issue Dec 9, 2024 · 0 comments

Collaborator

634750802 commented Dec 9, 2024 •

edited

Loading

This will make it possible for multiple clients to view the same chat streaming procedure (including stackvm, step info and text generation).

Implementation:

client 1 call chat api to generate chat
- Server creates a stream response to client 1
- Server creates a redis stream and duplicates each chunks from original stream
client N call the new read chat stream api to read chunks from redis stream and display streaming UI by the duplicated stream.

Related:

protocol: show "Generating" status while it is really generating #357

The text was updated successfully, but these errors were encountered:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment