Streaming Messages

[tracyboehrer]

Spec update, to include description of streaming messages, for example OpenAI streamed responses, to a channel. This is already implemented in Copilot Studio and Teams.

Streaming Text Messages

• Typing Activity is used to send an aggregate text value of a larger chunked message.
  • "Delta" type streaming is not supported in the inital version, but could be added later if required.
• Stream information will be included in an Entity in the Activity
  • Type: "streaminfo"
  • Properties
    • streamId: Required unique ID
    • streamSequence: Required incrementing integer for each Typing Activity sent, and final Message
    • streamType: Optional, default is "streaming"
      • "informative"
      • "streaming"
      • "final"
        • Only on final "message" Activity
    • streamResult: Optional, default "success", only appropriate on final message.
      • "success"
      • "timeout"
      • "error"
• Informative messages
  • Can be sent in any order
  • Text is in Activity.Text
  • TextFormat can be supplied
  • Increments the streamSequence
• Throttling
  • Outgoing Typing Activities should be throttled to a configurable interval
• Streaming should terminate via a configurable timeout
  • A final 'message' type Activity should be sent.
• Only Activity.Text is streamed.
  • Configurable Activity.TextFormat
• The final complete message is sent via a normal 'message' type Activity
  • This should be the complete buffered text.
  • Should contain streamInfo Entity with "streamType" equal to "final", and the "streamResult"
  • Can contain any other valid Activity content.
• Unsupported channels
  • Should buffer until the stream message is complete, and sent as a 'message" Activity
  • WebChat and Teams are the only supported channels currently.

Example:

// Sending the informative message
{
  "type": "typing",
  "text": "Getting the answer...",
  "textFormat": "markdown",
  "entities": [
      {
        "type": "streaminfo",
        "streamId": "a-00001",
        "streamType": "informative",
        "streamSequence": 1,
      }
  ]
}

// Sending the first chunk
{
  "type": "typing",
  "text": "A quick brown",
  "entities": [
      {
        "type": "streaminfo",
        "streamId": "a-00001",
        "streamType": "streaming",
        "streamSequence": 2,
      }
  ]
}

// Sending the second chunk
{
  "type": "typing",
  "text": "A quick brown fox jumped over the",
  "entities": [
      {
        "type": "streaminfo",
        "streamId": "a-00001",
        "streamType": "streaming",
        "streamSequence": 3,
      }
  ]
}

// Sending the last chunk
{
  "type": "typing",
  "text": "A quick brown fox jumped over the lazy dog.",
  "entities": [
      {
        "type": "streaminfo",
        "streamId": "a-00001",
        "streamType": "streaming",
        "streamSequence": 4,
      }
  ]
}

// Final message
{
  "type": "message",
  "text": "A quick brown fox jumped over the lazy dog.",
  "entities": [
      {
        "type": "streaminfo",
        "streamId": "a-00001",
        "streamType": "final",
        "streamSequence": 5,
      }
  ]
}

Other

• Bot Framework/MCS SDK
  • New StreamInfo Entity
  • Transcription
    • May want to optionally allow transcription of Typing Activities containing the "streaminfo" Entity
• This should probably be a proactive operation due to the indeterminate length of the streamed responses.
  • Mindful in handling Azure scale-down during a long operation.
  • Less of an immediate issue for MCS SDK since AsyncCloudAdapter is being used.