Package v1 AI Assistant for production release

[christad92]

Motivation

One of the many application areas of AI in OpenFn is to enable users to do more on the platform. By doing more, we mean writing jobs faster, building workflows faster, and building more adaptors when required among others. We have implemented a v0 of this assistant that leveraged an OpenAI GPT-based model that is deployed on Apollo to answer questions users might have. This feature was limited to superusers ONLY and we are now making this open to all users within the instance.

Design: https://www.figma.com/design/4hGWdO4sajnjcOwE81j8BK/App-(new)?node-id=4970-635&t=kVYLqPxptbSYr3PY-0

Requirements:

1. Remove the superuser flag:
   The feature should be available to all users with owner, editor, and admin roles in a project. Viewers can only see the chats but will not be able to make queries.
   • Move ai-chat user permission check to policies/users.ex #2405
2. Add a table for chat_session
   -- Project_id
   -- User_id
   -- Is_public (others in the project can see the session and default to true)
   -- Is_deleted
   -- Workflow_id
   -- Step_id
   -- Model
3. Add a table for chat_message so we can keep context for the workflow and count & limit usage
   -- Chat_session_id
   -- Timestamp
   -- user_id
   -- Is_public (to let other users in your project view this message and default to true)
   -- Question
   -- Answer
   -- Is_deleted
4. Polish CSS in AI Assistant panel:
   We want to align the UX and style of the panel to align with existing designs.
   -- Input section with the send button
   -- Colour of the buttons
   -- User profile placeholder for message callout
5. Add an empty state page if AI is not enabled in ENV or if the API key is missing. (We won't hide the tab).
   We want to be able to convert OSS users to sign up for cloud-hosted service if they don’t have AI set up but can click on a link to test the feature on https://app.openfn.org/.
6. Limit usage per project

• Limit usage of AI Assistant #2438

7. Chat Sessions are based on steps in a workflow and users can start new sessions or continue from a previous session.
   -- For each session, users should be able to scroll up to see the previously asked questions since the AI assistant is limited by plan, it is essential to ensure that users don’t waste credits by repeating the same questions.
   -- Users should be able to delete questions/answers in a session if they are the author of the session.
8. Add disclaimer text for users:
   -- AI can make mistakes and it is essential to tell users that they should validate the responses.
9. Display usage counts
   -- Implement a card that will show the usage summary of runs and AI credits in the current billing cycle.
   -- The card should show runs and AI credits in a progress bar
   -- The card should tell users when the plan will renew
   -- The card should have a link to take the user to the subscription page for the project. This will replace the existing subscription menu on the project scope.
   This plan usage card will be visible to ALL users

Nice to Haves:

1. Allow context in the data model

• Some sort of pointer to the previous response?
• How much of the history do we send?
• What other context do we include?

2. Add new features to the workflow panel (and build into github.com/openfn/apollo)

• Context settings (all default to false for security but could be turned on per workflow?) to “allow sending X if requested”
• Consider sending the current workflow code as context.
• Consider sending input data as context?
  --- Add a note to the user that it is advisable for them to ONLY do this with test data rather than real data which can include PII or sensitive business data.
  --- Joe Clark says could we scrub the data such that we will only send the shape of the data (keys/structure) but not the values.
• Consider sending output data as context?
• Consider sending LOGS as context.

3. Allow searching through history and across sessions
4. Allow users to be able to toggle if a session should be made public or private.

[josephjclark]

UPDATE: I've spun out a new issue for this: See #2411

ORIGINAL POST:
A couple of inputs from me - I'll come back and edit this comment, I'm just capturing some notes

The disclaimer needs to include:

• a link to our responsible AI policy (when its ready)
• A reminder to take all AI suggestions with a pinch of salt (and maybe a link to community?)
• A mention that we're using claude (btw I need to swap out the backend)
• A warning not to paste PPI into the chat box.

We need to think carefully about adding input and output data to the prompt because there's some serious data protection stuff. Maybe users ALWAYS have to click a button to attach state/output, and maybe ALWAYS have to confirm the action. Also it should clearly indicate to the user when input state is part of the prompt.

Some other stuff:

• If the workflow has used AI chat, there needs to be prominent icon in the workflow listing and maybe title (the icon links to the chat directly). This is both a) a disclosure that AI was used to build the workflow, and b) a short-cut to get to the chart. This will be an important part of our Responsible AI policy (which we'll be sharing for feedback in a couple of days)
• I'm thinking of being _really _ obnoxious with the disclaimer. Every time the panel is opened, the disclaimer is like a full-screen cover, and you have to click to dismiss it.
• Alternatively / as well, we should have a tiny text warning next to the input box (claude actually has a disclaimer on every output message)

[josephjclark]

We also need to support the streaming API in both apollo and Lightning, so that each token appears as its generated. This is a huge boost to the UX.

[midigofrank]

Is the chat session for the whole project? I think it makes sense to bring it down to the workflow, or even the Job itself

[taylordowns2000]

Hey @midigofrank , @stuartc , and @christad92 , was there a specific reason that requirement 9 was left out of this? This is quite important for non AI stuff also... users want to know what plan they're on, when it ends, and how much usage they have left.

Display usage counts
-- Implement a card that will show the usage summary of runs and AI credits in the current billing cycle.
-- The card should show runs and AI credits in a progress bar
-- The card should tell users when the plan will renew
-- The card should have a link to take the user to the subscription page for the project. This will replace the existing subscription menu on the project scope.
This plan usage card will be visible to ALL users

Unless there was a strong reason for omitting this, I think we should prioritize it quite high amongst the other issues in #2613

[midigofrank]

If I remember correctly, this was left out because it required much effort and also the AI credits count wasn't yet finalized. So it got out of scope for the AI issue at hand.

…

On Wed, Nov 27, 2024 at 10:49 AM Taylor Downs ***@***.***> wrote: Hey @midigofrank <https://github.com/midigofrank> , @stuartc <https://github.com/stuartc> , and @christad92 <https://github.com/christad92> , was there a specific reason that requirement 9 was left out of this? This is quite important for *non* AI stuff also... users want to know what plan they're on, when it ends, and how much usage they have left. Display usage counts -- Implement a card that will show the usage summary of runs and AI credits in the current billing cycle. -- The card should show runs and AI credits in a progress bar -- The card should tell users when the plan will renew -- The card should have a link to take the user to the subscription page for the project. This will replace the existing subscription menu on the project scope. This plan usage card will be visible to ALL users image.png (view on web) <https://github.com/user-attachments/assets/45f14315-8fc7-486a-a250-2b58b82496dc> Unless there was a strong reason for omitting this, I think we should prioritize it quite high amongst the other issues in #2613 <#2613> — Reply to this email directly, view it on GitHub <#2296 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJLYA766ZIEDW5N4HGUIWCT2CV2QBAVCNFSM6AAAAABK42UF5CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMBTGE2DAMJQHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[taylordowns2000]

Understood. I see this: #2438

Does that mean that this issue is now unblocked and ready to go as part of #2613 or are there other technical blockers?

[midigofrank]

Hmm, I don't think the limits for the ai credits have been set yet. #2438 paved the road for it.

But yes, I think we can work on adding that usage card with/without the AI credits limit.