[Copilot No. Series] Number series copilot #764
Conversation
…ut AI implementation business logic) Added new app components and capabilities for the No. Series Copilot app, including codeunits, page extensions, pages, and tables, to introduce AI features for generating number series. Additionally, new registers and installers were included to handle capability registration and app installation. This enhances the app's functionality and extensibility.
Update page and table numbers as there was overflow with the Base Application.
- Temporary adding Number Series Copilot Setup table and page to store aoai secrets, system prompt and functions prompt in the isolated storage. - Add Jupyter Notebook patterns to gitignore. - Updating codeunit, page, and table objects numbers.
The code refactor improves tool retrieval, adds tool ID handling, and updates UI elements. The changes aim to enhance the No. Series Copilot tool functionality and user experience. These alterations are crucial for integrating Azure Key Vault in future. Include tool calls to the chat history - Add support for tool call IDs in chat history, improving traceability and accountability for tool call results. The changes enable associating tool call IDs with tool messages in the chat history, enhancing the clarity and transparency of the interactions. This update aligns with the need to provide comprehensive context for tool call responses within the chat history.
- Introduce 'AddToolMessage' method and modify existing methods to capture tool-related data for chat messages. - Add 'HistoryToolCallIds' list to save tool call IDs for messages. - Create 'IsToolsList' method to check if a message contains tool-related information. - Update 'AOAIChatRoles.Enum.al' to define a new 'tool' chat role. These changes enable the tracking and differentiation of tool call results in chat history.
Enable support for returning valid JSON object as chat completion, adding guidelines for JSON production, and checking compatibility for JSON response format.
Improve json output format specification declaration and [TEMPORARY] implement no. series number limitations to avoid reaching token limit and timeout when dealing with many tables.
Consolidates tool invocation logic by using a dedicated function and refines prompt formats for better user experience. Also updates existing patterns to reflect more accurate numbering conventions. These changes aim to streamline tool usage and enhance prompt clarity.
Consolidate tool retrieval methods; refactor output format instructions. Improved modularity and maintainability.
Add new Json module to provide tools for working with JSON data, including reading, writing, and parsing JSON. Introduces codeunit for initializing JSON array and object, retrieving element counts, fetching objects by index, and getting values for specified record fields. Implements methods to interact with JSON data using .NET types. This is partly ported from Base Application codeunit 5459 "Json management"
Additionally, the visibility of response text on the No. Series Proposal Page is dynamically managed, depending on json response validation
…e number series pattern
Tools message parsing into `function name`, `arguments` and `tool call id`
Replace direct parsing with dedicated method for improved maintainability and readability. This enhances code structure and sets the stage for future extensibility. No related issues.
…f the tool response exceeds token limit. Ensure secure handling of sensitive data in chat messages by swapping text types with secure text types. Implement chunking of tool responses to avoid exceeding token limits, resulting in improved prompt generation for new number series. Indirectly address potential security vulnerabilities and enhance system functionality.
Change the dictionary to standardize chunk processing of tables in order to avoid hallucinations and effectively handle large volumes. This ensures accurate and complete model responses, enhancing overall functionality. The refactor also introduces new helper functions to process tool completions and validate generated number series.
…nd output examples from setup Introduce methods to retrieve tool 1 patterns and output examples from setup. Also, add labels and separators for better formatting. Improve error handling and optimize code readability.
Improved retrieval of tool 1 general instructions and output format for better modularity and maintainability.
…dle duplicate No. Series codes This commit refactors JSON manipulation by separating common JSON operations into a separate file and introduces new functionality to handle duplicate No. Series codes using random character generation. This change improves data integrity and ensures uniqueness of No. Series codes. It adds methods to replace or add properties in JSON objects and modify JSON arrays as necessary.
Replace text prompts for tool limitations, code guidelines, description guidelines, number guidelines, output examples, and output format prompts. Add validation for property lengths in generating number series.
Refactor JSON handling methods to return JSON objects and arrays as text, enhancing readability and modularity across the codebase, paving the way for simplified integration with other modules. This change sets the stage for easier maintenance and future expansions of the JSON-related functionalities. No related issues are involved.
| end; | ||
| end; | ||
|
|
||
| procedure ApplyProposedNoSeries(var NoSeriesGenerated: Record "No. Series Proposal Line") |
There was a problem hiding this comment.
same, i'd like the variable name to somewhat match the object name. Variables can add descriptors to make it more precise but the more they differ the more I think we simply need better names...
| var | ||
| NoSeries: Record "No. Series"; | ||
| begin | ||
| NoSeries.Init(); |
There was a problem hiding this comment.
NoSeries is a local record variable and not in a loop so there is nothing to Init ...
| var | ||
| NoSeriesLine: Record "No. Series Line"; | ||
| begin | ||
| NoSeriesLine.Init(); |
There was a problem hiding this comment.
same, don't init when not needed.
| NoSeries.Description := NoSeriesGenerated.Description; | ||
| NoSeries."Manual Nos." := true; | ||
| NoSeries."Default Nos." := true; | ||
| //TODO: Check if we need to add more fields here, like "Mask", "No. Series Type", "Reverse Sales VAT No. Series" etc. |
There was a problem hiding this comment.
having todo's in code worries me... is this just a note? should it be fixed before we ship? What if we don't fix it?
I dislike comments in code because they get outdated very quickly but I can live with them (docs are better :P) but todo's like this should not be in code.
| NoSeriesLine."Ending No." := NoSeriesGenerated."Ending No."; | ||
| NoSeriesLine."Warning No." := NoSeriesGenerated."Warning No."; | ||
| NoSeriesLine."Increment-by No." := NoSeriesGenerated."Increment-by No."; | ||
| //TODO: Check if we need to add more fields here, like "Allow Gaps in Nos.", "Sequence Name" etc. |
There was a problem hiding this comment.
I think it's important to set the "Allow Gaps" / Implementation field which depends on the purpose of the no. series.
I think we default to Normal / no gaps because it's the safe choice but we prefer Sequence / allow gaps because it has better perf.
src/Business Foundation/App/NoSeriesCopilot/src/Copilot/NoSeriesCopilotImpl.Codeunit.al
Outdated
Show resolved
Hide resolved
| FieldRef: FieldRef; | ||
| begin | ||
| RecRef.Open(NoSeriesGenerated."Setup Table No."); | ||
| if not RecRef.FindFirst() then |
There was a problem hiding this comment.
Since the assumption here seems to be that there is only 1 setup record and that record would have a blank primary key, should we not assert that more aggressively? Instead of finding the first record maybe we should 'get' the record with the blank prim. key. ... or check that the record we find has a blank value.
| } | ||
| } | ||
|
|
||
| procedure GetEndpoint() Endpoint: Text[250] |
There was a problem hiding this comment.
All these methods should be add to a codeunit somewhere. Logic should reside in codeunits not tables, pages etc...
I get the appeal of these getters but they are not used consistently. Sometimes they are used, sometimes not. They are also not using the Rec. as I'd expect but instead do their own Get... yes, these will likely hit the cache but still, we have Setup.Get all over the place now instead of getting the record once only..
src/Business Foundation/App/NoSeriesCopilot/src/Copilot/NoSeriesCopilotImpl.Codeunit.al
Outdated
Show resolved
Hide resolved
src/Business Foundation/App/NoSeriesCopilot/src/Copilot/NoSeriesCopilotImpl.Codeunit.al
Outdated
Show resolved
Hide resolved
| AOAIChatMessages.DeleteMessage(AOAIChatMessages.GetHistory().Count); // remove the last message, as it is not needed anymore | ||
| AOAIChatMessages.DeleteMessage(AOAIChatMessages.GetHistory().Count); // remove the tools message, as it is not needed anymore | ||
|
|
||
| Sleep(1000); // sleep for 1000ms, as the model can be called only limited number of times per second |
There was a problem hiding this comment.
is 1000ms really the right number? do we have any plan to finetune this a little more?
src/Business Foundation/App/NoSeriesCopilot/src/Copilot/NoSeriesCopilotImpl.Codeunit.al
Outdated
Show resolved
Hide resolved
| ApplyProposedNoSeries(); | ||
| end; | ||
|
|
||
| local procedure GenerateNoSeries() |
There was a problem hiding this comment.
these methods should be moved to a codeunit... and we should have a oneliner here at most..
|
Proposing numerical sequences for use in production settings diverges from those in demonstrations. It's important to evaluate the long-term viability of each series and distinguish between various types of documents. I have seen so many systems where the partner has started with Contoso/Cronus number series and belive that this is the way to go. |
…ditional function to get results in json, instead of just prompting it. X5 increase in quality and stability. Improve the way tool prompts are generated for adding and changing number series, ensuring clarity and efficiency in user interactions. These changes aim to enhance the prompt building process based on specified tables and patterns, leading to a more streamlined user experience and accurate data handling.
Added
It has prompt guides and a placeholder now
I added the
Since I changed the approach from "Get Intent > Run intent function > Add result to chat history > get final answer" to "Get Intent > Run intent function > Get Result > Make standalone AI call without chat history" this is not relevant anymore. But thanks for reimplemening the function calling. Much easier to work now.
I've removed token counting inside of the functions calling. Instead, it's chunking requests based on number series to generate. As now i use standalone calls, instead of sending chat history from the beginning and I build the final prompt, I'm pretty in control of used tokens. According to the testing each call consumes average 15% of prompt tokens and 50% of completion tokens.
There was a discussion on this, but seems we don't have other ways, currently how to find setup tables. Some additional checks could be that
I followed your advice, and instead of "ask in the next Json format" approach I use now 3rd function and get Json array as input. This was a really good advice, which allowed to increase correctness from 10 tables to 50. However, in final version of the copilot I left 40 tables, as 50 works in 9 cases out of 10, according to my tests. And 40 seems to work always and usually cover most of the scenarios.
This is a good idea, but not in this version. However, after moving to function call, instead of json output, and additional prompt finetuning i see much reduce in wrong generations. We also discussed this, outside of this thread, and decided to add Also, i implemented a logic, that if number series is already set up in the "[area] Setup" table, then we don't need to generate it once again. Meaning, that if user asks "Generate me number series for purchase module", accepts the first result and then asks once again, there will be no generations as it's already configured. This will reduce the number of "duplications" errors and also quality will increase.
My tests say that approx. 38 seconds to generate 40 number series.
Answered above
Thank you for support in this journey ;) |
Added a new Codeunit for managing notifications, refactored notification handling to include centralized notification sending and recalling, and implemented handling for chat completion errors. This update enhances the error notification system and ensures consistent error messaging. This commit solves some microsoft#764 reviews
Improve setup table retrieval by filtering and checking metadata, ensuring correct setup table identification and validation. Enhances data accuracy and reliability.
|
😅 "I worked really hard (c)" and now we have improved version of No Series Copilot.
This was achieved, due to new way of generating the final json array of number series. Thanks @darjoo for advice. |
Adjusted function to correctly remove the last character in the AreasText variable to ensure proper string formatting. This change ensures accurate text representation.
Summary
Introducing the first [draft] version of a "Number Series Copilot" for Business Central, an innovative feature designed to simplify the creation of number series through natural language commands. This allows users to effortlessly set up number series for specific entities like customers, vendors, or even across all entities, by simply describing their needs or specifying a format example.
Current version is intent to create number series for NEW company, where no Number Series exist yet.
Number.Series.Copilot.-.New.company.Scenario.mp4
This flow was tested on the
Set up numbers series for the new companylike user promptsPlease follow this diagram (green flow), to get the idea of the ongoing process.
Key ideas:
Setup:
No. Series with Copilot Setupto set up Azure Open AI connection and promptsgpt-35-turbo-1106Work Item(s)
Fixes #659
Fixes AB#503148