Add StreamingLLM support to studio2 chat #2060

monorimet · 2024-01-09T07:34:59Z

adds in call to gen_external_params for quantized weights generation
separates weights management from model init
separates vmfb loading from compilation (switch to vmfbRunner)
switches api test to use a smaller model

apps/shark_studio/api/utils.py

dan-garvey · 2024-01-09T17:41:00Z

should we adapt to just use the turbine llm runner?

monorimet · 2024-01-09T18:43:58Z

should we adapt to just use the turbine llm runner?

sounds OK to me, but will need some work to integrate with the UI
might help with maintenance since turbine seems to be the favorite for new features/dev workflows... we just need to have an option to run with SRT and its bleeding edge flags etc

monorimet · 2024-01-17T18:10:00Z

issue filed for the edge case preventing us from running the api test on a small model with externalized weights iree-org/iree#16138

dan-garvey · 2024-01-17T17:31:07Z

apps/shark_studio/api/llm.py

@@ -99,30 +175,57 @@ def __init__(

    def compile(self) -> None:


I'd prefer not moving all the compile code into this non reusable place, unless this is part of a migration intended to deprecate the shark api

mistake, should just be model arch/api-specific compile flags

dan-garvey · 2024-01-17T19:07:45Z

apps/shark_studio/tests/api_test.py



 class LLMAPITest(unittest.TestCase):
-    def testLLMSimple(self):
+    def test01_LLMSmall(self):


weird naming?

forces a certain ordering of tests, but won't be truly necessary until more tests are added.

dan-garvey · 2024-01-17T19:08:15Z

apps/shark_studio/web/ui/chat.py

    cli=False,
 ):
    global language_model
+    print("Prompt prefix: ", prompt_prefix)


dan-garvey · 2024-01-17T19:08:49Z

apps/shark_studio/web/ui/chat.py

@@ -63,7 +72,7 @@ def chat_fn(
    prefill_time = 0
    is_first = True
    for text, exec_time in language_model.chat(history):
-        history[-1][-1] = text
+        history[-1][-1] = f"{text}{E_SYS}"


ideally f"{text}{E_SYS}" would happen at the api layer no?

I thought the same but struggled to reproduce good results after moving this to api layer. IIRC it had to do with how we yield from the underlying function call.

raikonenfnu · 2024-01-17T19:24:30Z

Is there a way we can specify the vmfb and safetensor path?

monorimet · 2024-01-17T20:38:14Z

Would be good to put in an option to set self.vmfb_path instead of renaming the target file. Will update in a few.

monorimet · 2024-01-18T05:05:31Z

Specifying vmfb path etc. will take a genuine CLI option interface or an option in the UI, both of which are significant changes and should come in a follow-up.

@dan-garvey I would prefer not to touch the finely balanced prompt handling until SDXL is finished, and we need this patch to clear the turbine CI.

dan-garvey

Please file issue for prompt handling, this is how we got where we were in 1.0

monorimet · 2024-01-19T01:04:18Z

Please file issue for prompt handling, this is how we got where we were in 1.0

#2073 filed

* Streaming LLM * Update precision and add gpu support * (studio2) Separate weights generation for quantization support * Adapt prompt changes to studio flow * Remove outdated flag from llm compile flags. * (studio2) use turbine vmfbRunner * tweaks to prompts * Update CPU path and llm api test. * Change device in test to cpu. * Fixes to runner, device names, vmfb mgmt * Use small test without external weights.

* (WIP): Studio2 app infra and SD API UI/app structure and utility implementation. - Initializers for webui/API launch - Schedulers file for SD scheduling utilities - Additions to API-level utilities - Added embeddings module for LoRA, Lycoris, yada yada - Added image_processing module for resamplers, resize tools, transforms, and any image annotation (PNG metadata) - shared_cmd_opts module -- sorry, this is stable_args.py. It lives on. We still want to have some global control over the app exclusively from the command-line. At least we will be free from shark_args. - Moving around some utility pieces. - Try to make api+webui concurrency possible in index.py - SD UI -- this is just img2imgUI but hopefully a little better. - UI utilities for your nod logos and your gradio temps. Enable UI / bugfixes / tweaks * Studio2/SD: Use more correct LoRA alpha calculation (#2034) * Updates ProcessLoRA to use both embedded LoRA alpha, and lora_strength optional parameter (default 1.0) when applying LoRA weights. * Updates ProcessLoRA to cover more dim cases. * This bring ProcessLoRA into line with PR #2015 against Studio1 * Studio2: Remove duplications from api/utils.py (#2035) * Remove duplicate os import * Remove duplicate parse_seed_input function Migrating to JSON requests in SD UI More UI and app flow improvements, logging, shared device cache Model loading Complete SD pipeline. Tweaks to VAE, pipeline states Pipeline tweaks, add cmd_opts parsing to sd api * Add test for SD * Small cleanup * Shark2/SD/UI: Respect ckpt_dir, share and server_port args (#2070) * Takes whether to generate a gradio live link from the existing --share command line parameter, rather than hardcoding as True. * Takes server port from existing --server_port command line parameter, rather than hardcoding as 11911. * Default --ckpt_dir parameter to '../models' * Use --ckpt_dir rather than hardcoding ../models as the base directory for checkpoints, vae, and lora, etc * Add a 'checkpoints' directory below --ckpt_dir to match ComfyUI folder structure. Read custom_weights choices from there, and/or subfolders below there matching the selected base model. * Fix --ckpt_dir possibly not working correctly when an absolute rather than relative path is specified. * Relabel "Custom Weights" to "Custom Weights Checkpoint" in the UI * Add StreamingLLM support to studio2 chat (#2060) * Streaming LLM * Update precision and add gpu support * (studio2) Separate weights generation for quantization support * Adapt prompt changes to studio flow * Remove outdated flag from llm compile flags. * (studio2) use turbine vmfbRunner * tweaks to prompts * Update CPU path and llm api test. * Change device in test to cpu. * Fixes to runner, device names, vmfb mgmt * Use small test without external weights. * HF-Reference LLM mode + Update test result to match latest Turbine. (#2080) * HF-Reference LLM mode. * Fixup test to match current output from Turbine. * lint * Fix test error message + Only initialize HF torch model when used. * Remove redundant format_out change. * Add rest API endpoint from LanguageModel API * Add StreamingLLM support to studio2 chat (#2060) * Streaming LLM * Update precision and add gpu support * (studio2) Separate weights generation for quantization support * Adapt prompt changes to studio flow * Remove outdated flag from llm compile flags. * (studio2) use turbine vmfbRunner * tweaks to prompts * Update CPU path and llm api test. * Change device in test to cpu. * Fixes to runner, device names, vmfb mgmt * Use small test without external weights. * Formatting and init files. * Remove unused import. * Small fixes * Studio2/SD/UI: Improve various parts of the UI for Stable Diffusion (#2074) * Studio2/SD/UI: Improve various parts of the UI of Shark 2 * Update Gradio pin to 4.15.0. * Port workarounds for Gradio >4.8.0 main container sizing from Shark 1.0. * Move nod Logo out of the SD tab and onto the top right of the main tab bar. * Set nod logo icon as the favicon (as current Shark 1.0). * Create a tabbed right hand panel within the SD UI sized to the viewport height. * Make Input Image tab 1 in the right hand panel. * Make output images, generation log, and generation buttons, tab 2 in the right hand panel * Make config JSON display, with config load, save and clear, tab 3 in the right hand panel * Make gallery area of the Output tab take up all vertical space the other controls on the tab do not. * Tidy up the controls on the Config tab somewhat. * Studio2/SD/UI: Reorganise inputs on Left Panel of SD tab * Rename previously added Right Panel Output tab to 'Generate'. * Move Batch Count, Batch Size, and Repeatable Seeds, off of Left Panel and onto 'Generate' Tab. * On 'Generate' tab, rename 'Generate Image(s)' button to 'Start', and 'Stop Batch' button to 'Stop'. They are now below the Batch inputs on a Generate tab so don't need the specificity. * Move Device, Low VRAM, and Precision inputs into their own 'Device Settings' Accordion control. (starts closed) * Rename 'Custom Weights Checkpoint' to 'Checkpoint Weights' * Move Checkpoint Weights, VAE Model, Standalone Lora Weights, and Embeddings Options controls, into their own 'Model Weights' Accordion control. (starts closed) * Move Denoising Strength, and Resample Type controls into their own 'Input Image Processing' Accordion. (starts closed) * Move any remaining controls in the 'Advanced Options' Accorion directly onto the left panel, and remove then Accordion. * Enable the copy button for all text boxes on the SD tab. * Add emoji/unicode glphs to all top level controls and Accordions on the SD Left Panel. * Start with the 'Generate' as the initially selected tab in the SD Right Panel, working around Gradio issue #7805 * Tweaks to SD Right Tab Panel vertical height. * Studio2/SD/UI: Sizing tweaks for Right Panel, and >1920 width * Set height of right panel using vmin rather than vh, with explicit affordances for fixed areas above and below. * Port >1920 width Gradio >4.8 CSS workaround from Shark 1.0. * Studio2/SD: Fix sd pipeline up to "Windows not supported" (#2082) * Studio2/SD: Fix sd pipeline up to "Windows not supported" A number of fixes to the SD pipeline as run from the UI, up until the point that dynamo complains "Windows not yet supported for torch.compile". * Remove separate install of iree-runtime and iree-compile in setup_venv.ps1, and rely on the versions installed via the Turbine requirements.txt. Fixes #2063 for me. * Replace any "None" strings with python None when pulling the config in the UI. * Add 'hf_auth_token' param to api StableDiffusion class, defaulting to None, and then pass that in to the various Models where it is required and wasn't already being done before. * Fix clip custom_weight_params being passed to export_clip_model as "external_weight_file" rather than "external_weights" * Don't pass non-existing "custom_vae" parameter to the Turbine Vae Model, instead pass custom_vae as the "hf_model_id" if it is set. (this may be wrong in the custom vae cast, but stops the code *always* breaking). * Studio2/SD/UI: Improve UI config None handling * When populating the UI from a JSON Config set controls to "None" for null/None values. * When generating a JSON Config from the UI set props to null/None for controls set to "None". * Use null rather string 'None' in the default config --------- Co-authored-by: Ean Garvey <[email protected]> * Studio2/SD/UI: Further sd ui pipeline fixes (#2091) On Windows, this gets us all the way failing in iree compile of the with SD 2.1 base. - Fix merge errors with sd right pane config UI tab. - Remove non-requirement.txt install/build of torch/mlir/iree/SRT in setup_venv.ps1, fixing "torch.compile not supported on Windows" error. - Fix gradio deprecation warning for `root=` FileExplorer kwarg. - Comment out `precision` and `max_length` kwargs being passed to unet, as not yet supported on main Turbine branch. Avoids keyword argument error. * Tweak compile-time flags for SD submodels. * Small fixes to sd, pin mpmath * Add pyinstaller spec and imports script. * Fix the .exe (#2101) * Fix _IREE_TARGET_MAP (#2103) (#2108) - Change target passed to iree for vulkan from 'vulkan' to 'vulkan-spriv', as 'vulkan' is not a valid value for --iree-hal-target-backends with the current iree compiler. Co-authored-by: Stefan Kapusniak <[email protected]> * Cleanup sd model map. * Update dependencies. * Studio2/SD/UI: Update gradio to 4.19.2 (sd-studio2) (#2097) - Move pin for gradio from 4.15 -> 4.19.2 on the sd-studio2 branch * fix formatting and disable explicit vulkan env settings. --------- Co-authored-by: Stefan Kapusniak <[email protected]> Co-authored-by: Stanley Winata <[email protected]> Co-authored-by: gpetters-amd <[email protected]> Co-authored-by: gpetters94 <[email protected]>

monorimet added 2 commits January 8, 2024 23:25

Streaming LLM (WIP)

f6b249a

Update precision and add gpu support

353d50f

monorimet requested review from dan-garvey and raikonenfnu January 9, 2024 07:35

monorimet added 3 commits January 9, 2024 10:54

(studio2) Separate weights generation for quantization support

ee53971

Small fixes to prompts, weights gen

79460ef

Adapt prompt changes to studio flow

8283054

dan-garvey requested changes Jan 9, 2024

View reviewed changes

apps/shark_studio/api/utils.py Outdated Show resolved Hide resolved

Final commit

a7b2231

Update utils.py

f456e87

monorimet marked this pull request as ready for review January 9, 2024 18:47

monorimet added 8 commits January 9, 2024 13:05

Remove outdated flag from llm compile flags.

df78dec

(studio2) use turbine vmfbRunner

163241a

fix streaming

3ac56ad

bugfixes

a099d76

Fix stopping

32e712b

tweaks to prompts

ead1715

Update CPU path and llm api test.

c195dd6

Fix formatting.

a788e12

monorimet requested a review from dan-garvey January 16, 2024 17:42

monorimet added 4 commits January 16, 2024 11:44

Remove unused function.

c48c562

Change device in test to cpu.

643214d

Fixes to runner, device names, vmfb mgmt

dccd585

Only run small tests.

1ed5543

monorimet force-pushed the streaming-llm branch 2 times, most recently from 1c7896d to 6428316 Compare January 17, 2024 17:04

monorimet requested a review from aviator19941 January 17, 2024 18:11

monorimet force-pushed the streaming-llm branch from 15e76eb to f8f61f8 Compare January 17, 2024 18:12

Use small test without external weights.

9ff8fa7

monorimet force-pushed the streaming-llm branch from f8f61f8 to 9ff8fa7 Compare January 17, 2024 18:24

dan-garvey requested changes Jan 17, 2024

View reviewed changes

monorimet and others added 3 commits January 17, 2024 13:15

remove redundant compile flags

ed780b0

remove print of prompt prefix

29c40f4

Fix flags.

0c6e5ca

dan-garvey approved these changes Jan 19, 2024

View reviewed changes

monorimet merged commit 05b4982 into main Jan 19, 2024
2 checks passed

monorimet deleted the streaming-llm branch January 19, 2024 01:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add StreamingLLM support to studio2 chat #2060

Add StreamingLLM support to studio2 chat #2060

monorimet commented Jan 9, 2024 •

edited

Loading

dan-garvey commented Jan 9, 2024

monorimet commented Jan 9, 2024

monorimet commented Jan 17, 2024

dan-garvey Jan 17, 2024

monorimet Jan 17, 2024

dan-garvey Jan 17, 2024

monorimet Jan 17, 2024

dan-garvey Jan 17, 2024

monorimet Jan 17, 2024

dan-garvey Jan 17, 2024

monorimet Jan 17, 2024

raikonenfnu commented Jan 17, 2024 •

edited

Loading

monorimet commented Jan 17, 2024

monorimet commented Jan 18, 2024

dan-garvey left a comment

monorimet commented Jan 19, 2024

		@@ -99,30 +175,57 @@ def __init__(

		def compile(self) -> None:

Add StreamingLLM support to studio2 chat #2060

Add StreamingLLM support to studio2 chat #2060

Conversation

monorimet commented Jan 9, 2024 • edited Loading

dan-garvey commented Jan 9, 2024

monorimet commented Jan 9, 2024

monorimet commented Jan 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

raikonenfnu commented Jan 17, 2024 • edited Loading

monorimet commented Jan 17, 2024

monorimet commented Jan 18, 2024

dan-garvey left a comment

Choose a reason for hiding this comment

monorimet commented Jan 19, 2024

monorimet commented Jan 9, 2024 •

edited

Loading

raikonenfnu commented Jan 17, 2024 •

edited

Loading