Integrate Whisper CPP and write a wrapper module in Aprapipes

[joiskash]

IMPORTANT: All PRs must be linked to an issue (except for extremely trivial and straightforward changes).

Fixes #321

Description

1. Added a custom port of whisper cpp to vcpkg
2. Added a WhisperStreamTransform module that can load a module and process raw PCM audio streams.

Alternative(s) considered
Could have integrated other ASR modules like MMS by meta, Kaldi etc. But this is a pure c++ implementation of whisper.
TODO: Enable GPU version

Have you considered any alternatives? And if so, why have you chosen the approach in this PR?

Type

Type Choose one: Feature

Screenshots (if applicable)

20231222-1614-50.5028935.mp4

Checklist

• I have read the Contribution Guidelines
• I have written Unit Tests
• I have discussed my proposed solution with code owners in the linked issue(s) and we have agreed upon the general approach

[kumaakh]

Please address the comments

[kumaakh]

Ok so this PR is doing 2 unrelated tasks then... 1. Migrating to vs2022 2. Adding whisper.cpp Can you first raise a PR for 1.

…

On Wed, Dec 27, 2023, 11:12 AM Kashyap Jois ***@***.***> wrote: *IMPORTANT: All PRs must be linked to an issue (except for extremely trivial and straightforward changes).* Fixes #321 <#321> *Description* 1. Added a custom port of whisper cpp to vcpkg 2. Added a WhisperStreamTransform module that can load a module and process raw PCM audio streams. *Alternative(s) considered* Could have integrated other ASR modules like MMS by meta, Kaldi etc. But this is a pure c++ implementation of whisper. TODO: Enable GPU version Have you considered any alternatives? And if so, why have you chosen the approach in this PR? *Type* Type Choose one: Feature *Screenshots (if applicable)* https://github.com/Apra-Labs/ApraPipes/assets/48797364/9d9e0fe7-093c-448f-9340-180a1d77b638 *Checklist* - I have read the Contribution Guidelines <https://github.com/Apra-Labs/ApraPipes/wiki/Contribution-Guidelines> - I have written Unit Tests - I have discussed my proposed solution with code owners in the linked issue(s) and we have agreed upon the general approach ------------------------------ You can view, comment on, or merge this pull request online at: #324 Commit Summary - 8e23a6e <8e23a6e> Add custom port vcpkg for whisper - fb29351 <fb29351> Add whisper stream - 4ed21e2 <4ed21e2> Add whisper stream header - deab2d0 <deab2d0> Add whisper cpp to Cmake list - 9462f59 <9462f59> Add test frame type and minor changes - c55c40b <c55c40b> Add whisper to vcpkg - 870862c <870862c> Add vcpkg custom overlay ports to thirdparty - ca4a6e4 <ca4a6e4> Modify with whisper option - 482f02c <482f02c> Send whisper output as text frames - d12edf5 <d12edf5> revert changes to sound record test - 3275afd <3275afd> Add whisper UT - acd8a1f <acd8a1f> Fix PS to remove whisper from vcpkg json File Changes (15 files <https://github.com/Apra-Labs/ApraPipes/pull/324/files>) - *M* base/CMakeLists.txt <https://github.com/Apra-Labs/ApraPipes/pull/324/files#diff-39632a26106d4456aaea786080d87f45b82c5557c943509ac13adfe4f5e40331> (31) - *M* base/fix-vcpkg-json.ps1 <https://github.com/Apra-Labs/ApraPipes/pull/324/files#diff-108fa56af17edc4832a9a60f2da2893d4a9d97d1520c754279fae2e490aaa60a> (12) - *M* base/include/FrameMetadata.h <https://github.com/Apra-Labs/ApraPipes/pull/324/files#diff-115465558e03e51e12483e2ad85633b6dad58c11da6207617d174e0996861dde> (3) - *M* base/include/Mp4WriterSinkUtils.h <https://github.com/Apra-Labs/ApraPipes/pull/324/files#diff-b3a7c1e318d8509718a493184c6d752657146b041fa99390527db2c0f6081026> (3) - *A* base/include/WhisperStreamTransform.h <https://github.com/Apra-Labs/ApraPipes/pull/324/files#diff-71676037027429c7ed55bc971d5ddd05a94b6103284e8dfdcbf088ecd76809d1> (50) - *A* base/src/WhisperStreamTransform.cpp <https://github.com/Apra-Labs/ApraPipes/pull/324/files#diff-159a6c84279995c9144164221805378f3705ca9f80b3c43329edfbe8f708a1dc> (180) - *M* base/test/sound_record_tests.cpp <https://github.com/Apra-Labs/ApraPipes/pull/324/files#diff-d9e343178f14bd88ece459e221fe62ad95d26a5b6958a5626a76f31d9ca2a718> (42) - *A* base/test/whisper_asr_tests.cpp <https://github.com/Apra-Labs/ApraPipes/pull/324/files#diff-7901c86f236b754236887a01d999e10feb80d290728270467b0e12e96303292e> (58) - *M* base/vcpkg.json <https://github.com/Apra-Labs/ApraPipes/pull/324/files#diff-390712685125a9163b675b1920dcf6285333ebfad24878cc15939ed173e7720d> (102) - *A* data/whisper/models/.gitattributes <https://github.com/Apra-Labs/ApraPipes/pull/324/files#diff-fe4c27734f20effed3f7a3cc4c6b1e2079c3ca63d40d770d6970686fd24c86c7> (1) - *A* data/whisper_asr_test.pcm <https://github.com/Apra-Labs/ApraPipes/pull/324/files#diff-224ddf225063503561f2980ea75c16392cc7ca0d0546e76136ef50013b8e3370> (0) - *A* thirdparty/custom-overlay/whisper/portfile.cmake <https://github.com/Apra-Labs/ApraPipes/pull/324/files#diff-3ce03f71738e53bacf331247916f52eefa32a1955687d9ebefa9a6a85a436683> (29) - *A* thirdparty/custom-overlay/whisper/usage <https://github.com/Apra-Labs/ApraPipes/pull/324/files#diff-7888d0ea9ddfd34ec2a9ba22beb62c42ce38a90c3f721507744a4b4100be3f36> (4) - *A* thirdparty/custom-overlay/whisper/vcpkg.json <https://github.com/Apra-Labs/ApraPipes/pull/324/files#diff-324913f25a116299ae5179dc71058eef30e59b5a255884a3371037ffa603f533> (17) - *M* vcpkg <https://github.com/Apra-Labs/ApraPipes/pull/324/files#diff-ca4b9abdd99374b1e975427f9c9162f723d8125473a06ca29059630d347474b3> (2) Patch Links: - https://github.com/Apra-Labs/ApraPipes/pull/324.patch - https://github.com/Apra-Labs/ApraPipes/pull/324.diff — Reply to this email directly, view it on GitHub <#324>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABWFNAWD3GEGLKJ6OL7GT5LYLRCFNAVCNFSM6AAAAABBEQVTU2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGA2TONJUGU3TAOI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[joiskash]

Ok so this PR is doing 2 unrelated tasks then...

1. Migrating to vs2022
2. Adding whisper.cpp Can you first raise a PR for 1.

Apra-Labs/vcpkg#4 I have raised a PR for this in Apra-Labs vcpkg

[kumaakh]

2 small changes required: move the Constructor body to CPP
fix the alloc/free for the vector

[github-actions]

Test Results Windows-cuda

  1 files  ±0    1 suites  ±0   12m 34s ⏱️ +2s
404 tests +4  297 ✅ +4  107 💤 ±0  0 ❌ ±0 
297 runs  +4  190 ✅ +4  107 💤 ±0  0 ❌ ±0 

Results for commit 42df5de. ± Comparison against base commit 110a2e2.

♻️ This comment has been updated with latest results.

[github-actions]

Test Results Linux-CudaT

  1 files  ±0    1 suites  ±0   10m 30s ⏱️ +6s
408 tests +4  241 ✅ +4  167 💤 ±0  0 ❌ ±0 
241 runs  +4   74 ✅ +4  167 💤 ±0  0 ❌ ±0 

Results for commit 42df5de. ± Comparison against base commit 110a2e2.

♻️ This comment has been updated with latest results.

[github-actions]

Test Results Linux_ARM64

  1 files  ±0    1 suites  ±0   10m 55s ⏱️ +12s
430 tests +4  266 ✅ +4  164 💤 ±0  0 ❌ ±0 
266 runs  +4  102 ✅ +4  164 💤 ±0  0 ❌ ±0 

Results for commit 42df5de. ± Comparison against base commit 110a2e2.

♻️ This comment has been updated with latest results.

[mraduldubey]

format using vscode ctrl + K, F command