Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove capture page, and from the moment one start speaking, a memory in progress should appear. #606

Closed
josancamon19 opened this issue Aug 16, 2024 · 4 comments
Assignees

Comments

@josancamon19
Copy link
Contributor

josancamon19 commented Aug 16, 2024

Is your feature request related to a problem? Please describe.

  • I want to make sure memories are not lost
  • A better onboarding experience, when having your first memory automatically created.
  • UI simplification, removing one whole tab, and having memories as main.

Describe the solution you'd like
This requires the backend to determine when a memory is created.

  • How to determine new memory id.
  • So the websocket once it receives transcript, it should schedule a 2 minutes event that finishes the conversation.
  • Once the conversation is finished, the memory object should be sent back in the websocket to the app.
  • The mobile UI should handle this by having in progress memories + removing capture/page.dart.
  • Websocket reconnection has to handle "memory in progress" scenarios.
@kodjima33 kodjima33 moved this to Backlog in omi TODO Aug 16, 2024
@josancamon19
Copy link
Contributor Author

The idea here @beastoin is to modify the websocket behavior a little bit.

  1. The backend should have the 2 minutes timer determining when to create a memory, not the app.
  2. The app would handle receive events from the websocket and act accordingly, this will be either new segments of a memory, a new memory was created, etc.
  3. Post processing should be handled on the backend side, this means we should be able to keep a temporal version of the file while a memory is in progress.
  4. capture/page.dart should be removed, instead we should have only memories page, and when a new memory starts, it should appear there, on top as in-progress. -> on memory detail opened, it should show summary in progress, but show the current transcript as it goes.

@beastoin
Copy link
Collaborator

interesting! lemme 🏄‍♀️

@josancamon19
Copy link
Contributor Author

More context.

Image

@beastoin
Copy link
Collaborator

PR: #684

beastoin added a commit that referenced this issue Sep 5, 2024
Issue: #606

**Status:** 🏄‍♀️  Ready on staging, waiting for review

## Key results
- [x] No more Capture Page. User can see the new memory immediately
after speaking, in progress, no summary til completed, show in the
memory page `p1`
- [x] Backend handle create memory logic, when, what, how. Frontend just
needs stream audio to Backend, that's all.
- [x] No more post-processing request from Frontend.

## Todos
- [x] Remove capture page
- [x] UI for processing memory on Memory page
- [x] Handle create memory logic from Backend
  - [x] New memory watch
  - [x] Message event via socket
  - [x] Backward compatible with the old-app version
  - [x] Support geolocation, emotional feedback
- [x] Handle post-processing logic from Backend
  - [x] Capture audio frame, rebuild audio wav
  - [x] Do post-processing 
- [x] Fix the weird slowlyyyy on talking speed with opus, guess that is
the re-frames problem, double check another codec pcm8, pcm16
- [x] Double check `.photos` from Frontend whenever creating new memory
in capture page
- [x] Maintain the old-logic for OpenGlass on creating new memory
- [x] Enhance UX
- [ ] Should leave a check on
  - [x] Mic recording
  - [x] initiateWebsocket / resetState on provider
  - [ ] Openglass memory

## Deploy plan
 - [ ] Deploy backend > api
 - [ ] Deploy app > ios, android

---


















<!-- This is an auto-generated comment: release notes by OSS
Entelligence.AI -->
### Summary by Entelligence.AI

- New Feature: Introduced a new function `getProcessingMemoryById` to
retrieve processing memories by ID, enhancing the memory retrieval
process.
- New Feature: Added support for handling server message events in the
WebSocket stream, improving real-time data handling.
- New Feature: Implemented a new function `processMemoryContent` that
processes messages related to a server memory, providing better memory
content management.
- Refactor: Adjusted tab navigation and UI elements on the home page for
improved user experience.
- Refactor: Updated the `process_audio_dg` function in the backend for
more efficient transcript segment handling.
- New Feature: Introduced a new `LiteTranscriptWidget` class for
alternative UI rendering of transcripts.
<!-- end of auto-generated comment: release notes by OSS Entelligence.AI
-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

No branches or pull requests

2 participants