The Playback Rewrite

[nielsvanvelzen]

🚀 What is The Playback Rewrite?

The goal of the rewrite is to create new playback code from scratch for video playback, audio playback and live tv. The new code should be easier to maintain and be prepared for a migration to Jellyfin 11. New features like SyncPlay can be added without much trouble. All code is Kotlin-first and uses the new SDK. It should be used by all our official Android apps for consistent behavior between (Android-) devices when playing files.

The rewrite is a very long project because I'm just one person volunteering to work on Jellyfin in my spare time. There is no hard deadline or specific version goals. This issue contains a task list of the bigger features/changes.

Status updates will be posted occasionally in the comments. Please read the bottom paragraph of this issue before commenting yourself.

🏆 Tasks

This list will change over time, some features might not be available on the initial release of the new playback code. When possible issues/pull requests will be linked for tracking purposes. There is no particular order for this list, new features might be added before restoring older features.

The amount of checked boxes does not indicate progress or completeness of the rewrite. Some changes required to make this happen have been running in production since version 0.15.0.

🏗️ Preparations

Various tasks that need to be completed to work on the rewrite.

• Create new Kotlin-first SDK
• Add new SDK
• Support multiple Gradle modules when building - Add (empty) playback module and CODEOWNERS #1072
• Add a way to enable the new playback module - Move useExternalPlayer and getPlaybackActivityClass functions to new PlaybackLauncher interface #1064

🎆 Proof of Concept

• Design architecture - Add playback rewrite backend and plugin systems #2401
• Add playlist support - everything is a playlist, even single item playback - Add playback rewrite backend and plugin systems #2401
• Add bare-bones player UI
• Add ExoPlayer backend - Add ExoPlayer backend to playback rewrite #2427
• Load video from server
• Report playback progress to server - Add PlaySessionService to playback rewrite #2428

🔄 Restoring features

• Support transcoding/remuxing
• Add remote control API - Add PlaySessionService to playback rewrite #2428
  • Support multiple items in PlayMessage POST /Sessions/{sessionId}/Playing only starts playing the first itemId parameter provided #2394
• Add intros
• Add chapter support
• Move "next up" screen to new playback code

🛠️ Integration (ATV)

• Use new playback module for audio (music, podcasts, audiobooks etc.) - Split MediaManager into interface and implementation #2453, Add option to enable playback rewrite for audio #2455
• Use new playback module for video (movies, series etc.)
• Use new playback module for Live TV
• Delete old playback code

🛠️ Integration (mobile)

• Make player UI touch compatible
• Release playback module to Maven Central (same versioning as the app)
• Replace native player with playback module
• Restore missing features

💥 Adding new features

• Add smart codec detection for directplay, remux and transcoding
• Add system preferences to overwrite codec support manually
• Add playback speed controls
• Add Media Session integration - Add MediaSessionPlugin to playback rewrite #2457
  • Use ImageProvider to proxy images
• Add Picture in Picture support
• Add gappless playback - Add ExoPlayer backend to playback rewrite #2427
• Add SyncPlay
• Add player stats (similar to YouTube's "stats for nerds") for debugging

Server blockers

Some features cannot be made or won't work great without changes to the server. This is a list of them.

• Add "speed" property to playback state API's jellyfin#8971
  Required to properly implement speed controls, it would also need changes to the dashboard of the web client
• Syncplay GroupUpdate class not properly added in OpenAPI spec jellyfin#6052
  Required to implement SyncPlay

❗ This issue

Please do not reply to this issue for any questions about the playback rewrite. The comment sections is meant for our contributors to give updates or discuss the implementation. For anything else, please create a new issue or discussion. We may delete your comment if we consider it off topic.

[mueslimak3r]

I was looking intro the code for trailer playback last night and I see that the object for external trailers just uses links videos (eg: a link to a youtube video). I'm not sure how feasible it would be, but it seems like a good design for a re-written player might be for it to just play video links. Then other classes would be queried for stuff specific to jellyfin like subtitle object classes, mediastream info, etc.

With the current player it's entirely designed around using jellyfin specific classes like BaseItemDto and its MediaStreams and SubtitleInfos, etc. Because of this it's not possible to make exoplayer play an arbitrary youtube link, even though exoplayer is capable of doing so.

[nielsvanvelzen]

it seems like a good design for a re-written player might be for it to just play video links.

The current implementation already does this, it never uses the Jellyfin API directly to make it easier to update if the ever API changes. This doesn't mean that YouTube video is supported though, the implementation will only allow video from the Jellyfin server. If we want to support YT a server plugin should deal with that, not the client.

Then other classes would be queried for stuff specific to jellyfin like subtitle object classes, mediastream info, etc.

My idea is to create some kind of "lookup" service where the playback code queries for data. This service would be implemented in the app code. I haven't worked on this part yet.

it's not possible to make exoplayer play an arbitrary youtube link, even though exoplayer is capable of doing so.

I don't think this is possible without using the proprietary YouTube API.

[nielsvanvelzen]

Let me give all of you another (quick) status update about the famous playback rewrite. I've been very busy the past few weeks (might even be months, didn't check) on a few key points:

1. 10.9 compatibility. This work happens in multiple places, from manually validating each API change in the release to updating the SDK and updating the app. This work is now done and the current 0.16.z releases of the app are fully compatible.
2. Refactoring the app to fully use the SDK. I recreated the layer that talks with the API a few years back, this is our "Kotlin SDK". However, the app still used our old "Java Apiclient" in most places. With all new additions in 10.7, 10.8 and 10.9 I could no longer not use the SDK as all newly added features could not be used. This migration took up most of my time and I can now say that almost all code is using the SDK! The first (real) 10.9 feature I added is support for normalization gain in music playback (Implement normalization gain for music tracks #3660)!
3. Rethinking some core components of the playback code. Most of this is now implemented and this should make it easier to add video playback related functionality.
4. Add video support. The initial changes are already merged, and development builds have an option to enable it. It's highly experimental but it does work, you can play video now.

While all of this is great news, I'm also working towards the next release of the app, 0.17.0. The current release started its beta cycle almost a year ago so it's time for a new one. Considering the new video playback is still very early it will not be ready for 0.17. The experimental option for it will also most likely be hidden for the release (still debating this part).

This was the quick status update for now, I didn't bother too much with the formatting or even re-reading it so forgive me for any mistakes. Hoping to give the next status update a lot quicker!

Happy watching!

If you appreciate my work, you can show your support with a donation through GitHub sponsors.