Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

perf: Optimization of ZIP streaming decompression #1057

Open
wants to merge 59 commits into
base: dev/exp
Choose a base branch
from
Open

perf: Optimization of ZIP streaming decompression #1057

wants to merge 59 commits into from

Conversation

iamscottxu
Copy link
Contributor

@iamscottxu iamscottxu commented Aug 29, 2024 •
edited
Loading

#1037 的优化，此分支可能不稳定，不要合并，如需合并合并#1037。

  • 将重试机制从单次压缩包内文件下载前移到单次数据流读取，当单个压缩包内的文件下载到一半失败时无需将其整个文件重新下载；✔️
  • 将域名解析和解析的IP地址可用性检测从单次压缩包内文件下载前移到整个压缩包下载阶段，并均衡地分配服务器节点（如果DNS服务器返回多个可用IP地址）；✔️
  • 升级时，先获取删除文件列表和差异文件，根据删除文件列表删除文件后，边下载，边解压，边合并。（部分实现）✔️；
  • 升级时，先获取差异文件列表，边下载，边解压，边合并。

@iamscottxu
Added support for ZIP compressed package streaming download
238d5a1 
Signed-off-by: scottxu <[email protected]>
@iamscottxu
Merge branch 'main' of https://github.com/Scighost/Starward
038243a 
Signed-off-by: scottxu <[email protected]>

# Conflicts:
#	src/Starward.Language/Lang.resx
#	src/Starward.Language/Lang.zh-CN.resx
@iamscottxu
Fixed Chinese translation error for streaming download options
248af32 
Signed-off-by: scottxu <[email protected]>
@iamscottxu
Adjusted the conditions for automatic deletion of ZIP central directo…
60e59c1 
…ry files.

Signed-off-by: scottxu <[email protected]>
@iamscottxu
Fixed Chinese translation error for streaming `Default Game Installat…
7eb4b97 
…ion Path Description`

Signed-off-by: scottxu <[email protected]>
@iamscottxu
remove an unnecessary judgment if(FileInfo.Exists)FileInfo.Delete;
ad36ad1 
Signed-off-by: scottxu <[email protected]>
@iamscottxu
fixed a bug where the ZipStreamDownload resource file could not be found
ddceb6b 
Signed-off-by: scottxu <[email protected]>
@iamscottxu
HttpClientHandler has been deprecated and replaced SocketsHttpHandler
ab656c8 
Signed-off-by: scottxu <[email protected]>
@iamscottxu
Merge branch 'SocketsHttpHandler' into streamdownload
ccf9d53
@iamscottxu
fixed a bug where streaming download tasks could run in parallel and …
eff47d8 
…cause unexpected behavior after a traditional download mode task was started

Signed-off-by: scottxu <[email protected]>
@iamscottxu
fixed a bug that could cause a download task to end prematurely when …
6f97c05 
…a streaming download was retried

Signed-off-by: scottxu <[email protected]>
@iamscottxu
fixed a bug where exceptions might not be caught when streaming downl…
be2caed 
…oads

Signed-off-by: scottxu <[email protected]>
@iamscottxu
fixed a bug where exceptions during streaming downloads would be inco…
ec1257d 
…rrectly caught as `TaskCanceledException`

Signed-off-by: scottxu <[email protected]>
@iamscottxu
fixed a bug where an error occurred when an HttpServerNotSupportedPar…
d161ba2 
…tialDownloadException was thrown

Signed-off-by: scottxu <[email protected]>
@iamscottxu
HTTP LastModified is no longer required to be present when streaming …
cbc0e68 
…ZIP downloads

Signed-off-by: scottxu <[email protected]>
@iamscottxu
resolve domain names in advance before streaming downloads
28b8c7d 
Signed-off-by: scottxu <[email protected]>
@iamscottxu
fixed a bug in ContentType validation when ZIP streaming download
5fb6513 
Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
fixed a bug where cancel exceptions were not caught when streaming do…
7fadea2 
…wnloads

Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
added Canceling status because streaming downloads take too long to c…
cb7be67 
…ancel

Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
disk space pre-allocation and read/write cache optimization for strea…
81770d3 
…ming download

Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
optimized the thread scheduling logic of streaming download mode, and…
cb8247a 
… fixed the bug in which the progress was displayed incorrectly after the download was paused and restarted

Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
fixed a bug where central directory files were not closed after task …
50ea759 
…completion or cancellation when streaming downloads

Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
optimized the code for streaming download
4784d5e 
Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
removed redundant code for streaming downloads
d97bc30 
Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
adjusted the code format
e1cc38a 
Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
optimized the code of streaming download speed limiter
d2d7196 
Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
Merge branch 'main' into streamdownload
1def457 
Signed-off-by: OsakaRuma <[email protected]>

# Conflicts:
#	src/Starward.Language/Lang.es-ES.resx
#	src/Starward.Language/Lang.it-IT.resx
#	src/Starward.Language/Lang.ja-JP.resx
#	src/Starward.Language/Lang.ko-KR.resx
#	src/Starward.Language/Lang.ru-RU.resx
#	src/Starward.Language/Lang.th-TH.resx
@iamscottxu
added "Cancelling" message
daa4bbd 
Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
globally limit the maximum number of connections per server and optim…
f33702f 
…ize streaming download speed

Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
Merge branch 'main' into streamdownload
101e7b1
@iamscottxu iamscottxu mentioned this pull request Aug 29, 2024
Open
@iamscottxu
submit missing file
aa03f15 
Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
removed redundant code
61eeaa0 
Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
optimized the code of stream download
23aa9fd 
Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
Merge branch 'streamdownload' into streamdownload_optimization
81c3f1d 
Signed-off-by: OsakaRuma <[email protected]>

# Conflicts:
#	src/Starward.Core.ZipStreamDownload/FastZipStreamDownloadTask.partial.cs
@iamscottxu
Merge branch 'main' into streamdownload
cf4ce68
@iamscottxu
Merge branch 'streamdownload' into streamdownload_optimization
3e6b2e6
@iamscottxu
optimized the speed of DNS resolution during streaming downloads.
d7bce53 
Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
optimized the number of threads that need to be started when there ar…
6a7d922 
…e few files in the compressed package during streaming download

Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
Merge branch 'main' into streamdownload
dc09431
@iamscottxu
fixed the bug that method CurrentTaskFinished was executed multiple…
57da19b 
… times by multiple threads (Scighost#1083) (Scighost#1082)

Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
handling exception, deleted voice code
e64c24c 
Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
Merge branch 'bug_fix' into streamdownload
640f41e 
Signed-off-by: OsakaRuma <[email protected]>

# Conflicts:
#	src/Starward/Services/Download/InstallGameService.cs
@iamscottxu
fixed typo of variable name
631be21 
Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
fixed typo in variable name
cc0a99d 
Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
Merge branch 'bug_fix' into streamdownload
b9f2ebd 
Signed-off-by: OsakaRuma <[email protected]>

# Conflicts:
#	src/Starward/Services/Download/InstallGameService.cs
@iamscottxu
fixed a bug where only the first step of the task could be run if the…
9ad0944 
…re were multiple steps of the task when installing the game

Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
Merge branch 'bug_fix' into streamdownload
671d245
@iamscottxu
Merge branch 'main' into streamdownload
a5b1980
@iamscottxu
fixed bug where task would not continue after streaming download comp…
7b8df5c 
…leted

Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
Merge branch 'main' into streamdownload
2ea68fe 
Signed-off-by: OsakaRuma <[email protected]>

# Conflicts:
#	src/Starward.Language/Lang.de-DE.resx
#	src/Starward.Language/Lang.es-ES.resx
#	src/Starward.Language/Lang.it-IT.resx
#	src/Starward.Language/Lang.ja-JP.resx
#	src/Starward.Language/Lang.ko-KR.resx
#	src/Starward.Language/Lang.ru-RU.resx
#	src/Starward.Language/Lang.th-TH.resx
#	src/Starward.Language/Lang.vi-VN.resx
#	src/Starward/Services/Download/InstallGameManager.cs
#	src/Starward/Services/Download/InstallGameService.cs
@iamscottxu
Added judgment to skip relevant code when streaming download without …
b72c87c 
…using speed limiter.

Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
because the "pausing" state was added, some codes were restored
1656da6 
Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
fixed a bug where speed limiting didn't work when streaming downloads
0674c97 
Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu
Merge branch 'dev/exp' into streamdownload
533129e 
Signed-off-by: OsakaRuma <[email protected]>

# Conflicts:
#	src/Starward/Services/Download/InstallGameService.cs
@iamscottxu iamscottxu force-pushed the streamdownload_optimization branch from a4cf2c7 to 533129e Compare October 17, 2024 12:57
@iamscottxu
Optimized the update logic during streaming download. First download …
e698cb1 
…the list of files to be deleted, delete the files in the list, then download the files to a temporary folder, and finally move the files.

Signed-off-by: OsakaRuma <[email protected]>
@iamscottxu iamscottxu force-pushed the streamdownload_optimization branch from 1f6ff36 to e698cb1 Compare October 19, 2024 08:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@iamscottxu