-
-
Notifications
You must be signed in to change notification settings - Fork 443
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fixed: Error on opening a split zim file on external storage. #3803
Conversation
@mgautierfr will release new version of |
@MohitMaliFtechiz I have implemented libzim 9.2.1 in java-libkiwix. Does that solve the problem? |
d4d6d89
to
ac7e0f6
Compare
@kelson42 Thanks for integrating the libzim Zim chunks:
This error was due to there is a
java.lang.Exception: Corrupted block detected
2024-05-03 12:11:37.364 14087-14279 System.err pid-14087 W at org.kiwix.libzim.Archive.getArticleCount(Native Method)
2024-05-03 12:11:37.365 14087-14279 System.err pid-14087 W at org.kiwix.kiwixmobile.core.reader.ZimFileReader.getArticleCount(ZimFileReader.kt:169)
2024-05-03 12:11:37.365 14087-14279 System.err pid-14087 W at org.kiwix.kiwixmobile.core.reader.ZimFileReader.toBook(ZimFileReader.kt:324)
2024-05-03 12:11:37.365 14087-14279 System.err pid-14087 W at org.kiwix.kiwixmobile.core.zim_manager.fileselect_view.adapter.BooksOnDiskListItem$BookOnDisk.<init>(BooksOnDiskListItem.kt:67)
2024-05-03 12:11:37.365 14087-14279 System.err pid-14087 W at org.kiwix.kiwixmobile.core.StorageObserver$convertToBookOnDisk$1.invokeSuspend(StorageObserver.kt:59)
2024-05-03 12:11:37.365 14087-14279 System.err pid-14087 W at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
2024-05-03 12:11:37.365 14087-14279 System.err pid-14087 W at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
2024-05-03 12:11:37.366 14087-14279 System.err pid-14087 W at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:279)
2024-05-03 12:11:37.366 14087-14279 System.err pid-14087 W at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85)
2024-05-03 12:11:37.367 14087-14279 System.err pid-14087 W at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
2024-05-03 12:11:37.367 14087-14279 System.err pid-14087 W at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source:1)
2024-05-03 12:11:37.367 14087-14279 System.err pid-14087 W at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
2024-05-03 12:11:37.367 14087-14279 System.err pid-14087 W at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source:1)
2024-05-03 12:11:37.367 14087-14279 System.err pid-14087 W at org.kiwix.kiwixmobile.core.StorageObserver.convertToBookOnDisk(StorageObserver.kt:57)
2024-05-03 12:11:37.367 14087-14279 System.err pid-14087 W at org.kiwix.kiwixmobile.core.StorageObserver.getBooksOnFileSystem$lambda-0(StorageObserver.kt:45)
2024-05-03 12:11:37.367 14087-14279 System.err pid-14087 W at org.kiwix.kiwixmobile.core.StorageObserver.$r8$lambda$J3Q4dPQVOFEOgCBh9ahA4Ca-Zak(Unknown Source:0)
2024-05-03 12:11:37.368 14087-14279 System.err pid-14087 W at org.kiwix.kiwixmobile.core.StorageObserver$$ExternalSyntheticLambda1.apply(Unknown Source:4)
2024-05-03 12:11:37.368 14087-14279 System.err pid-14087 W at io.reactivex.internal.operators.flowable.FlowableMap$MapConditionalSubscriber.onNext(FlowableMap.java:106)
2024-05-03 12:11:37.368 14087-14279 System.err pid-14087 W at io.reactivex.subscribers.SerializedSubscriber.onNext(SerializedSubscriber.java:100)
2024-05-03 12:11:37.368 14087-14279 System.err pid-14087 W at io.reactivex.internal.operators.flowable.FlowableWithLatestFrom$WithLatestFromSubscriber.tryOnNext(FlowableWithLatestFrom.java:94)
2024-05-03 12:11:37.368 14087-14279 System.err pid-14087 W at io.reactivex.internal.operators.flowable.FlowableWithLatestFrom$WithLatestFromSubscriber.onNext(FlowableWithLatestFrom.java:76)
2024-05-03 12:11:37.368 14087-14279 System.err pid-14087 W at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.onNext(FlowableSubscribeOn.java:97)
2024-05-03 12:11:37.368 14087-14279 System.err pid-14087 W at io.reactivex.internal.operators.flowable.FlowableCombineLatest$CombineLatestCoordinator.drainAsync(FlowableCombineLatest.java:374)
2024-05-03 12:11:37.369 14087-14279 System.err pid-14087 W at io.reactivex.internal.operators.flowable.FlowableCombineLatest$CombineLatestCoordinator.drain(FlowableCombineLatest.java:406)
2024-05-03 12:11:37.369 14087-14279 System.err pid-14087 W at io.reactivex.internal.operators.flowable.FlowableCombineLatest$CombineLatestCoordinator.innerValue(FlowableCombineLatest.java:250)
2024-05-03 12:11:37.369 14087-14279 System.err pid-14087 W at io.reactivex.internal.operators.flowable.FlowableCombineLatest$CombineLatestInnerSubscriber.onNext(FlowableCombineLatest.java:521)
2024-05-03 12:11:37.369 14087-14279 System.err pid-14087 W at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.onNext(FlowableSubscribeOn.java:97)
2024-05-03 12:11:37.369 14087-14279 System.err pid-14087 W at io.reactivex.internal.subscriptions.DeferredScalarSubscription.complete(DeferredScalarSubscription.java:132)
2024-05-03 12:11:37.370 14087-14279 System.err pid-14087 W at io.reactivex.internal.operators.flowable.FlowableFromCallable.subscribeActual(FlowableFromCallable.java:50)
2024-05-03 12:11:37.370 14087-14279 System.err pid-14087 W at io.reactivex.Flowable.subscribe(Flowable.java:14935)
2024-05-03 12:11:37.370 14087-14279 System.err pid-14087 W at io.reactivex.Flowable.subscribe(Flowable.java:14882)
2024-05-03 12:11:37.370 14087-14279 System.err pid-14087 W at io.reactivex.internal.operators.flowable.FlowableSubscribeOn$SubscribeOnSubscriber.run(FlowableSubscribeOn.java:82)
2024-05-03 12:11:37.370 14087-14279 System.err pid-14087 W at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)
2024-05-03 12:11:37.370 14087-14279 System.err pid-14087 W at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)
2024-05-03 12:11:37.371 14087-14279 System.err pid-14087 W at java.util.concurrent.FutureTask.run(FutureTask.java:264)
2024-05-03 12:11:37.371 14087-14279 System.err pid-14087 W at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:307) Apart from this, the test case of reading splitted zim is also passing locally. |
I just saw that this issue with zim-tool has now been fixed openzim/zim-tools#401 |
@MohitMaliFtechiz The bug of creating surnumerous empty chunk has been fixed, see openzim/zim-tools#401. But, the fact that the libzim does not deliver properly if there is such an empty chunk is a bug. Please open an issue for that at openzim/libzim. Regarding the workaround, you have developed to handle this libzim bug. Please keep it and put clear comment it should be removed once the libzim bug fixed. Open a dedicated issue to do that in next release. ... and the put this PR in review so we can mergw. |
@kelson42 Okay.
I have just captured the error thrown by the libzim to not crash the rendering process ac7e0f6 there is no other code changes I have done, okay I am putting the comments for this.
Please merge this PR after the latest java-libkiwix is integrated into the project. |
Agree with @kelson42, libzim should be able to read split file even with a empty part at the end (or in the middle). I personally can open such split file without any issue (not using kiwix-android)
|
app/src/androidTest/java/org/kiwix/kiwixmobile/reader/ZimFileReaderWithSplittedZimFileTest.kt
Outdated
Show resolved
Hide resolved
@mgautierfr The error was throwing when I was trying to get the mediaCount and articleCount which are the
If it is not a libzim error, and you can open the zim file of this type. Then it is probably the error of |
This may be a zstd error (facebook/zstd#2852). If it is the case, it means that we are passing corrupted data to be decompressed. It would be surprising that libzim pass a wrong value (assuming file is actually not corrupted), especially that a zim split made with However, your test split zim file creator is (potentially) splitting in the middle of cluster and I have identified a race condition. Are you sure the error is with a extra empty part or it is with the testing split file ? |
@mgautierfr The problem is with the empty part, the test case is passing successfully without any error with the latest bindings of java-libkiwix, The test case opens the zim file and reads the content, there is no error in the test case. However,
We can improve this test part to avoid this type of situation. |
Ok, let's me try to resume the issue:
Are we agree on this ? |
@mgautierfr Let me correct few things then you can resume the issue.
I am opening the
Libzim
No, in version 9.2.1 it working and in version 9.2.0 it is failing to open the zim file. |
@MohitMaliFtechiz I have released java-libkiwix 2.1.1, configured in kiwix-android and rebase this PR… but the CI is still red! |
Thank @MohitMaliFtechiz for the explanation. |
@mgautierfr I am using zim-tools 3.1.0
libzim 7.2.0
+ libzstd 1.4.8
+ liblzma 5.2.5
+ libxapian 1.4.18
+ libicu 70.1.0 |
@MohitMaliFtechiz Please redo the operation wirh latest version of zim-tools, your version is already a bit old... maybe there a bug arounf this |
@mgautierfr Here are the all details from beginning.
acerpc@acerpc-Aspire-Lite-AL15-51:~/Documents$ zimsplit wikipedia_ar_all_nopic_2024-03.zim
opening new file wikipedia_ar_all_nopic_2024-03.zimaa
opening new file wikipedia_ar_all_nopic_2024-03.zimab
opening new file wikipedia_ar_all_nopic_2024-03.zimac
opening new file wikipedia_ar_all_nopic_2024-03.zimad
WARNING: Part wikipedia_ar_all_nopic_2024-03.zimad is bigger that max part size. (3187373325>2147483648)
opening new file wikipedia_ar_all_nopic_2024-03.zimae
acerpc@acerpc-Aspire-Lite-AL15-51:~/Documents$ sha256sum wikipedia_ar_all_nopic_2024-03.zim
f744022ed8208a50a1ce191cd5af946debc4012d36c41f7c338e8ae904b86303 wikipedia_ar_all_nopic_2024-03.zim
acerpc@acerpc-Aspire-Lite-AL15-51:~/Documents$ sha256sum wikipedia_ar_all_nopic_2024-03.zimaa
8fd91f8f328ca1a5c77129fd7e4d7435f663fa632e9c0078c3d38d0bccdea366 wikipedia_ar_all_nopic_2024-03.zimaa
acerpc@acerpc-Aspire-Lite-AL15-51:~/Documents$ sha256sum wikipedia_ar_all_nopic_2024-03.zimab
eb1892b3bfee9876626a12bff57d90c6663ab143e74d88e8ebddc6084f167e82 wikipedia_ar_all_nopic_2024-03.zimab
acerpc@acerpc-Aspire-Lite-AL15-51:~/Documents$ sha256sum wikipedia_ar_all_nopic_2024-03.zimac
33729314abe6b640d918770cfd9146a13b49eac537a52b00e6568e3a358d1ce5 wikipedia_ar_all_nopic_2024-03.zimac
acerpc@acerpc-Aspire-Lite-AL15-51:~/Documents$ sha256sum wikipedia_ar_all_nopic_2024-03.zimad
b17e5f6fbbc140486cab4c19adc8970364796acdbd06a8a9439e501d02cfb040 wikipedia_ar_all_nopic_2024-03.zimad
acerpc@acerpc-Aspire-Lite-AL15-51:~/Documents$ sha256sum wikipedia_ar_all_nopic_2024-03.zimae
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 wikipedia_ar_all_nopic_2024-03.zimae
acerpc@acerpc-Aspire-Lite-AL15-51:~/Documents$ md5sum acerpc@acerpc-Aspire-Lite-AL15-51:~/Documents$ md5sum wikipedia_ar_all_nopic_2024-03.zim
66bb7f6ca0008b304231040a53b71ab4 wikipedia_ar_all_nopic_2024-03.zim
acerpc@acerpc-Aspire-Lite-AL15-51:~/Documents$ md5sum wikipedia_ar_all_nopic_2024-03.zimaa
981bff8a335e0e13ac2a9fd1af25df43 wikipedia_ar_all_nopic_2024-03.zimaa
acerpc@acerpc-Aspire-Lite-AL15-51:~/Documents$ md5sum wikipedia_ar_all_nopic_2024-03.zimab
4ff359e0dbdd1af19ce39353f198e31d wikipedia_ar_all_nopic_2024-03.zimab
acerpc@acerpc-Aspire-Lite-AL15-51:~/Documents$ md5sum wikipedia_ar_all_nopic_2024-03.zimac
6504996d96eacc34e9c504c1ca4d0acb wikipedia_ar_all_nopic_2024-03.zimac
acerpc@acerpc-Aspire-Lite-AL15-51:~/Documents$ md5sum wikipedia_ar_all_nopic_2024-03.zimad
a909742cd7df2e8aa7dffe9f9d74dd3f wikipedia_ar_all_nopic_2024-03.zimad
acerpc@acerpc-Aspire-Lite-AL15-51:~/Documents$ md5sum wikipedia_ar_all_nopic_2024-03.zimae
d41d8cd98f00b204e9800998ecf8427e wikipedia_ar_all_nopic_2024-03.zimae cksum acerpc@acerpc-Aspire-Lite-AL15-51:~/Documents$ cksum wikipedia_ar_all_nopic_2024-03.zim
3675923895 9262630633 wikipedia_ar_all_nopic_2024-03.zim
acerpc@acerpc-Aspire-Lite-AL15-51:~/Documents$ cksum wikipedia_ar_all_nopic_2024-03.zimaa
1205357161 2147459163 wikipedia_ar_all_nopic_2024-03.zimaa
acerpc@acerpc-Aspire-Lite-AL15-51:~/Documents$ cksum wikipedia_ar_all_nopic_2024-03.zimab
3126437707 2147376980 wikipedia_ar_all_nopic_2024-03.zimab
acerpc@acerpc-Aspire-Lite-AL15-51:~/Documents$ cksum wikipedia_ar_all_nopic_2024-03.zimac
3803321215 1780421165 wikipedia_ar_all_nopic_2024-03.zimac
acerpc@acerpc-Aspire-Lite-AL15-51:~/Documents$ cksum wikipedia_ar_all_nopic_2024-03.zimad
2606185103 3187373325 wikipedia_ar_all_nopic_2024-03.zimad
acerpc@acerpc-Aspire-Lite-AL15-51:~/Documents$ cksum wikipedia_ar_all_nopic_2024-03.zimae
4294967295 0 wikipedia_ar_all_nopic_2024-03.zimae
|
@kelson42 The latest version of The latest version of |
You can get the last version of zim-tools here : https://download.openzim.org/release/zim-tools/ But anyway, the checksum of you file are same as mine, so it is not a problem of zim-tools version. |
You can get the last version of zim-tools here : https://download.openzim.org/release/zim-tools/ But anyway, the checksum of you file are same as mine, so it is not a problem of zim-tools version. I have add a simple (java) test opening split files (with an extra 0 part) on my computer and it works. So it seems it is a android only issue. |
Can you try to run the application under strace (https://source.android.com/docs/core/tests/debug/strace) please ? |
@mgautierfr Thanks. |
97029bf
to
5069816
Compare
@mgautierfr Thanks for your testing, I have also added the test case in Android with 0 bytes zim file, and my test case is passed without any error as well, it is returning the |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3803 +/- ##
============================================
- Coverage 53.30% 53.24% -0.06%
+ Complexity 1319 1313 -6
============================================
Files 292 292
Lines 11067 11060 -7
Branches 1471 1471
============================================
- Hits 5899 5889 -10
+ Misses 4195 4194 -1
- Partials 973 977 +4 ☔ View full report in Codecov by Sentry. |
@MohitMaliFtechiz The handling of this edge case is dlowing down the release. Please create dedicated issue to continue the work, so we can merge this one. |
@kelson42 I have created a separate ticket for this edge case #3827. |
@MohitMaliFtechiz Please don't call a PR by the error... but by the solution! Here this PR has an issue title! |
* Added test cases for testing that the ZimFileReader can open the splitted zim file or not.
…icleCount` because it was throwing `java.lang.Exception` with splitted zim file when the zim file is corrupted. If there is any zim file is corrupted and libzim throws this exception then our UI will not show the other zim file on `LocalLibraryScreen` due to the getting zim files process being crashed.
…unt this will necessary to catch the all exception thrown by libzim to prevent the aborting of the rendering process of other zim files.
Fixes #3605
articleCount
because it was throwingjava.lang.Exception
with splitted zim file when the zim file is corrupted. If there is any zim file is corrupted and libzim throws this exception then our UI will not show the other zim file onLocalLibraryScreen
due to the getting zim files process being crashed.