Zim-tools version 3.3.0 splitted zim files giving Corrupted block detected error when there is a 0 size extra ZIM chunk.

[MohitMaliFtechiz]

Describe the bug

• For full details please see Fixed: Error on opening a split zim file on external storage. #3803.
• But the main issue that we are facing is the Corrupted block detected error when getting the getArticleCount() and getMediaCount() with zim-tools's splitted zim files(When there is a 0size size extra part) However, we can read the zim file content. If we delete that 0 size extra part from our storage then there is no error and it starts giving the value from these methods.
• We have written the test cases in Fixed: Error on opening a split zim file on external storage. #3803 for testing this scenario where the 0 size part is present in the storage, but the test is passing because, with that zim file, there is no error in getting the getArticleCount() and getMediaCount(). Also, if we split any zim file with this test case and tries to load it in our application then these methods retuns the value.
• One more issue we have detected of this type of error, when we tries to save the bookmark then the application crashes with these logs, when we tries to use the update(Archive archive) method of the Book class.

 Process: org.kiwix.kiwixmobile, PID: 17268
                                                                                                    java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
                                                                                                    	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:562)
                                                                                                    	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:878)
                                                                                                    Caused by: java.lang.reflect.InvocationTargetException
                                                                                                    	at java.lang.reflect.Method.invoke(Native Method)
                                                                                                    	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:552)
                                                                                                    	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:878) 
                                                                                                    Caused by: java.lang.Exception: Corrupted block detected
                                                                                                    	at org.kiwix.libkiwix.Book.update(Native Method)
                                                                                                    	at org.kiwix.kiwixmobile.core.main.CoreReaderFragment.toggleBookmark(CoreReaderFragment.kt:1758)
                                                                                                    	at org.kiwix.kiwixmobile.core.main.CoreReaderFragment_ViewBinding$6.doClick(CoreReaderFragment_ViewBinding.java:121)
                                                                                                    	at butterknife.internal.DebouncingOnClickListener.onClick(DebouncingOnClickListener.java:26)
                                                                                                    	at android.view.View.performClick(View.java:7729)
                                                                                                    	at android.view.View.performClickInternal(View.java:7706)
                                                                                                    	at android.view.View.-$$Nest$mperformClickInternal(Unknown Source:0)
                                                                                                    	at android.view.View$PerformClick.run(View.java:30484)
                                                                                                    	at android.os.Handler.handleCallback(Handler.java:959)
                                                                                                    	at android.os.Handler.dispatchMessage(Handler.java:100)
                                                                                                    	at android.os.Looper.loopOnce(Looper.java:232)
                                                                                                    	at android.os.Looper.loop(Looper.java:317)
                                                                                                    	at android.app.ActivityThread.main(ActivityThread.java:8501)

Expected behavior
It should return the value from the getArticleCount() and getMediaCount() methods with zim-tools splitted zim file as well.

[mgautierfr]

This use case is really strange.

• zimsplit creates extra empty file => While it is a bug in zimsplit it should have no impact. This extra zim file "foo.zimafcould have simply being created withtouch foo.zimaf`
• Java unittesting is passing => As expected, libzim is able to read split zim file (even with extra empty file). However, it is run on java linux, not android.
• Trying to open split zim file with extra empty => getArticleCount fails.
• Corrupted block detected is probably raise by zstd about corrupted data.

So I feel like either:

• Having a extra empty part make android itself behaves differently
• This is not related to extra empty part but simply corrupted file.

Can you run sha256sum wikipedia_ar_all_nopic_2024-03.zim* on android itself and be sure sha256 are the same than for file in linux ?

[MohitMaliFtechiz]

Can you run sha256sum wikipedia_ar_all_nopic_2024-03.zim* on android itself and be sure sha256 are the same than for file in linux ?

@mgautierfr The sha256sum is the same on both android and Linux.

Linux sha256sum

sha256sum wikipedia_ar_all_nopic_2024-03.zim*

f744022ed8208a50a1ce191cd5af946debc4012d36c41f7c338e8ae904b86303  wikipedia_ar_all_nopic_2024-03.zim
8fd91f8f328ca1a5c77129fd7e4d7435f663fa632e9c0078c3d38d0bccdea366  wikipedia_ar_all_nopic_2024-03.zimaa
eb1892b3bfee9876626a12bff57d90c6663ab143e74d88e8ebddc6084f167e82  wikipedia_ar_all_nopic_2024-03.zimab
33729314abe6b640d918770cfd9146a13b49eac537a52b00e6568e3a358d1ce5  wikipedia_ar_all_nopic_2024-03.zimac
b17e5f6fbbc140486cab4c19adc8970364796acdbd06a8a9439e501d02cfb040  wikipedia_ar_all_nopic_2024-03.zimad
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  wikipedia_ar_all_nopic_2024-03.zimae

On Android

sha256sum wikipedia_ar_all_nopic_2024-03.zim*

8fd91f8f328ca1a5c77129fd7e4d7435f663fa632e9c0078c3d38d0bccdea366  wikipedia_ar_all_nopic_2024-03.zimaa
eb1892b3bfee9876626a12bff57d90c6663ab143e74d88e8ebddc6084f167e82  wikipedia_ar_all_nopic_2024-03.zimab
33729314abe6b640d918770cfd9146a13b49eac537a52b00e6568e3a358d1ce5  wikipedia_ar_all_nopic_2024-03.zimac
b17e5f6fbbc140486cab4c19adc8970364796acdbd06a8a9439e501d02cfb040  wikipedia_ar_all_nopic_2024-03.zimad
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  wikipedia_ar_all_nopic_2024-03.zimae

Having a extra empty part make android itself behaves differently.

Might be possible, if the extra empty part is created by other systems (e.g. here kiwix-tools), and extra part is working in test case because it is created by the android itself. But we need to ensure that, this is the case here.

[kelson42]

@MohitMaliFtechiz As far as I know title is wrong, it is having problem only if you have a 0 sized surnumerous ZIM chunk (which is not always the case)? You confirm?

[MohitMaliFtechiz]

@kelson42 You are right. The problem only happens when there is a 0 size extra zim file.

[MohitMaliFtechiz]

Can you try to run the application under strace (https://source.android.com/docs/core/tests/debug/strace) please ? #3803 (comment)

@mgautierfr Here is the debug stack trace of the application, when this error happens.

DebugStaceRTraceOfCorruptedBlockDetected.txt

[kelson42]

@mgautierfr Any news on this?

[MohitMaliFtechiz]

@mgautierfr With the new java-libkiwix now it is giving full logs instead of the Corrupted block detected error. With an extra empty part now the application started crashing when rendering the content, and without the extra part, all things are working fine. According to these logs, it is failing to read the .zimd part which is the last part for the zim file and the extra empty part is zimae.

FATAL EXCEPTION: DefaultDispatcher-worker-19
                                                                                                    Process: org.kiwix.kiwixmobile, PID: 11803
                                                                                                    java.lang.Exception: Cannot read chars.
                                                                                                     - File part is /storage/37B9-D7D4/Documents/wikipedia_ar_all_nopic_2024-03.zimad
                                                                                                     - File part size is 3187373325
                                                                                                     - File part range is 6075257308-9262630633
                                                                                                     - size_to_get is 1024
                                                                                                     - total size is 1024
                                                                                                     - Reading offset at 6075257161
                                                                                                     - logical local offset is 18446744073709551469
                                                                                                     - physical local offset is 18446744073709551469
                                                                                                     - error is Cannot read file
                                                                                                    : Invalid argument
                                                                                                    	at org.kiwix.libzim.Item.getData(Native Method)
                                                                                                    	at org.kiwix.kiwixmobile.core.reader.ZimFileReader$streamZimContentToPipe$1.invokeSuspend(ZimFileReader.kt:364)
                                                                                                    	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
                                                                                                    	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
                                                                                                    	at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)
                                                                                                    	at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:103)
                                                                                                    	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
                                                                                                    	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
                                                                                                    	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
                                                                                                    	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)

[mgautierfr]

@MohitMaliFtechiz Thanks for the trace. I may have found something.
Can you test with the libzim(and libkiwix!) libraries from https://tmp.kiwix.org/ci/dev_preview/fix_locate/ please ?

[MohitMaliFtechiz]

@mgautierfr Thanks for providing the binaries, I can confirm that this issue is fixed by this solution.

[MohitMaliFtechiz]

This issue is not fully fixed since we have not integrated the latest binaries in the application. We should keep this ticket until the new java-libkiwix is not integrated into the project with openzim/libzim#903 this fix.

[kelson42]

@MohitMaliFtechiz The issue is fixed from libzim perspective, if you want to keep track at Android level, I recomment to open an issue there. Otherwise, I agree we should release libzim soon and use it in next Kiwix Android release.

[kelson42]

@MohitMaliFtechiz Sorry, did not realised it was in kiwix-android.