PKG-8 not reported

[maria-messerschmidt]

I have attempted to trigger the error message for PKG-8 which states that encrypted files should be stored and not compressed. I have not been able to trigger this error message, and my attempts have brought up a few issues,

This issue is raised in relation to PKG-8, but of course we also have POL_1 which should fail on ALL encrypted files. This is not always triggered either, so I am raising that as a separate issue.

As far as I can tell, the trigger for PKG-8 is that a file in the zip archive should be registered in the manifest, be encrypted and not be stored. I tried a couple of scenarios to trigger this, and they produced a variety of results, but no PKG-8.

Scenario 1: generating APP-2 error
I encrypted the styles.xml file which already existed in the zip archive and is registered in the manifest.

E001a.ods

This should have triggered PKG-8 since the file is in the manifest as:

<manifest:file-entry manifest:full-path="styles.xml" manifest:media-type="text/xml"/>

And the attributes of the zip entry include encryption and not stored:

Path = styles.xml
Folder = -
Size = 20310
Packed Size = 3190
Modified = 2024-04-30 08:01:14.0000000
Created =
Accessed =
Attributes =
Encrypted = +
Comment =
CRC = 7BBE0528
Method = pkAES-256 Deflate
Characteristics = NTFS StrongCrypto : Encrypt StrongCrypto UTF8
Host OS = FAT
Version = 51
Volume Index = 0
Offset = 19770

The validator doesn't actually validate the file though based on the log:

C:\odf\odf-validator-main>odf-validator.bat -p "filer\testfiler\E001a.ods"
APP-1: [INFO] Validating filer\testfiler\E001a.ods.
APP-2: [ERROR] Unsupported feature encryption used in entry styles.xml

I can see a couple of issues with this error message:

1. It is not PKG-8 or POL_1.
2. APP-2 should be “Supplied path %s IS NOT an existing file", so it is unclear how we are getting this result here.
3. Since this is an APP-x message, what will happen when this is validated using API?
4. Above is the result for running the validator with profile. Without profile, the log looks like this:

C:\odf\odf-validator-main>odf-validator.bat "filer\testfiler\E001a.ods"
APP-1: [INFO] Validating filer\testfiler\E001a.ods.
org.apache.commons.compress.archivers.zip.UnsupportedZipFeatureException: Unsupported feature encryption used in entry styles.xml
at org.apache.commons.compress.archivers.zip.ZipUtil.checkRequestedFeatures(ZipUtil.java:147)
at org.apache.commons.compress.archivers.zip.ZipFile.getInputStream(ZipFile.java:953)
at org.openpreservation.format.zip.ZipFileProcessor.getEntryInputStream(ZipFileProcessor.java:116)
at org.openpreservation.odf.pkg.PackageParserImpl.processEntry(PackageParserImpl.java:129)
at org.openpreservation.odf.pkg.PackageParserImpl.processZipEntries(PackageParserImpl.java:109)
at org.openpreservation.odf.pkg.PackageParserImpl.parsePackage(PackageParserImpl.java:100)
at org.openpreservation.odf.pkg.PackageParserImpl.parsePackage(PackageParserImpl.java:70)
at org.openpreservation.odf.validation.ValidatingParserImpl.parsePackage(ValidatingParserImpl.java:74)
at org.openpreservation.odf.validation.Validator.validatePackage(Validator.java:107)
at org.openpreservation.odf.validation.Validator.validate(Validator.java:83)
at org.openpreservation.odf.apps.CliValidator.validatePath(CliValidator.java:68)
at org.openpreservation.odf.apps.CliValidator.call(CliValidator.java:60)
at org.openpreservation.odf.apps.CliValidator.call(CliValidator.java:35)
at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
at picocli.CommandLine.execute(CommandLine.java:2170)
at org.openpreservation.odf.apps.CliValidator.main(CliValidator.java:87)

Scenario 2: not generating error message related to encryption
Here, I encrypted the thumbnail instead. Again, this was already in the ODS-package and registered in the manifest.
E001e.ods

Again, this should have triggered PKG-8 since the file is in the manifest:

<manifest:file-entry manifest:full-path="Thumbnails/thumbnail.png" manifest:media-type="image/png"/>

And the attributes of the zip entry include encryption and not stored:

Path = Thumbnails\thumbnail.png
Folder = -
Size = 15196
Packed Size = 15110
Modified = 2024-04-30 08:01:14.0000000
Created =
Accessed =
Attributes = A
Encrypted = +
Comment =
CRC = 8B1062C5
Method = pkAES-256 Deflate
Characteristics = NTFS StrongCrypto : Encrypt StrongCrypto
Host OS = FAT
Version = 51
Volume Index = 0
Offset = 747

Now the validator is happy to start validating the file and produces this log:

C:\odf\odf-validator-main>odf-validator.bat -p "filer\testfiler\E001e.ods"
APP-1: [INFO] Validating filer\testfiler\E001e.ods.
APP-5: [INFO] DNA ODF Spreadsheets Preservation Specification Profile report for filer\testfiler\E001e.ods.
MIM-1: mimetype [ERROR] The "mimetype" file SHALL be the first file of the zip file.
DOC-3: mimetype [INFO] OpenDocument MIMETYPE application/vnd.oasis.opendocument.spreadsheet detected
POL_2: E001e.ods [ERROR] Standard Compliance | The file MUST comply with the standard "OASIS Open Document Format for Office Applications (OpenDocument) v1.3".
NOT VALID, 2 errors, 0 warnings and 1 info messages.

Again, some issues with this result:

1. Not PKG-8 or POL_1
2. Although there are errors, they are not related to encryption at all, i.e. the file is allowed through. The error is just because of the mimetype file position, and this can be corrected.
3. Why is there a discrepancy in how this is treated compared to scenario 1? They are two different files, but the encryption and compression is the same.
4. In this case, the error messages (except of course for POL_2) are the same when validator is run without the profile.

Scenario 3: Adding a new random file to the zip archive
So in the first two scenarios, I used files that were already present in the zip archive. For this one, I tried adding the random extra file random.png (which is encrypted) to the root of the zip archive. I didn't add it to the manifest.

E001f.ods

This generated the following:
C:\odf\odf-validator-main>odf-validator.bat -p "filer\testfiler\E001f.ods"
APP-1: [INFO] Validating filer\testfiler\E001f.ods.
APP-5: [INFO] DNA ODF Spreadsheets Preservation Specification Profile report for filer\testfiler\E001f.ods.
MIM-1: mimetype [ERROR] The "mimetype" file SHALL be the first file of the zip file.
DOC-3: mimetype [INFO] OpenDocument MIMETYPE application/vnd.oasis.opendocument.spreadsheet detected
MAN-1: random.png [ERROR] The manifest SHALL contain an entry for every file in the package, zip entry random.png has no corresponding manifest file entry.
POL_2: E001f.ods [ERROR] Standard Compliance | The file MUST comply with the standard "OASIS Open Document Format for Office Applications (OpenDocument) v1.3".
NOT VALID, 3 errors, 0 warnings and 1 info messages.

This will produce an error message since the file is not in the manifest, but no errors related to encryption.

Scenario 4: Adding random.png to the manifest
So I added random.png to the manifest:
<manifest:file-entry manifest:full-path="random.png" manifest:media-type="image/png"/>

E001g.ods

and tried again:
C:\odf\odf-validator-main>odf-validator.bat -p "filer\testfiler\E001g.ods"
APP-1: [INFO] Validating filer\testfiler\E001g.ods.
APP-5: [INFO] DNA ODF Spreadsheets Preservation Specification Profile report for filer\testfiler\E001g.ods.
MIM-1: mimetype [ERROR] The "mimetype" file SHALL be the first file of the zip file.
DOC-3: mimetype [INFO] OpenDocument MIMETYPE application/vnd.oasis.opendocument.spreadsheet detected
POL_2: E001g.ods [ERROR] Standard Compliance | The file MUST comply with the standard "OASIS Open Document Format for Office Applications (OpenDocument) v1.3".
NOT VALID, 2 errors, 0 warnings and 1 info messages.

And now it gets through (again except for the mimetype error).

Scenario 5: Trying with a file names signatures in META-INF/
Since there are some special restrictions on the META-INF/ folder, I tried adding an encrypted file named signatures to this folder. In this folder, it doesn't need to be in the manifest, so I didn't add it there.

E001h.ods

The result was like scenario 1:
C:\odf\odf-validator-main>odf-validator.bat -p "filer\testfiler\E001h.ods"
APP-1: [INFO] Validating filer\testfiler\E001h.ods.
APP-2: [ERROR] Unsupported feature encryption used in entry META-INF/signatures

And without the profile, it also reacted the same as in scenario 1.

The signatures file is actually the same file as random.png - I just renamed it.

Summary
I have not been able to generate PKG-8, but my attempts showed that there is an APP-2 error message in some cases, and the file slips through in other cases.