Error: undecodable files from output are not same as files from source.

[vampyresqvid]

Hello,

I have a project with 16 sets of DPX files, and four of the sets have given me this error. The others processed as expected. I'm running rawcooked with the --all flag, default license. One example of the error output below:

"Decoding was checked, issues detected, see below.

Error: undecodable files from output are not same as files from source.
GRADED DPX/Painting in Object4396.dpx
GRADED DPX/Painting in Object4399.dpx
GRADED DPX/Painting in Object4400.dpx
GRADED DPX/Painting in Object4401.dpx
GRADED DPX/Painting in Object4402.dpx
GRADED DPX/Painting in Object4403.dpx
GRADED DPX/Painting in Object4404.dpx
GRADED DPX/Painting in Object4405.dpx
GRADED DPX/Painting in Object4406.dpx
GRADED DPX/Painting in Object4407.dpx
... "

I'm attaching mediainfo output for one of the files the error message flagged, and for one of the files it did not flag. There are a number of differences between them, but I don't know what might be causing the problem. I'm curious if you can explain why this error is occurring and if there's anything I can do about it.

Let me know if any other information would be helpful. Thank you.

PaintingInObject0280.txt
PaintingInObject4400.txt

[JeromeMartinez]

Thanks for the traces. Unfortunately I don't see an obvious difference which would have an impact on the reversibility.
Is it possible to have one of the files? here or [email protected]. I don't need the data part, having e.g. the 16 first KB should be enough if there is an issue with the content itself.

[JeromeMartinez]

I can not reproduce the issue with the file you send to me. Tried with RAWcooked 21.09 and FFmpeg 4.4.
Can you reproduce the issue with a single file on your machine? If not, can you provide he full directory (the smallest one with an issue...)

jerome@MacMini1100 355 % rawcooked --all GRADED\ DPX                              
Using default license.
Consider to support RAWcooked: https://MediaArea.net/RAWcooked

Track 1:                                                    
  Painting in Object%04d.dpx
 (4400 --> 4400)
  DPX/Raw/RGB/10bit/FilledA/BE

ffmpeg version 4.4 Copyright (c) 2000-2021 the FFmpeg developers
  built with Apple clang version 12.0.5 (clang-1205.0.22.9)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.4_2 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-avresample --enable-videotoolbox
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
Input #0, image2, from 'GRADED DPX/Painting in Object%04d.dpx':
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: dpx, gbrp10le(gbr/unknown/unknown), 4096x3112, 24 fps, 24 tbr, 24 tbn, 24 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (dpx (native) -> ffv1 (native))
  File GRADED DPX.rawcooked_reversibility_data -> Stream #0:1
Press [q] to stop, [?] for help
Output #0, matroska, to 'GRADED DPX.mkv':
  Metadata:
    encoder         : Lavf58.76.100
  Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), gbrp10le(gbr/unknown/unknown, progressive), 4096x3112, q=2-31, 200 kb/s, 24 fps, 1k tbn
    Metadata:
      encoder         : Lavc58.134.100 ffv1
  Stream #0:1: Attachment: none
    Metadata:
      mimetype        : application/octet-stream
      filename        : RAWcooked reversibility data
frame=    1 fps=0.4 q=-0.0 Lsize=    1113kB time=00:00:00.00 bitrate=9118104.0kbits/s speed=0.000443x    
video:1112kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.119202%
2.7 MiB/s                              

Info: Reversibility data created by RAWcooked 21.09.
Info: Uncompressed file hashes (used by reversibility check) present.

Reversability was checked, no issue detected.

[JeromeMartinez]

@vampyresqvid can you reproduce the issue with a single file on your machine? If not, can you provide he full directory (the smallest one with an issue...)

[vampyresqvid]

Hello, Sorry for the delay, I was out for a long weekend. I was able to get the same error with this limited number of 14 DPX files from the same directory: https://www.dropbox.com/sh/o9xqdi7oy623bqy/AACTl0jXr-WbLn15BlQk7Wy8a?dl=0

…

-Dan

On Mon, Sep 27, 2021 at 4:17 PM Jérôme Martinez ***@***.***> wrote: @vampyresqvid <https://github.com/vampyresqvid> can you reproduce the issue with a single file on your machine? If not, can you provide he full directory (the smallest one with an issue...) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#355 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEVL52X36I27FWK2FDPZKM3UEDGPPANCNFSM5EMUSDYA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[JeromeMartinez]

Thank you.
Actually you need the first file listed as having a problem and the previous one, e.g. 4395 and 4396, it is enough for creating the issue.

Good news: issue is in the decoder. So created MKV files will be correctly decoded after a patch.

The issue comes from the difference between the encoder (currently a third party tool, FFmpeg) and the decoder (home made), and the algorithm between both tools is not same in this use case.
Use case is different "flavors" of DPX between files in the same sequence, e.g. some files are in little endian (written by Premiere Pro) and some others are in big endian (written by Photoshop). I guess that some files where edited, and the decoder uses the first file as the reference then write correctly the headers but use the "flavor" of the first file as the reference (which is wrong).

The encoder (FFmpeg) behaves correctly, and we need to adapt the decoder (RAWcooked) for such case.