support unsupported message types

[Meteor0id]

There are a few unsupported message types.
I guess these are work in progress? Things like profile name changes can be important to include in exports.

[bepaald]

Are we talking about the import from desktop, or the html export here?

I believe HTML export should now output most (but not all) status messages (I think it outputs profile name changes, does it not?).

On the import from desktop: The main problem here is that I do not have these message types in my own database. A second problem is some status messages are simply complicated, though I don't think the profile name change is (but certainly all group updates are going to be a pain, don't expect them soon).

Anyway, I'm in need of some new SIM cards for my dev-phones, when I get them I will work on some of these message types. When I run --importfromdesktop, I only have two unsupported message types:

Warning: Unsupported messagetype 'group-v2-change'. Skipping.
Warning: Unsupported messagetype 'timer-notification'. Skipping message.

Maybe you have a longer list of message types for me to work on?

[Meteor0id]

Just scrolling through my log I see these:

Warning: Unsupported messagetype 'profile-change'. Skipping message.
Warning: Unsupported messagetype 'change-number-notification'. Skipping message.
Warning: Unsupported messagetype 'keychange'. Skipping message.
Warning: Unsupported messagetype 'group-v2-change'. Skipping.
Warning: Unsupported messagetype 'delivery-issue'. Skipping message.
Warning: Unsupported messagetype 'timer-notification'. Skipping message.
Warning: Unsupported messagetype 'verified-change'. Skipping message.
Warning: Unsupported messagetype 'group-v1-migration'. Skipping message.
Warning: Unsupported messagetype ''. Skipping message.

Note I edited this message to add any others I have ran into

I think things like group v1 migration are a low priority, but things like change number, profile change, delivery issue, verified change, key change and timer notification are all relevant when reviewing an exported chat, these impact security and or the conversation itself.

[bepaald]

Just a quick update on these message types:

Warning: Unsupported messagetype 'profile-change'. Skipping message.
Warning: Unsupported messagetype 'keychange'. Skipping message.
Warning: Unsupported messagetype 'timer-notification'. Skipping message.
Warning: Unsupported messagetype 'verified-change'. Skipping message.
Warning: Unsupported messagetype 'group-v1-migration'. Skipping message.
Warning: Unsupported messagetype ''. Skipping message.

These should all be supported now (timer-notifications only in non-group threads, for groups these are group-v2-change). Though the empty type '' is obviously a bit problematic, the only ones of these I've seen are from your backup and were all timer-notifications. If you see any of these still being skipped, let me know.

Warning: Unsupported messagetype 'change-number-notification'. Skipping message.
Warning: Unsupported messagetype 'delivery-issue'. Skipping message.

I plan on doing these later. But the change-number one will be a while, I don't want to mess with my testing setup as it is now so it will be whenever all the other importfromdesktop stuff is mostly finished (though I may need a reminder). The delivery-issue could happen anytime as soon as I figure out how to get force one to happen. If you have this message (not just in Desktop, but in your Android backup as well) you could help me out with it.

Warning: Unsupported messagetype 'group-v2-change'. Skipping.

Well, I won't say it'll never happen, but... well....

Problem is, each of these messages contain not only the actual change, but also both the previous and the current group-state (which is a big structure). All this info is not available in the Desktop message. I would need to piece this this together by tracking all the group changes in both desktop and android backup to try and figure out what the previous and current state were at the time of that message. Honestly I don't know if it's even possible, but it would be incredibly hard in any case.

As for exporting to HTML: that function only cares about the actual 'change' part of the group-v2-change message, which shouldn't be to hard (though it translates to dozens of actual messages (changing group title/description/avatar, adding/removing members, inviting, accepting/declining invites, changing member roles, changing timers, and many more), so I do plan to add these, one by one, to --addincompletedataforhtmlexport in the future.

[Meteor0id]

so I do plan to add these, one by one, to --addincompletedataforhtmlexport in the future.

Is it necessary to diffferentiate between complete html and incomplete html? After all, complete html without these messages is per defenition incomplete as well. You could just ignore them when compiling from html back into an backup file (leading to an incomplete back up file which is unavoidable anyway when building from a html export, right?)

Not sure if I am making sense, just thinking out loud questioning if the destinction incompletehtml makes sense.

[bepaald]

[...]complete html and incomplete html[...]

I'm not exactly sure what you mean, so I'm not sure this response will be satisfactory, but: There is no 'complete' or 'incomplete' HTML. The --exporthtml function simply works on backup files, and its output is always complete (or, as complete as I have implemented it up until now). The export function has no idea where its input comes from (whether foreign stuff was added to it) as long as it's a Signal Android backup file.

--addincompletedataforhtmlexport is only a modifier flag for the --importfromdesktop option. The Desktop import just imports messages from the Desktop into a Signal Android backup file. It does not know whether you plan to export it to a new backup, to html, xml, csv or raw directory, or all of those at once after it is done importing.

[Meteor0id]

3 more (and maybe more exist but these I happened to notice)

1. the status "sessionEnded" which appears as "Secure session reset" is also not suupported yet
   In html export these currently show as empty messages with only a date.

2. the status "incomingError" which appears as "Error handling incoming message" is also not suupported yet
   In html export these currently show as empty messages with only a date.

3. Any quoted story shows up as some quoted emojis in Signal Desktop but is an empty message in the exported html.
   

[bepaald]

• I've added the end session type (locally)
• Assuming this message was in the backup originally (and not imported and half-handcrafted by this program): in the HTML file, every message starts with a <div id="msg-NNN" class="type-NNN msg .... For any unsupported type (that will mostly show up as an empty message), the type-NNN would be very useful to have.
• I have zero experience with stories, never seen them, but I'll make some to reproduce this and work on it. It's somewhere on my seemingly endless to-do-list :)

[Meteor0id]

Both the incoming message errors I have looked at and the story messages show the following classes: type-10485780 msg msg-incoming

I assuming this means these failed incoming messages are also story messages.

[bepaald]

type-10485780

Well this to me indicates that the type in the Desktop database was simply incoming, after which the function imported it as a normal incoming message. I did find the flag for END_SESSION (I think, you will have to tell me if it worked) so hopefully they are now imported correctly. Also, I had two END_SESSION messages in my Android backup so I could see the type and correctly set it.

For the "incomingError", I think I found the data in the Desktop database (though I'm not sure), but I don't know what to translate it to for the Android backup yet.

On a somewhat related note, maybe you can run --runsqlquery "SELECT DISTINCT (type & 0x1F) from messages" sometime on your Android backup, you could have some types I don't have and tell me what they look like.

[Meteor0id]

On a somewhat related note, maybe you can run --runsqlquery "SELECT DISTINCT (type & 0x1F) from messages" sometime on your Android backup, you could have some types I don't have and tell me what they look like.

How exactly? If I run it right now it tells me the table messages does not exist

---

I have no clue what we are looking at here but were you looking for this?

 * Executing query: SELECT DISTINCT (type & 0x1F) FROM message
(type & 0x1F)
0
1
2
3
7
9
12
14
15
20
23
22

I might be able to find back an older android backup with much more messages in it

[bepaald]

Yeah thanks! My mistake, in the dekstop database the table is called 'messages', on Android it's 'message'. You can imagine how many times a day I make that mistake 😭

Anyway, nothing in there I don't have myself, I was sort of hoping for 5 or 6, which I suspect I would need for the "incomingError" desktop message

[Meteor0id]

How can I run this same test on an older backup which does not have a message table?

[bepaald]

That depends on the database version.

You will at least need to replace 'message' with 'mms'. If 'type' does not yet exist in the 'mms' table, it was still called 'msg_box'.

Then, if still present, you would need to run the same query on 'sms' (and using 'type' again, no 'msg_box').

[Meteor0id]

Thanks, I was already on it but had not yet posted here.

My older Android backup does have more types:

 * Executing query: SELECT DISTINCT (type & 0x1F) FROM mms
(type & 0x1F)
0
1
2
3
7
9
12
13
14
15
17
20
23
22
24

[Meteor0id]

type 14 = Bob changed their phone number
type 13 = does not get displayed on Signal Desktop

---

Stickers are currently also shown as empty messages without a distinct type.

[bepaald]

These are the types of the messages in Android right? Does the type 13 show up on Android? Also, I would be interested in the full type as well (SELECT DISTINCT type FROM [message/mms] WHERE (type & 0x1f) IS [13/14]).

There on my to-do-list, it'll be a while, and I might need more info.

---

Right! I hadn't even noticed! I will fix that, that should be doable since I have those messages in my Desktop database. Not sure when I'll have time, but it will be done.

[Meteor0id]

Does the type 13 show up on Android?

Don't know, would have to re-import the old backup file in which I have it. Wasn't planning on re importing but maybe soon.

 * Executing query: SELECT DISTINCT type FROM mms WHERE (type & 0x1f) IS 0
type
0
 * Executing query: SELECT DISTINCT type FROM mms WHERE (type & 0x1f) IS 1
type
1
 * Executing query: SELECT DISTINCT type FROM mms WHERE (type & 0x1f) IS 2
type
2
 * Executing query: SELECT DISTINCT type FROM mms WHERE (type & 0x1f) IS 3
type
3
 * Executing query: SELECT DISTINCT type FROM mms WHERE (type & 0x1f) IS 7
type
7
 * Executing query: SELECT DISTINCT type FROM mms WHERE (type & 0x1f) IS 9
type
9
 * Executing query: SELECT DISTINCT type FROM mms WHERE (type & 0x1f) IS 12
type
12
 * Executing query: SELECT DISTINCT type FROM mms WHERE (type & 0x1f) IS 13
type
13
 * Executing query: SELECT DISTINCT type FROM mms WHERE (type & 0x1f) IS 14
type
14
 * Executing query: SELECT DISTINCT type FROM mms WHERE (type & 0x1f) IS 15
type
15
 * Executing query: SELECT DISTINCT type FROM mms WHERE (type & 0x1f) IS 17
type
17
 * Executing query: SELECT DISTINCT type FROM mms WHERE (type & 0x1f) IS 20
type
20
2097684
2105364
2113556
8388628
10485780
10551316
10616852
10747924
11075604
11206676
14680084
4305453076
 * Executing query: SELECT DISTINCT type FROM mms WHERE (type & 0x1f) IS 23
type
87
10485783
10551319
10747927
11075607
11206679
 * Executing query: SELECT DISTINCT type FROM mms WHERE (type & 0x1f) IS 22
type
8214
16406
10493974
10502166
14680086
 * Executing query: SELECT DISTINCT type FROM mms WHERE (type & 0x1f) IS 24
type
10485784

---

I noticed that "you have verified from another device" and the opposite you have removed verification on another device" (not exact quotes but whatever) are being omitted in the export. In cases where this is included in Desktop but not present in the Android backup it might still be useful to just turn it into a status message "You have removed the verified status for this Bob on one of your devices which was not Signal Desktop". That way the info is still kept in the export.

[bepaald]

Don't know, would have to re-import the old backup file in which I have it. Wasn't planning on re importing but maybe soon.

Right, thanks, but not necessary, I think I can figure it out.

The output is useful, thank you.

---

Not sure I fully get what you're saying, but I'll look at this when I have time later.

[bepaald]

Stickers are currently also shown as empty messages without a distinct type.

I've attempted to now import these from the desktop db. It has had limited testing, I hope it works.

[Meteor0id]

FRAME 98281 (100.0%)... Read entire backup file...

done!
Trying to match conversation (1/140) (type: private)
 - Importing 4309 messages into thread._id 1108
Getting value of field 'sticker_id' (idx 5). Value as string: 
Type: Dn Requested type: x
terminate called after throwing an instance of 'std::bad_any_cast'
  what():  bad any_cast

[bepaald]

Thanks, I guess the sticker data doesn't always look like it does in my database.

I've had it print out some info on the sticker data (I'd like to see that) when this error is about to occur, and then skip it. i don't think the sticker will get imported correctly, but it might still work. At least it will hopefully not crash anymore.

[Meteor0id]

Still crashing, but a little later on

FRAME 98280 (100.0%)... �[2K
FRAME 98281 (100.0%)... Read entire backup file...

done!
Trying to match conversation (1/140) (type: private)
 - Importing 4309 messages into thread._id 1108
 === Row 1/1 ===
json_extract(json, '$.sticker') : {"packId":"c4xxxxxxxxxxxxxxxxxxxxxxxxxxxxx5","stickerId":9,"packKey":"koxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+E=","data":{"size":10908,"contentType":"image/webp","width":512,"height":512,"path":"7c\\7cxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx600"}}
 === Row 1/1 ===
json_extract(json, '$.sticker') : {"packId":"b5xxxxxxxxxxxxxxxxxxxxxxxxxxxxxd","stickerId":95,"packKey":"mFxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxXQ=","data":{"size":17412,"contentType":"image/webp","width":512,"height":512,"path":"fb\\fbxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxb42"}}
 === Row 1/1 ===
json_extract(json, '$.sticker') : {"packId":"b5xxxxxxxxxxxxxxxxxxxxxxxxxxxxxd","stickerId":94,"packKey":"mFxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxXQ=","data":{"size":16426,"contentType":"image/webp","width":512,"height":512,"path":"5a\\5axxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx1ba"}}
 === Row 1/1 ===
json_extract(json, '$.sticker') : {"packId":"b5xxxxxxxxxxxxxxxxxxxxxxxxxxxxxd","stickerId":85,"packKey":"mFxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxJXQ=","emoji":"😑","data":{"size":20478,"contentType":"image/webp","width":512,"height":512,"path":"17\\17xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxf7"}}
 === Row 1/1 ===
json_extract(json, '$.sticker') : {"packId":"c4xxxxxxxxxxxxxxxxxxxxxxxxxxxxa5","stickerId":9,"packKey":"kxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxL+E=","emoji":"😰","data":{"size":10908,"contentType":"image/webp","width":512,"height":512,"path":"22\\22xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxf5"}}
 === Row 1/1 ===
json_extract(json, '$.sticker') : {"packId":"eexxxxxxxxxxxxxxxxxxxxxxxxxxxxbx","stickerId":3,"packKey":"TxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxLY=","emoji":"😄","data":{"size":14526,"contentType":"image/webp","width":512,"height":512,"path":"fe\\fexxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx6"}}
 === Row 1/1 ===
json_extract(json, '$.sticker') : {"packId":"eexxxxxxxxxxxxxxxxxxxxxxxxxxxxba","stickerId":81,"packKey":"T1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxY=","emoji":"😏","data":{"size":25412,"contentType":"image/webp","width":512,"height":512,"path":"c4\\c4xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxc1"}}
Skipping conversation, conversation has no messages (2/140)
Trying to match conversation (3/140) (type: group)
Warning: Chat partner was not found in recipient-table. Attempting to create.
         NOTE THE RESULTING BACKUP CAN MOST LIKELY NOT BE RESTORED
         ON SIGNAL ANDROID. IT IS ONLY MEANT TO EXPORT TO HTML.
Succesfully created new recipient for group (id: 6018).
Warning: Failed to find matching thread for conversation, creating. (id: __signal_group__v2__!c6xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx2f)
 - Importing 2281 messages into thread._id 1177
Warning: Unsupported message type 'group-v2-change'. Skipping... (this warning will be shown only once)
Warning: Unsupported message type 'timer-notification (in group)'. Skipping... (this warning will be shown only once)
Getting value of field 'sticker_id' (idx 5). Value as string: 77xxxxxxxxxxxxxxx06
Type: NSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE Requested type: x
terminate called after throwing an instance of 'std::bad_any_cast'
  what():  bad any_cast

Also you might want to look at rare stickerpacks with a custom number of emoji. Some people manually edited sticker packs to show suggestions for more or fewer emoji:
https://signalstickers.com/pack/ee7c8f7152bb5203d88c8d2cfb9d71ba

[Meteor0id]

I noticed you made another change yesterday after I tested, here is the new result, again an error unfortunately

FRAME 98280 (100.0%)... �[2K
FRAME 98281 (100.0%)... Read entire backup file...

done!
Trying to match conversation (1/140) (type: private)
 - Importing 4311 messages into thread._id 1108
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string: construction from null is not valid

But this is to be expected right, we'll just have to try a few times and it pays off if the tool ends up supporting all kinds of variations and old backup files.

[bepaald]

I had another go at this, hope it works now. If you get more output (=== Row 1/1 === etc...) or crashes let me know.

[Meteor0id]

Stickers now seem to work well, thanks again for your work on this tool. The export tool still shows them in a message bubble when exported to html but that's just a html style thing.

[bepaald]

Hm, that's a weird one, I can't explain. Stickers should not have a bubble, and in my exports they don't (neither imported ones or original ones).

Do all stickers have a bubble? If you look at the source of the html of one with a bubble, there should be a message id in a comment above it, could you tell me this? Then I'll use that to try to print some info during import to try and find out why the sticker isn't being recognized as such.

[Meteor0id]

<!--Message: _id:27523,type:10485783-->
<div class="msg msg-outgoing msg-sender-6061">
            <div class="attachment">
              <div class="msg-img-container">
                <input type="checkbox" id="zoomCheck-35572-1683749256405">
                <label for="zoomCheck-35572-1683749256405">
                  <img src="media/Attachment_35572_1683749256405.bin" alt="Image attachment">
                </label>
              </div>
            </div>
            <div class="footer">
              <span class="msg-data">May 10, 2023 22:07:36</span>
              <div class="checkmarks checkmarks-sent"></div>
            </div>
          </div>

Do all stickers have a bubble?

So far yes, all the ones I have seen while looking at a few random chats

---

I also keep wondering why those checkmarks show "sent" rather than "read" like they should in this case (they are read in Signal Desktop, and the chat i looked at does not exist anymore in the Android backup). That's on all messages though not just sticker ones.
No idea if that's related at all but who knows.

Edit:
The checkmarks are an issue with Signal Android not doing the same thing as Signal Desktop: https://community.signalusers.org/t/show-the-same-checkmarks-on-android-as-shown-on-androud-for-messages-i-have-send-in-an-empty-group/53251
Sorry about this dustraction lets get back to the bubbles around stickers.

[Meteor0id]

Dealing with thread 1234
 ==== 
 INFO ON IMPORTED STICKER (htmlexport)
(no results)
Dealing with thread 1235

[bepaald]

Thanks, there was nothing before that? I was also hoping for (more relevant) info during the desktop-import-part... Could you confirm the _id is still 27523?

(I just realized maybe I'm reordering the id's between the import and export... That complicates things, I'll need to have a think about it).

(Maybe you've noticed I made more adjustments btw, I don't always have time to respond to all messages, but feel free to keep it coming, I just add everything to the list to fix or consider fixing, and respond when I have time).

[Meteor0id]

I did not expect you needed me to compare one export with the other so that test group I have already deleted. I can't compare the id.

There is nothing else in the output related to stickers

I have compared the sticker message of another chat to the sticker message in a much older backup before stickers were even being exported properly, and the id has remained unchanged. I will keep this chat and this message around so you can take its ID for the next test.

<!-- Message: _id:11727,type:10485783 -->
<div class="msg msg-outgoing msg-sender-702">
            <div class="attachment">
              <div class="msg-img-container">
                <input type="checkbox" id="zoomCheck-31364-1617319494559">
                <label for="zoomCheck-31364-1617319494559">
                  <img src="media/Attachment_31364_1617319494559.bin" alt="Image attachment">
                </label>
              </div>
            </div>
            <div class="footer">
              <span class="msg-data">Apr 02, 2021 01:24:54</span>
              <div class="checkmarks checkmarks-sent"></div>
            </div>
          </div>

[bepaald]

I did not expect you needed me to compare one export with the other so that test group I have already deleted. I can't compare the id.

Well the old _id was 27523, you posted it just e few messages ago. But it was for nothing, my fault, I forgot the message id's get reordered after the desktop import is complete, so I apologize, we have to do this again. If you now run it (with --addincompletedataforhtmlexport) the reordering is skipped. If you could run again and let me know a sticker-message id, I will make another update (to match this new id) and then - hopefully - we get some more info during importing.

[Meteor0id]

Noting about stickers in the output. Here is a sticker

<!--Message: _id:218862,type:10485783-->
<div class="msg msg-outgoing msg-sender-702">
            <div class="attachment">
              <div class="msg-img-container">
                <input type="checkbox" id="zoomCheck-31364-1617319494559">
                <label for="zoomCheck-31364-1617319494559">
                  <img src="media/Attachment_31364_1617319494559.bin" alt="Image attachment">
                </label>
              </div>
            </div>
            <div class="footer">
              <span class="msg-data">Apr 02, 2021 01:24:54</span>
              <div class="checkmarks checkmarks-sent"></div>
            </div>
          </div>

[bepaald]

Thanks, now if you run it again, there should hopefully be some output during importing of that particular message.

[Meteor0id]

Trying to match conversation (87/141) (type: group)
 - Importing 125 messages into thread._id 995
 ====== 
  INFO ON MSG_ID 218862 (desktop import):
  issticker: 1
 === Row 1/1 ===
            path : d4\d4a75155357be5c637129d3e511bd6a5ead20ad2795b013f2e30a25565418c67
    content_type : image/webp
            size : (NULL)
   sticker_emoji : 🤨
  sticker_packid : c4c42d9a066ef552a36d470c402320a5
      sticker_id : 190
       file_name : (NULL)
upload_timestamp : 0
           flags : 0
         pending : 0
      cdn_number : 0
 === Row 1/1 ===
  packkey : koWQoMicqLsiSxgkKczV6fuw+qUX6FEzEcIwxEl5L+E=
   packid : c4c42d9a066ef552a36d470c402320a5
stickerid : 190
    emoji : 
Inserted sticker data into part:
 === Row 1/1 ===
 sticker_pack_id : (NULL)
sticker_pack_key : (NULL)
      sticker_id : -1
   sticker_emoji : (NULL)
   ========
Trying to match conversation (88/141) (type: group)

Dealing with thread 995
 ==== 
 INFO ON IMPORTED STICKER (htmlexport)
 === Row 1/1 ===
            _id : 31364
      unique_id : 1617319494559
             ct : image/webp
      file_name : (NULL)
   pending_push : 0
sticker_pack_id : (NULL)
Dealing with thread 997

[bepaald]

I had another go at this yesterday before work, maybe it's fixed?

[Meteor0id]

The background is now fixed.

I ran more tests and I think if a chat contains only one or more sticker messages and nothing else the chat will not get shown on index.html while of course it should

[bepaald]

The background is now fixed.

Good to hear

I ran more tests and I think if a chat contains only one or more sticker messages and nothing else the chat will not get shown on index.html while of course it should

Hm, that sounds weird. I have created a new group between my devphones, only exchanged two stickers (and even deleted the 'created new group' message), but I can't reproduce. This is only a problem on the index? The actual conversation html is created? Can I assume these are threads that are imported from the desktop, and don't exist on backup?

[Meteor0id]

Will test again later but this was desktop import not in android backup

[Meteor0id]

Unable to reproduce myself. Might have been some mistake on my previous test. Let's disregard and assume it all works.

I did notice index.html does not yet show (sticker) as last send message preview for a chat where a sticker was the last message. That's all. Everything else about stickers works as expected as far as I tested.

Edit:
Just curious, does de backup contain every full stickerpack? Current behavior of the export when clicking a sticker is to enlage it, but maybe in some future we could display the pack to which it belongs? or add something like --html-stickerpack-overview to print all stickerpacks? Not at all an urgent feature but a would be nice some day thing.

[bepaald]

Unable to reproduce myself. Might have been some mistake on my previous test. Let's disregard and assume it all works.

👍

I did notice index.html does not yet show (sticker) as last send message preview for a chat where a sticker was the last message. That's all. Everything else about stickers works as expected as far as I tested.

I should have this done locally, but it was much more involved than I thought, I should test it some more before I push it. Not at home right now, but hopefully tomorrow.

[bepaald]

Ok, done I think.

[Meteor0id]

Still encountering a few, but I guess these are expected. Not sure yet if they occur only on Desktop or also in my android backup but I could maybe have a look at the info you need?

[Warning]: Unsupported message type 'timer-notification (fromSync=true)'. Skipping...
[Warning]: Unsupported message type 'delivery-issue'. Skipping message. (this warning will be shown only once)

And this one but who cares:
[Warning]: Unsupported message type 'group-v2-change'. Skipping... (this warning will be shown only once)

[bepaald]

[Warning]: Unsupported message type 'group-v2-change'. Skipping... (this warning will be shown only once)
[Warning]: Unsupported message type 'timer-notification (fromSync=true)'. Skipping...

These are indeed expected. The first one is too difficult (maybe even impossible) to correctly translate, the second one is definitely impossible.

[Warning]: Unsupported message type 'delivery-issue'. Skipping message. (this warning will be shown only once)

This one is simply unknown. I don't know if it can be supported or not, because I have never seen one.

I've now changed the warning message for types that may be possible to support to unhandled message type, instead of unsupported, just to be able to tell them apart. I also try, when --verbose is added, to print the title of the thread and the date of the message along with the raw message data of the unhandled message. If you can find a corresponding message in your Android backup, then I could probably support the message-type.

[Meteor0id]

this one has no identifier or type or content whatsover in html but in the app it looks like "you accepted an invite" which is stange because I created this group myself, but ok.

Odd duplicate, and although I can't recommand Starbucks you now know where to find a toilet in Leiden. It's in the back you can easily get there but make sure the tall blond lady (the owner) doens't notice you or she will tell you no.

[bepaald]

this one has no identifier or type or content whatsover in html but in the app it looks like "you accepted an invite" which is stange because I created this group myself, but ok.

You mean there is nothing in the actual source code? Every message the exporter writes to the file should be prepended with a comment:

The type would be interesting information. Probably also interesting would be the output of --runsqlquery "SELECT body,message_extras FROM message WHERE _id = XXX, where XXX should be the _id from the comment (if this is an older database message_extras will not exist and must be removed from the query). If there is no comment in the HTML, something is certainly going wrong.

As the screenshot shows, I thought I was already supporting these invitation messages though there may be variants, so I'm curious what's going on. Lastly, in case this is an imported (from Desktop) message, it should probably fall under the unsupported 'group v2 update' types and the bug would simply be that it isn't skipped like it should be.

Odd duplicate, and although I can't recommand Starbucks you now know where to find a toilet in Leiden. It's in the back you can easily get there but make sure the tall blond lady (the owner) doens't notice you or she will tell you no.

Lol, well I'm not in Leiden very often, but I'll keep it in mind.

I can not imagine what could lead to a doubling like this. Is it possible this is simply a message duplicated by importing from Desktop? There is of course a little deviation in the timestamps, but that is not uncommon at all. If it's not, I assume you can confirm the message does not appear twice in the app? And if you look at the HTML comments for these two messages and take the _ids and run --runsqlquery "SELECT body FROM message WHERE _id = id1 OR _id = id2", what do you get?

Thanks!

[bepaald]

Note, if the messages are imported from a desktop database, it's probably even more important to get the message details from there, since it is likely that something went wrong during translation from Desktop to Android.

There is now a way to actually track messages from the desktop database to the exported HTML. It is a bit of a complicated procedure, but it's better than nothing. To do this, add both the -v/--verbose and the new --skipmessagereorder flags when doing the --importfromdesktop. Now when you find the message _id in the HTML source code (in a comment above any message), you can link this _id to a message in the desktop. Example HTML:

          <!-- Message: _id:1211,type:10485780 -->
          <div class="msg msg-incoming deleted-msg">
            <div>
              <pre>This message was deleted.</pre>
            </div>
            <div class="footer">
              <span class="msg-data">Mar 07, 2024 19:44:09</span>
            </div>
          </div>

For each message it has imported, the tool will have printed lines like the following:

Message 117/130: 'incoming' (rowid: 611)
Handling reactions...done
Inserting message...
Running query: "INSERT INTO message (thread_id, date_sent, date_received, date_server, type, body, read, from_recipient_id, to_recipient_id, m_type, quote_id, quote_author, quote_body, quote_missing, quote_mentions, shared_contacts, remote_deleted, view_once, quote_type) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) RETURNING _id"
Inserted new row into table 'message'. New _id: 1211
done
Inserting attachments...done
Inserting reactions...done
Inserting mentions...done

Here we see that the created message — the one that has New _id: 1211 — originated from Desktop message with rowid 611. Because one needs to search through the programs output after running the command, and the --verbose flag causes a lot of output to be written, it may be helpful to save the output (for example by using -l/--logfile [output.log]).

Now that we know the rowid of the message in the Desktop database, we can get all message info by running:

signalbackup-tools --rundtsqlquery "SELECT * FROM messages WHERE rowid = 611"

There may be private info in this last commands' output, so censor as needed.