Some emails blank when fetched from iCloud

[ybizeul]

Many incoming emails are blank in the UI, sending samples now

PHP version: PHP 8.2.22
FreeScout version: 1.8.151
Database: MySQL / PostgreSQL
Are you using CloudFlare: Yes / No
Are you using non-official modules: Yes / No

[ybizeul]

I just saw 1.8.152 might address similar issues, hold on

[freescout-help]

Fixed in FreeScout v1.8.152.

[ybizeul]

Is there a way to clear Message with such Message-ID has been fetched before: [email protected] so I can see if it fetches email correctly ?

I deleted it Freescout but it still sees it at something fetched before, I also cleared cache.

[freescout-help]

Only by sending a new email.

[ybizeul]

Ok I found one that's not importing correctly, sent you an email

[freescout-help]

importing correctly

What do you mean? Can you attach some screenshots? Are you using POP3 or IMAP?

[ybizeul]

It's just blank, that's IMAP

[freescout-help]

We can't reproduce the issue on FreeScout v1.8.152

[ybizeul]

I'm running 1.8.152 now.

What's next ? how can we troubleshoot this ?

[freescout-help]

Send a sample email to [email protected] but not EML file and not forwarding.
And answer ALL our questions: #4202 (comment)

[ybizeul]

I just sent the raw email content using curl to freescout support, hopefully it'll exhibit the issue.

What do you mean? Can you attach some screenshots? Are you using POP3 or IMAP?

Not sure what's missing. I attached a screenshot, said it's IMAP

[freescout-help]

Something is wrong with your email. Other email clients also can not show it properly:

[ybizeul]

Yes I guess i couldn't outsmart smtp when using curl to send it, but my mail client correctly parsed the original

[ybizeul]

I think I managed to send a correct one :

[freescout-help]

FreeScout v1.8.152 successfully fetched it:

[ybizeul]

So it's either something in the original headers, that I had to strip, or something with iCloud IMAP ?

[freescout-help]

It's hard to tell.

You can try to put EML file into /storage/logs/email.eml, run php artisan freescout:parse-eml --mailbox=YOUR-MAILBOX-ID and check the output.

[ybizeul]

It seems to be parsing Text Body, HTML Body, Date etc correctly.

Here is what I'm trying right now.

• Delete email in Freescout
• Delete reference in database

delete from threads where message_id='FR2P281MB29570501D66929BBC36BEFDB858B2@FR2P281MB2957.DEUP281.PROD.OUTLOOK.COM';

• Mark email as unread
• Fetch in System / Tools with debug.

What I'm getting :

[2024-08-28 14:36:31] Fetched: 1
[2024-08-28 14:36:31] 1) NAbox Help
[2024-08-28 14:36:32] Thread successfully created: 2268
[...]
>> TAG3 SELECT "NAbox"
<< * 554 EXISTS
<< * 0 RECENT
<< * OK [UNSEEN 549]
<< * OK [UIDVALIDITY 1659880285]
<< * OK [UIDNEXT 574]
<< * FLAGS (\Answered \Flagged \Draft \Deleted \Seen \Recent $MailFlagBit0 $MailFlagBit1 $MailFlagBit2 $Forwarded Redirected $NotJunk NotJunk)
<< * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \Recent $MailFlagBit0 $MailFlagBit1 $MailFlagBit2 $Forwarded Redirected $NotJunk NotJunk \*)]
<< TAG3 OK [READ-WRITE] SELECT completed (took 2 ms)
>> TAG4 UID SEARCH SINCE "25-Aug-2024" UNSEEN
<< * SEARCH 552
<< TAG4 OK SEARCH completed (took 5 ms)
>> TAG5 UID FETCH 552 (FLAGS)
<< * 549 FETCH (UID 552 FLAGS (\Answered))
<< TAG5 OK FETCH completed (took 4 ms)
>> TAG6 UID FETCH 552 (RFC822.HEADER)
<< * 549 FETCH (UID 552 RFC822.HEADER {12053}
[ Here, full headers of the email are displayed ]
<< TAG6 OK FETCH completed (took 36 ms)
>> TAG7 UID FETCH 552 (RFC822.TEXT)
<< * 549 FETCH (UID 552 RFC822.TEXT "")
<< TAG7 OK FETCH completed (took 181 ms)

Now, with an email that "works", doing the same thing, here goes :

>> TAG7 UID FETCH 574 (RFC822.TEXT)
<< * 555 FETCH (UID 574 RFC822.TEXT {3284}
<< 
<< --Apple-Mail=_ABE49CB5-3F28-4089-B773-D5B67CFE8374
<< Content-Transfer-Encoding: 7bit
<< Content-Type: text/plain;

It looks like I'm onto something right ? On the first email, the server seems to return an empty RFC822.TEXT, I'm not sure what 549 is compared to 555

Finally, I enabled debug in Apple Mail on my side and looked at the trace. The mail is fetched with the following IMAP command I believe :

63.176 UID FETCH 575 (X-APPLE-REMOTE-LINKS BODYSTRUCTURE BODY.PEEK[])
* 556 FETCH (UID 575 MODSEQ (394336005050597) X-APPLE-REMOTE-LINKS () BODYSTRUCTURE (("text" "plain" NIL NIL NIL "7bit" 0 0 NIL NIL NIL NIL)("application" "pkcs7-signature" ("NAME" "smime.p7s") NIL NIL "base64" 2880 NIL ("ATTACHMENT" ("FILENAME" "smime.p7s")) NIL NIL) "signed" ("PROTOCOL" "application/pkcs7-signature" "BOUNDARY" "Apple-Mail=_35798986-B89C-42EA-A20B-B33563FC9AE7" "MICALG" "sha-256") NIL NIL NIL) BODY[] {7675}

From here I'm thinking, any way I can check and change RFC822.TEXT in Freescout to something else like BODYSTRUCTURE ?

[ybizeul]

Ok, so using BODY[TEXT] instead of RFC822.TEXT here

freescout/overrides/webklex/php-imap/src/Connection/Protocols/ImapProtocol.php

Line 700 in f218fe1

$result = $this->fetch(["$rfc.TEXT"], is_array($uids)?$uids:[$uids], null, $uid);

Now a previous email I never got to display does show up correctly.

I'll let it sit for a few days and report if any side effect pops up

More references :
zammad/zammad#4589
https://stackoverflow.com/questions/75444763/python-imaplib-fetching-icloud-mail-rfc822-not-working/75734707#75734707

[freescout-help]

Can you share the changes you've made in the code.

[ybizeul]

        $result = $this->fetch(["$rfc.TEXT"], is_array($uids)?$uids:[$uids], null, $uid);

to

        $result = $this->fetch(["BODY[TEXT]"], is_array($uids)?$uids:[$uids], null, $uid);

From what I see, this method is called in two locations :

freescout/overrides/webklex/php-imap/src/Query/Query.php

Line 237 in f218fe1

$contents = $this->client->getConnection()->content($uids, "RFC822", $this->sequence);

and

freescout/overrides/webklex/php-imap/src/Message.php

Line 498 in f218fe1

$contents = $this->client->getConnection()->content([$sequence_id], "RFC822", $this->sequence === IMAP::ST_UID);

And both are using "RFC822" parameter so there doesn't seem to a variable for this.

I commented on this exact issue here : Webklex/php-imap#510

[freescout-help]

Try with the attached /overrides/webklex/php-imap/src/Connection/Protocols/ImapProtocol.php file.
It uses BODY[TEXT] if IMAP Server for the mailbox is set to imap.mail.me.com.

ImapProtocol.zip

[ybizeul]

<< * 549 FETCH (UID 552 BODY[TEXT] {168647}

Looking good !

[freescout-help]

@CoE2013 Your issue is different as you have body in the DB. Create a new issue.

[mitchplze]

I believe I'm having the same issue. Related at least.

Posted in #4219 (deep link)