Skip to content

Email

Jump to bottom Edit New page
Neil Kolban edited this page Mar 15, 2016 · 18 revisions

An active fork of this node is in flight at:

https://github.com/nkolban/node-red-nodes

The Email node provides both input and output of emails. The node is available on NPM at:

As of 2016-03-06, the node has dependencies on:

Reading emails

The current implementation uses the IMAP protocol to interact with a back-end IMAP provider. The very last email seen is received and the message added to an emitted event. If the email body is plain text, it is added to msg.payload. If the email body is HTML, it is added to msg.html. The subject line of the email is propagated in msg.topic. The headers for the email are available in msg.header. The posting date of the email is available in msg.date. The sender email address is available in msg.from.

Future enhancements

There are a number of potential enhancements available to the email node that can be addressed. These include:

  • Support of IMAP that is not over SSL/TLS.
  • Support for self-certified SSL certificates.
  • Support for attachment processing. Attachments should appear as an array in msg.attachments.
  • Support for POP3 as well as IMAP.
  • Support for more than just last message processing.

Design Notes

Parsing incoming emails

Consider an email message that is sitting on an email server. It appears that an email is transmitted over SMTP and received by the email server and stored. It also appears that the email is saved in a certain format. That email can then be retrieved by a client application using IMAP and/or POP3.

Question: Is the email message retrieved via IMAP and POP3 the same format/content? If I retrieve the "blob-o-data" that represents my email, do I get the same think with both IMAP and POP3 retrievers?

Answer: NK 2016-03-07 - After testing, the answer is yes. Whether the email is received in parts via IMAP or as a complete entity using POP3, the result in both cases is identical and can be fed into mailparser for processing.

Question: For POP3 clients, is it acceptable to delete/destroy a message once received and processed?

Answer: NK 2016-03-14 - Yes. This is the normal model of POP3 interactions. An email agent gets the email from the email server and it is deleted.

Question: For attachments, how should the attachment be stored in the attachments array?

Answer: NK 2016-03-06 - I vote for the direct output of a record from the mailparser package. It seems rich and un-opinionated. In addition, we should always assume a more specific package has more appreciation for their domain that we do. As such, if they added a property they had a reason to do so (we should assume) and it should not be for us to question why or, unless we have a good reason to think otherwise, exclude it from our exposure of data.

Question: Since IMAP appears superior to POP3, why bother supporting POP3 at all?

Answer: NK 2016-03-08 - IMAP provides the ability to keep a copy of the email on the server and the message is only deleted when requested. Basically, POP3 is a read-once while IMAP is read-many and delete when no longer needed. For end-users reading email manually, this is desirable. I can read an email on my PC and the same email on my phone and when I choose to delete it from either, it is deleted in both. IMAP provides richer function. POP3 is simpler, we download the email to the email client and then it is deleted from the server. No other email client can connect to the server and see it again. For automated email processing, this may be exactly what is required. We don't want to ever see the same email twice and POP3 appears to provide exclusive access to an email account. This may actually prove to be beneficial.

There is a second possibility that may also be extremely relevant and that is the notion of simply exposing choice. If the effort (design, development and maintenance) of supporting POP3 in addition to IMAP is small enough, why would we choose not to support POP3 in addition to IMAP?

My vote is to provide support for both with the initial default being IMAP if not explicitly changed to POP3 by user configuration/selection.

Proposed design

Here we discuss the best so far notion for the design for processing incoming emails. The design is in the form of synchronous pseudo code which, when implemented in JavaScript, will become event based.

// Called periodically to check email.  Farms out the request to the correct email input technology
function checkEmail() {
  if (useIMAP) {
    checkIMAP();
  } else if (usePOP3) {
    checkPOP3();
  }
}

function checkIMAP() {
  connect to email server using IMAP;
  if (there are new messages) {
    for (each new message) {
      retrieve the new message;
      processNewMessage(message);
    }
  }
}

function checkPOP3() {
  connect to email server using POP3;
  if (there are new messages) {
    for (each new message) {
       retrieve the new message;
       processNewMessage(message);
    }
  }
}

function processNewMessage(message) {
  parse the email message into its parts;
  build an outgoing Node-RED object from the email message;
  send the message onwards for processing down the flow;
}

With the above in mind, the next thing to consider is how the above are to be processed. This breaks into two parts ... POP3 and IMAP.

Related forum posts and issues:

Related NPM packages

Technical tasks

  • How do we retrieve an email using POP3?

Examining the documentation for poplib, the high level is:

var client = new POP3Client(...);
// Called when the client had connected to the POP3 server ...
client.on("connect", function() {
  client.login(...); // Login to the server
});

// Called when we have logged in ...
client.on("login", function(..., rawdata) {
  client.list(); // List the messages
});

// Called with a list response ...
client.on("list", function(...) {
  client.retr(...); // Retrieve a message
});

// Called when a message has been retrieved ...
client.on("retr", function(...) {
  client.dele(...); // Delete a message
});

// Called when a message has been deleted ...
client.on("dele", function(...) {
});
  • How do we flag IMAP messages?

When we open an inbox we get back a list of supported flags. These will include \Deleted, \Seen, \Draft, \Answered and \Flagged. After processing an email it will always be flagged as \Seen however we can also flag it as \Deleted or \Answered. Using the imap module, we can set flags using the addFlags method.

  • What does the response from a POP3 'list' request look like?

The event for list contains status, msgCount, msgNumber and data. Of these, examination seems to show that the important one is msgCount. This is the number of messages in the mailbox. If 0, then there are no messages.

  • What does the response from a POP3 retr request look like?

The event for retr contains status, msgNumber and data. The data is the body of the email ... this is truly the whole body and includes headers and payload.

  • What does a mailparser parsed attachment look like?

An example attachment includes:

{
  contentType: "text/plain",
  charset: "utf-8",
  fileName: "application.ini",
  transferEncoding: "base64",
  contentDisposition: "attachment",
  generatedFileName: "application.ini",
  contentId: "1ba3477561756fe5833025518a94ac1c@mailparser",
  checksum: "ac03f68312dcd4b5633a7e399dcf911b",
  length: 611,
  content:
  {
    type: "Buffer",
    data: [59,32,52, ..., 52,53,10]
  }
}
  • mailparser works against POP3 retrieved messages, can it work against IMAP?

Yes. Testing has shown that using the IMAP library, if we request both HEADER and TEXT parts and then invoke mailparser.write(message) for each message part received, we can then call mailparser.end() and the parsing occurs as desired. This means that both POP3 and IMAP can both leverage mailparser.

Critical decisions

  • Rename header to headers - The msg.header property contains the email headers. It is suggested that this be renamed to msg.headers.

Major work items before pull-request

  • Handle multiple messages available in POP3.
Add a custom footer
Add a custom sidebar
Clone this wiki locally