Proposal: Flip email queuing paradigm around (breaking change)

[joshuadavidthomas]

Status Quo

Right now each application connects to a central database and uses a database router to make sure sent emails end up in that central database. This has been working fine, but there are some downsides:

• All of our applications are relatively simple with only have one main database, so introducing a second database with the required router adds unnecessary complexity. Testing in particular has been a sore spot.
• Having each application responsible for connecting to the central database spreads the complexity around, which on one hand is nice -- it's a small amount of mental overhead per application -- but spreading it amongst the distributed apps means a bunch of small bits of complexity adds up.
• It's not hard to see that this probably doesn't scale very well. We haven't reached that point (and may never), but each new application using the relay adds more and more mental overhead.
• Once an application queues up an email to send, there's no way to see the status of the email from within the application. You need to go to the central database to do that. We could add some admin to the app to enable this, but then every application would be able to see all the emails sent across all applications using the service which a) is a pain and b) potentially not the most secure. We could get around this by adding something like an application_id to the Message model and have each application register itself with a unique id. Then in the admin filter just the application's emails, but I don't love that as it's adding and spreading even more complexity around (see the previous point). How are the ids generated? How is each application registered with the relay service? Etc.
• Version upgrades of the library are non-trivial, requiring coordination across all applications and the relay service. Especially since they are all sharing a database, migrations in particular are a big pain point. Some of this was alleviated by the recent addition of the library's version in the metadata of the messages.

I wrote this library to solve a very specific problem we have at work. We have our applications running where they cannot access our corporate internal SMTP server. Using an external ESP was not feasible as the various spam and security features of our email provider were flagging our applications' emails.

I chose a database queue because I wanted something robust that was (relatively) simple to run. At the time I was not interested in running another service e.g. a message queue like Redis or RabbitMQ. But that may change in the future and adding support for that in this library as it's currently designed would be non-trivial. (YAGNI of course, so this is not the ultimate goal of what I'm proposing, but would be a nice side-effect.)

Proposal

I think instead of each application and the relay service using on central database, instead it should be flipped around the other way. Each application uses its own default database for queuing emails and the relay service connects to each one to get queued emails to send.

When I first had this thought, I dismissed it as a bit silly and me looking for a way to over engineer something (which I am wont to do). But as I've sat with the idea, I do think it solves a lot of the problems with the current state of things:

• It would simplify using the library at the application level. Instead of spreading the complexity around all the applications, it would be centralized in the relay service.
• Each application would have access to their own queue since it would just be in its own database. No need to worry about seeing other application's emails or introducing a way to associate and filter based on the application.
• Version upgrades would no longer need to be so tightly rolled out. We would need to specify which versions of the library we support, and as long as the applications stick to that it should be fine. The relay service would be responsible for handling different versions in the case of any potential changes to the message schema. (Am I seriously proposing something like LTS versions for an emailing library? Yes, I think I am. 🙃 Though in all seriousness sticking to SemVer +-1 would probably be the most straightforward.)
• Using something other than a database queue would be easier to implement, provided the relay service can communicate with the new queue.
• The relay service could be its own full blown Django application, with a fancy dashboard and everything. ✨

Using this approach would make the relay service much more complex than it currently is, so it's not all upside. The service would need to connect to each application's database somehow, so it would need to know where the database is and would need to be accessible from wherever the service is running. It would shift the complexity of the version upgrades from a coordination issue to a logic issue within the service itself. But those tradeoffs may be worth it. 🤷‍♂️

Gathering Feedback

This would be quite a big breaking change from how the library works and I'm not totally sold on it yet, but something needs to change and this is what I've come up with at the moment.

To be honest, I'm not even sure how many if any users actually use this library. According to pypistats.org it does get a non-zero amount of installs, but I have no way how many of those are our own vs automated bot services vs actual users.

I'm opening this issue to gather feedback for this proposal and pinning it. I will more than likely be following through with this, but I still need time to consider all the sides to it. If there are users that have issues with this proposal, this is the place to post them. I'm open to any honest feedback, but since this library is primarily written to solve a very specific problem for the company I work for, I can't promise anything other than I will listen.

[joshuadavidthomas]

After thinking on this, I wonder if both paths can't coexist, at least for a little while. So keep the EmailDatabaseRouter for the applications for now, but expand the relay service to be able to read from other database queues. That would certainly ease the transition from a central queue to each application having it's own queue.

[joshuadavidthomas]

It would make this not a breaking change for now, which would be nice. We could mark the router as deprecated, slated for removal in some future release. It could give people time to migrate, if there are people using this application besides @westerveltco. Selfishly, it would make it easier for me to migrate all of our apps to the new way of handling the email queue.

[jefftriplett]

Let's queue up some time to discuss it next week.

I think instead of each application and the relay service using on central database, instead it should be flipped around the other way. Each application uses its own default database for queuing emails and the relay service connects to each one to get queued emails to send.

"Each application uses its own default database for queuing emails" I thought we were already doing this.

"relay service connects to each one to get queued emails to send." - I personally wouldn't want to do this.

I think we should figure out what is problematic or weird about the existing setup, and ideally, every email should be sent immediately. If there is an issue, having a queued backup with some history is reasonable. We should also think about the volume of email sent per project. Are we looking at 2, 20, 200, or 2000 emails per day, and is that worth keeping constant connections open?

Maybe you want the service behind the firewall to only talk with one service outside of the firewall that everything else can relay through which would cut down the O(n) connection issues on either side?