C#, .Net and Azure

Email automation

2020/05/16

If you’ve followed my blog then you know I’m a fan of automation and the programming model Azure Functions enable.

Some of my personal automation is built around emails, and SendGrid’s Inbound Parse feature is a real enabler when it comes to automating emails.

The fact that I can host the automation for pennies a month using a SaaS service that I don’t need to update/patch is just the cherry on top.

So far, I have used Inbound Parse to:

Overall, I have been really happy with all of these systems, except for the fact that I have to set each up on a custom (sub)domain.

The reason for that is that SendGrid only allows one webhook per domain so I ended up having to host each webhooks on a separate domain (marcstan.net, bugs.marcstan.net, etc.).

This makes sense from their perspective because they do retries for up to 72 hours in case of failures and with multiple webhooks guaranteeing exactly once delivery is impossible to achieve.

I recently added two more services that deal with emails and instead of having to setup more subdomains and SendGrid records I wanted all of my functions to run on a single domain.

Because each system really only needs access to 4-5 parts of the emails (from, to, subject, body and maybe attachments) I decided to build an abstraction that:

Conceptually this system needed to process the incoming emails, determine which webhooks to call (filtering) and invoke the webhooks (forwarding) while also taking care of retries.

Email fanout

Of course I built it as another Azure function that I open sourced on Github as well.

Using a rule engine (much like a regular inbox) you can decide which emails should be processed by which action.

email fanout

Example action processing of the fanout system

In the example above the fanout system first determines which targets to call (based on the rules & filters) and then executes each action.

The Archive action writes emails into a storage account (as sort of a poor mans email inbox backup).

The second action forwards newsletters to another webhook which then posts them to a matrix room.

And the third action forwards emails to a private inbox.

Of course, these are just examples. I implemented various actions that can be configured & customized.

Rules, filters & actions

In the system a rule is defined as zero or more filters and one or more actions.

With no filter, each email is simply sent to the respective actions whereas using filters (similar to mailbox filters) allows forwarding certain emails to specific actions. Read more on filters on Github.

Actions then allow to perform specific tasks.

So far, I have built 4 actions:

The Webhook action is especially helpful because targets now no longer need to parse the (complicated) Inbound Parse format but can instead rely on a simplified model that provides them with all the essential information of the email. The Webhook action even allows minor modifications (e.g. drop attachments & body to only forward the subject/sender to the webhook).

The actions are documented on github as well.

Retries & delivery guarantees

Because each target may fail individually, I needed to add support for retries and keep track of the success status.

As mentioned earlier exactly once delivery is not really possible in a distributed system so I opted for at least once delivery (accepting that webhooks might be triggered multiple times).

I also decided to go the simplest route possible for retries: reusing the retry mechanism that SendGrid already offers.

For each received email I evaluate all the configured rules and determine which targets should receive a message based on their respective filters.

Each target is then called in parallel and marked as successful/failed in a status table. If > 0 actions failed I respond with (400) Bad Request to SendGrid which triggers their retry mechanism.

On the next SendGrid retry the fanout system only re-executes the actions that weren’t successful before (based on the status table). This behaviour continues until either all actions succeed (in which case (200) OK is sent to SendGrid and they stop retrying) or SendGrid stops retrying after 72 hours at which point the email is simply not delivered to all targets.

In theory each target should receive the notification exactly once as I mark each action as successful as soon as it completes. Only once all actions are processed will SendGrid receive the overall success/failure notification. Future retries should thus have the previous success/failure state available in the status table and react accordingly.

In practice I have received notifications exactly once when all targets succeeded on first try and when one or more targets failed, I have received notifications mostly once and rarely twice (so far I had two duplicate notifications across ~100 notifications).

Personally, I am okay with rare duplicate messages as I’d rather have duplicate than lost messages.

From vs. Reply-To

Interesting sidenote: I didn’t know about this feature of the email spec until I accidently hit reply on a newsletter recently. To my surprise the To field was filled with a different email than the newsletter was sent from (support@example.com vs. noreply@example.com).

Turns out an email sender can set the Reply-To header to another address and when a recipient hits reply the alternate address is used as the target. Neat!

Based on this new knowledge I decided to add the Email action type to allow responding directly from a private inbox (which lets me retire my email-relay).

Email action

With my email-relay it is possible to receive emails from a custom domain in a private inbox (Gmail, Outlook, ..) and to directly reply to them from the inbox but have the email be sent from the domain (without needing to purchase a mail package).

The reply feature was a bit clunky to use because you basically responded to your own domain (with the original sender address in the subject), the email-relay would then use Inbound parse to detect that

A) It’s you (the owner of the domain) sending from your registered private email B) It is a reply to an existing email C) the address the email was sent to should be used as the sender D) The actual recipient is encoded in the subject

and would then parse the recipient address from the subject and send a new email (from the domain, to the recipient, stripping out your private email address).

Since I barely used the feature (and if I did use it it just felt strange to have the actual target email in the subject) and armed with my new knowledge about the Reply-To feature I decided to build a different type of action directly into the fanout system that is much simpler:

When someone sends me an email to my domain his email is still repackaged and sent from my domain to my private email but now with the original sender in the Reply-To field.

When I now hit reply from my private inbox the original sender is injected in the to field, allowing me to reply directly to the sender. More details here.

This feels much more natural (and less complicated).

Just beware that the email is now actually sent from the private inbox instead of the domain - with the email relay your private email was never exposed as the actual email was always sent from the domain (using SendGrid).

However, my personal email is all over my git commits anyway so there really isn’t any necessity to hide it when someone sends me an email to the domain.

Still, some people prefer sending emails from a custom domain - in which case it is still possible to hook-up the email relay behind the email fanout (using Forward action) - or purchase a proper mail package at some domain host. ;)


For me personally I now have a solution that

You can find the source on Github.