Businesses rely on transactional emails for everything from order confirmations to one-time passwords. They plan an important role in many workflows. And for customers, there are few things as frustrating as not being able to create an account or reset your password because you haven’t received an email yet. Both sides expect speed and reliability. So, we set out to determine which email delivery platform was fastest. 

In this post, we’ll show you how we used Multiple’s load testing platform to evaluate email delivery times and API response times for five popular email delivery platforms: Amazon SES, MailChimp, Mailgun, Postmark, and Twilio SendGrid. Most importantly, we’ll crown a champion.

Testing Methodology

To determine who has the fastest transactional email API, first we had to determine the metrics that each provider would be evaluated on. We decided on email delivery time and API response time.

• Email delivery time: The number of milliseconds from API call to receiving an email.
• API response time: The number of milliseconds from API call to response.

Initially, we tried measuring email delivery time using IMAP. However, we encountered two issues: timestamps only had second-resolution, and the server’s clock was out of sync with UTC by several seconds. Both impacted the accuracy of the test, so we created an SMTP server that inserted the email subject and timestamp into a Redis database whenever a message was received. Each email had a unique email so that we could track it.

For each provider, we emulated 20 virtual users (VUs) concurrently making API calls for two minutes. All tests were run on Multiple’s load test platform at approximately the same time from the same location to reduce variance.

Example Code

NPM Dependencies

"dependencies": {
   "mailgun.js": "9.2.1",
   "form-data": "4.0.0",
   "uuid": "9.0.0",
   "ioredis": "5.3.2"
}

Require Statements

const formData = require('form-data');
const IORedis = require('ioredis');
const Mailgun = require('mailgun.js');
const uuid = require('uuid');

Environment Variables

MAILGUN_API_KEY=YOUR_API_KEY
REDIS_IP_ADDRESS=YOUR_REDIS_IP
EMAIL_DOMAIN=YOUR_DOMAIN
FROM_EMAIL=YOUR_FROM_EMAIL
TO_EMAIL=YOUR_TO_EMAIL

VU Init

async function vuInit(ctx) {
  const mailgun = new Mailgun(formData);
  const mg = mailgun.client({ username: 'api', key: ctx.env.MAILGUN_API_KEY });
  const redis = new IORedis(6379, ctx.env.REDIS_IP_ADDRESS);

  const checkInboxForMessageSubject = async subject => {
    const receivedAt = await redis.get(subject);
    return receivedAt ? parseInt(receivedAt, 10) : null;
  };

  return { mg, redis, checkInboxForMessageSubject };
}

VU Loop

async function vuLoop(ctx) {
  const { mg, checkInboxForMessageSubject } = ctx.vuInitData;
  const emailSubject = `Eamil ID: ${uuid.v4()}`;
  const emailSendTime = Date.now();
  await mg.messages.create(ctx.env.EMAIL_DOMAIN, {
    from: ctx.env.FROM_EMAIL,
    to: ctx.env.TO_EMAIL,
    subject: emailSubject,
    text: 'Running API Benchmarks with Multiple!',
    html: '
Running API Benchmarks with Multiple!
',
  });
  await new Promise(resolve => setTimeout(resolve, 5000));

  ctx.metric('Send Email API', Date.now() - emailSendTime, 'ms (API)');

  while (true) {
    const emailReceivedTimestamp = await checkInboxForMessageSubject(
      emailSubject,
    );

    if (emailReceivedTimestamp) {
      ctx.metric(
        'Receive Email',
        emailReceivedTimestamp - emailSendTime,
        'ms (EMAIL)',
      );
      break;
    }

    if (Date.now() - emailSendTime > 30000) {
      throw new Error('Email not received in 30 seconds');
    }

    await new Promise(resolve => setTimeout(resolve, 500));
  }
}

VU Cleanup

async function vuCleanup(ctx) {
  const { redis } = ctx.vuInitData;
  redis.disconnect();
}

Performance

Before we reveal the final results, let’s take a look at performance. The graph below is from our test of Twilio SendGrid.

Let’s walk through the results. There are a few things you’ll want to keep an eye on. 

1. Email delivery time: It's important to determine if the email delivery time is acceptable for your use case. While longer delivery times might be acceptable for marketing content, they are less so for one-time passwords.
2. API response time: The graph suggests that the Send Email API has response times around 5,000 ms. As a result, it makes sense to call this API asynchronously, potentially using a queue pattern.
3. Consistency: The graph shows a nearly flat line for API response times, indicating consistent performance. This is a good sign, you don’t want an unpredictable service. However, it's important to mention that the duration of this test was only two minutes. Performance may vary at different times of the day.

Results

Now it’s time to answer the question: who’s got the fastest transactional email API? 

Email Delivery Time Rankings

1. Mailchimp (570 ms)
2. Amazon SES (737 ms)
3. Postmark (1,118 ms)
4. Twilio SendGrid (1,404 ms)
5. Mailgun (2,697 ms)

API Response Time Rankings

1. Twilio SendGrid (5,091 ms)
2. Mailgun (5,118 ms)
3. Amazon SES (5,218 ms)
4. Postmark (5,263 ms)
5. Mailchimp (5,331 ms)

Parting Thoughts

Based on our tests, Mailchimp had the fastest email delivery time with AWS SES not far behind. The slowest performance was measured for Mailgun, with email delivery times roughly 4.5x slower than Mailchimp. API response times were all pretty close with Twilio SendGrid beating out Mailgun by 27ms.

However, speed shouldn’t be the only consideration when choosing an email provider. Support, reliability, features, and pricing should all factor into your decision. We hope that this benchmark study helps you to make an informed decision when choosing an email delivery platform. Try it out yourself with the samples in this post – we’re curious to see your results.

Is there another service or category you want us to benchmark? Let us know: hello@multiple.dev.