Archiving and sending to final SMTP server

In today’s digital landscape, where email communication plays a vital role in business operations, Keeping your email archive secure, compliant, and retrievable is crucial for any business. However, managing the large volume of email data can lead to operational difficulties, including regulatory compliance, maintaining an audit trail, and preventing data loss. That’s where Amazon Simple Email Service (SES) Mail Manager’s email archiving feature comes in.

In this blog post, we will explore how Amazon SES Mail Manager’s email archiving and search features can improve email security and compliance. If you’re a newcomer to Mail Manager, look at this blog post on Amazon SES Mail Manager. It provides valuable information on important features, such as Ingress Endpoint, Traffic Policy, Rule Sets, and SMTP Relay.

Problem Statement:

Imagine a scenario where a critical email from a key client is buried deep within your organization’s email archives, and you need to retrieve it for an important audit. The challenge of ensuring your business remains compliant with stringent data retention policies across every email communication for thousands of employees for a certain period or permanently.

Solution explained:

Amazon SES Mail Manager Email archiving is a powerful tool that addresses many of the challenges organizations face dealing with the difficulty and expense of archiving email at scale. Compliance and regulatory frameworks like GDPR, HIPAA, and SOX often require email archiving, which is a common objective identified by customers needing to comply with those regulatory frameworks. For regulated businesses, failure to comply with email archiving regulations can result in severe financial penalties and reputational damage.

Amazon SES Mail Manager securely archives and safeguards your emails, providing easy search and export functionality. It provides full-time, enterprise-level archiving without increasing the storage requirements of your mailbox server. The feature provides a reliable and efficient solution to address most compliance requirements. By automatically archiving the types of emails you specify, the service ensures that your organization maintains a comprehensive audit trail of its communications, enabling quick retrieval and review as needed.

The email archiving feature of Mail Manager provides organizations the ability to archive email while in transit rather than archiving at the user’s mailbox. Many organizations prefer archiving in transit for email archiving to meet compliance requirements and maintain comprehensive records. If you would like to learn more about in transit archiving, visit this blog –  Email Archiving with Mail Manager: Why To Archive In Transit vs At The Mailbox.

How email flows with an Amazon SES Mail Manager Email Archiving

For instructional purposes in this blog post, we’ll focus on how you can introduce Mail Manager archiving into your existing email infrastructure. We’ll cover how to seamlessly integrate Mail Manager with reference architectures. Later in the blog, we are going to explore Mail Manager’s archiving capabilities, including search, export and retention policies.

Current setup

Our example organization has an existing mail server (it might be a on-premises Microsoft Exchange Server, Microsoft 365, Google Workspace, etc). Their DNS is configured to route all email directly to this mail server. There is currently no archiving capacity within the existing email infrastructure, when needed, archiving is handled by individual mailbox users and PST files. While this method is suitable for personal email archiving, it fails to meet the organizations’ security requirements and compliance standards.

Figure 1: Example organization’s existing inbound email email flow.

Email Archiving in transit

Once we introduce Mail Manager into the current mail flow (see figure 1) the organization can readily archive all incoming messages.

Figure 2: Example organization’s proposed inbound email flow, with Mail Manager archiving in-transit prior to delivery.

In the new architecture, we’ve introduced Mail Manager into the organization’s inbound (see figure 2) email flow. This new flow leverages Mail Manager’s ability to archive email from Mail Manager Ingress endpoints. By using a Mail Manager Rule set, our example organization can selectively store and preserve emails that meet their configured criteria.

Mail Manager Email Archiving and Search and Export Capabilities

Mail Manager’s archive search capabilities are designed to be user-friendly and efficient. You can perform searches based on various criteria, such as sender, recipient, subject line, date range, or even specific keywords in the Subject line. The search results provide options to quickly review a single archived email in the console, or export the search results for a range of dates and other criteria to Amazon Simple Storage Service (S3), for review in external tools.

Let’s explore using Mail Manager’s archive search to find a specific email by the sender’s address:

Figure 3: Mail Manager’s archive search interface

From the search results, we can click on the results to review the email in the console:

Figure 4: Mail Manager’s archive search results

Once we’ve found the target email, it can be downloaded by clicking on “View details“. The image below shows an example message details page with information about the email, including message headers such as In-Reply-To, X-Original-Mailer and X-Mailer.

Figure 5: Mail Manager’s archived email detailed page

Mail Manager’s archive search history tab allows us to find archive searches created in the last 30 days, and view the search results, as shown in the image below:

Figure 6: Mail Manager’s archive search history

Mail Manager’s archive export history tab lists all of the archived email searches you exported to an Amazon S3 Bucket within the last 30 days.

Figure 7: Mail Manager’s archive Export History

Step by Step Setup:

Now that we have explained how Mail Manger can be inserted into our example organization’s email workflow to provide email archiving, let’s explore how you can implement Mail Manager’s archiving capabilities in your inbound email workflows.

Figure 8: End-to-End Email Archiving flow

The above diagram (Figure 3) illustrates the overall structure and components involved in this architecture and mail flow:

1. Mail senders look up the Mail Manager Ingress endpoint in DNS
2. The Ingress endpoint can be authenticated or open. Authenticated Ingress endpoints require senders to authenticate. Open Ingress endpoints do not require authentication and are generally used for accepting inbound mail by setting the domain’s MX record value to the Ingress endpoint hostname.
3. Traffic policies control which messages are allowed or rejected
4. Rule Set conditions control which actions are performed. These conditions control which messages trigger the Email Archive action.
5. Optionally, additional Rule Set actions can be performed, such as the SMTP Relay action to deliver email to a 3rd party SMTP server

Follow the steps below to configure Mail Manager in your AWS account to implement this architecture:

1. Log into the SES Console and select Mal Manager from the left navigation menu.
   1. Note, as of this writing, Mail Manager is generally available in the following AWS Regions: US East (N. Virginia), US West (Oregon), Europe (Ireland, Frankfurt), Asia Pacific (Tokyo, Sydney).
2. Under Mail Manger navigation, create an archive (or multiple archives for different use cases)
   1. Enter a unique name in the Archive name field.
   2. (Optional) Select a retention period in the Retention period field to override the default retention period of 180 days.
   3. (Optional) You can encrypt your archive either by entering your own AWS KMS key into the KMS key ARN field, or by selecting Create new key.
   4. Choose Create archive.
3. Under Mail Manger navigation, create a Traffic policy to determine the email you want to block or allow.
   1. Create Traffic policy.
   2. On the Create a traffic policy page, enter a unique name for your traffic policy.
   3. (Optional) If you want to discard any messages above a certain size, enter a value in bytes in the Maximum message size field.
   4. In Default action, choose whether the Traffic policy is to either Allow or Deny (block) messages that fall outside of (are not addressed by) the conditions of your policy statements.
   5. Select Add new policy statement to create a statement for your traffic policy.
   6. Choose either Allow or Deny (block) for the action to be taken when the statement’s conditions are met.
   7. Build a Rule condition by selecting an email protocol and a conditional operator for the value you enter.
      1. Select Add new condition if you want to add more conditions to this policy statement. To learn more about a condition property and its operators and valid values, see the Policy statement conditions reference.
   8. If you’re subscribed to an Email Add On, you’ll be able to select it here as an email protocol.
      1. If you want add more policy statements and conditions, repeat steps above.
   9. Select Create traffic policy.
4. Under Mail Manger navigation, create a rule set to perform actions on the email you allow in.
   1. Create rule set and enter a unique name for your rule set.
   2. Create new rule on the edit page.
   3. In the Rule details sidebar, enter a unique name for your rule.
   4. Select Add new condition to create a condition that the message must match; or check the EXCEPT in the case of: box followed by Add new exception to create a condition that the message must not match.
   5. Build the condition or exception by selecting an email property and a conditional operator for the value you enter.
      1. Select Add new condition or Add new exception if you want to add more conditions or exceptions to this rule. To learn more about a condition property and its operators and valid values, see the Rule conditions reference.
   6. Select Add new action to define the action to be taken when the rule’s conditions are matched and/or exceptions are not matched.
      1. To add more actions to be taken, select Add new action. To learn more about actions and their parameters, see the Rule actions reference.
   7. Create an Archive rule. Save rule set
5. Under Mail Manger navigation, Create your ingress endpoint and assign to it the traffic policy and rule set.
   1. Choose Ingress endpoints under Mail Manager.
   2. On the Ingress endpoints page, select Create ingress endpoint.
   3. On the Create new ingress endpoint page, enter a unique name for your ingress endpoint.
      1. Choose whether the ingress endpoint will be an Open or Authenticated endpoint.
   4. Select a Traffic policy to determine the email you want to block or allow.
   5. Select a Rule set containing the rule actions you want to perform on the email you allow in.
   6. Select Create ingress endpoint.
6. Configure your email environment to use the ingress endpoint.
   1. At the time you create an ingress endpoint, an “A” record for the endpoint will be generated and its value displayed on the ingress endpoint’s summary screen in the SES console. The way you use the value of this record depends on the type of endpoint you created and your use case.
   2. DNS providers have different procedures and interfaces for configuring email records. The key pieces of information you need to put into your DNS settings are listed in our documentation – https://docs.aws.amazon.com/ses/latest/dg/eb-ingress.html#eb-ingress-a-record
7. Under Mail Manger navigation, create an SMTP Relay to send mail on to your existing mail server.
   1. Choose SMTP relays under Mail Manager.
   2. On the SMTP relays page, select Create SMTP relay.
   3. On the Create SMTP relay page, enter a unique name for your SMTP relay.
   4. Depending on what type of SMTP Relay you want to configure, follow the respective instructions:
      1. inbound (non-authenticated)
      2. outbound (authenticated) SMTP relay
8. Update your DNS MX records to point to your new Mail Manager’s ingress point, instead of the existing mail server.

Note: Make sure that you have tested the steps above in your development environment and that you understand the steps before deploying into your production environment.

Conclusion

Amazon SES Mail Manager’s email archiving capabilities are designed for organizations that are seeking to enhance the security, compliance, and audit-ability of their email communications. By seamlessly integrating this feature into their existing email infrastructure, organizations can now archive all inbound messages in transit, ensuring a comprehensive, tamper-proof record of their email activities. The powerful search and export functionality of Mail Manager makes it easy to quickly locate and access specific emails when needed, whether for compliance audits, legal requests, or internal investigations.

This level of email visibility and control is particularly crucial for organizations operating in highly regulated industries like government, healthcare and finance, where the stakes for non-compliance can be severe. Beyond the compliance benefits, Mail Manager’s email archiving also helps to alleviate the operational headaches and expenses associated with traditional in-house archiving systems. By offloading this responsibility to AWS, organizations can focus their resources on their core business priorities, while still maintaining the security and accessibility of their critical email data.

If you’re looking to strengthen your email security posture, simplify your compliance efforts, and improve the overall management of your email archives, we encourage you to explore how Amazon SES Mail Manager’s email archiving capabilities can be seamlessly integrated into your existing email infrastructure. Take the first step towards a more secure, compliant, and efficient email management solution by contacting us today.

About the Authors

Sesan Komaiya

Sesan is a Solutions Architect at Amazon Web Services. He works with a variety of customers, helping them with cloud adoption, cost optimization and emerging technologies. Sesan has over 15 year’s experience in Enterprise IT and has been at AWS for 5 years. In his free time, Sesan enjoys watching various sporting activities like Soccer, Tennis and Moto sport. He has 2 kids that also keeps him busy at home.

Alexey Kurbatsky

Alexey is a Senior Software Development Engineer at AWS, specializing in building distributed and scalable services. Outside of work, he enjoys exploring nature thru hiking as well as playing guitar.

Jesse Thompson is an Email Deliverability Manager with the Amazon Simple Email Service team. His background is in enterprise development and operations, with a focus on email abuse mitigation and encouragement of authenticity practices with open standard protocols. Jesse’s favorite activity outside of technology is recreational curling.

Zip

Zip is a Sr. Specialist Solutions Architect at AWS, working with Amazon Pinpoint and Simple Email Service and WorkMail. Outside of work he enjoys time with his family, cooking, mountain biking, boating, learning and beach plogging.