Automating HTTP/S Redirects and certificate management at scale

Organizations today use many ways to drive traffic to their websites and applications. This is important for new feature launches, marketing campaigns, advertising, and so on. One common approach uses HTTP/S redirects, where you send a user from one domain, or Uniform Resource Locator (URL), to another. Redirects are incredibly useful tools when moving websites, setting up temporary microsites, adding a new page to a website, or performing maintenance. In this post, we show how to build a scalable and cost-effective solution to handle HTTP/S redirects at scale for multiple domains by using Application Load Balancer (ALB) to perform rule-based redirects, AWS Global Accelerator to get static IP addresses, and AWS Lambda to host redirect logic on serverless infrastructure. This method simplifies deployments while benefiting from the scale, availability, and reliability of AWS infrastructure and services.

Let’s take an example. You want to run a marketing campaign for a new product that doesn’t have a standalone website. You purchase a new domain for the product campaign, set up website infrastructure for this new domain, and then use the domain URL to send your end users to that new campaign page. This approach means that if you’re running marketing campaigns for multiple new product launches, then you may have to buy domains and setup website infrastructure for each of them, which is a costly, time-consuming, and non-scalable process. Alternatively, if you have a main website for your organization, then you can host the new products on different webpages within the main website and simply redirect users from the individual products’ domains to their respective webpages within the main website using HTTP/S redirects.

Choosing the second option lets you take advantage of existing infrastructure for your new product launches while reducing the load on your developers so they can focus on building amazing products.

Before diving into the solution walkthrough, let’s understand redirects and their use-cases.

Redirect types

In this section, we discuss a few common types of redirects and how they work.

DNS redirects:

This is a method of redirecting from one hostname to another through a DNS server during the DNS resolution process. This pattern is also known as a domain redirect. It’s the simplest form of redirection used by hosting companies and domain name registrars. As shown in figure 1, the client sends a DNS query for mail.example.com in step 1. This query is redirected to another domain mailserver-1.example.com in step 2. The client then sends a new DNS query for mailserver-1.example.com in step 3. The DNS server responds with an IP address (A record) in step 4.

Figure 1: DNS redirection

However, if you are looking for redirecting one domain to another at a webserver level (display the destination domain’s webpage), then DNS redirects is insufficient. You also need a redirect at the webserver level (also known as HTTP redirect) to accomplish this. We discuss HTTP redirects in the following section.

HTTP redirects:

This method lets you redirect your end users from one URL to another through a webserver after the DNS resolution process. This is also known as URL Forwarding. The concept of HTTP redirects is based on the status codes in the HTTP protocol.

There are two types of HTTP redirection, permanent and temporary, under the Redirection messages category with 3XX codes.

1. 301 redirects (Moved Permanently)

   In this type, you setup a permanent redirect where you move the website URL to another address. As shown in figure 2, the client sends a request for the www.example.com/old_url/ in step 1. The client gets an IP address for the web server in the DNS response in step 2. In step 3, the client sends a HTTP request for www.example.com/old_url/ to the web server. The web server responds back with HTTP 301 (Moved Permanently) pointing to a new URL www.example.com/new_url/ in step 4.

   

   Figure 2: HTTP 301 permanent redirection

2. 302 redirects (Found)

   In this type, you setup a temporary redirect where you move the website URL to another address temporarily, but you plan to restore the original URL. As shown in Figure 3, the client sends a request for www.example.com/current_url/ in step 1. The client gets an IP address for the web server in the DNS response in step 2. In step 3, the client sends a HTTP request for www.example.com/current_url/ to the web server. The web server returns a temporary redirect using HTTP 302 pointing to the new location www.example.com/temporary_url/ in step 4.

   

   Figure 3: HTTP 302 temporary redirection

It’s important that you choose the right type of redirect for your use-case.

Use case and walkthrough

Let’s go back to the marketing campaign example from the beginning of this post. Your organization registered under the main domain name example.com. You buy/acquire individual domains for each product that you sell, such as <product1>.com for Product 1, <product2>.com for Product 2, and so on. You register these domains via Amazon Route 53, or any other third-party domain registrar, and then configure these domains on individual servers to host the website for each product. You want to host these domains securely using TLS/SSL certificates. In this situation, to save on costs and time while maintaining security, you might want to redirect users who are trying to access your new domains to webpages within the main landing page of your organization. We show this in the following table and figure:

Figure 4: HTTP redirection at scale

After successful DNS resolution, the client sends an HTTP request for <product1>.com to the web server in step 1 of figure 4. In step 2, the webserver redirects the client to a specific path on their main web page example.com/product1. Similarly, this happens for other products like <product2>.com, <product3>.com, etc.

Existing solutions

There are two existing AWS solutions that can help address these challenges.

Solution 1: Amazon S3 Static Website Hosting

This solution lets you redirect from a root apex domain, such as <product1>.com, to another domain, such as <product1>.example.com. You must create an Amazon Simple Storage Service (Amazon S3) bucket with the same name as the domain to which you are trying to redirect your users, such as <product1>.example.com (static website). The end users access the static website using the Amazon S3 website endpoint. If you have a domain registered with Route 53 (acting as your DNS provider), then you can create a CNAME record that redirects www.<product1>.com to <product1>.example.com.

However, if you’re looking to redirect your apex domain <product1>.com to another domain, such as <product1>.example.com, then you can’t create a CNAME record as per RFC1035. Instead, you must use an Alias record within Route 53 to redirect the apex zone (there is no charge for DNS queries for Alias records). Therefore, in this solution, you first create an Amazon S3 bucket by the name <product1>.example.com, and then you create an Alias record for <product1>.com within Route 53 to redirect users to the Amazon S3 endpoint hosting the static website <product1>.example.com.

Pros:

• Best suited for static content.
• You can access your website using HTTP only.

Cons:

• End users can’t connect to Amazon S3 over HTTPS.
• Doesn’t scale well, as you must create a separate Amazon S3 bucket for each new domain.
• This solution doesn’t provide a static IP address, and thus isn’t suited for apex (or root) domain forwarding when you’re not using Route 53.

Solution 2: Amazon CloudFront

As we saw in the previous solution, you can only connect to static websites hosted on Amazon S3 through HTTP. To overcome this limitation and serve a static website hosted on Amazon S3 over HTTPS, you could use Amazon CloudFront. You must request an SSL certificate for the domain using AWS Certificate Manager (ACM) and associate it with your CloudFront distribution. For example, you can request an SSL certificate for <product1>.example.com and associate it with your CloudFront distribution. Then, you can enforce your users to use HTTPS to access the static website hosted on an Amazon S3 bucket through the CloudFront distribution.

Pros:

• You can access the website using HTTPS.
• Hosts up to 100 domains in the Alternate Domain Names (CNAMEs) field in each CloudFront distribution and configure separate forwarding rules using CloudFront functions.
• CloudFront provides caching capabilities for content, which reduces the load on your origin server and thereby reduces latency

Cons:

• You can associate a maximum of one SSL/TLS certificate with each CloudFront distribution. Therefore, you can only host up to 100 domains on each distribution using a Subject Alternative Name (SAN) certificate that can have up to 100 domain names per certificate.
• This solution doesn’t provide a static IP address, and thus isn’t suited for apex (or root) domain forwarding when you’re not using Route 53.

We’ve heard from our customers that they must redirect hundreds or even thousands of domains. These domains could use any DNS provider (either Route 53 or other third-party DNS providers). As we saw, the existing solutions don’t scale to hundreds and thousands of domains. In the next sections, we show you how to automate the redirects at scale.

High-level architecture/solution overview

You can redirect hundreds of their domains (websites) from old URLs to new URLs, with making minimal changes to your DNS. The solution proposed in this blog works regardless of which DNS provider (Route 53 or other third-party DNS providers) you are using to manage your domains.

Global Accelerator helps you improve the availability, performance, and security of your public applications. It provides two static public IPs that are used as points of entry for all HTTPS requests that must be redirected. The end user makes an HTTPS request for the web URL with the network layer destination IP of that of the Global Accelerator. Once the packet reaches the Global Accelerator, it’s forwarded to an ALB, which acts as its target Endpoint.

ALB is a type of load balancer that operates at the request level (layer 7), routing traffic to targets (EC2 instances, containers, IP addresses, and Lambda functions) based on the content of the request. An ALB is used to load balance HTTP/S requests, and it integrates with ACM to handle SSL/TLS certificate offloading for HTTPS requests. It can send traffic to a Lambda function by registering the function as a target of ALB.

Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers. We use a Lambda function here to parse HTTP requests for the URL being queried, which subsequently queries an Amazon DynamoDB table that has the mapping of all old URLs to new URLs. Then, the Lambda function inserts this new URL in an HTTP 301 response and sends it back to the client.

How it works

Figure 5: High-level architecture and packet flow

Here is the flow of events as shown in figure 5:

Step 1: HTTPS request from the user enters the AWS network through Global Accelerator at the nearest edge location to the user.
Step 2: Global Accelerator forwards the HTTPS request to ALB registered as an endpoint to the Global Accelerator.
Step 3: ALB authenticates the request against the SSL/TLS certificates managed by ACM and passes the query to the Lambda function.
Step 4: Lambda code is invoked by the query. In turn, the Lambda function queries the DynamoDB table for the corresponding new URL.
Step 5: DynamoDB sends a response to Lambda with the redirect URL.
Step 6: The Lambda function sends an HTTP 301 Permanent Redirect response back to the ALB.
Step 7: The ALB forwards the HTTP 301 Permanent Redirect response to the Global Accelerator.
Step 8: The Global Accelerator responds to the user with the HTTP 301 permanent redirect response.

At this point, the user reissues a new HTTPS query with the new URL back to its local DNS server. The local DNS server resolves to the new URL IP, and the user can see the new URL webpage.

Solution deployment and testing

Prerequisites

Before you set up this solution, you must:

1. Create a VPC with an Internet Gateway attached. Then, create two public subnets and two private subnets in two Availability Zones (AZs) in any AWS Region (such as us-west-1). Create appropriate route tables for the subnets and configure security groups and NACLs. Refer to the VPC with public and private subnets (NAT) page in our technical documentation for detailed steps.
2. Request public-certificates from ACM for the domains that you would like to redirect, or import third-party issued TLS/SSL certificates into ACM.
3. Set up a Gateway VPC endpoint for Amazon DynamoDB in the VPC and update the route tables of the two private subnets.

Solution deployment steps

Step 1: Create a DynamoDB table and create Items in it that have mappings of all the old URLs to the new URLs.

Figure 6: DynamoDB Table configuration

Step 2: Create a Lambda Function and associate it with the private subnets of the VPC that you created in the prerequisites.

Figure 7: Lambda code

Figure 8: Lambda VPC configuration

The Lambda function parses the HTTP request for the URL being queried, looks-up the DynamoDB table, and returns a response with the redirect URL. Here is a sample using JSON code for the Lambda Function:

import json
import boto3
from boto3.dynamodb.conditions import Key

def lambda_handler (event, context):   

    print (json.dumps (event, indent=4))
    print (event ['headers']['host'])
    client = boto3.resource ('dynamodb')
    table = client.Table ('BlogTable')
    OriginalURL = table.get_item (Key = {'URL_Query' : event ['headers']['host']})
    print (OriginalURL ['Item']['URL_Response'])
    response = {}
    response ["statusCode"] = 301
    response ["headers"] = {'Location' : 'https://'+OriginalURL['Item']['URL_Response']}
    data = {}
    response ["body"] = json.dumps (data)
    return response

Step 3: Create a target group with Lambda type and register the Lambda function with it.

Figure 9: ALB Target Group configuration

Step 4: Create an ALB with the following configuration:

• Choose the public subnets of the VPC that you created in the prerequisites
  

  Figure 10: Application Load Balancer network configuration

• Choose HTTPS listener on port 443, and use the certificates that you created or imported into ACM in the prerequisites.
  

  Figure 11: ALB Listeners configuration

  

  Figure 12: Application Load Balancer certificates configuration

• Choose your Lambda-based target group that you created in Step 3.
  

  Figure 13: ALB Listener rules configuration

Step 5: Configure a standard Global Accelerator with TCP 443 as the listener and the ALB that you created in Step 4 as the endpoint.

Figure 14: AWS Global Accelerator listener configuration

Figure 15: AWS Global Accelerator endpoint configuration

Step 6: If you’re using Route 53 as your DNS provider, then create a Public Hosted Zone for the domain that you’re looking at redirecting. Then, create an ALIAS record for the Public Hosted Zone and point it to your Global Accelerator’s DNS name.

Figure 16: Route 53 configuration

Testing

After completing these steps, you’re ready to test the solution. You can use a simple CURL command to verify the domain redirection, as shown here:

curl -I https://product1.com

HTTP/2 301 
server: awselb/2.0
date: Sun, 12 Feb 2023 19:51:16 GMT
content-type: application/octet-stream
content-length: 2
location: https://example.com/product1

Optional: Automating the addition of domains to SSL certificates

To make this solution scalable, we must automate the addition of new domains to existing certificates, or add new individual certificates for the new domains. This is achieved using a Bash script that:

1. Checks if there is an existing certificate to which the domain can be added. If yes, then it adds the domain to the certificate.
2. If no certificates are found, then it adds a new certificate and the domain to it.

Here’s what this looks like in code:

#!/bin/bash

carn=$(aws acm list-certificates | jq -r --arg domain "$1" '.CertificateSummaryList[] | select(.DomainName==$domain) | .CertificateArn' | tr -d '"')
san=$(aws acm describe-certificate --certificate-arn $carn | jq '.Certificate.SubjectAlternativeNames[]' | tr -d '"')
new_san_arn=$(aws acm request-certificate --domain-name $1 --validation-method DNS --subject-alternative-names $san $2 | jq '.CertificateArn' | tr -d '"')
echo "Certificate Validation in progress..."

aws acm wait certificate-validated --certificate-arn $new_san_arn
echo "Certificate Issued !"

arn=$(aws elbv2 describe-load-balancers | jq -r --arg myalb "$3" '.LoadBalancers[] | select(.LoadBalancerName==$myalb) | .LoadBalancerArn' | tr -d '"')
larn=$(aws elbv2 describe-listeners --load-balancer-arn $arn | jq '.Listeners[].ListenerArn' | tr -d '"')
echo "Adding New Certificate"

aws elbv2 add-listener-certificates --listener-arn $larn --certificates CertificateArn=$new_san_arn
sleep 5s

certarn=$(aws elbv2 describe-listener-certificates --listener-arn $larn | jq '.Certificates[].CertificateArn'| tr -d '"')

for i in $certarn; do 
        if [ $i = $new_san_arn ]
        then 
                echo "Deleting Old Certificate"
                aws elbv2 remove-listener-certificates --listener-arn $larn --certificates CertificateArn=$carn
        fi
 
done

certarn=$(aws elbv2 describe-listener-certificates --listener-arn $larn | jq '.Certificates[].CertificateArn'| tr -d '"')

for i in $certarn; do 
        if [ $i = $carn ]
        then 
                echo "Error in Deleting Old Certificate"
                exit 0
        fi
 
done

echo "Process Completed"

Other considerations

1. Multi-Region deployment: For multi-Region deployments, the previous solution can be expanded as follows:
   1. You can add ALBs in various Regions as endpoint groups for your Global Accelerator to improve the performance for your end users.
   2. Use Amazon DynamoDB global tables to make your service global. This will allow database changes made in one Region to reflect across all Regions, and thus maintain database consistency across Regions.
2. Pointing your domains from third-party DNS providers to AWS resources: If you use a DNS provider other than Route 53 for your domain name, then you might be able to point your domain to services hosted on AWS. Check if your DNS provider supports CNAME records on a root domain using techniques such as CNAME flattening, ANAME records, or WebHops redirection. For the domain that you’re trying to redirect, you can either create a CNAME record and point it to your Global Accelerator’s DNS name, or create an A record and point it to the static IP addresses of your Global Accelerator.
3. Scaling: The solution described in this post scales well with the number of domains and SSL/TLS certificates quotas that we have in place. Refer to the documentation for the respective services to learn more about their service quotas. The default limit is currently 25 certificates on ALB, and the default limit for the number of domains that the ACM provided certificate can contain is currently 10 (using SAN). As of this writing, you can have up to 100 domains per ACM-issued certificate. This would let you host up to 2500 hostnames using a single ALB without including wildcards. To scale this further, you need to add additional ALBs.
4. Caching: This solution involving Global Accelerator doesn’t provide caching capabilities.

Cleanup

You have now successfully deployed and tested the solution. To avoid ongoing charges for the resources that you created to implement this solution, delete:

• Lambda Functions
• DynamoDB Tables
• ALB
• Target Group(s)
• Accelerators in Global Accelerator
• Route 53 Public Hosted Zones and their records
• Certificates in ACM

Conclusion

In this post, we showed how you can perform multiple domain redirects in a cost effective and scalable fashion with minimal changes to your DNS configuration by leveraging Global Accelerator, ALB, and Lambda. We also showed the automation of SSL/TLS certificate management for multiple domains using ALB and ACM.