What‘s new in Cloud Foundations network module: security groups, Route 53 Resolver, Transit Gateway connect attachment, Direct Connect gateway, combined inspection of north-south and east-west traffic, centralized inspection of ingress traffic, network deployment in non-main regions

The Chinese version^[1] of this blog post is published at the same time.

As Cloud Foundations (CF) is delivered to more and more customers, with respect to network deployments which require high flexibility, we are consistently enhancing network modules or developing new functions according to the various actual needs of customers, as a result of which they better meet most common and general network construction requirements. This article summarizes the recently added network construction features and provides a systematic introduction. You can read related articles to learn about the Cloud Foundations network module’s two sharing modes^[1], network traffic inspection^[2], and multiple networks construction^[3], etc., however we won’t go into detail in this article.

During the project delivery process, we found that Cloud Foundations previous network module was a good solution to common network construction problems such as infrastructural network construction, basic network traffic inspection, and multi-regional multi-account network connectivity. However, support is still incomplete or even lacking in the following areas for Amazon Web Services (AWS) resources. There is no support for security groups and their rules in the Amazon VPC (VPC); there is no support for connect attachment in the Amazon Transit Gateway (TGW); in terms of traffic inspection with gateway load balancer (GWLB), it provides separated inspection architecture for north-south and east-west using two VPCs, but lacks combined inspection architecture using a single VPC; there is insufficient support for Internet gateway (IGW) route table in centralized inspection for ingress traffic; there is no support for Amazon Route 53 Resolver, Amazon Direct Connect (DX) gateway, and the need to deploy networks only in regions other than the main region. Here’s a breakdown of the updates.

Security groups and their rules

Security groups are an essential part of building a secure network. This update allows the network module to define and deploy security groups and their ingress and egress rules in a standardized, convenient, automated, and batch manner to help you build a more secure network. In the network definition, we add groups property in vpc property to define security groups, the type is mapping. Security group properties:

Ingress and egress rule attributes:

Predefined protocols and their default ports:

Where:

1. If the protocol is *, then there is no need to define a port;
2. If the protocol is icmp, then the type is 8 for echo request;
3. If the protocol is tcp, udp, then you need to define the start and end port range. For a single port, you only need to set the first item in the array;
4. If the protocol is one of the those with a default port and no port is defined, then the default port is used.

It is worth mentioning that the properties and predefined values for the above security groups and their rules are the same with those defined by the Cloud Foundations Product Factory^[4]. Currently, the system configures a default security group for each VPC, which allows ingress access from the common traffic and the security group itself, and egress access to the world.

Examples

The following are examples of security group definitions for security group rules for different use cases.

1. Web server rules: In the following definition, * is for all traffic0.0.0/0, and the predefined protocol and its default port are used to avoid hardcoded ports;

   {
     "vpcs": {
       "security": {
         "cidr": "10.1.0.0/16",
         "groups": {
           "web": {
             "in": [
               { "cidr": "*", "protocol": "http" },
               { "cidr": "*", "protocol": "https" }]
           }
       }}
   }}
   

   JavaScript
2. Database server rules: The ingress rules in the following definition use vpc to refer to the current VPC CIDR range 10.1.0.0/16 to avoid hardcoding it. They use predefined protocols and their default database ports, reference the default security group ID through default. You can also directly use the security group ID;

   {
     "vpcs": {
   "security": {
         "cidr": "10.1.0.0/16",
         "groups": {
           "db": {
             "in": [
               { "protocol": "mssql",    "cidr": "vpc" },
               { "protocol": "mysql",    "cidr": "vpc" },
               { "protocol": "redshift", "cidr": "vpc" },
               { "protocol": "postgres", "group": "default" },
               {
                 "protocol": "oracle",
                 "group": "sg-0683eac6d1c08566b",	
                 "description": "Allow connections from EC2 security group"
               }],
             "out": [
               { "protocol": "http",  "cidr": "*"  },
               { "protocol": "https",  "cidr": "*"   }]
           }
       }}
   }}
   

   JavaScript
3. Rules to connect to instances from your computer: egress rule allows access to the world;

   {
     "vpcs": {
       "security": {
         "cidr": "10.1.0.0/16",
         "groups": {
           "ec2": {
             "in": [
               { "cidr": "54.222.61.34/32", "protocol": "ssh" },
               { "cidr": "54.222.61.34/32", "protocol": "rdp" }],
             "out": [{ "cidr": "*", "protocol": "*" }]
           }
       }}
   }}
   

   JavaScript
4. DNS server rules and Amazon EFS rules: The default protocol is tcp, only one bit needs to be set for single port; NFS allows the group itself to access;

   {
     "vpcs": {
       "security": {
         "cidr": "10.1.0.0/16",
         "groups": {
           "dns": {
             "in": [
               {  "cidr": "vpc", "ports": [53] },
               { "protocol": "udp", "group": "default", "ports": [53] },
               { "protocol": "nfs", "group": "this" }]
           }
       }}
   }}
   

   JavaScript

Route 53 Resolver

Cloud Foundations network module adds support for the Route 53 resolver to meet the need to resolve dynamic DNS queries between local networks or other VPCs and the current VPC. According to best practices, one inbound and one outbound endpoint are sufficient for all DNS queries, so the system creates only one inbound and outbound endpoint by default. The network definition adds resolver, a first-level resource property:

Inbound and outbound endpoint properties:

Rule properties:

Meanwhile, like VPC flow logs, the system automatically configures resolver query logging sent to the Network Account’s Amazon CloudWatch log group and the Logs Account’s network bucket, and binds them to the VPCs used by the forwarding rules. The cross-account logging configuration hereinbefore is one of the signs of the comprehensive integration of Cloud Foundations’ multi-account cloud environment. Refer to the following example:

1. The inbound and outbound endpoints of the resolver all query DNS through the hub VPC;
2. Both endpoints set the default security group and web security group, where the web security group is defined in the hub VPC;
3. The forwarding rule is used by the dev VPC;
4. The forwarding rule outbound endpoint uses the only outbound endpoint;

   {
     "resolver": {
       "in":  { "vpc": "security", "groups": ["default", "web"] },
       "out": { "vpc": "security", "groups": ["default", "web"] },
       "rules": {
         "example": {
           "domain": "site.company.com",
           "ips": ["123.45.67.89"],
           "vpcs": ["dev"]
         }
       }
     },
     "vpcs": {
       "security": {
         "cidr": "192.168.0.0/16",
         "groups": {
           "web": {
             "in": [
               { "cidr": "*", "protocol": "http" },
               { "cidr": "*", "protocol": "https" }],
             "out": [{ "cidr": "*", "protocol": "*" }]
           }
         },
         "subnets": [[[12, 0], [12, 1]], [], [[8,  3], [8,  4]]]
       },
       "dev": {
         "cidr": "10.0.0.0/16",
         "subnets": [[[12, 0], [12, 1]], [], [[8,  3], [8,  4]]]
       }
   }}
   

   JavaScript

The following figure shows the outbound endpoint of the above resolver definition. Notice that it is configured with two security groups:

The following figure shows the two types of query logging configurations automatically configured by the system, where the log group is in the current Network Account and the S3 network bucket is in the Logs Account:

Transit gateway connect attachment

In order to meet customers’ needs to connect to software-defined wide area networks (SD-WAN) by configuring transit gateways (TGW), Cloud Foundations network module adds support for transit gateway connect attachments and connect peers. This attachment supports the Generic Routing Encapsulation (GRE) tunnel protocol and enables dynamic routing through Border Gateway Protocol (BGP). It relies on an existing VPC or Direct Connect attachment as the transport attachment to transfer data. Refer to the official documentation for details. We add the cidrs attribute for the transit gateway to configure its CIDR ranges. Note the difference with the cidr attribute. Some properties of transit gateway:

Connect attachments and connect peers properties:

The following example defines a hub VPC where a third-party virtual device is placed, and a transit gateway connect attachment is configured to use the VPC attachment as the transport attachment:

{
  "vpcs": {
    "security": {
      "is_hub": true,
      "cidr": "192.168.0.0/16",
      "subnets": [[[12, 0], [12, 1]], [], [[8, 3], [8, 4]]]
    }
  },
  "tgw": {
    "asn": 64513,    "cidr": "10.0.0.0/8",
    "enabled": true, "cidrs": ["192.0.2.0/24"],
    "connects": {
      "vpc": {
        "attachment": "security",
        "cidrs": ["169.254.7.0/29"],
        "peer": { "asn": 64515, "enabled": true, "address": "192.168.0.11" }
      }
    }
}}

The following figure shows that the resource type deployed according to the above definition is Connect’s transit gateway attachment and its two connected peers. The transport attachment is a hub VPC attachment.

Direct Connect gateway

Amazon Direct Connect (DX) is a commonly used service to directly connect customer’s internal network to a Direct Connect location, bypassing the Internet. Cloud Foundations network module support to create Direct Connect gateways and associate to transit gateways. Afterwards, you can create a transit virtual interface to connect the DX connection to the DX gateway and then connect to the transit gateway. Like transit gateways, one region contains only one DX gateway. The network definition adds property dx, a first-level resource for Direct Connect gateway with attributes:

Overlapping allowed prefixes are not allowed when multiple transit gateways are associated with one Direct Connect gateway. The following example defines a Direct Connect gateway with an ASN of 64512, a transit gateway with an ASN of 64513, and a common CIDR range of 10.1.0.0/16. After deployment, you can view the activation, association, and configuration of the Direct Connect gateway in the Direct Connect console.

{
  "dx":  { "asn": 64512, "enabled": true },
  "tgw": { "asn": 64513, "enabled": true, "cidr": "10.1.0.0/16" }
}

The following figure shows the status of the Direct Connect gateway and its associated transit gateway:

So far, Cloud Foundations network module supports four types of first-level attributes, namely:

1. vpcs: virtual private cloud definition;
2. tgw: transit gateway definition;
3. resolver: Route 53 Resolver definition;
4. dx: Direct Connect definition;

Except for VPC, only one resource can be defined per region for the other three resources.

Combined inspection of east-west north-south traffic

In our previous blog on inspecting network traffic in all directions^[3], we showed six common hub-spoke network topology architectures that can be defined and deployed with one-click using Cloud Foundations network module, namely:

1. VPC peering connection;
2. Centralized egress to internet;
3. North-south traffic inspection;
4. East-west traffic inspection;
5. Separated inspection of east-west and north-south traffic;
6. Multi-regional TGW peering connection;

The 5^th architecture above uses a hub VPC for north-south traffic inspection, and another VPC to inspect east-west traffic, i.e., north-south and east-west traffic are inspected “separately” by two VPCs. For this scenario, you need to set up a security appliance in each VPC, which can be the same or different. For example, use AWS Network Firewall to inspect east-west traffic, and use GWLB with a third-party firewall to inspect north-south traffic. In practice, we find that there is still a lack of a common architecture where the same security appliance is used to check north-south and east-west traffic separately in the hub VPC, i.e., combined inspection of east-west and north-south traffic. “Combined” here means combining the hub VPC and the inspection VPC of the 5^th architecture into one hub VPC, and also combining the two security appliances into one. Specifically, the routes for north-south and east-west traffic of the pre-inspection TGW route table in the 5^th architecture are migrated to the intra-subnet route tables of the hub VPC in the following diagram. For security appliance, you can choose GWLB with a third-party device.

Major configurations are:

1. Hub VPC sets intra, private and public subnets;
2. Hub VPC the first private subnet contains appliance north-south and east-west endpoints zonally;
3. Hub VPC route tables for intra subnets route full traffic to zonal appliance north-south endpoints, common traffic to zonal appliance east-west endpoints;
4. Hub VPC route tables for private subnets route full traffic to zonal NAT gateway, common traffic to TGW;
5. Hub VPC route tables for public subnets route common traffic to zonal appliance north-south endpoints, full traffic to IGW;
6. Spoke VPC does not set public subnets while route tables for intra and private subnets route full traffic to TGW;
7. TGW pre-inspection route table pre：
   1. All spoke VPC attachments are associated to this table;
   2. The hub VPC attachment is not propagated to this table;
   3. Add a route for full traffic to the hub VPC attachment;
8. TGW post-inspection route table post：
   1. The hub VPC attachment is associated to this table;
   2. All spoke VPC attachments are propagated to this table;
   3. Route other CIDR traffic to peering connection, Direct Connect or VPN attachments manually.

To this end, Cloud Foundations network module updated the GWLB properties, added same_ns_ew to control whether to set up a separate endpoint for east-west traffic inspection, and added same_subnet to control whether the GWLB is in a separate subnet. If the GWLB is not in the same subnet with its endpoints, then the hub VPC needs to prepare at least two private subnets:

Target group properties:

Compared to the 3^rd architecture above^[3], the following configuration just adds gwlb.same_ns_ew = false, Additionally, the configuration places the GWLB and its endpoint on a different subnet, and sets gwlb.same_subnet = false. At this point, the hub VPC needs to configure at least two private subnets.

{
  "vpcs": {
    "security": {
      "is_hub": true,
      "cidr": "192.168.0.0/16",
      "nat": { "enabled": true },
      "igw": { "enabled": true },
      "gwlb": {
        "enabled":    true, "same_ns_ew":  false, 
        "cross_zone": true, "same_subnet": false,
        "groups": { "main": { "health": { "enabled": true } } }
      },
      "subnets": [
        [[12, 0], [12, 1]], [[8, 1], [8, 2]],
        [[8,  3], [8,  4]], [[8, 5], [8, 6]]]
      },
    "dev": {
      "cidr": "10.0.0.0/16",
      "subnets": [[[12, 0], [12, 1]], [], [[8, 3], [8, 4]]]
    },
    "prod": {
      "cidr": "10.1.0.0/16",
      "subnets": [[[12, 0], [12, 1]], [], [[8, 3], [8, 4]]]
    }
  },
  "tgw": {
    "enabled": true,
    "cidr": "10.0.0.0/8",
    "tables": {
      "pre": { "associations": ["dev", "prod"], "routes": { "*": "security" } },
      "post": { "associations": ["security"], "propagations": ["dev", "prod"] }
    }
}}

As can be seen from the intra subnet route table of the hub VPC in the below figure, all traffic is routed to the north-south endpoint, and common traffic is routed to the east-west endpoint. The above two endpoints are in the same GWLB. This makes it possible to inspect the east-west and north-south traffic separately in the same security appliance in the same VPC.

Centralized inspection of ingress traffic

Cloud Foundations network module enhances support for centralized inspection of ingress traffic, extends route table definitions associated with Internet gateways, enables to specify and route subnet traffic to zonal security appliance endpoints. Here, we’ll take the 5^th architecture in the previous blog on network traffic inspection in all directions^[3] , e.g. “separated inspection of east-west and north-south traffic”, as an example, to explain how to upgrade the ingress spoke VPC to include traffic inspection. First, add properties to the internet gateway (IGW):

The subnet index refers to the VPC subnet array property definition, i.e., the array contains intra, public, and private subnets in order. The 1^st is the intra subnet, the 2^nd is the public subnet, and from the 3^rd on are the private subnets. If the application load balancer (ALB) is in the first and second private subnets, the corresponding subnets array is [2, 3].

Ingress traffic is usually inspected by security appliances in the public subnet, so it is necessary to set whether the security appliance is placed there. When an endpoint is placed in the public subnet, there is usually no need to set a network address translation (NAT) gateway, and private subnet traffic routing is slightly different from both the hub VPC and the east-west inspection VPC. The network module adds the following property to the VPC’s network firewall and gateway load balancer:

The following is an updated diagram for the 5^th architecture of “separated inspection of east-west and north-south traffic”. Among them, the ingress spoke VPC design refers to the “Centralized ingress deployment model” in this blog post^[5]:

Major configurations for the ingress spoke VPC are:

1. Ingress spoke VPC sets intra, public and private subnets;
2. Ingress spoke VPC route tables for public subnets route full traffic to IGW;
3. Ingress spoke VPC route tables for private subnets route full traffic to zonal appliance endpoints, common traffic to TGW;
4. Ingress spoke VPC route tables for IGW route private subnet traffic to zonal appliance endpoints;

The major points of the following configuration update are mainly at the ingress spoke VPC. The IGW associates the route table and routes to the first private subnet with an index of 2; the GWLB is deployed in the public subnet; a target group is set up and health check is enabled. In terms of network security appliance, egress hub VPC and ingress spoke VPC both use GWLB with third-party firewalls, while east-west traffic inspection spoke VPC uses AWS Network Firewall. Readers can flexibly utilize various security appliances at those inspection VPCs according to the business situation.

{
  "vpcs": {
    "inspect": {
      "nfw": { "enabled": true },
      "cidr": "192.168.0.0/16",
      "subnets": [[[12, 0], [12, 1]], [], [[8, 3], [8, 4]]]
    },
    "egress": {
      "is_hub": true,
      "cidr": "10.0.0.0/16",
      "nat": { "enabled": true },
      "igw": { "enabled": true },
      "gwlb": {
        "enabled": true,    "same_ns_ew": true,
        "cross_zone": true, "same_subnet": false,
        "groups": { "main": { "health": { "enabled": true } } }
      },
      "subnets": [
        [[12, 0], [12, 1]], [[8, 1], [8, 2]],
        [[8,  3], [8,  4]], [[8, 5], [8, 6]]
      ]
    },
    "ingress": {
      "cidr": "10.1.0.0/16",
      "igw": {
        "enabled": true,
        "table": { "enabled": true, "subnets": [2] }
      },
      "gwlb": { 
        "enabled": true, "in_public": true,
        "groups": { "main": { "health": { "enabled": true } } }
      },
      "subnets": [[[12, 0], [12, 1]], [[8, 1], [8, 2]], [[8, 3], [8, 4]]]
    },
    "dev": {
      "cidr": "10.2.0.0/16",
      "subnets": [[[12, 0], [12, 1]], [], [[8,  3], [8,  4]]]
    },
    "prod": {
      "cidr": "10.3.0.0/16",
      "subnets": [[[12, 0], [12, 1]], [], [[8,  3], [8,  4]]]
    }
  },
  "tgw": {
    "enabled": true,
    "cidr": "10.0.0.0/8",
    "tables": {
      "pre": {
        "associations": ["ingress", "dev", "prod"],
        "routes": { "*": "egress", "tgw": "inspect" }
      },
      "post": {
        "associations": ["inspect", "egress"],
        "propagations": ["ingress", "dev", "prod"]
      }
    }
  }
}

As can be seen from the routes in the IGW route table below, traffic to the first private subnet CIDR range is routed to the security appliance endpoint according to the availability zone. If you need to route to multiple subnets, simply set up multiple corresponding subnet indexes.

Non-main region network deployment

In delivery practices, we have encountered situations where customers require that no network resources be deployed in the main region to save costs, but mainly to deploy network and workload resources in the governing regions. Owing to technical or regulatory constraints, there will indeed be situations where the “major region” for hosting networks and workloads differs from the main region set when Cloud Foundations was deployed. On the other hand, global resources that are not regional, such as Amazon IAM, need to be provisioned in advance in the main region. Therefore, the network module added enabled property to the VPC resource to control resource deployment. This attribute of VPC is the only similar property of all network resources that is true by default:

Assuming that the main region is Ningxia and the major (governing) region is Beijing. In the following definition, the main region only creates flow log and read-only roles by turning off the switch, and creates network resources in the major (governing) region. Any VPC that needs to be shared to member accounts must first create relevant roles in those accounts. VPCs that do not need to be shared are managed only by the Network Account, such as the hub VPCs, inspection VPCs, etc., then there is no need to create roles in advance and are managed centrally in the major (governing) region.

{
  "vpcs": {
    "sandbox": { "enabled": false, "accounts": ["156165649349"] }},
  "cn-north-1": {
    "vpcs": {
      "security": { "is_hub": true, "cidr": "192.168.0.0/16" },
      "sandbox": { "cidr": "10.1.1.0/24", "accounts": ["156165649349"] }
    }
}}

Conclusion

This article describes the enhancements and new features that Cloud Foundations has made to the network module in various aspects in 2024. These initiatives are all aimed at solving a variety of complex network construction issues encountered during actual project deliveries. As it is called “Rome wasn’t built in a day,” Cloud Foundations network module’s support for complex AWS network resources was not built in a day either, and it is still under active construction. Based on the concept of “working backwards” and the “from customer to customer” mentality, we will actively listen to customer needs and continue to meet the ever-changing and generally applicable network architecture requirements. In the next step, we will focus on strengthening comprehensive support and construction capabilities in network security and network firewalls.

References

1. Blog post: “Use Cloud Foundations to holistically plan and one-click deploy two network sharing models in multi-account organizations on the cloud”, February 2023
2. Blog post: “Use Cloud Foundations to plan and design multi-regional hub-spoke network topology on the cloud and one-click deploy east-west south-north traffic inspection separated or combined”, November 2023
3. Blog post: “Use Cloud Foundations network-sharing products to plan, design and one-click deploy cross-regional multiple cloud networks on multiple network accounts”, February 2024
4. Blog post: “Use Cloud Foundations Product Factory to plan, design and one-click deploy infrastructural cloud resources such as multi-account access control and permission policies”, March 2024
5. Blog post (in Chinese): “Network Firewall 部署小指南（二）路由设计”, October 2024

本篇作者