NetDevSecOps to modernize AWS networking deployments

NetDevOps orchestrates and automates network changes to shorten the network delivery lifecycle, treats the network as code to allow for version control, and tests changes to make sure of quality and stability. This increased speed helps organizations better serve their customers and compete more effectively in the market. The post NetDevOps: A modern approach to AWS networking deployments summarized its principles and components. In this post, we expand on how to implement NetDevOps to build and maintain modern network architectures. We incorporate DevSecOps best practices to integrate security into DevOps. With NetDevSecOps, you can deliver a secure and compliant network infrastructure.

Considerations

Every organization is unique, and there is no single correct approach to implementing NetDevOps. The following section lists some considerations that you must keep in mind during your decision making. These considerations will help you define a framework for releasing changes with confidence, or rollback without causing disruptions.

1/ Infrastructure as Code (IaC) options – AWS offers several IaC options, including AWS CloudFormation, AWS Cloud Development Kit (AWS CDK) and Terraform. When deciding on your approach for IaC, ask these questions:

2/ Deployment strategies – Define how you want to deploy your network stack. Organizations follow different deployment strategies based on their business model. Some may choose to deliver a network stack that is fully tested as a service catalog product. Others may implement them using Customization of Control Tower (CfCT) as part of the account provisioning process of Control Tower. You may also deploy network stack through continuous integration/continuous development (CI/CD) pipelines. The CI/CD pipelines can be based on AWS Code Pipeline, AWS CDK Pipelines, or custom pipelines, such as Jenkins, Gitlab, etc.

3/ Define unit tests – When an organization uses pipelines to provision application and infrastructure, infrastructure testing plays a key role in mitigating problems related to any misconfiguration. The infrastructure team is aware of the organization’s policies. Therefore, it’s essential to write and integrate unit tests as part of the provisioning pipeline. Based on the IaC option you choose, you will create unit test policy tools and unit tests. Cfn-guard and Cloudformation hooks can be used for CloudFormation, Open Policy Agent (OPA) checks, and Hashicorp sentinel can be written for Terraform. With the AWS CDK, your infrastructure can be as testable as any other code you write. In order to implement testing when using the AWS CDK, follow the testing constructs documentation. Upon unit test failures, the pipeline deployment fails.

4/ Security and compliance tools integration – NetSecOps introduces security to the NetDevOps practice by integrating security assessments throughout the CI/CD process. It makes security a shared responsibility among all network team members that are involved in building the network stack for your organization. The network development team collaborates with the security team before they write any code. Likewise, network operations teams continue to monitor the network infrastructure for security issues after deploying it. As a result, you can secure your network while ensuring compliance. Depending on the IaC option you chose, you can integrate your CI/CD pipelines with security tools. For example, cfn_nag, cfn_lint, cfn_guard for CloudFormation, and cdk_nag tool for AWS CDK. If you are using Terraform, then tfsec and terrascan tools can be included as part of the pipeline. There are also cloud agnostic security tools, such as Checkov or Cloud custodian, that can be integrated with the pipeline to govern compliance and security requirements.

5/ Defining stages – You must define the CI/CD of your network as a pipeline, where new network configurations or required changes are submitted as code on one end, pass over a series of stages such as build, test, staging, and validation, before they are deployed in production. Every stage acts like a gatekeeper, assessing a specific part of the code. As the code progresses through the network pipeline, we assume that code in latter stages is higher quality, because it has been verified more times. Problems uncovered in an early stage stop the code from progressing through the pipeline. The network operations team immediately receives the results from the tests, and they stop all further builds and releases if the software doesn’t pass the stage.

6/ Multi-account support – As you adopt AWS, you must determine how to meet your business, governance, security, and operational requirements. The use of multiple AWS accounts plays an important role in how you meet those requirements. We recommend that you create deployment pipelines in an account within Deployments OU that has permission to deploy networking resources across other OUs and accounts. This account comprises resources that support how you build, validate, promote, and release changes to your network.

7/ Multi-Region support – As part of the organization’s Business Continuity Plan (BCP), consider disaster recovery options to mitigate against natural and technical disasters that affect an entire AWS Region. Therefore, the organization should consider multi-Region networking to protect against these disasters.

8/ Monitoring network CI/CD pipelines – You must integrate monitoring that notifies the network operations team on a failure or a specific event. Amazon CloudWatch metrics, CloudWatch Logs, and Amazon EventBridge notification monitoring is setup for the pipeline.  If security and compliance issues are detected as part of provisioning the resource, those findings can be sent to AWS Security Hub. The Network Security Operations (NetSecOps) team can take action the Security Hub findings.

A practical example

Let’s consider a multi-account, multi-Region scenario where a business wants to centralize all ingress traffic to AWS. As part of compliance requirements (NIST 800-53 Rev 5 and PCI DSS version 3.2.1), you must implement a limits to network access controls – “Detect whether public routes exist in the route table for an Internet Gateway (IGW)”. We can use AWS Transit Gateway for this scenario with architecture as shown in figure 1.

Figure 1: A multi-account hub and spoke design leveraging Transit Gateway

To implement this use case, we take the considerations we highlighted earlier into account when defining our NetDevSecOps Pipeline as shown in figure 2. The results are:

• Consideration 1 (IaC) – We built our infrastructure using the AWS CDK, which is synthesized and deployed as part the stages in the deployment.
• Consideration 2 (Deployment strategies) – Under the hood, the AWS CDK pipeline construct uses AWS CodePipeline and AWS CodeBuild, which is used as part of continuous delivery strategy. Here,we test our infrastructure code and deploy it to our Test and Production environments at different stages in the process.
• Consideration 3 (Unit test) – We wrote a pytest to validate the resource property “AWS::CodePipeline::Pipeline” of the AWS CDK Synthesized template. This only deploys resources to the Primary and Disaster Recovery Regions.
• Consideration 4 (Security and compliance tools integration) – This compliance check validates that the Hub VPC subnet launched with a Public IP through cfn-nag, and publishes the findings to Security Hub.
• Consideration 5 (Defining stages) – The pipeline has Hub, Spoke, and Validate stages to provision the resources in the Hub and spoke environment. We use AWS VPC Reachability Analyzer to validate connectivity.
• Consideration 6 (Multi-account support) – The pipeline provisions resources to the Development and Production environment.
• Consideration 7 (Multi-Region support) – We provision resources in both the Primary and Disaster Recovery Regions.
• Consideration 8 (Monitoring) – We send compliance findings to Security Hub.

Figure 2: NetDevSecOps Pipeline in action

Prerequisites

You need the following:

1. Five AWS Identity and Access Management (IAM) accounts as part of AWS Organizations with administrator privileges for the services used in this solution.

• Development Account – Used to provision the NetDevOps Pipeline (This can also be a shared services or deployment account)
• Dev Hub Account – NetDevOps Pipeline provisions the Dev Hub Workload
• Dev Spoke Account – NetDevOps Pipeline provisions the Dev Spoke Workload
• Prod Hub Account – NetDevOps Pipeline provisions the Prod Hub Workload
• Prod Spoke Account – NetDevOps Pipeline provisions the Prod Spoke Workload

2. Turn on trusted access for AWS Resource Access Manager (AWS RAM) in the Organizations Management Account. Review these steps to activate it.

3. Turn on the Delegated administrator account for Security Hub in the Organizations Management Account. Review these steps to activate it.

4. Configure the AWS credentials of the Development account in the terminal. Install the AWS CDK Toolkit. If you’re unfamiliar with the steps, then review the AWS CDK workshop for help.

Deployment steps

Next, we deploy the code in this Git repository. We assume there is an AWS deployment account that has access to the Network Hub account and workload accounts for development and production environments. This deployment comprises two AWS CDK applications:

1. NetDevOps foundations application – This AWS CDK application provisions all foundations that are required as part of the NetDevOps Pipeline. This includes AWS CodeCommit Repository, Cross account IAM role, Amazon Simple Notification Service (Amazon SNS) Topic and AWS Lambda function used to send cfn-nag findings.

2. NetDevOps Pipeline application – This AWS CDK application provisions the NetDevOps Pipeline for network infrastructure, with all aforementioned stages in a multi-account and multi-Region setup.

NetDevOps foundations application

1. Clone the repository and bootstrap the application in your local machine by running the following AWS Command Line Interface (AWS CLI) commands from a terminal window.

git clone <repo url> NetDevopsPipeline cd NetDevopsPipeline 
python3 -m venv .venv 
source .venv/bin/activate 
pip install -r requirements.txt

2. Update accounts.py in the “netdevopspipeline” folder with development, hub, and spoke accounts in the development and production environment as shown in figure 3.

Figure 3: An example accounts.py file

3. Update cdk.json with your email to receive the SNS notification, Organizations ID, and then deploy the NetDevops Foundations Application.

cdk bootstrap aws://<Development account id>/<Primary Region>
cdk deploy -c infra_type=NetDevopsFoundation --require-approval=never

Note : Subscribe the SNS notification sent to the email address mentioned in cdk.json file.

NetDevOps Pipeline application

1. Clone the CodeCommit repository (network-devops-repo) created in the previous step.

git remote add codecommit https://git-codecommit.<Primary Region>.amazonaws.com/v1/repos/network-devops-repo

NOTE: The above step requires setting up Https git credentials. Review the steps to setup git credentials. Due to operational reason if you’re unable to setup git credentials then follow the steps mentioned for git-remote-codecommit setup.

2. Add the code to the repository by running the following commands:

git add .
git commit -m "Initial NetDevops pipeline repo commit"
git push --set-upstream codecommit main

3. Bootstrap the hub account and spoke account in the development and production environment by following these steps. Or, bootstrap the accounts using StackSet, following the steps highlighted in this post.

Make sure that the correct IAM account credentials are setup during the bootstrapping of each environment.

Hub Account (Development Environment)

cdk bootstrap aws://<Hub account id>/<Primary Region> —trust <Development account> --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess 
cdk bootstrap aws://<Hub account id>/<DR Region> —trust <Development account> --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess

Spoke Account (Development Environment)

cdk bootstrap aws://<Spoke account id>/<Primary Region> —trust <Development account> --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess
cdk bootstrap aws://<Spoke account id>/<DR Region> --trust <Development account> —cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess

Hub Account (Production Environment)

cdk bootstrap aws://<Hub account id>/<Primary Region> —trust <Development account> --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess
cdk bootstrap aws://<Hub account id>/<DR Region> —trust <Development account> --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess

Spoke Account (Production Environment)

cdk bootstrap aws://<Spoke account id>/<Primary Region> —trust <Development account> --cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess
cdk bootstrap aws://<Spoke account id>/<DR Region> --trust <Development account> —cloudformation-execution-policies arn:aws:iam::aws:policy/AdministratorAccess

4. To provision the NetDevOps Pipeline in the “Development account”, setup the IAM credentials of the pipeline and run the following command. The deployment of NetDevOps Pipeline with all stages is shown in figure 4.

cdk deploy --all --require-approval=never

Figure 4: An example NetDevSecOps pipeline with stages

Note that deployment takes 10-15 minutes because of the CodePipelines and CloudFormation template creation. Check the progress on the AWS CodePipeline Console on your Hub account Primary Region.

5. The NetDevOps Pipeline has a manual approval stage to validate the hub and spoke connectivity. An Amazon SNS notification is sent when VPC Reachability Analyzer reports that the pipeline is reachable as shown in figure 5. To provision resources in the production environment, review this notification and approve the manual stages.

Figure 5: An example analysis from VPC reachability analyzer

Now you’ve successfully implemented a multi-account, multi-Region NetDevOps Pipeline with the ability to identify one limit network access control.

Validating security and compliance checks

In this step, we validate security and compliance leveraging cfn-nag as part of the pipeline. This is done as part of the build stage (synth-and-deployment-validation):

1. Copy this snippet of code and replace the content of self.vpc present in the DevSpokeInfraStack.py file in the pipeline stack folder as shown in figure 6.

self.vpc = ec2.Vpc(
            self,
            "spoke_vpc",
            max_azs=2,
            cidr=cidr,
            enable_dns_hostnames=True,
            enable_dns_support=True,
            subnet_configuration=[
                ec2.SubnetConfiguration(
                    subnet_type=ec2.SubnetType.PRIVATE_ISOLATED,
                    name="private",
                    cidr_mask=24,
                ),
                ec2.SubnetConfiguration(
                    subnet_type=ec2.SubnetType.PRIVATE_ISOLATED,
                    name="tgw",
                    cidr_mask=28,
                ),
                ec2.SubnetConfiguration(
                    subnet_type=ec2.SubnetType.PUBLIC,
                    name="Public",
                    cidr_mask=24,
                    map_public_ip_on_launch=True
                )
            ],
            nat_gateways=0,
        )

Figure 6 : An example comparison for snippet of code for VPC subnet configuration

2. Push the code changes by running:

git commit -a -m "Adding Public subnet in the spoke stack" 
git push codecommit main

Once you push the changes, the pipeline gets invoked and goes to a failed state, and findings are sent to the Security Hub as shown in figure 7.

Figure 7: Findings sent to AWS Security Hub

Validating unit tests

The existing NetDevOps Pipeline only allows you to create resources in us-east-1 (Primary Region) and us-west-2 (Secondary Region). If the provisioning occurs in any other Region, then the pipeline fails at the build stage (synth-and-deployment-validation).

1. Update the accounts.py present in the NetDevOps Pipeline folder to provision in any Region other than us-east-1 (Primary Region) and us-west-2 (Secondary Region) as shown in figure 8.

Figure 8: An example of updated accounts.py file

Note that you must make sure the Hub and Spoke accounts are bootstrapped in both the development and production environments in the “us-east-2” Region.

2. Push the code changes by running the following commands:

git commit -a -m "Adding invalid region subnet in the accounts.py" 
git push codecommit main

Once the changes are pushed, the pipeline gets invoked, and the pipeline changes to the failed state at the build stage. When reviewing the build failure logs, you can notice it’s due to pytest failed validation as shown in figure 9.

Figure 9: An example pytest failure

Cleanup

To avoid unnecessary charges, delete the resources created during the deployment of NetDevOps Pipeline and testing. To cleanup the resources, perform the following steps in the order shown here:

1. The stacks created by the CodePipeline should be deleted manually from the CloudFormation console. Therefore, identify the stacks, and delete them in hub and spoke accounts in both the primary and DR Regions for the production and development environments. Also, delete the S3 buckets.

2. Destroy the NetDevOps Foundations and NetDevOps Pipeline Applications by running the following commands:

cdk destroy --all
cdk destroy -c infra_type=NetDevopsFoundation

Conclusion

NetDevOps helps you deliver applications and services faster. This means faster changes, and a shorter network delivery lifecycle. In this post, we showed how to implement NetDevSecOps in practice by leveraging AWS CDK. We showed how to include security and compliance checks as part of your network CI/CD. We removed the undifferentiated heavy lifting for setting up the CI/CD pipeline by using the provided templates to provision the required infrastructure. We hope that you’ve found this post informative and we look forward to hearing how you use this approach!

About the Authors

A correction was made on January 26, 2024 to remove comments from a copyeditor that should have been deleted prior to publication.