Observability for AWS App Runner VPC networking

With AWS App Runner, you can quickly deploy web applications and APIs at any scale. You can start with your source code or a container image, and App Runner will fully manage all infrastructure, including servers, networking, and load balancing for your application. If you want, App Runner can also configure a deployment pipeline for you.

This year, AWS announced Amazon Virtual Private Cloud (Amazon VPC) support in February and AWS X-Ray support in April for App Runner services.

With VPC support, applications can now connect to private resources in your Amazon VPC, such as an Amazon Relational Database Service (Amazon RDS), Redis, or Memcached caches in Amazon ElastiCache, or other private services hosted in your VPC.

With X-Ray support, you can take advantage of adding tracing to App Runner services without having to configure and set up the required sidecars or agents. You can trace your containerized applications in AWS X-Ray by instrumenting applications with the AWS Distro for OpenTelemetry (ADOT).

This blog post will help you to learn more about how these two capabilities work together in practice with a Node.js web application connected to an Amazon RDS database.

Architecture

Walkthrough

Prerequisites

For this walkthrough, you should have the following prerequisites:

An AWS account with full privileges to create and access the following resources:
- VPC
- Subnets
- Internet gateway
- NAT gateway
- Security group
- IAM role
- Amazon RDS database
- X-Ray
- Amazon CloudWatch
- App Runner
Basic knowledge of containers

Step 1: Preparing the Amazon VPC and subnets

Follow these steps to create a VPC called hotel-app-vpc with subnets to host your database and applications:

Open the Amazon VPC console.
On the VPC Dashboard, choose Launch VPC Wizard.
Enter the following information into the wizard and then choose Create VPC.
1. Name tag auto-generation: Enter “hotel-app.”
2. Availability Zones: Choose 3.
3. NAT gateways: Choose In 1 AZ.
4. VPC endpoints: Choose None.
5. All other settings: Leave as default.

It takes several minutes to create the VPC, subnets, and other configured resources. Once they are created, you can select the View VPC button at the end of the wizard and check the detail. Note down the name of the VPC and subnets, which you will use later.

Step 2: Create security groups

In order to deploy an application on App Runner that has outbound access to a VPC, you must first create an App Runner VPC connector by specifying one or more subnets and security groups to associate with the application.

With regards to the security groups, it is important to note that App Runner VPC connector is only used for outbound communication from your application. Thus, the inbound rules of the security group(s) are not relevant and are effectively ignored. What matters are the outbound rules, which should allow communication to the desired destination endpoints. Read Deep Dive on AWS App Runner VPC Networking if you’re interested in more detail.

We will create two security groups with corresponding security group rules within the VPC created in Step 1:

hotel-app-svc-sg: The security group for App Runner VPC connector

Rule	IP version	Type	Protocol	Port range	Destination
Inbound	N/A	N/A	N/A	N/A	N/A
Outbound	IPv4	All Traffic	All	All	0.0.0.0/0

hotel-app-rds-sg: The security group for your Amazon RDS database, which is the desired destination resource for your application communication

Rule	IP version	Type	Protocol	Port range	Source
Inbound	N/A	MYSQL/Aurora	TCP	3306	sg-0f7ff683254111fb2 / hotel-app-svc-sg

You must ensure the security group associated with the destination resource hotel-app-rds-sg have appropriate inbound rules to allow traffic from the VPC connector security group hotel-app-svc-sg.

Use these steps to create security groups using the console:

Open the Amazon VPC console. In the navigation pane, choose Security Groups, then choose Create security group.
Enter a name hotel-app-svc-sg and description for the security group (see the tables below for our suggested names). You cannot change the name and description of a security group after it is created.
From VPC, choose the VPC hotel-app-vpc. Leave security group rules empty for now; we will add them later on.
You can add tags now, or you can add them later. To add a tag, choose Add new tag and enter the tag key and value. Choose Create security group.
Repeat Steps 1–3 for another security group hotel-app-rds-sg
Add the security group rule by referring to the table above:
1. For Type, choose MYSQL/Aurora.
2. For Destination, choose hotel-app-svc-sg.
Choose Create security group.
Open the Amazon VPC console. In the navigation pane, choose Security Groups. You will see the two security groups created above hotel-app-svc-sg and hotel-app-rds-sg.
Select the security group hotel-app-svc-sg.
Choose Actions, Edit outbound rules.
Choose Add rule and do the following by referring to the table above:
1. For Type, choose All traffic.
2. For Destination, choose Anywhere-IPv4.
Choose Save rules.

Note down the name of the security groups, which you will use later when creating the Amazon RDS database and App Runner service.

Step 3: Preparing the Amazon RDS database

This step is to set up the Amazon RDS database to be hosted in a private VPC network.

Open the Amazon RDS console.
In the navigation pane, choose Databases.
Choose Create database and make sure that StandardCreate is chosen.
For Engine type, choose Amazon Aurora.
For Edition, choose Amazon Aurora with MySQL compatibility.
Expand Hide filters, select Show versions that support Serverless v2, and choose an available option.
For Templates, choose Dev/Test.
For DB cluster identifier, enter a name for the DB cluster, or leave the default name. Here we put hotel-app-database.
For Master username, keep default value admin.
To use an automatically generated password for the DB cluster, make sure that the Auto generate a password box is selected. To enter your password, clear the Auto generate a password box, and then enter the same password in Master password and Confirm password.
For DB instance class, choose Serverless.
For Connectivity, use the VPC hotel-app-vpc and the security group hotel-app-rds-sg created in Steps 1 and 2. You don’t need to make the database publicly accessible because you’re going to connect using private VPC networking.

UI image of the Connectivity vpc information

13. Keep all the other default values.

14. Choose Create database.

It takes several minutes for the RDS database to be provisioned. You can view the database list on the Amazon RDS Databases page. Choose hotel-app-database in the DB identifier list and view database detail:

UI image of the hotel-app-database related information

Note down the Writer instance endpoint name, password, and user information for later application connection.

Step 4: Preparing the application source code repository

With App Runner, you can deploy a new service from code hosted in a source code repository or using a container image. In this example, I use a private project that I have on GitHub.

It’s a very simple Node.js web application connecting to the database. The application stores hotel room information in a private MySQL/Aurora database on Amazon RDS. Room information can be retrieved and updated using HTTP(S) GET and POST requests to the app. The Amazon RDS database is only accessible from within the VPC. You can create tables, add items, and list items. This is the source code of the app (hotel-app/app.js):

const my_tracer = require('./create-a-tracer');
const AWS = require('aws-sdk');

var createError = require('http-errors');
var express = require('express');
var path = require('path');
var cookieParser = require('cookie-parser');
var logger = require('morgan');

var indexRouter = require('./routes/index');
var createRouter = require('./routes/create');
var addRouter = require('./routes/add');
var roomsRouter = require('./routes/rooms');

var app = express();

// view engine setup
app.set('views', path.join(__dirname, 'views'));
app.set('view engine', 'jade');

app.use(logger('dev'));
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.use(cookieParser());
app.use(express.static(path.join(__dirname, 'public')));

app.use('/', indexRouter);
app.use('/create', createRouter);
app.use('/add', addRouter);
app.use('/rooms', roomsRouter);

// catch 404 and forward to error handler
app.use(function(req, res, next) {
  next(createError(404));
});

// error handler
app.use(function(err, req, res, next) {
  // set locals, only providing error in development
  res.locals.message = err.message;
  res.locals.error = req.app.get('env') === 'development' ? err : {};

  // render the error page
  res.status(err.status || 500);
  res.render('error');
});

module.exports = app;

Instrument your application

Instrument your application using OpenTelemetry. Depending on the specific ADOT SDK that you use, ADOT supports two approaches: auto-instrumentation and manual-instrumentation. For instructions, see Getting Started with the OpenTelemetry JavaScript SDK on Traces Instrumentation. I will use manual-instrumentation here as an example. If you want to see how auto-instrumentation works, you can read another blog post.

To trace your application via X-Ray, the following OpenTelemetry JavaScript packages must be installed in your application’s main directory.

Install OpenTelemetry JavaScript packages

hotel-app/package.json:

{
  "name": "hotel-app",
  "version": "0.0.0",
  "private": true,
  "scripts": {
    "start": "node ./bin/www"
  },
  "dependencies": {
    "cookie-parser": "~1.4.4",
    "debug": "~2.6.9",
    "express": "~4.16.1",
    "http-errors": "~1.6.3",
    "jade": "~1.11.0",
    "morgan": "~1.9.1",
    "mysql": "^2.18.1",
    "postcss": "^8.4.6",
    "postcss-cssnext": "^3.1.1",
    "postcss-nested": "^5.0.6",
    "@opentelemetry/api": "^1.0.3",
    "@opentelemetry/sdk-trace-node": "^0.25.0",
    "@opentelemetry/sdk-metrics-base": "^0.25.0",
    "@opentelemetry/exporter-collector-grpc": "^0.25.0",
    "@opentelemetry/instrumentation": "^0.25.0",
    "@opentelemetry/instrumentation-http": "^0.25.0",
    "aws-sdk": "^2.875.0",
    "@opentelemetry/id-generator-aws-xray": "^0.25.0",
    "@opentelemetry/propagator-aws-xray": "^0.25.0",
    "opentelemetry-instrumentation-aws-sdk": "^0.24.2",
    "@opentelemetry/instrumentation-mysql": "^0.23.0"
  }
}

Sending traces to X-Ray

To send trace data to X-Ray via the ADOT, you must configure the X-Ray ID generator, X-Ray propagator, and gRPC exporter on the global tracer provider.
hotel-app/create-a-tracer.js:

'use strict'

// OTel JS - API
const { trace } = require('@opentelemetry/api');

// OTel JS - Core
const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');

// OTel JS - Core - Exporters
const { CollectorTraceExporter } = require('@opentelemetry/exporter-collector-grpc');

// OTel JS - Core - Instrumentations
const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http');
const { MySQLInstrumentation } = require('@opentelemetry/instrumentation-mysql');
const { AwsInstrumentation } = require('opentelemetry-instrumentation-aws-sdk');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions')

// OTel JS - Contrib - AWS X-Ray
const { AWSXRayIdGenerator } = require('@opentelemetry/id-generator-aws-xray');
const { AWSXRayPropagator } = require('@opentelemetry/propagator-aws-xray');


const tracerProvider = new NodeTracerProvider({
  resource: Resource.default().merge(new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: "hotel-app"
  })),
  idGenerator: new AWSXRayIdGenerator(),
  instrumentations: [
    new HttpInstrumentation(),
    new MySQLInstrumentation(),
    new AwsInstrumentation({
      suppressInternalInstrumentation: true
    }),
  ]
});

// Expects Collector at env variable `OTEL_EXPORTER_OTLP_ENDPOINT`, otherwise, http://localhost:4317
tracerProvider.addSpanProcessor(new SimpleSpanProcessor(new CollectorTraceExporter()));

tracerProvider.register({
  propagator: new AWSXRayPropagator()
});

module.exports = trace.getTracer("hotel-app");

Inject tracer in the main code of the app (hotel-app/app.js):

const my_tracer = require('./create-a-tracer');

Set up Amazon RDS database connection

Update rdsUrl, password, and user with the RDS database endpoint, password and user information in hotel-app/rds.js from Step 3.

var mysql = require('mysql');

var rdsUrl = 'replace with endpoint to the RDS';
var password =  'replace with password to the RDS';
var user = 'replace with user to the RDS';

// mysql connection pool
var rdsPool = mysql.createPool({
    connectionLimit : 12,
    host: rdsUrl,
    password: password,
    user: user
});

module.exports.pool = rdsPool;
module.exports.url = rdsUrl;

Step 5: Set up an IAM role

App Runner uses an AWS Identity and Access Management (IAM) role to authorize access to other AWS services. In this case, App Runner uses the role to allow the application to send tracing data to X-Ray.

Open the IAM console.
In the navigation pane of the IAM console, choose Roles, and then choose Create role.
For Select trusted entity, choose Custom trust policy. Follow the instructions for instance roles in How App Runner works with IAM in the App Runner Developer Guide, and input the role trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "tasks.apprunner.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

4. Choose Next.

5. For Add permissions:

a. Attach an AWS-managed policy AWSXRayDaemonWriteAccess that allows App Runner to send tracing data to X-Ray.

b. Attach an AWS-managed policy AmazonRDSDataFullAccess that allows App Runner to integrate with the Amazon RDS database.

6. Choose Next.

7. For Role name, enter hotel-app-instance-role.

8. Review the role and then choose Create role.

Your IAM role will look like the following screenshot:

hotel-app-instance IAM role UI image

Step 6: Create and configure your App Runner service to enable VPC networking and X-Ray tracing

Now that your code is instrumented and you’ve created the necessary role and VPC resources, create your App Runner service and configure it to enable VPC networking and X-Ray tracing.

Open the App Runner console.
Navigate to the App Runner Services
Choose Create service.
For repository type, choose Source code repository.
For Connect to GitHub, choose your GitHub repository and the branch to use.
For the Deployment settings, I chose Manual. Optionally, you could have selected the Automatic deployment trigger to have every push to this branch deploy a new version of your service.
Choose Next.
Then, configure the build. This is a very simple application, so I pass the build and start commands in the console:
1. Runtime: Nodejs 14
2. Build command: npm install
3. Start command: npm start
  For more advanced use cases, you can add an apprunner.yaml configuration file to your repository as in this sample application.
Choose Next.
In the Service settings section, for Service name, enter hotel-app.
In the Security section, for Instance role, choose the IAM role that was created earlier, hotel-app-instance-role.
For AWS KMS key, choose Use an AWS-owned key.
For Networking, select the new option to use a Custom VPC for outgoing network traffic and then add a new VPC connector.

For Add new VPC connector, enter VPC connector name hotel-app-svc-vpc. To add a new VPC connector, you can select the VPC, subnets, and the hotel-app-svc-sg security group created in Steps 1 and 2. In this way, the App Runner service will be able to connect to the Amazon RDS database.
For the newly introduced Observability section, enable Tracing with AWS X-Ray.
Keep all the other default values.
Choose Next.
Review all the settings and then choose Create & deploy.
On the service dashboard page, in the Service overview section, monitor the service status. The service is live and ready to process requests when the status turns to Running.

Step 7: Run application

After a few minutes, the service is running, and the Observability tab shows Tracing On.

Scre4enshot of the Observability tab showing Tracing ON

Select the Default domain URL to open a new tab in your browser. The application is up and running and connecting to the Amazon RDS database with VPC networking.

You can

Select /create to create a table in the database if one doesn’t already exist. (You must call this API first if this is the first time using the app.)
Select /add to insert a new room data item in the database.
Select /rooms to display rooms in the database.

Welcome to AP Hotel! with above options displayed

Room list table

It works!

Step 8: View tracing data in X-Ray

On the Observability tab, choose View service map.

You are redirected to the CloudWatch console. Notice that HTTP requests and AWS SDK requests are instrumented. The following image shows an X-Ray service map with traces collected from the example application.
UI screenshot of CloudWatch Service map

Choose View traces.
The console shows the CloudWatch Traces page.

Suppose that you’ve made a few API calls. Some of them returned 4xx Error. To determine which group of requests caused the service 4xx responses, query the X-Ray traces and filter by the HTTP status code. To troubleshoot the cause of the error, on the Traces list, narrow the results to the group of bad requests by HTTP method. Then review the details of each trace segment. This information can help you to troubleshoot issues.

Clean Up

Delete the App Runner service hotel-app.
Delete the App Runner VPC connector hotel-app-svc-vpc.
Delete the IAM role created earlier hotel-app-instance-role.
Delete the Amazon RDS database hotel-app-datastore.
Delete the security groups hotel-app-svc-sg and hotel-app-rds-sg.
Delete the NAT gateway of the VPC hotel-app-vpc.
Detach the internet gateway from the VPC hotel-app-vpc.
Delete the internet gateway.
Delete the VPC hotel-app-vpc.

Resources to learn more

There are more resources to learn about AWS App Runner VPC and X-Ray supports, and we encourage you to check them out!

Container Day – Docker Con | How Developers can get to production web applications at scale easily
Containers from the Couch | AWS App Runner X-Ray Integration Feature Launch
Containers from the Couch | AWS App Runner adds support for Amazon VPC!
Containers from the Couch | AWS App Runner VPC Support – Launch Demo
AWS Blog | Tracing an AWS App Runner service using AWS X-Ray with OpenTelemetry
AWS Blog | Deep Dive on AWS App Runner VPC Networking
AWS Blog | New for App Runner – VPC Support
✨ AWS App Runner Public Roadmap at GitHub ✨ – See the latest roadmap items and leave your requests and feedback to the App Runner team directly!

Happy reading!

Follow me on Twitter @pymhq

Containers