Containers

Using Gloo as an Ingress Gateway for AWS App Mesh

NOTICE: October 04, 2024 – This post no longer reflects the best guidance for configuring a service mesh with Amazon ECS and Amazon EKS, and its examples no longer work as shown. For workloads running on Amazon ECS, please refer to newer content on Amazon ECS Service Connect, and for workloads running on Amazon EKS, please refer to Amazon VPC Lattice.

——–

As part of their organization’s digital transformation, more and more customers are electing to use a managed Kubernetes service, like Amazon EKS, as their container-orchestration system of choice to deploy, scale, and manage microservices.

As the number of microservices grow within an application, it becomes difficult to pinpoint the exact location of errors, re-route traffic after failures, and safely deploy code changes. A service mesh, like AWS App Mesh, makes it easy to run services by providing consistent visibility and network traffic controls for services built across multiple types of compute infrastructure such as Amazon EC2, Amazon EKS, and AWS Fargate.

Before service mesh, developers had to use a language-specific library to implement features like backoffs, or retries, now they can decouple these features from the application and spend more time on the business logic and less time on non-functional requirements.

But operation or SRE teams still need to create and manage access to the applications from outside the Kubernetes cluster. Ingress controller has become a popular solution to solve this type of challenge.

Today, we will take a look at Gloo (from solo.io) which is a feature-rich, Kubernetes-native ingress controller, and API Gateway based on Envoy. Gloo is exceptional in its function-level routing, its support for legacy apps, microservices, and serverless applications. It can easily run on AWS using AWS App Mesh.

In this blog post:

  • We will create an EKS cluster with add-ons to create and manage AWS App Mesh automatically.
  • We will deploy a sample application using AWS App Mesh.
  • We will install and configure Gloo as the Ingress Controller.
  • And finally we will use Gloo for a canary deployment.

Prerequisites

Before starting, we need to install the following tools on our local computer:

Getting started

We will start by creating an EKS cluster in the us-west-2 Region using a YAML config file:

#create eks config file
cat <<EoF > eks-config.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: GlooDemo
  region: us-west-2
  version: "1.13"

nodeGroups:
  - name: ng1-GlooDemo
    instanceType: m5.large
    desiredCapacity: 2
    iam:
      withAddonPolicies:
        autoScaler: true
        #allow access AWS app mesh
        appMesh: true
        #allow access to AWS X-Ray
        xRay: true
        #allow access to AWS CloudWatch
        cloudWatch: true
EoF

#create eks cluster
eksctl create cluster --auto-kubeconfig -f eks-config.yaml

#load the EKS cluster config
export KUBECONFIG=${HOME}/.kube/eksctl/clusters/GlooDemo

The cluster creation will take up to 15 minutes.

To simplify the creation and the management of AWS App Mesh, we will install two add-ons:

  • aws-app-mesh-controller-for-k8s that will manage AWS App Mesh resources for a Kubernetes cluster.
  • aws-app-mesh-inject that will be responsible for automatically inject the App Mesh container as a sidecar. To enable sidecar injection for a namespace, it is necessary to label the namespace with appmesh.k8s.aws/sidecarInjectorWebhook=enabled
#install aws-app-mesh-controller-for-k8s
kubectl apply -f https://raw.githubusercontent.com/aws/aws-app-mesh-controller-for-k8s/v0.1.1/deploy/all.yaml

Now we will confirm that the Kubernetes custom resources for AWS App Mesh were created with the following command:

kubectl get crd

NAME                               CREATED AT
eniconfigs.crd.k8s.amazonaws.com   2019-10-26T00:51:33Z
meshes.appmesh.k8s.aws             2019-10-26T00:59:52Z
virtualnodes.appmesh.k8s.aws       2019-10-26T00:59:52Z
virtualservices.appmesh.k8s.aws    2019-10-26T00:59:53Z

The aws-app-mesh-inject add-on needs to know the name of the mesh before being installed.
We will use color-mesh as our mesh name, and to help with visibility and tracing, we will also add the X-Ray container as a sidecar:

#export mesh name
export MESH_NAME="color-mesh"

#install X-Ray sidecar
export INJECT_XRAY_SIDECAR="true"
export ENABLE_STATS_TAGS="true"
export ENABLE_STATSD="true"

#install the sidecar injector
curl https://raw.githubusercontent.com/aws/aws-app-mesh-inject/master/scripts/install.sh | bash

Deploying a mesh connected sample application

The sample application consists of two components:

  • ColorGateway – A simple HTTP service written in Go that is exposed to external clients and that responds to http://service-name:port/color. The gateway responds with a color retrieved from color-teller and a histogram of colors observed at the server that responded up to the point when you made the request.
  • ColorTeller – A simple HTTP service written in Go that is configured to return a color. Multiple variants of the service are deployed. Each service is configured to return a specific color.

ColorGateway isn’t aware of the multiple variants of the ColorTeller component. AWS App Mesh will expose one ColorTeller virtual service with three virtual nodes and three routes.

To deploy the sample application, apply the following file to your Kubernetes cluster with the following command.

#install sample application
kubectl apply -f https://raw.githubusercontent.com/fmedery/blogs/GlooPost1/eks-appmesh-gloo/color.yaml

You can verify all the deployed objects using the following command (notice that two extra containers are running in each pod: one for AWS X-Ray and one for AWS App Mesh.

kubectl -n appmesh-demo get all
NAME                                     READY   STATUS    RESTARTS   AGE
pod/colorgateway-957f89b6b-wrcsb         3/3     Running   0          45s
pod/colorteller-759fc757cc-72kbw         3/3     Running   0          44s
pod/colorteller-black-6c5dd7689c-lxqzl   3/3     Running   0          43s
pod/colorteller-blue-58dbf546d5-gz9t7    3/3     Running   0          43s

NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/colorgateway        ClusterIP   10.100.209.207   <none>        9080/TCP   45s
service/colorteller         ClusterIP   10.100.8.175     <none>        9080/TCP   44s
service/colorteller-black   ClusterIP   10.100.100.216   <none>        9080/TCP   44s
service/colorteller-blue    ClusterIP   10.100.41.247    <none>        9080/TCP   43s

NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/colorgateway        1/1     1            1           46s
deployment.apps/colorteller         1/1     1            1           45s
deployment.apps/colorteller-black   1/1     1            1           44s
deployment.apps/colorteller-blue    1/1     1            1           44s

NAME                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/colorgateway-957f89b6b         1         1         1       46s
replicaset.apps/colorteller-759fc757cc         1         1         1       45s
replicaset.apps/colorteller-black-6c5dd7689c   1         1         1       44s
replicaset.apps/colorteller-blue-58dbf546d5    1         1         1       44s

NAME                              AGE
mesh.appmesh.k8s.aws/color-mesh   48s

NAME                                            AGE
virtualnode.appmesh.k8s.aws/colorgateway        48s
virtualnode.appmesh.k8s.aws/colorteller         48s
virtualnode.appmesh.k8s.aws/colorteller-black   47s
virtualnode.appmesh.k8s.aws/colorteller-blue    47s

NAME                                                       AGE
virtualservice.appmesh.k8s.aws/colorgateway.appmesh-demo   46s
virtualservice.appmesh.k8s.aws/colorteller.appmesh-demo    47s

Installing Gloo

Now that our sample application has been installed, we will install Gloo ingress gateway.

#install gloo gateway using the gloo command line interface
glooctl install gateway

We can get an overview of all of the resources running using the command below (notice that Gloo automatically provisioned an AWS Elastic Load Balancer).

kubectl get all -n gloo-system
NAME                                    READY   STATUS      RESTARTS   AGE
pod/discovery-799484bcc4-nr2v2          1/1     Running     0          49s
pod/gateway-certgen-kr5gv               0/1     Completed   0          56s
pod/gateway-proxy-v2-6476c4759f-v82vm   1/1     Running     0          48s
pod/gateway-v2-8679b5cb57-t4smr         1/1     Running     0          49s
pod/gloo-5578cbc65b-dqqk9               1/1     Running     0          50s

NAME                       TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)                      AGE
service/gateway            ClusterIP      10.100.7.155     <none>                                                                    443/TCP                      51s
service/gateway-proxy-v2   LoadBalancer   10.100.37.255    aada551fcf78c11e9b9fe02975132263-1637777701.us-west-2.elb.amazonaws.com   80:30454/TCP,443:32704/TCP   50s
service/gloo               ClusterIP      10.100.222.125   <none>                                                                    9977/TCP,9988/TCP,9966/TCP   51s

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/discovery          1/1     1            1           49s
deployment.apps/gateway-proxy-v2   1/1     1            1           48s
deployment.apps/gateway-v2         1/1     1            1           49s
deployment.apps/gloo               1/1     1            1           50s

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/discovery-799484bcc4          1         1         1       49s
replicaset.apps/gateway-proxy-v2-6476c4759f   1         1         1       48s
replicaset.apps/gateway-v2-8679b5cb57         1         1         1       49s
replicaset.apps/gloo-5578cbc65b               1         1         1       50s

NAME                        COMPLETIONS   DURATION   AGE
job.batch/gateway-certgen   1/1           7s         57s

NAME                              AGE
mesh.appmesh.k8s.aws/color-mesh   2m

Now, let’s have a look at the pods:

  • pod/discovery-799484bcc4-nr2v2: this component is responsible for dynamically discovering services to which Gloo can route (upstreams).
  • pod/gateway-proxy-v2-6476c4759f-v82vm: this is the Envoy proxy
  • pod/gateway-v2-8679b5cb57-t4smr: this component allows users to configure an Envoy Proxy and also generates configuration that the Gloo control plane can use to generate Envoy configuration through xDS
  • pod/gloo-5578cbc65b-dqqk9: an event-driven component responsible for generating configuration for and serving the core xDS services and configuration of custom Envoy filters.
    Because of this decoupling, developers contributing to Gloo can easily add support for other architectures like Knative for example.

Allowing Ingress connectivity using Gloo

Let’s dig a little deeper into two of Gloo core concepts Virtual Service and Upstream:

  • Virtual Service defines a set of route rules that live under a domain or set of domains. Route rules consist of a matcher, which specifies the kind of function calls to match (requests and events, are currently supported), and the name of the destination (or destinations) to route them to.
  • Upstream defines destinations for routes. Upstreams tell Gloo what to route to. Upstreams are automatically discovered by Gloo.

We will first verify that the ColorGateway Virtual Service was discovered by Gloo.

glooctl get upstreams|grep colorgateway

| appmesh-demo-colorgateway-9080    | Kubernetes | Accepted | svc name: colorgateway |
| appmesh-demo-colorgateway-v1-9080 | Kubernetes | Accepted | svc name: colorgateway |

We will create a Gloo Virtual Service called colorgateway and a route that will redirect the path /appmesh/color to the virtual service appmesh-demo-colorgateway-9080 using /color as his path.

#create a virtual service called colorgateway
glooctl create virtualservice --name colorgateway

#add a route to the virtual service
glooctl add route \
            --name colorgateway \
            --path-prefix /appmesh/color \
            --prefix-rewrite /color \
            --dest-name appmesh-demo-colorgateway-9080
+-----------------+--------------+---------+------+----------+-----------------+---------------------------------------------+
| VIRTUAL SERVICE | DISPLAY NAME | DOMAINS | SSL  |  STATUS  | LISTENERPLUGINS |                   ROUTES                    |
+-----------------+--------------+---------+------+----------+-----------------+---------------------------------------------+
| colorgateway    | colorgateway | *       | none | Accepted |                 | /appmesh/color ->                           |
|                 |              |         |      |          |                 | gloo-system.appmesh-demo-colorgateway-9080  |
|                 |              |         |      |          |                 | (upstream)                                  |
+-----------------+--------------+---------+------+----------+-----------------+---------------------------------------------+

Now we will try to connect to the application.

#verify that the load balancer URL exists
#creating a load balancer can take a few minutes
glooctl proxy url

http://aa1706959c82111e9bdfc02f316f0629-1493319117.us-west-2.elb.amazonaws.com:80

#test the connectivity
for i in {1..10}; do curl $(glooctl proxy url)/appmesh/color; echo; done
{"color":"blue", "stats": {"black":0.3,"blue":0.13,"white":0.57}}
{"color":"white", "stats": {"black":0.29,"blue":0.13,"white":0.58}}
{"color":"white", "stats": {"black":0.28,"blue":0.12,"white":0.6}}
{"color":"black", "stats": {"black":0.31,"blue":0.12,"white":0.58}}
{"color":"black", "stats": {"black":0.33,"blue":0.11,"white":0.56}}
{"color":"white", "stats": {"black":0.32,"blue":0.11,"white":0.57}}
{"color":"blue", "stats": {"black":0.31,"blue":0.14,"white":0.55}}
{"color":"blue", "stats": {"black":0.3,"blue":0.17,"white":0.53}}
{"color":"black", "stats": {"black":0.32,"blue":0.16,"white":0.52}}
{"color":"black", "stats": {"black":0.34,"blue":0.16,"white":0.5}}

We can now connect to the ColorGateway from outside of the EKS cluster using the route /appmesh/color

Canary Deployment using Gloo weighted destinations

With this section, we will introduce another important concept of Gloo, Upstream Group:

  • Upstream Group is a top-level object, that let you logically groups upstreams , giving you the ability to address them as a group in distinct VirtualServices

This is a common requirement for Canary deployments where you want all calling routes to forward traffic equally across the two service versions.

For this example, we will start by deploying a new version of the gateway called colorteller-gatewayV2.

#deploy colorteller-gatewayV2
kubectl apply -f https://raw.githubusercontent.com/fmedery/blogs/GlooPost1/eks-appmesh-gloo/color-gateway-v2.yaml

#verify if Gloo was able to discover it
glooctl get upstream | grep gatewayv2
|appmesh-demo-colorgatewayv2-9080   | Kubernetes| Accepted| svc name: colorgatewayv2|
|appmesh-demo-colorgatewayv2-v2-9080| Kubernetes| Accepted| svc name: colorgatewayv2|

Now we will create an upstream group called upstreamgroup-gateway and add routes that will split the traffic between colorteller-gateway (80%) and colorteller-gatewayv2 (20%).

#delete the virtual service previously created
glooctl delete vs --name colorgateway

#recreate the colorgateway virtual service with an upstream group has its backend
kubectl apply -f https://raw.githubusercontent.com/fmedery/blogs/GlooPost1/eks-appmesh-gloo/01_canary.yaml

upstreamgroup.gloo.solo.io/upstreamgroup-gateway created
virtualservice.gateway.solo.io/colorgateway configured

#verify the upstreamgroup has been created
glooctl get upstreamgroup --name upstreamgroup-gateway
+-----------------------+----------+--------------+----------------------------------+
|    UPSTREAM GROUP     |  STATUS  | TOTAL WEIGHT |             DETAILS              |
+-----------------------+----------+--------------+----------------------------------+
| upstreamgroup-gateway | Accepted | 100          | destination type: Upstream       |
|                       |          |              | namespace: gloo-system           |
|                       |          |              | name:                            |
|                       |          |              | appmesh-demo-colorgateway-9080   |
|                       |          |              | weight: 80   % total: 0.80       |
|                       |          |              |                                  |
|                       |          |              |                                  |
|                       |          |              | destination type: Upstream       |
|                       |          |              | namespace: gloo-system           |
|                       |          |              | name:                            |
|                       |          |              | appmesh-demo-colorgatewayv2-9080 |
|                       |          |              | weight: 20   % total: 0.20       |
+-----------------------+----------+--------------+----------------------------------+

#verify that we have a route pointing to the upstreamgroup
 glooctl get vs --name colorgateway
+-----------------+--------------+---------+------+----------+-----------------+-----------------------------------+
| VIRTUAL SERVICE | DISPLAY NAME | DOMAINS | SSL  |  STATUS  | LISTENERPLUGINS |              ROUTES               |
+-----------------+--------------+---------+------+----------+-----------------+-----------------------------------+
| colorgateway    |              | *       | none | Accepted |                 | /appmesh/color -> upstream group: |
|                 |              |         |      |          |                 | upstreamgroup-gateway.gloo-system |
+-----------------+--------------+---------+------+----------+-----------------+-----------------------------------+
#test the new route
for i in {1..10}; do curl $(glooctl proxy url)/appmesh/color; echo; donex`
{"color":"white", "stats": {"black":0.32,"blue":0.35,"white":0.33}}
{"color":"white", "stats": {"black":0.31,"blue":0.34,"white":0.34}}
{"color":"blue", "stats": {"black":0.31,"blue":0.36,"white":0.34}}
{"colorV2":"white", "statsV2": {"black":0.24,"blue":0.57,"white":0.19}}
{"color":"blue", "stats": {"black":0.3,"blue":0.37,"white":0.33}}
{"color":"black", "stats": {"black":0.31,"blue":0.36,"white":0.33}}
{"color":"blue", "stats": {"black":0.31,"blue":0.37,"white":0.32}}
{"color":"black", "stats": {"black":0.32,"blue":0.37,"white":0.32}}
{"colorV2":"blue", "statsV2": {"black":0.23,"blue":0.59,"white":0.18}}
{"color":"blue", "stats": {"black":0.31,"blue":0.38,"white":0.31}}

When we are confident that the new version is behaving as expected, we can increase the traffic sent to colorteller-gatewayV2.

#update the weight of each route to 50%
kubectl apply -f https://raw.githubusercontent.com/fmedery/blogs/GlooPost1/eks-appmesh-gloo/02_canary.yaml
upstreamgroup.gloo.solo.io/upstreamgroup-gateway configured

#verify the weight for each route has been updated to 50%
glooctl get upstreamgroup --name upstreamgroup-gateway
+-----------------------+----------+--------------+----------------------------------+
|    UPSTREAM GROUP     |  STATUS  | TOTAL WEIGHT |             DETAILS              |
+-----------------------+----------+--------------+----------------------------------+
| upstreamgroup-gateway | Accepted | 100          | destination type: Upstream       |
|                       |          |              | namespace: gloo-system           |
|                       |          |              | name:                            |
|                       |          |              | appmesh-demo-colorgateway-9080   |
|                       |          |              | weight: 50   % total: 0.50       |
|                       |          |              |                                  |
|                       |          |              |                                  |
|                       |          |              | destination type: Upstream       |
|                       |          |              | namespace: gloo-system           |
|                       |          |              | name:                            |
|                       |          |              | appmesh-demo-colorgatewayv2-9080 |
|                       |          |              | weight: 50   % total: 0.50       |
+-----------------------+----------+--------------+----------------------------------+

#test the route again
for i in {1..10}; do curl $(glooctl proxy url)/appmesh/color; echo; done
{"colorV2":"white", "statsV2": {"black":0.22,"blue":0.57,"white":0.22}}
{"color":"blue", "stats": {"black":0.31,"blue":0.38,"white":0.31}}
{"color":"white", "stats": {"black":0.3,"blue":0.38,"white":0.32}}
{"colorV2":"white", "statsV2": {"black":0.21,"blue":0.54,"white":0.25}}
{"colorV2":"blue", "statsV2": {"black":0.2,"blue":0.56,"white":0.24}}
{"color":"black", "stats": {"black":0.31,"blue":0.37,"white":0.31}}
{"colorV2":"white", "statsV2": {"black":0.19,"blue":0.54,"white":0.27}}
{"colorV2":"white", "statsV2": {"black":0.19,"blue":0.52,"white":0.3}}
{"colorV2":"black", "statsV2": {"black":0.21,"blue":0.5,"white":0.29}}
{"color":"black", "stats": {"black":0.32,"blue":0.37,"white":0.31}}

Now we will route all the traffic to colorteller-gatewayv2 by removing colorteller-gateway from the upstream group:

# update the upstream group
kubectl apply -f https://raw.githubusercontent.com/fmedery/blogs/GlooPost1/eks-appmesh-gloo/03_canary.yaml

#verify the upstreamgroup
glooctl get upstreamgroup --name upstreamgroup-gateway
+-----------------------+----------+--------------+-------------------------------------+
|    UPSTREAM GROUP     |  STATUS  | TOTAL WEIGHT |               DETAILS               |
+-----------------------+----------+--------------+-------------------------------------+
| upstreamgroup-gateway | Accepted | 100          | destination type: Upstream          |
|                       |          |              | namespace: gloo-system              |
|                       |          |              | name:                               |
|                       |          |              | appmesh-demo-colorgatewayv2-9080    |
|                       |          |              | weight: 100   % total: 1.00         |
+-----------------------+----------+--------------+-------------------------------------+

#test one last time
for i in {1..10}; do curl $(glooctl proxy url)/appmesh/color; echo; done
{"colorV2":"black", "statsV2": {"black":0.24,"blue":0.48,"white":0.28}}
{"colorV2":"blue", "statsV2": {"black":0.23,"blue":0.5,"white":0.27}}
{"colorV2":"white", "statsV2": {"black":0.23,"blue":0.48,"white":0.29}}
{"colorV2":"blue", "statsV2": {"black":0.22,"blue":0.5,"white":0.28}}
{"colorV2":"black", "statsV2": {"black":0.24,"blue":0.48,"white":0.27}}
{"colorV2":"white", "statsV2": {"black":0.24,"blue":0.47,"white":0.29}}
{"colorV2":"white", "statsV2": {"black":0.23,"blue":0.46,"white":0.31}}
{"colorV2":"black", "statsV2": {"black":0.25,"blue":0.44,"white":0.31}}
{"colorV2":"blue", "statsV2": {"black":0.24,"blue":0.46,"white":0.3}}
{"colorV2":"blue", "statsV2": {"black":0.24,"blue":0.47,"white":0.29}

Gloo is now routing all the traffic hitting /appmesh/color to colorteller-gatewayv2.

Conclusion

In this post, we demonstrated how Gloo can transparently interact with AWS App Mesh and how easily developers can create and manipulate access to their applications. We also showcase how to use Gloo to create a canary deployment.

To learn more about Gloo advanced features, follow this link.

In an upcoming post, we will take a look at another solo.io product: supergloo, the Service Mesh Orchestration Platform.