Manage Amazon OpenSearch Service Visualizations, Alerts, and More with GitHub and Jenkins

Amazon OpenSearch Service is a fully managed service for search and analytics. It allows organizations to secure data, perform searches, analyze logs, monitor applications in real time, and explore interactive log analytics. With its scalability, reliability, and ease of use, Amazon OpenSearch Service helps businesses optimize data-driven decisions and improve operational efficiency. AWS handles the heavy lifting of managing the underlying infrastructure, including service installation, configuration, replication, and backups, so you can focus on the business side of your application.

OpenSearch Service stores different types of stored objects, such as dashboards, visualizations, alerts, security roles, index templates, and more, within the domain. As your user base and number of Amazon OpenSearch Service domains grow, tracking activities and changes to those saved objects becomes increasingly difficult. It is advised to discourage contributors from making changes directly to the production OpenSearch Service domain and instead implement a gatekeeper process to validate and test the changes before moving them to OpenSearch Service.

In this post, we present a solution to deploy stored objects using GitHub and Jenkins while preventing users making direct changes into OpenSearch Service domain.

Solution overview

The solution consists of three personas:

Contributor – The contributor creates, updates, or deletes stored objects like alerts or dashboards
User – This persona uses the dashboard without changing any stored objects
Approver – The approver reviews and approves the stored object changes in GitHub pull requests

The following diagram illustrates the architecture of making stored object changes directly in the OpenSearch Service domain. As organizations scale their OpenSearch service domain usage, managing and tracking changes to stored objects across multiple domains become increasingly challenging.

The following diagram illustrates solution architecture, which manages stored objects using a continuous integration and delivery (CI/CD) pipeline.

Proposed Solution

The workflow to deploy stored object changes consists of the following steps:

Contributor commits changes as JSON files to GitHub repositories.
OpenSearch Service admin (approver) reviews the change, approves the pull request and merges changes to the repository.
A new commit invokes a build job in Jenkins.
Jenkins retrieves JSON files from the GitHub repository and performs validation.
Jenkins runs an OpenSearch Service API to deploy changes.

CI/CD automates code integration, testing, and deployment: CI focuses on frequent code merges and automated testing, and CD makes sure code is always deployment-ready, often pushing updates directly to production. This leads to faster, more reliable software releases.

In the following sections, we demonstrate how to build a CI/CD pipeline for deploying OpenSearch Service security roles using GitHub and Jenkins. By using a multi-branch pipeline in Jenkins, you can create pipelines for different types of stored objects. You can adapt the proposed architecture with the tools of your choice.

Prerequisites

To implement this solution, you need an AWS account. If you don’t have one, you can create an account. You also need an Amazon OpenSearch Service domain. You can use an existing domain or create a new domain.

It is recommended to deploy the OpenSearch Service domain in a VPC to make it secure.

Create an IAM role

In this step, you create an AWS Identity and Access Management (IAM) role in AWS for Jenkins to use when connecting to Amazon OpenSearch Service for push changes. Then you map this role to Amazon OpenSearch Service domain security roles.

Complete the following steps:

On the IAM console, create a policy called CodeCommitPolicy with the following permissions (replace the account ID and Amazon OpenSearch Service domain name):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "es:DescribeDomain",
            "Resource": "arn:aws:es:*:{your-account-id}:domain/*"
        },
        {
            "Effect": "Allow",
            "Action": "es:ESHttp*",
            "Resource": "arn:aws:es:*:{your-account-id}:domain/{domain-namedomain}/*"
        }
    ]
}

Assign this policy to the IAM role CodeCommitRole.

In the trust policy, specify that Amazon Elastic Compute Cloud (Amazon EC2) can assume this role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Make a note of the role ARN.
Open the Amazon OpenSearch Service dashboard using the OpenSearch Dashboards URL.

In the navigation panel, choose Security, then choose Roles.
Map the IAM role from the previous steps against the security_manager role of the Amazon OpenSearch Service domain to enable API invocation.

You can map the IAM role to the Amazon OpenSearch Service domain role based on the type of stored object you’re deploying.

Launch an EC2 instance

Note : Make sure to deploy the EC2 instance for hosting Jenkins in the same VPC as the OpenSearch domain.

Complete the following steps to set up an EC2 instance for installing Jenkins:

Launch an EC2 instance with the latest Amazon Linux 2023 AMI.
In the advanced details, choose IAM instance profile and map the role CodeCommitRole that you created earlier against this EC2 instance.
Open port 22 for SSH connections and port 8080, which is used by the Jenkins service.
Python comes bundled with the latest Amazon Linux AMIs. Make sure the Python version is later than 2.7.0:
```
python3 --version
```

Install and configure Jenkins on an EC2 server

In this step, you install and configure Jenkins on an EC2 server. Complete the following steps on the Jenkins EC2 instance terminal created in the previous step:

Connect to your instance using System manager:
```
sudo dnf update
```

Add the Jenkins repo using the following command:

sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo

Import a key file to enable installation from the package:

sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io-2023.key

Upgrade all Yum packages:
```
sudo dnf upgrade
```

Install Java (Amazon Linux 2023):

sudo dnf install java-17-amazon-corretto -y

Install Jenkins:
```
sudo dnf install jenkins -y
```
Enable the Jenkins service to start at boot:
```
sudo systemctl enable Jenkins
```
Start Jenkins as a service:
```
sudo systemctl start Jenkins
```

Configure the Jenkins server

Jenkins should now be running on port 8080 of the EC2 instance. You can access it using a web browser by connecting to the public IPv4 DNS of the EC2 instance, which can be found on the EC2 instance details page on the Amazon EC2 console. By default, the Jenkins server runs on the HTTP protocol. To make it secure, you can run it behind Application Load Balancer or behind the NGINX proxy.

Open the Jenkins web console in your browser using the URL below, and replace the EC2 IP address.
http://EC2 IPv4 DNS:8080

When Jenkins opens for the first time , it will ask for the administrator password. Use the admin password found at the following location on the Ec2 instance to unlock it

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Then you can create an admin user name and password.

Create GitHub repository

In this step, you create a GitHub repository to store Amazon OpenSearch Service security roles in JSON format. Complete the following steps:

Go to github.com and log in or create an account. Note : If you’re deploying the Jenkins server inside a VPC, you can connect the repository accessible within the VPC. To connect to a public GitHub repository, use an Internet Gateway.
Choose New repository.
Provide the repository name as openSearch2 and create it as a private repository.
On the Settings menu, choose Developer Settings, Personal Access Token, Tokens (classic), and create a token, which will be used by Jenkins to fetch files from the GitHub repository.

Create a Jenkins job

In this step, you create a Jenkins job that will pick up new changes committed to the GitHub repository and validate and deploy the changes to Amazon OpenSearch Service.

Open Jenkins from a web browser using the user name and password created earlier, then complete the following steps on the Jenkins web interface:

On the Jenkins home page, choose New Item.
Enter the name of the pipeline and select Freestyle Project.
Choose OK to continue.
Provide a description of the Jenkins job.
In the General section, choose Advanced.
Set a custom workspace for Jenkins to store files from the GitHub repository:
```
/var/lib/jenkins/workspace/opensearch
```
In the source code management, choose Git.
Enter the URL of the GitHub repository you created (https://github.com/<accountowner>/opensearch.git).
In the Credential section, choose Add to add the access token created for the GitHub repository.
Leave the settings as default.
In the Build Triggers section, set Poll SCM as H/05 * * * *.

A real-life delivery pipeline will have more than one stage: one each for build, test, and deploy, where you can put your custom logic to validate the syntax of JSON files and business logic, and take a series of actions before deploying files into an OpenSearch Service domain. In this post, we concentrate on the deployment stage only.

Jenkinsfile (Declarative Pipeline)
pipeline {
    agent any
    options {
        skipStagesAfterUnstable()
    }
    stages {
        stage('Build') {
            steps {
                echo 'Building'
            }
        }
        stage('Test') {
            steps {
                echo 'Testing'
            }
        }
        stage('Deploy') {
            steps {
                echo 'Deploying'
            }
        }
    }
}

Enter the following command in the build steps and click on save to save Jenkins job.
```
/usr/bin/python3 /opensearch/roles.py
```

Place the Python script on the Jenkins server and install dependencies

In this step, you install Python packages and create the Python file on the Jenkins server. These files will read stored object JSON files, connect to Amazon OpenSearch Service domain, and call the create_role API of OpenSearch Service.

Run the following steps on the Jenkins server:

Install python3-pip:
```
sudo yum install python3-pip
```

Install the required Python packages:

sudo python3 -m pip install opensearch-py requests_aws4auth boto3

Create a new file called users.py in the /opensearch location of the Jenkins server and enter the following code into it:

from opensearchpy import OpenSearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth
import boto3
import os
import glob

host = 'search-XXXXXXX-XXXXXXXXXXXXX.us-east-1.es.amazonaws.com' # e.g. my-test-domain.us-east-1.es.amazonaws.com, this will be different for VPC hosted domain
region = 'us-east-1' # e.g. us-west-1
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

# Create the client.
search = OpenSearch(
    hosts = [{'host': host, 'port': 443}],
    http_auth = awsauth,
    use_ssl = True,
    verify_certs = True,
    http_compress = True, # enables gzip compression for request bodies
    connection_class = RequestsHttpConnection
)

pythonFiles = [file for dirs in os.walk('/var/lib/jenkins/workspace/opensearch2/', topdown=True)
                     for file in dirs[2] if file.endswith(".json")]
for r in pythonFiles:
    path = "/var/lib/jenkins/workspace/opensearch2/" + r
    print(path)
    role_name = r.rsplit('.', 1)[0]
    data = open(path, 'r').read()
    response = search.security.create_role(role_name, body=data)
    print(response)

Test the solution

To test the solution, push the new role file itadminrole.json into the GitHub repository and build a job that will create the role itadminrole in the Amazon OpenSearch service domain.

Jenkins will fetch the changes and run the job in the next scan. You can also run a build job manually from the Jenkins website, which will create the role itadminrole in an OpenSearch Service domain.

We have observed how to automate the build cycle, starting with pushing changes to security roles, followed by the OpenSearch admin’s approval and PR merge. This action invokes a Jenkins build job that handles validation, testing, and deployment.

Clean up

To avoid incurring future charges, delete the resources created while building this solution:

Conclusion

In this post, we demonstrated how to build a CI/CD pipeline for deploying Amazon OpenSearch Service security roles using GitHub and Jenkins. With this approach, you can help prevent users from directly committing changes to the production OpenSearch Service domain. The solution tracks changes made by each user, performs additional validation checks according to business rules, and conducts testing, before finally pushing the changes into the OpenSearch Service domain. Another advantage of this method is the ability to migrate or reuse the same stored objects across multiple Amazon OpenSearch Service domains by changing the endpoint of the domain, eliminating the need for manual export and import.

If you are struggling to manage frequent stored object changes in your OpenSearch Service domain—such as dashboards, roles, and alerts etc. Consider this solution to streamline changes and easily replicate stored objects across other OpenSearch Service domains. Give it a try and share your feedback or questions in the comment.

About the Author

Bharav Patel is a Specialist Solution Architect with Amazon OpenSearch service. He primarily works on Amazon OpenSearch Service and helps customers with key concepts and design principles of running OpenSearch workload on the cloud. Bharav likes to explore new places and try out different cuisines.

AWS Big Data Blog