AWS Security Blog

How to Control Access to Your Amazon Elasticsearch Service Domain

September 9, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details.

With the recent release of Amazon Elasticsearch Service (Amazon ES), you now can build applications without setting up and maintaining your own search cluster on Amazon EC2. One of the key benefits of using Amazon ES is that you can leverage AWS Identity and Access Management (IAM) to grant or deny access to your search domains. In contrast, if you were to run an unmanaged Elasticsearch cluster on AWS, leveraging IAM to authorize access to your domains would require more effort.

In this blog post, I will cover approaches for using IAM to set permissions for an Amazon ES deployment. I will start by considering the two broad options available for Amazon ES: resource-based permissions and identity-based permissions. I also will explain Signature Version 4 signing, and look at some real-world scenarios and approaches for setting Amazon ES permissions. Last, I will present an architecture for locking down your Amazon ES deployment by leveraging a proxy, while still being able to use Kibana for analytics.

Note: This blog post assumes that you are already familiar with setting up an Amazon ES cluster. To learn how to set up an Amazon ES cluster before proceeding, see New – Amazon Elasticsearch Service.

Options for granting or denying access to Amazon ES endpoints

In this section, I will provide details about how you can configure your Amazon ES domains so that only trusted users and applications can access them. In short, Amazon ES adds support for an authorization layer by integrating with IAM. You write an IAM policy to control access to the cluster’s endpoint, allowing or denying Actions (HTTP methods) against Resources (the domain endpoint, indices, and API calls to Amazon ES). For an overview of IAM policies, see Overview of IAM Policies.

You attach the policies that you build in IAM or in the Amazon ES console to specific IAM entities (in other words, the Amazon ES domain, users, groups, and roles):

  1. Resource-based policies – This type of policy is attached to an AWS resource, such as an Amazon S3 bucket, as described in Writing IAM Policies: How to Grant Access to an Amazon S3 Bucket.
  2. Identity-based policies – This type of policy is attached to an identity, such as an IAM user, group, or role.

The union of all policies covering a specific entity, resource, and action controls whether the calling entity is authorized to perform that action on that resource

A note about authentication, which applies to both types of policies: you can use two strategies to authenticate Amazon ES requests. The first is based on the originating IP address. You can omit the Principal from your policy and specify an IP Condition. In this case, and barring a conflicting policy, any call from that IP address will be allowed access or be denied access to the resource in question. The second strategy is based on the originating Principal. In this case, you are required to include information that AWS can use to authenticate the requestor as part of every request to your Amazon ES endpoint, which you accomplish by signing the request using Signature Version 4. Later in this post, I provide an example of how you can sign a simple request against Amazon ES using Signature Version 4. With that clarification about authentication in mind, let’s start with how to configure resource-based policies.

How to configure resource-based policies
A resource-based policy is attached to the Amazon ES domain (accessible through the domain’s console) and enables you to specify which AWS account and which AWS users or roles can access your Amazon ES endpoint. In addition, a resource-based policy lets you specify an IP condition for restricting access based on source IP addresses. The following screenshot shows the Amazon ES console pane where you configure the resource-based policy of your endpoint.

Important note that applies throughout this post: Policies that contain es:* are for illustrative purposes only. Please ensure that you restrict actions appropriately for the desired users.

Screenshot of configuring the resource-based policy

In the preceding screenshot, you can see that the policy is attached to an Amazon ES domain called recipes1, which is defined in the Resource section of the policy. The policy itself has a condition specifying that only requests from a specific IP address should be allowed to issue requests against this domain (though not shown here, you can also specify an IP range using Classless Inter-Domain Routing [CIDR] notation).

In addition to IP-based restrictions, you can restrict Amazon ES endpoint access to certain AWS accounts or users. The following code shows a sample resource-based policy that allows only the IAM user recipes1alloweduser to issue requests. (Be sure to replace placeholder values with your own AWS resource information.)

{
  "Version": "2012-10-17",
  "Statement": [{
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111111111111:user/recipes1alloweduser"
      },      
      "Action": "es:*", 
      "Resource": "arn:aws:es:us-west-2:111111111111:domain/recipes1/*" 
    }   
  ] 
}

This sample policy grants recipes1alloweduser the ability to perform any Amazon ES–related actions (represented by “Action”:”es:*”) against the recipes1 domain.

For the preceding policy, you must issue a Signature Version 4 signed request; see Examples of the Complete Version 4 Signing Process (Python) for more information. Because those examples are in Python, I am including the following code for Java developers that illustrates how to issue a Signature Version 4 signed request to an Amazon ES endpoint. The sample code shown breaks down the signing process into three main parts that are contained in the functions: generateRequest(), performSigningSteps(), and sendRequest(). Most of the action related to signing takes place in the performSigningSteps() function, and you will need to download and refer to the AWS SDK for Java to use classes such as AWS4Signer that are used in that function.

By using the SDK, you hand over all the heavy lifting associated with signing to the SDK. You simply have to set up the request, provide the key parameters required for signing (such as service name, region, and your credentials), and call the sign method on the AWS4Signer class. Be sure that you avoid hard-coding your credentials in your code.

/// Set up the request
private static Request<?> generateRequest() {
       Request<?> request = new DefaultRequest<Void>(SERVICE_NAME);
       request.setContent(new ByteArrayInputStream("".getBytes()));
       request.setEndpoint(URI.create(ENDPOINT));
       request.setHttpMethod(HttpMethodName.GET);
       return request;
}

/// Perform Signature Version 4 signing
private static void performSigningSteps(Request<?> requestToSign) {
       AWS4Signer signer = new AWS4Signer();
       signer.setServiceName(SERVICE_NAME);
       signer.setRegionName(REGION);      

       // Get credentials
       // NOTE: *Never* hard-code credentials
       //       in source code
       AWSCredentialsProvider credsProvider =
                     new DefaultAWSCredentialsProviderChain();

       AWSCredentials creds = credsProvider.getCredentials();

       // Sign request with supplied creds
       signer.sign(requestToSign, creds);
}

/// Send the request to the server
private static void sendRequest(Request<?> request) {
       ExecutionContext context = new ExecutionContext(true);

       ClientConfiguration clientConfiguration = new ClientConfiguration();
       AmazonHttpClient client = new AmazonHttpClient(clientConfiguration);

       MyHttpResponseHandler<Void> responseHandler = new MyHttpResponseHandler<Void>();
       MyErrorHandler errorHandler = new MyErrorHandler();

       Response<Void> response =
                     client.execute(request, responseHandler, errorHandler, context);
}

public static void main(String[] args) {
       // Generate the request
       Request<?> request = generateRequest();

       // Perform Signature Version 4 signing
       performSigningSteps(request);

       // Send the request to the server
       sendRequest(request);
}

public static class MyHttpResponseHandler<T> implements HttpResponseHandler<AmazonWebServiceResponse<T>> {

       @Override
       public AmazonWebServiceResponse<T> handle(
                        com.amazonaws.http.HttpResponse response) throws Exception {

               InputStream responseStream = response.getContent();
               String responseString = convertStreamToString(responseStream);
               System.out.println(responseString);

               AmazonWebServiceResponse<T> awsResponse = new AmazonWebServiceResponse<T>();
               return awsResponse;
       }

       @Override
       public boolean needsConnectionLeftOpen() {
               return false;
       }
}

public static class MyErrorHandler implements HttpResponseHandler<AmazonServiceException> {

       @Override
       public AmazonServiceException handle(
                        com.amazonaws.http.HttpResponse response) throws Exception {
               System.out.println("In exception handler!");

               AmazonServiceException ase = new AmazonServiceException("Fake service exception.");
               ase.setStatusCode(response.getStatusCode());
               ase.setErrorCode(response.getStatusText());
               return ase;
         }

       @Override
       public boolean needsConnectionLeftOpen() {
               return false;
         }
}

Keep in mind that your own generateRequest method will be specialized to your application, including request type and content body. The values of the referenced variables (shown in red) are as follows.

private static final String SERVICE_NAME = "es";
private static final String REGION = "us-west-2";
private static final String HOST = "search-recipes1-xxxxxxxxx.us-west-2.es.amazonaws.com";
private static final String ENDPOINT_ROOT = "https://" + HOST;
private static final String PATH = "/";
private static final String ENDPOINT = ENDPOINT_ROOT + PATH;

Again, be sure to replace placeholder values with your own AWS resource information, including the host value, which is generated as part of the cluster creation process.

How to configure identity-based policies
In contrast to resource-based policies, with identity-based policies you can specify which actions an IAM identity can perform against one or more AWS resources, such as an Amazon ES domain or an S3 bucket. For example, the following sample inline IAM policy is attached to an IAM user.

{
 "Version": "2012-10-17",
 "Statement": [
  {
   "Resource": "arn:aws:es:us-west-2:111111111111:domain/recipes1/*",
   "Action": ["es:*"],
   "Effect": "Allow"
  }
 ]
}

By attaching the preceding policy to an identity, you give that identity the permission to perform any actions against the recipes1 domain. To issue a request against the recipes1 domain, you would use Signature Version 4 signing as described earlier in this post.

With Amazon ES, you can lock down access even further. Let’s say that you wanted to organize access based on job functions and roles, and you have three users who correspond to three job functions:

  • esadmin: The administrator of your Amazon ES clusters.
  • poweruser: A power user who can access all domains, but cannot perform management functions.
  • analyticsviewer: A user who can only read data from the analytics index.

Given this division of responsibilities, the following policies correspond to each user.

Policy for esadmin

{
 "Version": "2012-10-17",
 "Statement": [
  {
   "Resource": "arn:aws:es:us-west-2:111111111111:domain/*",
   "Action": ["es:*"],
   "Effect": "Allow"
  }
 ]
}

The preceding policy allows the esadmin user to perform all actions (es:*) against all Amazon ES domains in the us-west-2 region.

Policy for poweruser

{
 "Version": "2012-10-17",
 "Statement": [
  {
   "Resource": "arn:aws:es:us-west-2:111111111111:domain/*",
   "Action": ["es:*"],
   "Effect": "Allow"
  },
  {
   "Resource": "arn:aws:es:us-west-2:111111111111:domain/*",
   "Action": ["es: DeleteElasticsearchDomain",
              "es: CreateElasticsearchDomain"],
   "Effect": "Deny"
  }
 ]
}

The preceding policy gives the poweruser user the same permission as the esadmin user, except for the ability to create and delete domains (the Deny statement).
Policy for analyticsviewer

{
 "Version": "2012-10-17",
 "Statement": [
  {
   "Resource":
    "arn:aws:es:us-west-2:111111111111:domain/recipes1/analytics",
   "Action": ["es:ESHttpGet"],
   "Effect": "Allow"
  }
 ]
}

The preceding policy gives the analyticsviewer user the abiity to issue HttpGet requests against the analytics index that is part of the recipes1 domain. This is a limited policy that prevents the analyticsviewer user from performing any other actions against that index or domain.

For more details about configuring Amazon ES access policies, see Configuring Access Policies. The specific policies I just shared and any other policies you create can be associated with an AWS identity, group, or role, as described in Overview of IAM Policies.

Combining resource-based and identity-based policies

Now that I have covered the two types of policies that you can use to grant or deny access to Amazon ES endpoints, let’s take a look at what happens when you combine resource-based and identity-based policies. First, why would you want to combine these two types of policies? One use case involves cross-account access: you want to allow identities in a different AWS account to access your Amazon ES domain. You could configure a resource-based policy to grant access to that account ID, but an administrator of that account would still need to use identity-based policies to allow identities in that account to perform specific actions against your Amazon ES domain. For more information about how to configure cross-account access, see Tutorial: Delegate Access Across AWS Accounts Using IAM Roles.

The following table summarizes the results of mixing policy types.

Table showing the results of mixing policy types

One of the key takeaways from the preceding table is that a Deny always wins if one policy type has an Allow and there is a competing Deny in the other policy type. Also, when you do not explicitly specify a Deny or Allow, access is denied by default. For more detailed information about combining policies, see Policy Evaluation Basics.

Deployment considerations

With the discussion about the two types of policies in mind, let’s step back and look at deployment considerations. Kibana, which is a JavaScript-based UI that accompanies Elasticsearch and Amazon ES, allows you to extract valuable insights from stored data. When you deploy Amazon ES, you must ensure that the appropriate users (such as administrators and business intelligence analysts) have access to Kibana while also ensuring that you provide secure access from your applications to your various Amazon ES search domains.

When leveraging resource-based or identity-based policies to grant or deny access to Amazon ES endpoints, clients can use either anonymous or IP-based policies, or they can use policies that specify a Principal as part of the Signature Version 4 signed requests. In addition, because Kibana is JavaScript, requests originate from the end user’s IP address. This makes unauthenticated, IP-based access control impractical in most cases because of the sheer number of IP addresses that you may need to whitelist.

Given this IP-based access control limitation, you need a way to present Kibana with an endpoint that does not require Signature Version 4 signing. One approach is to put a proxy between Amazon ES and Kibana, and then set up a policy that allows only requests from the IP address of this proxy. By using a proxy, you only have to manage a single IP address (that of the proxy). I describe this approach in the following section.

Proxy-based access to Amazon ES from Kibana

As mentioned previously, a proxy can funnel access for clients that need to use Kibana. This approach still allows nonproxy–based access for other application code that can issue Signature Version 4 signed requests. The following diagram illustrates this approach, including a proxy to funnel Kibana access.

Diagram of funneling access for clients that need to use Kibana

The key details of the preceding diagram are described as follows:

  1. This is your Amazon ES domain, which resides in your AWS account. IAM provides authorized access to this domain. An IAM policy provides whitelisted access to the IP address of the proxy server through which your Kibana client will connect.
  2. This is the proxy whose IP address is allowed access to your Amazon ES domain. You also could leverage an NGINX proxy, as described in the NGINX Plus on AWS whitepaper.
  3. Application code running on EC2 instances uses the Signature Version 4 signing process to issue requests against your Amazon ES domain.
  4. Your Kibana client application connects to your Amazon ES domain through the proxy.

To facilitate the security setup described in 1 and 2, you need a resource-based policy to lock down the Amazon ES domain. That policy follows.

{
 "Version": "2012-10-17",
 "Statement": [
  {
   "Resource":
    "arn:aws:es:us-west-2:111111111111:domain/recipes1/analytics",
   "Principal": {
        "AWS": "arn:aws:iam::111111111111:role/allowedrole1"
   },
   "Action": ["es:ESHttpGet"],
   "Effect": "Allow"
  },
  {
   "Effect": "Allow",
   "Principal": {
     "AWS": "*"
   },
   "Action": "es:*",
   "Condition": {
     "IpAddress": {
       "aws:SourceIp": [
         "AAA.BBB.CCC.DDD"
       ]
     }
   },
   "Resource":
    "arn:aws:es:us-west-2:111111111111:domain/recipes1/analytics"
  }
 ]
}

This policy allows clients—such as the app servers in the VPC subnet shown in the preceding diagram—that are capable of sending Signature Version 4 signed requests to access the Amazon ES domain. At the same time, the policy allows Kibana clients to access the domain via a proxy, whose IP address is specified in the policy: AAA.BBB.CCC.DDD. We strongly recommend that the EC2 instance running the proxy be configured with an Elastic IP. This way, you can replace the instance when necessary and still attach the same public IP to it. For added security, you can configure this proxy so that it authenticates clients, as described in Using NGINX Plus and NGINX to Authenticate Application Users with LDAP.

Conclusion

Using the techniques in this post, you can grant or deny access to your Amazon ES domains by using resource-based policies, identity-based policies, or both. As I showed, when accessing an Amazon ES domain, you must issue Signature Version 4 signed requests, which you can accomplish using the sample Java code provided. In addition, by leveraging the proxy-based topology shown in the last section of this post, you can present the Kibana UI to users without compromising security.

If you have questions or comments about this blog post, please submit them in the “Comments” section below, or contact:

– Jon and Karthi

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.