Amazon Managed Grafana – Getting Started

September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details.

Amazon Managed Grafana is a fully managed and secure data visualization service that enables customers to instantly query, correlate, and visualize operational metrics, logs, and traces for their applications from multiple data sources. Amazon Managed Grafana is based on the open source Grafana project, a widely deployed data visualization tool popular for its extensible data source support. Developed together with Grafana Labs, Amazon Managed Grafana manages the provisioning, setup, scaling, and maintenance of Grafana, eliminating the need for customers to do this themselves. Customers also benefit from built-in security features that enable compliance with governance requirements, including single sign-on, fine-grained data access control, and audit reporting. Amazon Managed Grafana is integrated with AWS data sources that collect operational data, such as Amazon CloudWatch, Amazon Elasticsearch Service, Amazon Timestream, AWS IoT SiteWise, AWS X-Ray, and Amazon Managed Service for Prometheus (AMP), and provides plug-ins to popular open-source databases, third-party ISV monitoring tools, as well as other cloud services. With Amazon Managed Grafana you can easily visualize information from multiple AWS services, AWS accounts, and Regions in a single Grafana dashboard.

You can also perform an in-place upgrade to Grafana Enterprise to get access to additional features and plugins. With Grafana Enterprise, you can consolidate your data from AppDynamics, DataDog, Dynatrace, New Relic, MongoDB, Oracle Database, ServiceNow, Snowflake, Splunk, and Wavefront. Additionally, you can access support and training content directly from Grafana Labs to help you explore and adopt Grafana’s advanced features easily. Click here to learn more.

Before you begin

To use Amazon Managed Grafana in a flexible and convenient manner, we chose to leverage AWS Single Sign-On (SSO) for user management. AWS SSO is available once you’ve enabled AWS Organizations. In order to check whether an AWS account is part of an AWS Organization, head over to https://console.aws.amazon.com/organizations/ and you should see a view akin to the following and if you do not see AWS Organizations activated go ahead and create one:

Existing AWS Organization

Amazon Managed Grafana integrates with AWS SSO so that you can easily assign users and groups from your existing user directory such as Active Directory, LDAP, or Okta within the Amazon Managed Grafana workspace and single sign on using your existing user ID and password. This allows you to enforce existing login security requirements for your company such as two-factor authentication and password complexity.

If you don’t have an existing user directory or do not want to integrate with external identity providers through the Security Assertion Markup Language (SAML) 2.0 standard, you can create local users and passwords within AWS SSO and use them to sign in to the Grafana workspace:

SSO console: add user

With the preparations around AWS SSO and AWS Organizations out of the way, we’re ready to get into the core Amazon Managed Grafana setup.

Setting up Amazon Managed Grafana on your AWS account

You can easily spin up on-demand, autoscaled Grafana workspaces (virtual Grafana servers) that enable you to create unified dashboards across multiple data sources. Before we can use Amazon Managed Grafana for the following example, we need to set it up. In the following we’re using the AWS console to walk you through the required steps and comment on things to consider when performing each step.

After you hit the Create workspace button in the right upper corner of the Amazon Managed Grafana console landing page, give your new workspace a name and optionally a description:

Create new Amazon Managed Grafana workspace

Next, you need to define user and data access permissions. Unless you have a use case where you have to or want to manage the underlying IAM role and policy yourself (to have fine-grained control for other AWS services), we suggest that you let Amazon Managed Grafana manage the creation of IAM roles and policies for accessing your AWS Services data.

In this step you also have to enable AWS Single Sign-On (SSO) for Amazon Managed Grafana since this is how we manage user authentication to Grafana workspaces:

Configure Amazon Managed Grafana workspace settings

If you haven’t set up users via AWS SSO as mentioned above you can use the inline experience offered by Amazon Managed Grafana and click Create user at this step:

Enable SSO, create user inline

Then fill in the details in the form, that is, provide their email address and first as well as last name:

Create SSO user

You should quickly get a confirmation mail with the invitation:

Invitation mail for SSO user

Back to the Amazon Managed Grafana workspace setup: We’re almost done, we now need to tell Amazon Managed Grafana the data sources from which we want to consume and visualize data. To be able to try out everything, we have selected all of the following, but you may want to restrict to the necessary subset for your use case:

Amazon Managed Grafana workspace permissions

As usual, you will have a final opportunity to review your settings and then confirm the creation of the Amazon Managed Grafana workspace:

Amazon Managed Grafana workspace creation review

Once the workspace is created, you can assign users access to the Grafana workspace. You can either assign an individual user from AWS SSO, or you can choose to assign a user group:

Amazon Managed Grafana workspace, assign user

Now we’re done from an administrative point of view, so let’s switch gears. We can use the SSO user from above to gain access to the Amazon Managed Grafana workspace. Now you can click on the Grafana workspace URL from the workspace details page, as shown in the image above, to log into your Grafana workspace using the SSO credentials (user grafana-user) and you should see Amazon Managed Grafana enabled and ready for you to access:

SSO landing page with Amazon Managed Grafana listed

Now all you have to do is to click on the Amazon Managed Grafana icon and you’re logged into your Grafana workspace:

Grafana landing page

The setup is completed with this and we’re ready to use Amazon Managed Grafana. Let’s start with consuming data from Prometheus.

Integration with Amazon Managed Service for Prometheus (AMP) and other data sources

Amazon Managed Grafana supports a variety of datasources such as Amazon Managed Service for Prometheus, Amazon CloudWatch, AWS X-Ray, Amazon Elasticsearch, Amazon Timestream, AWS IoT SiteWise plugin,  and several others. The full-list of data sources can be found here.

Amazon Managed Grafana can auto-discover the accounts and resources you have for the AWS data-sources and auto-configure based on the permissions setup using CloudFormation.

Auto-detected AWS data sources

Provisioning a datasource

This option discovers the accounts and resources you have for the six AWS Services that Amazon Managed Grafana natively integrates with. Based on the permissions you granted during workflow creation, you can now just check the box for the account or resource you want to add, and Amazon Managed Grafana will automatically configure the data source with the right IAM role permissions without you having to manually copy and paste.

Manually configuring an AWS data source

You can also manually configure the data sources by following the steps below:

In this example, we are connecting to an Amazon Managed Prometheus (AMP) data source:

Connecting to AMP datasource manually

Once connected, you will be able to create dashboard panels by selecting the newly connected data source and query metrics using PromQL. Here is a screenshot showing metrics from an EKS cluster queried on an AMP data source.

EKS Pod Metrics dashaboard

Out-of-the-box Dashboards

Several AWS datasource plug-ins come with many out-of-the-box dashboards to help you get started quickly. You can find them under the Dashboards tab as shown below.

List of out-of-the-box dashboards for Amazon CloudWatch

You can easily import a dashboard by clicking the Import button. Below you can see a screenshot of the Amazon EC2 dashboard.

EC2 Dashboard

Creating custom dashboards

Grafana supports a variety of panel visualizations to create dashboards using a variety of data sources. Below you can see a dashboard showing AWS X-Ray Trace data from the One Observability Demo application. You can use AWS X-Ray filter expressions to create dashboard panels in Grafana to visualize trace data as shown below.

Custom dashboard showing AWS X-Ray trace data

You can also investigate a single trace to see the segment timeline by simply clicking on a Trace ID. Grafana also provides deep links to the AWS X-Ray console where you can use tools such as X-Ray Analytics.

You can take advantage of a large collection of pre-built Grafana dashboards built by the community that can be easily imported into your Grafana workspace and provide a domain specific quick start to visualizing and investigating observability data for a variety of popular data sources.

Conclusion

With Amazon Managed Grafana (and AMP), you can now use Grafana and Prometheus without having to worry about the operational management of maintaining infrastructure resources, and let AWS take care of the undifferentiated heavy lifting.

More importantly, you can rely on open standards such as CNCF OpenTelemetry along with our own AWS Distro for OpenTelemetry that, together with our open source-based, fully managed services for Grafana and Prometheus, enable you to build powerful observability on AWS.

Authors

Imaya Kumar Jagannathan

Imaya is a Senior Solution Architect focused on Amazon CloudWatch and AWS X-Ray. He is passionate about Monitoring and Observability and has a strong application development and architecture background. He likes working on distributed systems and is excited to talk about micro-service architecture design. He loves programming on C#, working with Containers and Serverless technologies.

Michael Hausenblas

Michael is an Open Source Product Developer Advocate in the AWS container service team covering open source observability and service meshes. Before AWS, Michael worked at Red Hat, Mesosphere, MapR and as a PostDoc in applied research. Reach him on Twitter via @mhausenblas.