Choosing between the two export options of the Amazon QuickSight asset deployment APIs

Are you looking to deploy and manage your Amazon QuickSight assets using the QuickSight asset deployment APIs (also known as asset bundle APIs)? If so, you have two options for exporting your assets: QUICKSIGHT_JSON or CLOUDFORMATION_JSON format. In this post, we explore the pros and cons of each option to help you make the right choice for your organization’s asset needs. We also provide an overview of the API deployment options and some practical guidance on how to use the AWS CloudFormation format.

So if you’re ready to learn more about the QuickSight deployment APIs and discover which export option is best for your needs, then let’s get started!

Overview of the asset deployment options

The new QuickSight deployment APIs introduced in June 2023 provide a robust mechanism to automate and accelerate deployment of QuickSight assets like dashboards, datasets (including scheduled or incremental refreshes), themes, data sources, VPC connections, and more.

These APIs allow you to create bundles of assets that can be easily deployed across accounts and environments without complex coding or infrastructure management. The APIs offer the same secure access controls as the other QuickSight APIs.

Key benefits of the new APIs include:

• Batch deployment of large volumes of assets faster compared to individual uploads. This streamlines setup of QuickSight environments.
• Rapid iteration and experimentation by enabling straightforward deployment of related assets together.
• Simplified updates and changes by deploying all updated assets in one process.
• Version control of assets, allowing collaboration while retaining production control.

To learn about more use cases, refer to Automate and accelerate your Amazon QuickSight asset deployments using the new APIs.

Overall, the APIs abstract away the complexity of asset relationships and management. Teams can focus on content creation, and the APIs handle consistent, automated deployments across environments. The APIs’ batching, versioning, and automation capabilities accelerate deployment workflows for QuickSight assets at scale.

Previously, you needed to have an in-depth understanding of QuickSight asset relationships to be able to deploy, back up, or replicate assets at scale in an automated manner. To learn more, refer to New Amazon QuickSight API Capabilities to Accelerate Your BI Transformation.

To facilitate, expedite, and simplify assets deployment, six new APIs have been introduced:

• Import APIs:
  • StartAssetBundleExportJob
  • DescribeAssetBundleExportJob
  • ListAssetBundleExportJobs
• Export APIs:
  • StartAssetBundleImportJob
  • DescribeAssetBundleImportJob
  • ListAssetBundleImportJobs

The StartAssetBundleExportJob has a required export format parameter that can take two options: CLOUDFORMATION_JSON or QUICKSIGHT_JSON.

If you’re using the AWS Command Line Interface (AWS CLI), the API call looks like the following code:

aws quicksight start-asset-bundle-export-job \
 --aws-account-id ***accountID*** \
 --asset-bundle-export-job-id cfn-demo \
 --resource-arns "arn:aws:quicksight:us-west-2:***accountID***:dashboard/***dashboardID***" \
 --export-format QUICKSIGHT_JSON \ #or CLOUDFORMATION_JSON
 --include-all-dependencies \
 --region us-west-2

If you’re using the AWS SDK for Python (Boto3), the API call looks like the following code:

response = sourceQuickSight.start_asset_bundle_export_job(
AwsAccountId=SourceAccountId,
AssetBundleExportJobId="job-id",
ResourceArns=resourceARNs,
ExportFormat=QUICKSIGHT_JSON, #or CLOUDFORMATION_JSON
IncludeAllDependencies=True
)

Knowing the advantages and disadvantages of each of these two options will enable you to make the right decision for your specific use case.

Comparing the two options

Asset bundles allow faster deployments, granular control, simplified asset versioning, and unlimited assets per bundle—the bundle file size is the only limit, currently set to 20 MB compressed. This format is particularly beneficial for managing QuickSight assets like dashboards, where ease of version control and deployment speed are critical.

AWS CloudFormation provides robust automation, security, and reliability by treating infrastructure as code. Stacks enforce boundaries and change controls. It’s effective for managing data sources, integrating them seamlessly into the broader AWS ecosystem through its ability to manage stacks across accounts and regions.

For rapid iteration of dashboards, asset bundles are preferable for their simplicity and speed. For controlled integration of QuickSight data sources with AWS infrastructure like RDS, Athena and Redshift, CloudFormation stacks provide stability, control, compliance, and rigor. Both methods can be well-suited for production. Asset bundles excel at managing dashboards flexibly and rapidly. CloudFormation excels at managing data sources, integrating them with AWS infrastructure in a controlled, automated way. Organizations can use a hybrid approach: asset bundles for dashboards and CloudFormation for data sources.

The following table summarizes the features of each option.

QUICKSIGHT_JSON

Choosing this option results in the API call creating one file with the .qs extension. This file format is used for exporting data from Amazon QuickSight, typically containing the data structure and contents from a QuickSight analysis or dashboard. It is a .zip file that contains separate folders for each asset type. Each folder contains a single JSON file for each asset with the resourceId as the file name. This folder structure makes it straightforward to commit the contents into a version control system like Git, so you can get the benefits of a complete version history.

This option has some clear advantages in terms of flexibility:

• No limit on assets – There is no restriction on the number of assets like dashboards and datasets that can be deployed using asset bundles. As of this writing, only the overall compressed bundle size has a soft limit of 20 MB. Bigger files can be uploaded in Amazon Simple Storage Service (Amazon S3).
• Flexible asset grouping – Any asset can be included in one or more bundles, allowing logical grouping and reusable bundles. This provides flexibility in managing assets.
• Granular control – Each asset in the bundle can be individually configured and customized. This allows fine-grained control over all aspects of an asset during deployment.
• Faster deployments – The asset bundle files are lightweight and quick to deploy compared to heavier infrastructure deployment files. This enables faster iterations. This is especially useful when deploying large numbers of assets quickly as part of regular maintenance processes or during times when data transfer speeds may be low due to network issues or problems with internet service providers
• Simple structure – The folder-based structure of asset bundles makes it straightforward to identify and track metadata for each asset because each one is filed in its separate JSON file. This improves understandability.
• Version control – Asset versions can be properly labeled (for example, dev or test) and deployed as needed. This makes reverting changes by redeploying older versions simpler.
• Workflow automation – The migration workflow includes exporting assets from your source account, importing them into your target account, and sharing them with yourself and others in the QuickSight target account. You can accomplish this using the AWS SDK in your chosen programming language or the AWS CLI, with the AWS SDK offering a complete automated solution.
  • Note that at the time of this writing, one step requires downloading the JSON file to your local machine or your local cloud environment before uploading it or referencing it in the target cloud environment. You can programmatically automate this step in your AWS SDK script to streamline the migration. Additionally, you can use AWS CodePipeline to automate the entire process in the cloud environment. It’s important to note that this specific step cannot be automated using the AWS CLI.
  • Sharing the new assets with yourself and others in the QuickSight target account is possible through the QuickSight UI, AWS CLI, or programmatically via AWS SDK using update-dashboard-permissions (or similar for datasets, data sources, and analyses). Additionally, support for automatic permission updates is on the horizon.
• Rollback – The default behavior is rolling back all assets in the bundle if any of them fail. If disabled, the assets that already succeeded at the time of failure will remain as is.
• IAM role – It uses the caller’s current AWS Identity and Access Management (IAM) role.

CLOUDFORMATION_JSON

Choosing this option results in the API call creating a CloudFormation template as one JSON file. AWS CloudFormation gives you a straightforward way to model a collection of related AWS and third-party resources, provision them quickly and consistently, and manage them throughout their lifecycles by treating infrastructure as code. A CloudFormation template describes your desired resources and their dependencies so you can launch and configure them together as a stack. You can use a template to create, update, and delete an entire stack as a single unit, as often as you need to, instead of managing resources individually. You can manage and provision stacks across multiple AWS accounts and AWS Regions.

When you create the CloudFormation stack, you have the option of overriding parameters, which allows you to add additional infrastructure resources that you want to deploy (for example, IAM permissions, Amazon Redshift connections, or VPCs).

The following are the benefits of using a CloudFormation template:

• Infrastructure as code standard – AWS CloudFormation is the native infrastructure as code solution on AWS. Using it aligns with best practices for defining and provisioning infrastructure in a templated, repeatable way.
• Declarative – It defines what resources are needed rather than how they should be created. You describe the desired resources and their properties, and AWS CloudFormation takes care of provisioning and configuring those resources in the correct order.
• Developer productivity – It integrates well with several SDKs, such as Java, Python, and Typescript. AWS CloudFormation uses a JSON/YAML-based syntax that is readable and integrates with developer tools and SDKs. This makes it effortless for teams to collaborate on infrastructure code.
• Flexible – It can be used via both the AWS Management Console and AWS CLI.
• Integration – It integrates easily and seamlessly with your existing infrastructure by using parameters to pass on configurable values (such as networking, hostnames, subnets, or CDN) without requiring user input each time they deploy assets.
• Workflow automation – If offers workflow automation similar to QUICKSIGHT_JSON format. It’s important to note that the CloudFormation file can be edited, before deployment, to include the necessary permissions for the newly migrated QuickSight assets.
• Rollback capabilities – It allows for rollback capabilities in the event of an issue.
• Increased security – It offers an added layer of security when using stack policies to limit changes or updates to stacks so that you have oversight over changes to critical infrastructure.
• Reusability – Common or shared infrastructure patterns can be turned into reusable templates. Stacks can also be nested, allowing for modular architecture. You can create stacks with multiple assets and pass configurable values to them when deploying.
• Multi-Region and multi-account – StackSets allow you to manage a single CloudFormation template across multiple Regions and accounts from a central source.
• Assume custom role – AWS CloudFormation has a role-arn parameter, which allows it to assume a specified role when deploying the stack.

When deploying CloudFormation stacks, you should also be aware of some potential limitations:

• Slower deployments – CloudFormation templates are more complex and declare multiple interconnected resources. This means stack deployments take longer compared to deploying individual assets.
• Additional coding and scripting – Successfully creating and deploying CloudFormation templates requires solid programming knowledge and scripting skills, especially for advanced customization
• More upfront configuration – There is greater upfront effort in scripting out the template compared to ad hoc deployment of assets. The template needs to comprehensively capture the infrastructure.
• Difficult version comparisons – With a single large template file representing the full stack, comparing changes across template versions is harder than comparing changes to individual assets.
• Limits – AWS CloudFormation has limits on the number of resources per stack as well as the number of parameters and mappings. This can constrain template design. For example, only 200 parameters can be passed. For a complete list of AWS CloudFormation limits, refer to AWS CloudFormation quotas.
• One stack per resource – A resource can only belong to one stack at a time. Complex systems may need to be broken down into multiple interlinked stacks. Depending on the complexity and interdependency of your infrastructure, you might need to come up with a strategy in terms of how to organize resources together into multiple stacks.
  • A stack represents a collection of AWS resources that are created, updated, or deleted together as a single unit. Each stack is defined by a CloudFormation template, which specifies the resources and their configurations. Each stack comes with its own enforced isolation, stack boundaries, resource and dependency management, resource namespacing, and scoped updates.
  • For example, if you have two dashboards that use the same data source or dataset, then you either bundle all three in the same stack (but other dashboards won’t be able to use the same data source), or create one stack for the data source and two stacks for the dashboards.

How to use CLOUDFORMATION_JSON

The following steps walk you through using CLOUDFORMATION_JSON:

1. In the source account, list all the dashboards in the QuickSight account and identify the ARN or ARNs for the asset you want to deploy:

   aws quicksight list-dashboards —aws-account-id XXXXXXXXXXXX

2. From the response, copy the dashboard ARN you want to migrate:

   "Arn": "arn:aws:quicksight:us-east-1:XXXXXXXXXXXX:dashboard/61c3c8ea-e345-4b69-ba02-b0de361f23ab",

3. Export the assets bundle:

   aws quicksight start-asset-bundle-export-job \
    --aws-account-id XXXXXXXXXXXX \
    --asset-bundle-export-job-id cfn-demo \
    --resource-arns "arn:aws:quicksight:us-east-1:XXXXXXXXXXXX:dashboard/61c3c8ea-e345-4b69-ba02-b0de361f23ab" \
    --export-format CLOUDFORMATION_JSON \
    --region us-east-1
   

4. To modify the original configurations settings, use parameters in two steps: First, create an override file to list the desired changes for exporting assets. Then, use a different override file to define new parameter values when importing assets into the target account.
5. In this example, we first create a JSON file named cfnOverride.json. It contains settings for two modifications to be made during the export from the source account: a) applying a prefix to all assets imported into the target account, and b) changing the name of the dashboard. Later we will create another json file to specify the values for the overridden parameters.

   {
       "ResourceIdOverrideConfiguration": {
           "PrefixForAllResources": true
       },
       "Dashboards": [
           {
               "Arn": "arn:aws:quicksight:us-east-1:XXXXXXXXXXXX:dashboard/61c3c8ea-e345-4b69-ba02-b0de361f23ab",
               "Properties": [
                   "Name"
               ]
           }
       ]
   }

6. Import this file into your source account environment. If you’re using AWS CloudShell, choose Actions and Upload, then enter the following code in your terminal:

   aws quicksight start-asset-bundle-export-job \
   --aws-account-id XXXXXXXXXXXX  \
   --asset-bundle-export-job-id cfn-demo \
   --resource-arns "arn:aws:quicksight:us-east-1:XXXXXXXXXXXX:dashboard/61c3c8ea-e345-4b69-ba02-b0de361f23ab" \
   --export-format CLOUDFORMATION_JSON \
   --cloud-formation-override-property-configuration "file://cfnOverride.json" \
   --include-all-dependencies \
   --region us-east-1

7. Describe the export job to get the download file:

   quicksight describe-asset-bundle-export-job \
   --aws-account-id XXXXXXXXXXXX  \
   --asset-bundle-export-job-id cfn-demo \
   --region us-east-1

If successful (Status: 200), the response will contain the DownloadUrl.

8. Copy and paste the link into a browser (it will either download or open the JSON file in the browser) and save the file.
9. Go to your target account and upload this file into your cloud environment. If you’re using AWS CloudShell choose Actions and Upload.

Alternatively, you can use the below command to get the file from the download link in the target account:

wget -o CloudFormation_bundle.json '<DownloadUrl>'

Now you can deploy the CloudFormation stack in the target account.

10. Open a terminal in your target account and create the stack. Optionally, you can use AWS CloudShell. In the following code, the CloudFormation file downloaded in the previous step is named CloudFormation_bundle (with no extension):

    aws cloudformation create-stack \
     --stack-name 'cfn-demo' \
     --disable-rollback \
     --template-body file://CloudFormation_bundle \
     --region us-east-1

Note 1: CloudFormation limits template size to under 51,200 bytes. If you exceed this limit, upload the file to an S3 bucket and use —template-url in the create-stack statement, like this: --template-url https://YOURBUCKETNAME.s3.amazonaws.com/CloudFormation_bundle, instead of --template-body. Remember to use the Object URL, not the S3 URI.

Note 2: To modify the original configuration, create a second JSON file named overvalues_for_cfn_demo.json in the target account. This file should list the parameters you wish to change from the template, along with their new values. As show in the example below, prefix all asset IDs imported into the target account with cfn_ and rename the dashboard to “Florida Blue (Migrated)”. To find the parameter keys, refer to the top section of the CloudFormation_bundle template where the parameters are listed.

Top Partial View of the CFN template (named CoudFormation_bundle):

{
  "AWSTemplateFormatVersion" : "2010-09-09",
  "Description" : "Generated by AWS QuickSight for cfn1 started at 2023-12-15 20:10:40 UTC",
  "Parameters" : {
    "61c3c8eae3454b69ba020631FFName" : {
      "Default" : "FloridaBlue Demo",
      "Description" : "Name for dashboard:61c3c8ea-e345-4b69-ba02-b0de361f23ab",
      "Type" : "String"
    },
    "ResourceIdPrefixForAllResources" : {
      "Default" : "",
      "Description" : "Resource id prefix for all resources",
      "Type" : "String"
    }

Thus the override file, overvalues_for_cfn_demo.json, will look like this:

[
{
"ParameterKey": "ResourceIdPrefixForAllResources",
"ParameterValue": "cfn_"
},
{
"ParameterKey": "61c3c8eae3454b69ba020631FFName",
"ParameterValue": "Florida Blue (Migrated)"
}
]

Once the template and the file with the override values have been uploaded in the target account, you can now run the create-stack command as such:

aws cloudformation create-stack \
--stack-name 'cfn-demo' \
--disable-rollback \
--template-body file://CloudFormation_bundle \
--parameters file://overvalues_for_cfn_override.json
--region us-east-1

Remember, you can also use --template-url https://YOURBUCKETNAME.s3.amazonaws.com/CloudFormation_bundle if the template size is bigger than the allowed quota.

If successful, you should see a StackId in the response.

{
    "StackId": "arn:aws:cloudformation:us-east-1:XXXXXXXXXXXX:stack/cfn-demo/5e3530b0-36ec-11ee-bb26-0e7555d227cb"
}

11. In your target account, open the AWS CloudFormation console.
12. Choose Stacks in the navigation pane.

You should see the name of the stack as cfn-demo and the status of the deployment.

Note 3: For the target account to create a data source you either 1/ have to give access to the target account to the datasets and data sources used in the source account or 2/ override the configuration file to allow the new assets in the target account to use existing similar data sources. in the target account (or elsewhere).

For example, if the dashboard in the source account uses an S3 file as a data source, then you need to modify the bucket policies to allow the target account to access it. Navigate to the bucket, go to Permissions and Edit the Bucket Policies:

{
"Version": "2008-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::TARGET ACCOUNT ID:root",
]
},
"Action": "s3:*",
"Resource": [ "arn:aws:s3:::YOUR BUCKET NAME/FOLDER NAME/*"]
}
]
}

Next, go to QuickSight in the target account and make sure that it has access to the S3 bucket in the source Account:

Note 4: Currently there is no direct reference from the dashboard to the corresponding analysis, hence analyses are not included in the export job.

Now you can share the newly deployed assets with yourself and others. You can use QuickSight APIs (such as update-dashboard-permissions) in the AWS CLI or your preferred AWS SDK to share the assets with yourself or select users and groups. Alternatively, you can share these assets via the QuickSight UI:

13. In the target account, open the QuickSight console.
14. Choose the user profile icon and choose Manage QuickSight and Manage Assets.
15. Search for the dashboard you just deployed.
    
16. Choose SHARE and for User or Group, enter the email addresses of the users (yourself included) that you want to share the assets with.

Now you can see and access the assets you just deployed.

Conclusion

Both the CloudFormation templates and QuickSight asset bundles have their pros and cons for deploying QuickSight assets.

Rather than an either/or choice, the two options are complementary. QuickSight deployments should evaluate both and craft an approach utilizing each tool’s strengths while mitigating their limitations based on the specific requirements like security, reliability, iteration speed, and customization needs.

When deciding between the two options for asset deployment needs, consider carefully which option will best meet your organization’s requirements while providing reliable security and fast turnaround times. Depending on your specific project goals and timeline constraints, either option could be a perfect fit—so make sure you do your research before committing!

About the authors

Gilbert V Lepadatu is a QuickSight Sr. Specialist Solution Architect. With experience in two Fortune 100 companies, he has spearheaded data science, analytics, and automation initiatives, achieving significant revenue growth and operational efficiencies. Leveraging a PhD in Philosophy and two masters’ degrees, Gilbert has also contributed to academia, teaching classes on statistics and data visualization. His diverse skill set encompasses AWS services, data visualization, reporting and pipelines, data science and analytics, machine learning and communication across technical levels.

Zhao Pan is a software development manager for Amazon QuickSight. He is working to provide a delightful developer experience to our customers to automate and streamline their BI operations. He has 20 years of software development experience in various tech stacks. Prior to QuickSight he was a people and technical leader at ADP building a next-gen platform for human capital management. When he is not at his desk, he can usually be found in his garage building one contraption or another.