AWS Big Data Blog
Run Apache Spark 3.5.1 workloads 4.5 times faster with Amazon EMR runtime for Apache Spark
The Amazon EMR runtime for Apache Spark is a performance-optimized runtime that is 100% API compatible with open source Apache Spark. It offers faster out-of-the-box performance than Apache Spark through improved query plans, faster queries, and tuned defaults. Amazon EMR on EC2, Amazon EMR Serverless, Amazon EMR on Amazon EKS, and Amazon EMR on AWS Outposts all use this optimized runtime, which is 4.5 times faster than Apache Spark 3.5.1 and has 2.8 times better price-performance based on an industry standard benchmark derived from TPC-DS at 3 TB scale (note that our TPC-DS derived benchmark results are not directly comparable with official TPC-DS benchmark results).
We added 35 optimizations since the EOY 2022 release, EMR 6.9, that are included in both EMR 7.0 and EMR 7.1. These improvements are turned on by default and are 100% API compatible with Apache Spark. Some of the improvements since our previous post, Amazon EMR on EKS widens the performance gap, include:
- Spark physical plan operator improvements – We continue to improve Spark runtime performance by changing the operator algorithms:
- Optimized data structures used in hash joins for performance and memory requirements, allowing the use of more performant join algorithm for more cases
- Optimized sorting for partial window
- Optimized rollup operations
- Improved sort algorithm for shuffle partitioning
- Optimized hash aggregate operator
- More efficient decimal arithmetic operations
- Aggregates based on Parquet statistics
- Spark query planning improvements – We introduced new rules in the Spark’s Catalyst optimizer to improve efficiency:
- Adaptively minimize redundant joins
- Adaptively identify and disable unhelpful optimizations at runtime
- Infer more advanced Bloom filters and dynamic partition pruning filters from complex query plans to reduce amount of data shuffled and read from Amazon Simple Storage Service (Amazon S3)
- Fewer requests to Amazon S3 – We reduced requests sent to Amazon S3 when reading Parquet files by minimizing unnecessary requests and introducing a cache for Parquet footers.
- Java 17 as default Java runtime used in Amazon EMR 7.0 – Java 17 was extensively tested and tuned for optimal performance, allowing us to make it the default Java runtime for Amazon EMR 7.0.
For more details on EMR Spark performance optimizations, refer to Optimize Spark performance.
In this post, we share the testing methodology and benchmark results comparing the latest Amazon EMR versions (7.0 and 7.1) with the EOY 2022 release (version 6.9) and Apache Spark 3.5.1 to demonstrate the latest cost improvements Amazon EMR has achieved.
Benchmark results for Amazon EMR 7.1 vs. Apache Spark 3.5.1
To evaluate the Spark engine performance, we ran benchmark tests with the 3 TB TPC-DS dataset. We used EMR Spark clusters for benchmark tests on Amazon EMR and installed Apache Spark 3.5.1 on Amazon Elastic Compute Cloud (Amazon EC2) clusters designated for open source Spark (OSS) benchmark runs. We ran tests on separate EC2 clusters comprised of nine r5d.4xlarge instances for each of Apache Spark 3.5.1, Amazon EMR 6.9.0, and Amazon EMR 7.1. The primary node has 16 vCPU and 128 GB memory and eight worker nodes have a total of 128 vCPU and 1024 GB memory. We tested with Amazon EMR defaults to highlight the out-of-the-box experience and tuned Apache Spark with the minimal settings needed to provide a fair comparison.
For the source data, we chose the 3 TB scale factor, which contains 17.7 billion records, approximately 924 GB of compressed data in Parquet file format. The setup instructions and technical details can be found in the GitHub repository. We used Spark’s in-memory data catalog to store metadata for TPC-DS databases and tables. spark.sql.catalogImplementation is set to the default value in-memory. The fact tables are partitioned by the date column, which consists of partitions ranging from 200–2,100. No statistics were pre-calculated for these tables.
A total of 104 SparkSQL queries were run in three iterations sequentially and an average of each query’s runtime in these three iterations was used for comparison. The average of the three iterations’ runtime on Amazon EMR 7.1 was 0.51 hours, which is 1.9 times faster than Amazon EMR 6.9 and 4.5 times faster than Apache Spark 3.5.1. The following figure illustrates the total runtimes in seconds.
The per-query speedup on Amazon EMR 7.1 when compared to Apache Spark 3.5.1 is illustrated in the following chart. Although Amazon EMR is faster than Apache Spark on all TPC-DS queries, the speedup is much greater on some queries than on others. The horizontal axis represents queries in the TPC-DS 3 TB benchmark ordered by the Amazon EMR speedup descending and the vertical axis shows the speedup of queries due to the Amazon EMR runtime.
Cost comparison
Our benchmark outputs the total runtime and geometric mean figures to measure the Spark runtime performance by simulating a real-world complex decision support use case. The cost metric can provide us with additional insights. Cost estimates are computed using the following formulas. They factor in Amazon EC2, Amazon Elastic Block Store (Amazon EBS), and Amazon EMR costs, but don’t include Amazon S3 GET and PUT costs.
- Amazon EC2 cost (include SSD cost) = number of instances * r5d.4xlarge hourly rate * job runtime in hours
- 4xlarge hourly rate = $1.152 per hour
- Root Amazon EBS cost = number of instances * Amazon EBS per GB-hourly rate * root EBS volume size * job runtime in hours
- Amazon EMR cost = number of instances * r5d.4xlarge Amazon EMR cost * job runtime in hours
- 4xlarge Amazon EMR cost = $0.27 per hour
- Total cost = Amazon EC2 cost + root Amazon EBS cost + Amazon EMR cost
Based on the calculation, the Amazon EMR 7.1 benchmark result demonstrates a 2.8 times improvement in job cost compared to Apache Spark 3.5.1 and a 1.7 times improvement when compared to Amazon EMR 6.9.
| Metric | Amazon EMR 7.1 | Amazon EMR 6.9 | Apache Spark 3.5.1 |
| Runtime in hours | 0.51 | 0.87 | 1.76 |
| Number of EC2 instances | 9 | 9 | 9 |
| Amazon EBS Size | 20gb | 20gb | 20gb |
| Amazon EC2 cost | $5.29 | $9.02 | $18.25 |
| Amazon EBS cost | $0.01 | $0.02 | $0.04 |
| Amazon EMR cost | $1.24 | $2.11 | $0.00 |
| Total cost | $6.54 | $11.15 | $18.29 |
| Cost Savings | Baseline | Amazon EMR 7.1 is 1.7 times better | Amazon EMR 7.1 is 2.8 times better |
Run OSS Spark benchmarking
For running Apache Spark 3.5.1, we used the following configurations to set up an EC2 cluster. We used one primary node and eight worker nodes of type r5d.4xlarge.
| EC2 Instance | vCPU | Memory (GiB) | Instance Storage (GB) | EBS Root Volume (GB) |
| r5d.4xlarge | 16 | 128 | 2 x 300 NVMe SSD | 20GB |
Prerequisites
The following prerequisites are required to run the benchmarking:
- Using the instructions in the emr-spark-benchmark GitHub repo, set up the TPC-DS source data in your S3 bucket and your local computer.
- Build the benchmark application following the steps provided in Steps to build spark-benchmark-assembly application and copy the benchmark application to your S3 bucket. Alternatively, copy spark-benchmark-assembly-3.5.1.jar to your S3 bucket.
This benchmark application is built from branch tpcds-v2.13. If you’re building a new benchmark application, switch to the correct branch after downloading the source code from the GitHub repo.
Create and configure a YARN cluster on Amazon EC2
Follow the instructions in the emr-spark-benchmark GitHub repo to create an OSS Spark cluster on Amazon EC2 using Flintrock.
Based on the cluster selection for this test, the following are the configurations used:
Run the TPC-DS benchmark for Apache Spark 3.5.1
Complete the following steps to run the TPC-DS benchmark for Apache Spark 3.5.1:
- Log in to the OSS cluster primary using
flintrock login $CLUSTER_NAME. - Submit your Spark job:
- The TPC-DS source data is at
s3a://<YOUR_S3_BUCKET>/BLOG_TPCDS-TEST-3T-partitioned. Check the prerequisites on how to set up the source data. - The results are created in
s3a://<YOUR_S3_BUCKET>/benchmark_run. - You can track progress in
/media/ephemeral0/spark_run.log.
- The TPC-DS source data is at
Summarize the results
When the Spark job is complete, download the test result file from the output S3 bucket s3a://<YOUR_S3_BUCKET>/benchmark_run/timestamp=xxxx/summary.csv/xxx.csv. You can use the Amazon S3 console and navigate to the output bucket location or use the Amazon Command Line Interface (AWS CLI).
The Spark benchmark application creates a timestamp folder and writes a summary file inside a summary.csv prefix. Your timestamp and file name will be different from the one shown in the preceding example.
The output CSV files have four columns without header names:
- Query name
- Median time
- Minimum time
- Maximum time
Because we have three runs, we can then compute the average and geometric mean of the runtimes.
Run the TPC-DS benchmark using Amazon EMR Spark
For detailed instructions, see Steps to run Spark Benchmarking.
Prerequisites
Complete the following prerequisite steps:
- Run aws configure to configure your AWS CLI shell to point to the benchmarking account. Refer to Configure the AWS CLI for instructions.
- Upload the benchmark application to Amazon S3.
Deploy the EMR cluster and run the benchmark job
Complete the following steps to run the benchmark job:
- Use the AWS CLI command as shown in Deploy EMR Cluster and run benchmark job to spin up an EMR on EC2 cluster. Update the provided script with the correct Amazon EMR version and root volume size, and provide the values required. Refer to create-cluster for a detailed description of the AWS CLI options.
- Store the cluster ID from the response. You need this in the next step.
- Submit the benchmark job in Amazon EMR using add-steps in the AWS CLI:
- Replace <cluster ID> with the cluster ID from the create cluster response.
- The benchmark application is at
s3://<YOUR_S3_BUCKET>/spark-benchmark-assembly-3.5.1.jar. - The TPC-DS source data is at
s3://<YOUR_S3_BUCKET>/BLOG_TPCDS-TEST-3T-partitioned. - The results are created in
s3://<YOUR_S3_BUCKET>/benchmark_run.
Summarize the results
After the job is complete, retrieve the summary results from s3://<YOUR_S3_BUCKET>/benchmark_run in the same way as the OSS benchmark runs and compute the average and geomean for Amazon EMR runs.
Clean up
To avoid incurring future charges, delete the resources you created using the instructions in the Cleanup section of the GitHub repo.
Summary
Amazon EMR continues to improve the EMR runtime for Apache Spark, leading to a performance improvement of 1.9x year-over-year and 4.5x faster performance than OSS Spark 3.5.1. We recommend that you stay up to date with the latest Amazon EMR release to take advantage of the latest performance benefits.
To keep up to date, subscribe to the Big Data Blog’s RSS feed to learn more about the EMR runtime for Apache Spark, configuration best practices, and tuning advice.
About the author
Ashok Chintalapati is a software development engineer for Amazon EMR at Amazon Web Services.
Steve Koonce is an Engineering Manager for EMR at Amazon Web Services.