Load testing the Pragma backend game engine to 1 million concurrent users on AWS

In this blog, we will dive into the significance of load testing a game backend, and how AWS for Games partner, Pragma, successfully load tested one million concurrent users (CCU) on their managed game backend platform that runs on AWS.

Creating smooth gameplay is just one part of the game creation equation. Every game has a complex web of elements that play important roles in delivering a perfect player experience. All games, and especially online and multiplayer, require a robust and efficient game backend to help deliver the experience players expect. Well-designed backends scale to accommodate the growing player base.

How a managed game backend works for you

Imagine you are crafting an epic adventure or a multiplayer experience and you want to focus your creative energy on designing the gameplay, characters, and storyline. That’s where a game-changing concept can significantly simplify the technical aspects of bringing your game to life: a managed game backend solution. Managed game backend solution providers are the trusted partner that take care of the technical complexities of server infrastructure, player management, in-game economy, and multiplayer support. They streamline the development process and ensure scalability, security, and reliability for your game.

It’s important to ensure that a game backend can withstand the heaviest of player traffic and demand. This is where load testing comes into play. Load testing simulates heavy player traffic and mimics peak usage scenarios. It helps the developers identify and rectify performance bottlenecks, crashes, or latency issues before they impact the player experience. And leveraging cloud services for setting up a load test environment is a cost-effective approach. The flexibility of AWS, combined with multiple deployment options like API/CLI, Infrastructure as Code, and global reach, allows developers to set up and tear down resources easily when testing is done.

Game backend solutions empower game developers to save time and money while building reliable and engaging games. These solutions especially benefit those who are just starting their game development, as they provide the tech support and infrastructure needed to bring their gaming visions to life. This accelerates development timelines and reduces upfront infrastructure costs, sparing studios from the resource intensive task of building and maintaining their own backend.

Pragma backend engine

Pragma is a venture-backed backend game engine founded by the engineering leaders who built the platforms for some of the largest live service games, including League of Legends, Fortnite, Destiny 2, and Plants vs. Zombies 2. Pragma powers services like accounts, matchmaking and player data for the world’s most ambitious live service games. The engine-based offering from Pragma gives studios full control, customizability, and the confidence that they can implement any feature they want.

Pragma Engine runs in two modes. Both modes share the same engine capabilities in terms of metrics, operability, authentication and authorization, and service-to-calls. The two modes are:

• The social backend provides account and social services
• The game backend provides the game loop, player data, and related services

Pragma’s load testing environment

The load testing environment was set up based on the targeted CCU of one million players. The main components in the test environment were:

• Load test clients with custom configurations and logic were deployed on Amazon Elastic Compute Cloud (EC2) The load test clients were responsible for simulating traffic that represented synthetic player behavior and interactions.
• Amazon Application Load Balancer(ALB) was used to distribute incoming load from the clients to the Game and Social gateways
• Game Gateway and Social Gateway Nodes were separately deployed on EC2 instances. Both gateways were responsible for routing player requests to the specific backend services
• Pragma engine, containing the backend services was run on multiple EC2 instances
• Amazon Managed Grafana to visualize our load test metrics

The load test was conducted using two strategies: first, stress-testing each component and service individually to identify bottlenecks and issues, followed by targeted fixes based on the findings. The second strategy involved removing throttles and gradually increasing the load to assess the system as a whole. Essentially, one approach focused on isolating issues in individual services, while the other provided a holistic evaluation of the entire system’s performance.

The approach was methodical, beginning with a smaller setup that deployed the platform on a single node. Progressing forward, it evolved into a multi-node deployment, strategically spreading services such as matchmaking, accounts, inventory, and more, across multiple instances to manage the expected load for each service effectively.

The incoming traffic load across the gateways was distributed using the ALB. The listener checked for client requests and routed them to the registered game and social gateways. Separately, Amazon Relational Database Service (RDS) MySQL instances were set up for both social and game gateways. Connection pooling efficiently managed application connections, reducing the overhead of creating and handling concurrent connections.

Metrics were gathered through a custom OpenTelemetry(OTel) collector and visualized on Managed Grafana. The collection of data and metrics from the system provided real-time visibility into its performance. This helped assess resource constraints on the platform and service instances. Consequently, the developers were able to make data-driven decisions, fostering effective optimizations and performance improvements. AWS also offers a distribution of the OpenTelemetry project, known as AWS Distro for OpenTelemetry (ADOT), which can be used to collect data and metrics from your applications.

While this load testing architecture has proven to be effective for Pragma, its crucial to recognize what worked for them may not be a one-size-fits-all solution. Every system is unique with its own set of intricacies and various types of users. Therefore, to successfully load test your system, it’s important to understand the key components of your system and plan how you’re going to simulate user behavior.

“When considering infrastructure for our Pragma game backend we knew that we wanted to focus on robustness, predictability, speed to solution, and ease of setup. For these reasons, it seemed natural to go with AWS. During our load test, EC2, RDS, ALB, and Managed Grafana were all essential services,” says Nik Palmer, a Technical Product Manager at Pragma. “Leveraging these services removed friction, meaning we could just focus on scaling Pragma from one thousand to one million simulated CCU, from a single node to a 34 EC2 instance multi-node topology.”

Planning for load testing

You can plan for effective load testing by setting clear objectives, understanding your user’s behavior, identifying critical scenarios and defining performance metrics with acceptable thresholds. This will help you choose the right tools and the correct setup to gain the right measurements. In order to simulate player behavior, create realistic load testing scenarios, prepare test data and configure your environment to mimic production environment.

Ready to dive into load testing on AWS? Check out this prescriptive guide. It will help you explore different methods, uncover key questions, and understand the impact to avoid potential pitfalls. Additionally, discover AWS testing policies, several tools and their applicability through the guide.

Conclusion

In the ever-evolving game world, where player expectations are higher than ever before, the significance of properly testing a game backend cannot be overstated. As we’ve explored throughout this article, load testing is the safety net that guarantees that when game is at its busiest, the backend remains robust and responsive, ultimately enhancing player satisfaction and the game’s overall success. By understanding why load testing matters, why you should do it, and how to do it, you’re not just preparing for launch, you’re laying a resilient foundation for your game.