Zero performance compromise: Gearbox Entertainment goes remote with AWS and Perforce

Gearbox Entertainment, founded by Randy Pitchford, is an award-winning game development studio and publisher known for creating games for smash hit game franchises including Half-Life, Borderlands, and Brothers in Arms. The company’s creative capabilities have seen it excel not just in video games, but also in comic books, action figures, apparel, art prints, novels, non-fiction reference titles, and even media content for film and television.

In 2020, when the COVID-19 pandemic led the world to put home working restrictions in place, Gearbox was faced with a challenge, how could it ensure that the business could continue running effectively while safeguarding its team? Having already established a solid relationship with AWS, leveraging its services to power its latest title, Borderlands 3, and to support its build pipeline by using Amazon S3 to manage its backup for Perforce, Gearbox enlisted the help of AWS and Perforce to move game production into the cloud. And while it had already been considering moving to the cloud to enable its teams to collaborate and work remotely, the transition didn’t come without its fair share of concerns.

“The primary problem was dealing with the source control system that resided in our offices on the Perforce server system,” says Jeff Macfee, IT Director at Gearbox. “Pre-pandemic, we already had complaints about performance and noticeable delays, so when the restrictions hit and we had to go remote, we were worried that this would exacerbate the problem.”

CATCH-A-RIIIIIIIIIIIIIIDE

The company’s Perforce server system was located in the Gearbox Entertainment on-premises data centre at its head offices in Frisco, Texas. The team needed a reliable, scalable solution that would allow it to smoothly flow data between home setups and Perforce, without delays or bandwidth throttling. “The source control is the code for the game. Without it we have no game, which makes it the most important service we maintain,” says Macfee. “You tend to treat that kind of property very carefully, and it’s easy to become risk averse. Any changes or disruption were quickly discouraged from a risk perspective. We were also worried about the usual challenges that come with moving into the AWS cloud such as cost. We didn’t want to wake up the morning after we had migrated our system to discover a $2 million bill.”

The team was aware of the volumes of data and traffic that the company generated and the need for consistently optimized performance, and how these requirements could impact costs if they moved to the cloud. “Our developer and build workflows were growing and required teams to pull hundreds of gigabytes a day. We were already juggling bandwidth limitations. We pretty quickly realized we needed a distributed infrastructure to accommodate development and automation demand on Perforce.” explains Alexander Hollis, DevOps Engineer at Gearbox Entertainment.

The company had seen a significant uptick in automation workflows that required teams to pull on hundreds of gigabytes a day. With everyone on the same network, the pressure of these automation tools and the demands on the system meant that Gearbox was already juggling bandwidth limitations. There was also concern that the move would potentially put its source control at risk, or that the migration could result in unexpected delays that would set projects back by weeks or even months. “Then the pandemic hit and we knew we had to go remote,” says Macfee. “Because we’d already considered the risks and the routes to this migration, it helped us to move more efficiently and tackle the challenges more effectively.”

There’s always room for more loot

Gearbox Entertainment spent a significant amount of time unpacking the pros and cons of the move and realised that by leveraging AWS, it had a stable route to reliable remote working by providing the agility and scale needed to ensure access to systems, optimized performance, and reliable backups. The global network of AWS regions also offered Gearbox the global reach it required to ensure team productivity.“AWS allowed us to expose our resources in a safe way, without risking pipe congestion—performance is critical for our developers when they’re working with the source code,” explains Austin Daelemans, DevOps Engineer, at Gearbox Entertainment. “The move to AWS gave us the leverage we needed to expand the pipe at a moment’s notice.”

Cost concerns were also resolved fairly quickly. The team weighed the costs of the service against the literal costs of running the same service on-premise, assigned values to the intangible elements such as support and scale, added in the time that would be spent managing these physical assets, and then factored in the constraints of running the system on site. AWS was not only comparably more cost effective, but it also provided Gearbox with more time to focus on creating amazing game experiences, and optimizing performance of its architecture. It could advance the company to a more mature stage, where it could scale and adapt at speed. “We’re a small IT team, in a building using every available area it can. So we run into limitations with people, space, and cooling. Infrastructure investment is nice, but it takes forever and we don’t have the people to manage and maintain it. With AWS, these issues disappear. We get what we pay for—we have the same human beings on my team, but they’re seeing results faster.” Jeff Macfee, IT Director at Gearbox Entertainment.

When the team started the move, they had three servers on-premise—one user and build facing, and two performing maintenance and backup. Everything was physical hardware on premise, so the team started by migrating an Edge server into AWS. Once that had been done successfully, they put a standby in the cloud with the goal of failover. This now meant they had one server on premise with the others located in the cloud, giving them the ability to distribute the load more evenly and spin up additional servers on demand.

“We started experimenting with the move in mid-April 2020 and had full production running with AWS by September,” says Hollis. “We put out our first Edge fast, and then we used the learnings from this to move the boss of the entire infrastructure—the source control—afterwards. We were really concerned about the risk when we started, but once we saw how well it worked, we were committed.”

The architecture used to make the move included Terraform—a platform that had already been put in play by the Gearbox team for testing and iterating. This was bolstered by AWS DataSync, which allowed the team to move its backups into live storage. By collaborating with Perforce, it experimented to make the move and ensure that performance and minimal risk remained a priority. Amazon Elastic File System was a natural fit for sharing and storing file data, compute was done in Amazon EC2 and the team leveraged Amazon Elastic Block Store to manage compute and performance.

Armaments here!

“Costs were within assessments and we’ve been adapting as we evolve so our costs have evolved along with this,” says Macfee. “The traditional performance issues have gone away. The contention between the development and the build, with all the build processes running all the time and all the developers trying to use it—this contention is starting to disappear. There are people who’ve worked here from the beginning who have never seen this before. Problems we thought would exist forever are going away, and that’s a huge win.”

AWS has provided Gearbox Entertainment with the ability to manage source control in the cloud, at an AAA level. And today, Gearbox continues to deliver exceptional game experiences, at scale, without hefty investments into more hardware, more space, and increasing management costs. “We can easily access intelligent automation tools, and we have the ability to create new Edge servers and expand our infrastructure rapidly, on demand,” concludes Daelemans. “Plus, the improved reliability and the predictability of costs means we can expand, add partners, and adapt systems accurately. It also made us look around at other services. Now that we have source control in the cloud, we can build in the cloud to be closer to it and bring the same benefits to Perforce maintenance processes and scale. We have these granular insights that we can use to optimize performance, scale, and costs.”