Pearl Abyss: Massive Scale using Azure SQL Database

  • [アーティクル]

Pearl Abyss is a Korea based game development studio established in 2009.  Black Desert Online, a PC-massively multiplayer online role-playing game (MMORPG), is their main title in production published across the globe.

On February 28, 2018, Pearl Abyss launched Black Desert Mobile on Microsoft Azure virtual machines utilizing 7,500 cores and 46 instances of Azure SQL Database in Korea Central.

Black Desert Mobile became the most successful game of the year, which set forth to be the strongest Azure gaming reference to date worldwide with milestones including:

  • 5 million registrations in one month
  • 3 million downloads at launch
  • #1 app by revenue in the Apple app store
  • #2 app by revenue in Google Play

Using Azure SQL Database, Pearl Abyss was able to accommodate 500,000 queries per second during the first hour after launch, generating 1GB/second of transaction log during that time.  This blog post will briefly walk through Pearl Abyss’ experience using Azure SQL Database to achieve massive performance and scale.

Exploring Database as a service (DBaaS) options

Pearl Abyss started their exploration of database platform options almost a year prior to the Black Desert Mobile launch.  Initially they considered development using an end-to-end infrastructure as a service (IaaS) approach, but they felt strongly that adopting a self-managing, lower-touch platform as a service (PaaS) solution would better support projected gaming volume and scalability requirements.

To help make their decision, Pearl Abyss benchmarked and compared the performance of Azure SQL Database against another popular database as a service (DBaaS) provider.  They created a database schema and simulated data set that represented the average user’s set of gaming activities.  This simulator ran representative workloads against both Azure SQL Database and the competing DBaaS provider. Pearl Abyss reported that Azure SQL Database outperformed the competing DBaaS provider and that other factors such as reduced maintenance requirements, read-only replicas, and point-in-time recovery allowed them to confidently move forward with Azure SQL Database as a data tier solution.

Architecting a MMORPG data tier

Pearl Abyss organizes their data tier into authentication servers and field servers, scaling by player “realm.” Figure 1 below shows a high-level overview of the application and data tier topology and connection flow.

[caption id="attachment_7475" align="alignnone" width="1099"] Figure 1: Gaming Application and Data Tier Topology[/caption]

A load balancer is used to connect a player to one server across a set of authentication servers hosted on Azure Virtual Machines.  Each VM connects to an Azure SQL database, and if the player is appropriately authenticated, a separate Azure SQL database is used to determine the player-realm mapping. Game activity is then handled within the appropriate realm on associated Azure VMs which in turn have their own Azure SQL Database to track game, realm and player state.  Additionally, for every realm there is a P15 Azure SQL Database used to log all critical activities and player inventory status.

This architecture enables extreme performance and scale, accommodating 500,000 queries per second at launch and resulting in a generation of 1 GB of log per second during the first hour of launch.  Azure SQL Database accommodated this burst of activity without issue and there were no specific server or database resource bottlenecks at launch.

Beyond performance and scalability

Pearl Abyss achieved the required scale and performance they anticipated, but this was not the only benefit that drew them to Azure SQL Database.  The following sections describe other aspects that have contributed towards the successful launch and ongoing supportability of Black Desert Mobile.

Leveraging Columnstore Index Compression

All player activity within Black Desert Mobile is recorded.  This ongoing activity recording allows players to exit and resume play with all actions preserved in the expected state, including an accounting of the correct inventory, player level, and starting location.

Pearl Abyss uses a combination of In-Memory OLTP and Clustered Columnstore Indexes (CCI) to capture logged player events.  Data is moved periodically from the In-Memory OLTP tables to the CCI tables, with 12 billion player events generated per day and hundreds of billions of rows preserved per month.  The player activity data in Black Desert Mobile is very compression-friendly, and results in significant storage reduction and reduced I/O operations. Across realms, the CCI compressed logging data size is 20TB.  Without using CCI compression, the footprint across realms would be 400 TB of data.

Scaled supportability experience

If a player calls customer support to report any issues, the customer service operator will look up the customer’s information and then research the player’s game-play history.  Oftentimes the customer service operator must research recent game history, however they are directed to never risk hurting performance or risk concurrency issues with live game play activity while researching an issue.

This meant that in the past when they used SQL Server as a data tier, they would use periodic database restores for root cause analysis rather than operating against read/write production databases.

In Azure SQL Database, Premium tier databases are automatically provisioned with read scale-out replicas to support the availability SLA.  These read-only replicas are provisioned with the same performance level as the read-write replica used by the regular database connections.  Pearl Abyss customer service operators now use Azure SQL Database read-only replicas for researching player-reported issues.  Read-only workloads from customer support are then naturally separated from regular player read-write replicas and don’t impact live game play.

Point in time recovery

Azure SQL Database automatically provides automated database backups.  With these automated database backups, you can restore a backup to an earlier point in time.  This allows for either database replacement or data recovery.  Pearl Abyss saw this as automatic coverage as a significant benefit, allowing extra assurance that they could recover databases in the event of an administrative error with minimal-to-no data loss, or in the event that a customer support operator needed to research a reported customer issue, restore to a specific point in time based on the reported incident.

Next Steps

Pearl Abyss and numerous other customers are using Azure SQL Database to support their mission critical applications and services.  To learn more about Azure SQL Database and building scalable, low-maintenance cloud solutions see: What is SQL Database? Introduction to SQL Database.

Want to get started but don’t know where to begin? Create your first SQL Database in Azure with your free Azure account.