Versioning Control the Game
What an adventure. Let's quickly define what Version Control and Branches are, and then we will dive into how they are related to Aether Story.
Version Control are systems that are used by developers to track changes and edits to their work over time, allowing them to undo mistakes, and work in branches, and merge edits by various developers together. This is probably an over simplication, but in essence, version control is like having a history and audit log of the work you do.
Branches allow you to work on different versions of your code or data. Such as copying a file and being able to edit it as a standalone set of work. The great thing about branches though is that they have the ability to be merged together and conflicts to be resolved. Again an over simplication.
With these terms out of the way what does this have to do with Aether Story? Well, Aether Story is a very complex project, it is more than just a single git repo, it is several. But it is also so much more than just code repos. It also has a large library of binary assets (such as images, sounds, music). And it also uses a relational database to design the game's content.
Aether Story is a data driven project. As such most of the game's content is designed and created within a database. And this is where the adventure began. The need to version control the game's data.
For the longest time I was in search of a way to version control the databases data. All solutions found up to this point were schema version control. Meaning, they only tracked the changes to the structure of the database, not the data itself. Not exactly what we needed. But this changed when we found Dolt.
Dolt: A Data Version Control Solution
Marketed as a drop-in relacement to MySql, it states that it can version control the schema and the data. The exact thing we were looking for.
The first step was to explore Dolt's functionalities and to test if they would work with our use-cases. We want to be able to stage different patches (Alphas) into their own branches, and be able to maintain and edit each independently.
Auto Incrementing ID Across Branches
Our first challenge came from the need for each record created across the various branches to have unique auto-incrementing ID's. Imagine if we wanted to merge together two branches of the game and all the new items added to each branch collided. It would be a nightmare. So we reached out to the very kind creators of Dolt and asked them if they had a recommended solution to this problem. We learned that Dolt is a relatively new product and that they did not yet have a solution for this. They added it to their issue tracker and said they would hopefully get to it.
But we couldn't wait. We took it into our own hands for our own use-case, we devised a Redis memory cache that all of our Insert statements referenced to look up the next available auto-increment ID independent of branches. And it worked.
Transactions and Commits
We were up and running. Or so we thought. The next issue we discovered is that due to the nature of Dolt, with how each user's commit is considered a transaction, when we started testing our code the transactions started causing inconsistent data changes.
The solution to this problem was two fold. First, we made it so the tool and editor for the game used a single Dolt connection. This solved most of the problems. Since the editor API is ran on a server, and multiple users use the API at the same time, we needed them to share a single database connection. Otherwise, each connection would not be transactional, and would be overwriting each other. Ideally Dolt would have each user have their own database instance running, and you would merge everyone's work together into a master database. But this is no where close to how we are setup, so wasn't an option.
The second thing we needed to do was to copy the Dolt database over to a traditional MariaDB instance when we wanted to spin up a dev server to test the actual game. This is because the game's code base needs multiple database connections (pooling), and when using multiple connections with Dolt caused data to be missing or inconsistent. In retrospect, at the time of writing this post, I just realized that another solution to cloning the database would be to simply commit the Dolt database each time we want to test the game, as the game would at least be synchronized around the commit point which would hopefully be servicable.
Branches
Once we got the data version controlled our adventure was not yet over. It became clear that we also needed different versions of the game's service and binary assets to also exist on the dev server in parallel.
The solution to this was to create a Branch Manager that not only managed the branch's Dolt databases, but also the Node.JS processes, and cloned the asset directories. It was a lot of work, but we got there: a fully functional Branch Manager that allows developers to toggle which branch of the game they are editing, and even spin up a test server and connect to it and test their changes in their own branch instance. All of this is as easy as clicking a button to the developers.
The Branch Manager for Aether Story - allowing us to manage all aspects of branches including node.js services, database branches, and asset directories.
Conclusion
And with that, a 40 day adventure through versioning control the game, we were successful. We now are able to release patches once again, and we now have a new work flow to maintain different branches in parallel. And it is largely thanks to Dolt and the hardwork by Mei and myself.