Introduction

At it Matters Games we have at least three people working on one project at the same time. On larger projects we had even more than 10 people being involved during development peaks. Sometimes people are even working from home. How can we make sure that the project is still running and deployable even when the developers are working on totally different features at the same time and code is added? How does one make sure that no one breaks different parts of the code or introduces unexpected side effects?

Traditionally software companies are using a dedicated quality assurance (QA) team. Once every change is integrated into the project, the testers will execute different predefined sets of tests, resulting in various test and bug reports. All reports will be added to a bug tracker. If any bugs occur developers will take care of them. Once bugs are fixed the QA will test again. This QA feedback loop will hopefully lead to a “bug free” project within a couple of days or even weeks.

In most cases every project has a dedicated QA and bug-fixing phase. A couple of developers will take care of any occurring problems, blocking them for any other task or from developing new features. Although very important, this phase can be very cost intensive and a grind for developers. We had projects were this phase would go on for month, leaving behind exhausted and disappointed developers.

Extreme Programming Practices

Extreme programming, also called XP, was developed by Kent Beck in 1996 (see also Extreme Programming Explained).  It describes practices like for example Pair Programming, Refactoring, Collective Ownership and Test-Driven Development:

Extreme Programming Practices

Using Continuous Integration (CI) and Test-Driven Development (TDD) reduces feedback loops to minutes, giving developers time to work on new tasks and features. Both will reduce the lengthy building and testing phases, also reducing the amount of people being involved in those processes. This adds measurable value to the business and improves the quality of projects.
All practices were meant to add value to a company and improve the quality of the software. They should reduce the feedback loops and make sure we are building the thing right. In the following sections I will introduce two practices we at it Matters Games are using to increase the success of our projects.

Continuous Integration

CI is also one practice from XP. Continuous Integration makes sure that builds are reproducible and reliable. In this practice developers commit their code and assets to a remote version control repository (like Git). This is best done a couple of times per day. Every time a new commit is done, the continuous integration server (like Jenkins) will pull a fresh copy of the project sources from the remote repository, build the project and execute all tests. If the build or any of the tests fail, a email or chat message is send to every developer. It’s preferable to combine this with a “stop and fix” attitude. Team members have to stop whatever they are doing and fix the problem introduced with the latest change. This makes sure that the project is always in a deployable condition. Problems should not accumulate until the QA phase is starting.

Automated Testing and Test-Driven Development

Sometimes developer think TDD is only about unit testing. But TDD does not define the granularity to which the tests should be written. There are many different type of tests. They can overlap and even conflict with each other. TDD is an evolution of concepts from Test First. Test First helps developers to think out ideas, concentrate on one problem at a time, and be precise about defining responsibilities and functionalities of modules, classes and even functions before writing them. TDD will give developer the confidence that their code is working as expected. Tests running in milliseconds, like unit tests, or in seconds, like integration tests, create very quick feedback loops during the development process. Thinking about what to test first on a module or class will help developers write code that is clean, simple and satisfies the requirements defined beforehand. It also helps avoiding overengineering and reduces the complexity of code. Simple and easy to read code is always easier to maintain and will reduce the probability of bugs.  A lot more on this topic can be found in Kent Beck’s book: Test Driven Development: By Example.

As a product grows, more and more features are added. Therefore, more and more features have to be tested by the QA, increasing the time it takes the QA team to test every feature manually. Automated testing of the correctness of your code can reduce the QA feedback loop from weeks to minutes. With a click of a button an entire system is tested and gives the assurance that the product is still deployable for production even after the last change was made. This avoids long QA phases for the business. Using automated testing also reduces the risk of writing more code based on erroneous code, reducing the bug-fixing costs even further.

But no matter what tests you perform, they should be automated if possible. Testing should always be a matter of just one simple button click in every developers environment. This will increase the frequency and also the likelihood of performing the tests. It’s best to integrate them into your CI pipeline.

Best Practices in Game Development

Combine TDD with CI

Combining TDD with CI can make a whole QA team redundant. This can drastically reduce the costs and the time it takes to bring a game to market. At it Matters Games we are using Unity3D as our main game engine. Running Unity3D in headless mode enables us to build and deploy app packages using a simple command line. At our office we have one Jenkins master server and several so called “slave nodes”. Jenkins calls the unity command line to build and deploy our projects every time a developer has committed a change to a project’s repository. To increase parallelism and speed of builds, build jobs are spread across slave nodes trough the network. Furthermore, since Unity3D 5.3 the nunit framework and integration tests are part of the Unity3D editor. With a simple command line tests are executed and written into a nunit compatible xml file. So we are able to perform tests using our CI server. The status of the builds and the tests is notified by mail and into a slack channel. Developers are always aware of successful and more importantly failing builds.

Avoid Manual Steps

Building and testing the final package of a game should never include any additional manual steps except e.g. starting the master build. Manual steps are prone to introduce errors. For example, never force the developer to change the output XCode project manually before building and exporting the final ipa file (like adding additional frameworks, changing provisioning profiles or setting compiler flags). Also build artifacts should never be deployed manually to an artifact repository. Best let the CI server deploy build artifacts to a repository automatically. This reduces the chance of sending a wrong version of a project to the customer. If you bulk build for multiple platforms like Google, Amazon and iOS like we do, make sure you are building the same revision of the repository. Think about “checkout once” for all versions of the game. Clean the workspace before building. This will remove the possibility of intermediate build files interfering with the current build. Archive additional build artifacts like test results and log files. This will help developers track errors and failures.

Know Your Tools

Additionally it should be very easy for developers to build and deploy the game in their local environment. If it takes developers more than just a click of a button to setup, build or test different versions of the game, they will not do it. They will not debug and test their changes before committing them to the repository. Building locally should be as easy as building using CI. Developers should be familiar with the build pipeline and the build scripts. Make sure everybody knows the build tools and how to use them.

Write Tests

There should always be time for writing and automating tests, but never add a task to the sprint called e.g. “Write Unit Tests”. Writing tests should be part of each and every coding task. If you create a separate task for testing, it will be the first task that is obsoleted by the project manager when it comes to timing issues. Be pragmatic with testing. It is not easy to decouple Unity3D code by using e.g. dependency injection and therefore allow for unit testing. There is no interface for the MonoBehaviour which allows for mocking. It can take a lot of effort to create tests for them. Always try to utilize methods that add more value to the project and create a quicker feedback loop with respect to the scope and the budget of the project. If it means doing TDD, than do it. If it means doing something completely different, use that method instead. Be agile and adopt to new practices if they work better for you.

Be Lazy

Be as lazy as possible. This does not mean you should lay back and let others do the work. This simply means adding automation wherever possible. Manual steps introduce errors, so avoid them. Think about scripts that can automate project setup, builds, platform switches, asset integration, testing and so on. This will give developers more time to concentrate on more important tasks: creating a quality and fun game.

More to Come

In the next couple of weeks I will describe how we at it Matters Games set up our CI pipeline in detail. I will describe:

• how to create a master server and add build slaves to it
• how to use Unity3D together with Jenkins
• how to deploy builds
• how to run tests on Jenkins and how to use Jenkins Pipelines with Unity3D projects.

– Ulrich Kaminski is the Technical Director of it Matters Games. He graduated from the University of Magdeburg and holds a diploma of engineering in computational visualistics (CV). During his 5 years at Reakktor Media, Ulrich gained extensive knowledge in AAA development. Since 2012 Ulrich has been working at it Matters Games, successfully developing and shipping games to a global market.