Title: What Do People Do All Day?

Author: Martin Moene

Date: 06 March 2015 21:03:22 +00:00 or Fri, 06 March 2015 21:03:22 +00:00

Summary: Thaddaeus Frogley shares his day to day activities as a games programmer.

Body: 

The inspiration behind the series...

This article was inspired – you could even say ‘requested’ – by Chris Oldwood’s post on accu-general: A plea – more articles on what kind of programming you do Hi All, Something I’ve tried to do in my own articles in C Vu and Overload is to try and give some context to the kind of programming I do, because I know it’s different from what many others do. In particular I’m interested to know what kinds of constraints, or lack of them others have to put up with. For example, I remember an accu-general thread about unit testing and how that might/might not be as easy to apply in the gaming industry. Those in the embedded market have historically avoided C++, when perhaps it’s certain features of C++ they couldn’t afford. Floating-point maths is apparently a no-no in finance, unless you’re doing large volumes of risk calculations and then performance trumps precision (well, in the bits of finance I worked in). How does the inability to patch a video game because it’s delivered on a read-only cartridge affect the development process? I don’t know about anybody else in ACCU, but I want to know more about the kinds of stuff other people do. And in particular what makes it different to what I do. I appreciate it’s often tricky to know what’s different (unconscious incompetence) but in those cases when you have had to make a trade-off – what was it and why? When have you read a blog post or  book about some cool technique and then shouted at it because it has no place in your industry/organisation/etc? I’m sure the editors of our C Vu and Overload journals would be more than happy to receive more content... Hopefully, this will become a series.

I work as a programmer for BossAlien, an award-winning game development studio based in Brighton, UK. We are part of the broader NaturalMotion team based in Oxford and largely operate as an independent studio.

I am currently the lead programmer on the client team of CSR Racing, a popular drag racing game for mobile phones and tablets. We operate as a live-game team on CSR Racing, meaning we work on updates, listen closely to player feedback, add content and features and fix bugs. My time is split between team leadership work, programming and what we refer to as ‘data-wrangling’.

The team

Our office of about 60 staff in Brighton maintains a relaxed informal atmosphere and internal work culture.

The programmers have a range of seniority, from fresh graduates all the way through to grizzled veterans, with years of experience distributed in exponential steps.

We have a mostly open plan office on a single floor, with two meeting/conference rooms, teleconferencing facilities, a kitchen and a shower. Some people, myself included, use standing desks. There is music played in the office which draws from a shared playlist, with different parts of the office having separate control over volume. People play musical instruments, nerf guns and board games at work.

The software

The game client is written mostly in C# (221kloc) and is based on Unity3D, a very popular game engine. In addition to C#, we work with Objective C (25kloc) on iOS, and Java (34kloc) on Android. The server side, both player facing APIs, and internal tools are written in PHP and C#, but that is handled by the infrastructure team, so I don’t have to worry about it. We also use a number of 3rd party SDKs, libraries and plug ins.

In addition to the code itself, the game also consists of ‘prefab’ files, which describe much of the visual and logical structure of the game, including the UI, in a proprietary format specific to Unity3D. Most of these are created using the GUI tools that make up the Unity3D editor, but total 1304kloc! Finally, there are the binary assets: audio, graphics, textures, geometry and other non-textual data. In total the working copy is ~7GB, and the built deliverable is close to 300MB per platform.

We use git to manage all the source assets and the build products. We have recently had to take corrective action and reorganise to reduce the size of the full repo, which had ballooned to 35GB. This was done by purging the build products from the main repo. We have started to use git submodules to support reuse and to logically separate other parts of the repo.

We use feature and release branches in git. As I write this, there are currently 49 branches on the remote. We also have, independent of git, a system for delivering game assets (art, and configuration files, we aim to be able to control as much as we can via data that we deliver live over S3), which has its own branching system. This means we often have a parallel branch structure. Our builds – asset system – exists in three stacks: Dev, QA, and Release. These various orthogonal configuration axis provide us with a lot of power and flexibility, but can be quite confusing for the uninitiated.

We avoid code ownership and try to share knowledge about systems around the team, using a mixture of code review, mentorship, regularly changing areas of responsibility and documenting work flows. Code is expected to be self-explanatory first and well commented second. We don’t have a fixed coding style or standard, instead using the convention that your code should be consistent with the code around it.

The builds for QA and for Release are produced by a build machine, using Pulse, and a set of custom build scripts. A change (code or art) can be tested in the editor in seconds. Certain asset changes need to be ‘bundled’, which can take a minute or so. A development build can be created and deployed to a device in the time it takes to make a cup of tea. Iteration times are very important and anything that slows the team down is considered a high priority problem.

The hardware

Most of the programmers have Apple laptops with external monitor(s), keyboard, mice and headphones. Specific individuals may have a PC or Windows laptop instead or as well depending on what their work requires. Most of the build machines are Mac Minis or retired laptops. People can take their laptops home and can work remotely if required. We have a single rack of servers for internal services, file shares, routers, etc., using AWS for the player facing servers. There is a pool of iOS and Android devices of for testing development builds on.

The process

We have to juggle a surprising number of challenges. Players in CSR Racing are always looking for new content and new things to do, so we are consistently adding new features based on demand. It’s critical for us to listen to and respond to player feedback. There are also new platform updates to be aware of, so we are always working hard to keep SDKs for integrations with external services up to date. We also have to be mindful to harden our system security as hackers sometimes look to cheat and exploit the game. As a competitive game, it’s critical that we maintain a consistent level of fairness.

We have a release and high level features plan, which is discussed regularly. The time line for this is maintained centrally across teams and is frequently updated and referred to, with notes about staff vacancies and other important calendar events. We closely monitor progress and shift resources to where they are needed in order to keep our commitments. We regularly discuss capacity and strive not to over commit. Overtime is rarely requested, and is never obligatory. Individual tasks and bug reports are kept track of using JIRA. Work is organised into releases of two types: Code Drops, and Data Pushes. With code drops we are gated by the platform holders and have roughly a 3-week latency from a Release Candidate to the update going live on App Store(s). A Data Push, on the other hand, contains no code (though some configuration files come close to a complexity that could be described as programming!), but game data that is pulled by clients with an internet connection from our S3 file store. This doesn’t go through a publisher approval process and can be subject to change as little as hours before it is published to players.

Whatever is pending release, our QA team is also constantly working on testing features in various stages of development. Email is an important tool for our asynchronous communications within the team, but we also have lots of face-to-face time, including regularly scheduled one-on-one chats.

We are able to release different code and data for each of the platforms we support, but we strive for feature parity and same day releases on all platforms.

Every day is different, but for me a typical day might start with reviewing the previous day’s commits, and setting up any relevant code reviews. I’ll likely then check my email, respond to questions about the project and do some code reviews. I’ll check JIRA and possibly triage any new bugs with the QA lead, and the project manager. If it’s a Tuesday or a Thursday, we’ll have a core team stand up meeting. I am responsible for the overall technical health of the project, so I have to play it by ear, identifying things that need doing, and either doing them, or making sure the right person does. This might be debugging and bug fixing, implementing new code, or helping plan code work. It could be auditing the branches in git and making sure the features get merged in where they are needed. Sometimes it’s testing, sometimes it’s setting up data to implement new features using data driven systems. We have kick-off meetings, and post-mortems for bigger chunks of work, so we can reflect on and improve our processes actively. Our general philosophy is to let people use the tools they prefer. I use Sublime, and TextWrangler. Some people use Xameran, others use MonoDevelop. The technology stack we use means debugging often has to be done via logging, as remote debugging the C# using MonoDevelop has stability problems. In addition to using git on the command line, we use SourceTree.

Recruitment is an important part of the company strategy, so group CV review is a regular feature of office life. Applicants who pass the CV screen are invited to do a remote code test, the results of which are discussed by the whole code team. The ones that pass that filter are invited to come in for a face to face interview.

The future

CSR Racing continues to be one of the world’s most popular racing games on mobile and tablet.

Our goal is to continue to delight players – both those new to CSR Racing and veterans.

While we do that, we try to improve the tools, workflows and the code itself so that all this can be done at the same rate, but with fewer people, freeing up more of the team to help with the development the company’s next game(s).

End

If you’d like to hear more about the development of CSR Racing, I will be talking about it on Wednesday the 22nd of April, at the ACCU 2015 Conference in Bristol.

Notes: 

More fields may be available via dynamicdata ..