XenoThreat Postmortem
03/10/2021 - 1:00 AM

On February 4, 2021, we launched Alpha 3.12.1: Assault on Stanton, which introduced the first dynamic event in the Star Citizen universe. The following is a postmortem from the senior developers themselves, detailing what was delivered and their thoughts on how it went.

XenoThreat Mission
What went well?
The initial design for XenoThreat was put together by Luke Pressley and I (Tony Z). We both have a strong working knowledge of the current state of the game and, after a few iterations, the final pitch felt solid and realistic. There was a good amount of documentation for how the event would be run, including a high-level breakdown of how to play it, what enemies should spawn, how the rewards worked, and how to test the screen overrides. This was all maintained throughout development, which worked as a quick reference for QA and other departments.

After a somewhat clunky start, the feedback loop on XenoThreat ended up being a pretty well-oiled machine. The process of gathering feedback from the PTU via the Player Experience Team and Feedback Reports, getting bugs in with the appropriate labels, and reviewing and triaging new bugs each day to get priority calls ended up being a very clean process. We made a call later in the process to have QA enter tracking tasks in JIRA as soon as the Feedback Report came in instead of waiting for internal reproductions. This allowed for more rapid iteration, so sometimes the developers were able to address the feedback even before QA got a chance to enter a proper bug for it.

Regarding the event itself, there were a number of features that were positively received. The shared income pool, social style distribution of tasks and gameplay types, and work-together nature of the cargo unloading were immensely popular with the community. This by itself was a huge part of the positive reception. Also, when everything was working smoothly, the entire feel and look was stunning. We are very happy with how the balance of design, VFX, and SFX came together. We created a strike-team dedicated to making the combat feel great, which made quick iteration more viable. The narrative that went along with the release was also quite cool, and the XenoThreat commander was effective and intimidating.

Other aspects of the event we found successful were large in-game income opportunities for players (all phases), FPS pirates populating the wreck sites (Phase 2), and team-oriented payout style (Phase 2). Many players appreciated the bonus of getting paid very well for doing the event in all phases - this further incentivized continued play and added replayability. Although there were some issues with the FPS AI characters, their general presence made going through the wreck sites more interesting for players and added to the varied style of gameplay. There were some detractors who wanted more income per-box in a more individualized payout system, but there were far more who liked the team-oriented style of payout, with many saying it encouraged cooperation.

While it was unfortunate we weren’t able to launch it at the end of the year, the decision to delay the event until 2021 benefited it massively. The difference between what we would have released before Christmas and what we did release are night and day when it comes to stability, performance, and quality of life. It was a great decision by the executive team.

What didn’t go so well?
Dynamic Events are a huge feat of management as their delivery relies on multiple departments. They require one owner with a clear vision and knowledge of all aspects to be on hand to answer questions and address feedback. In order to give XenoThreat the attention it required and deserved, a lot of folks had to juggle responsibilities. This type of thing can easily lead to other responsibilities sliding. If events of this scale are required several times a year, they cannot just be absorbed by an existing team as their other responsibilities can suffer.
The dialogue added a lot to the mission, but getting it in was a big undertaking with long lead times to deliver it and the mission triggers. There was insufficient time to get the actual lines into the fully functioning mission and iterate on appropriate changes. We did get placeholder lines in earlier but, due to the feature that triggered them still not being fully functioning and the mission not being completely finished, it was difficult to actually review them in situ. In the past, we have used programs like Visio to create flows of what lines trigger when and what orderings should occur. We didn't have time to do this for XenoThreat, so dialogue was implemented straight into the logic. This made the processes more ad-hoc, and extra flow-graph planning would have eased the process of designing the logic to support the dialogue triggers. Usurprisingly, much of the dialogue triggering had to be heavily woven into the mission logic for it to be fired at the right time, which meant that efforts to review the dialogue in context were unrealistic even if everything had been working and the mission finished when placeholder lines were delivered. I think we need to be more considerate about expectations on reviewing dialogue in the mix when that mix only really plays out during QA, PTU, and Evocati playtests.

More time spent prototyping would have been extremely beneficial, particularly for the Starfarer derelicts, as the feature requirements changed in the middle of development. Requests were made to add gravity to the interiors and to add targeting capability and UI support. We ended up shipping with one less version of the Starfarer derelicts than originally planned due to issues with zero-g. More prototyping would have also enabled us to better understand what balance was required to get to the point where we knew the combat worked as intended. There were times that the feedback could be red herrings, such as server performance being a cause instead of actual balance issues. This sometimes made it difficult to pinpoint where problems were occurring without extensive testing to verify.

Internal testing was sometimes especially challenging due to the difficulty of reproduction, breadth of repro steps, large number of testers required, and time-zone differences. Sometimes, the developers would get bugs without repro steps that they were unable to reproduce internally, while other times, the repro steps would be incredibly long and would take a massive amount of time to test. We need to be able to test the mission more easily so we can get a better understanding of the state of it at any given time. Later in development, we added new tools to bypass select aspects of the mission and modify internal parameters to help expedite the testing process. This should have been done earlier.

There was a lot of feedback from our community on aspects of the event that could have been better. Players weren’t made aware of how long Phase 3 would remain active and were surprised when it came to an end. Better information to manage expectations would have improved the situation. Event terminology regarding phases was confusing at times, with differences between how things were referred to internally and what was described to backers on Spectrum’s feedback threads.

We knew from the beginning that performance would be a limiting factor and that meant we wouldn’t be able to deliver a sufficient number of enemies to generate the level of challenge that we actually wanted in certain situations. As a result, some players found the Idris far too easy to kill and, when this happened, it derailed Phase 3 significantly. With the right loadouts (Typhoon torpedoes) a single Retaliator could drop the shields of an Idris and multiple players could kill it in just a few minutes. Rapid mission finishing and a long cooldown meant players trying to do the mission struggled to find a server with it active and, if they did, get there in time to participate. In addition, the Idris offered almost no threat to players even if they didn’t kill it quickly, it mostly acting as a bullet sponge with many players mentioning it holding still and continuously firing.

Hostility detection and CrimeStat determination are still too simplistic, with no modifier for shared mission participants whether the ship hit is targeted or any link to targeting at all. With so many ships in close proximity in a complex scenario, friendly fire was frequent and often resulted in players accidentally getting a CrimeStat, being dumped from the mission, and sent to prison upon death.

While we wanted to allow players complete freedom to choose which side to support, there was insufficient time. This caused problems when, inevitably, many PVP-oriented players took it upon themselves to attack players attempting the event. This was most prevalent in Phase 2. Since the event didn’t support this, however, there were few ways for mission participants to prevent or counter this, resulting in some strong opinions on either side. Most players just left those servers to find one without PVP. On our end, we're thinking about how we can support both play-styles.

Many players mentioned poor client framerate during both combat phases. In addition, there were numerous reports of mission-critical assets loading slowly including wreck sites, supply ships, XenoThreat fighters, and FPS AI (which loaded in after players started clearing cargo at times).

As opposed to Phase 2, Phase 3 was singularly focused on ship combat. This meant that alternative play styles, such as salvage and FPS combat, were not able to contribute, which caused considerable consternation and frustration amongst some players.

Players have very limited and shallow friend or foe identification systems (IFF), with red being either directly hostile to the player or someone with a CrimeStat and blue representing everyone else. There’s no way for players to tell who is on the mission, who is aggressive or hostile to others (without a comm array), and often who is in their party as the party markers are not always reliable. This was most critical on foot in the wreck, where no identification was used at all and hostile NPCs, hostile players, and cooperative players on the mission all appeared identically.

AI fighters wiggled, warped, teleported, flew erratically, and generally offered no threat, with players frequently calling them cannon fodder. In addition, players frequently mentioned how low their numbers seemed, limiting the sense of danger. AI fighters also seemed to lack aggressive behaviors, such as targeting cargo ships or bombers, and sometimes ignored incoming damage in the process.

Although some players commented that it was far better than previous patches, EVA remains an issue, especially on physics-grid transitions (entering and exiting a ship), with players frequently taking damage or even dying. Specifically reported were bad transitions that involved players falling over and/or taking damage, generally sloppy and imprecise motion in flight, and awkward loss of control when bumping into objects.

Torpedo spam dominated Phase 3 and was further encouraged by massive payouts for hits. The Idrises often failed to properly counter them and the simplistic nature of their use (lock, launch, done) contributed heavily to the “easy” feeling of Phase 3.

The law and hostility systems need significant work, especially with regards to potentially friendly fire and accidental damage. Included in this are how and why we apply a CrimeStat, how charges are pressed, how hostility is determined, and a thorough discussion of how to better identify friends and foes. That should include distinction and markers for players on the mission, counter-mission players (if appropriate), aggressive players, hostile players, criminals, and factions where appropriate.

What we’ll do differently to improve in the future:
There are several issues called out above that we are actively discussing and plan to address for future events. Over the coming weeks, we’ll look at each piece of feedback in turn and discern how to address them, which teams will work on them, and where they fit into the schedule. Issues with performance, AI, PVP design, and the law system are at the forefront of our minds to ensure these events are as good as they can be.

Tony Zurovec, Persistent Universe Game Director

AI Team
What went well?
XenoThreat was a great opportunity to have a company goal across multiple teams. This is common whenever we establish specific common goals (milestones or specific releases) and it usually brings several departments together.

For these types of event, we rely a lot on other teams. For example, the PU Mission Team was hugely responsive and reactive and we constantly had a key person to contact about the mission flow. This helped us understand the goals they were trying to achieve and adjust the logic and suggest different ways of approaching problems whenever we could. The collaboration has been fantastic.

These events also give us the opportunity to utilize both new features and make improvements to existing ones. For example, we were able to make the initial pass on capital ship combat, expose the new assignment for requesting docking, and improve the current mastergraph flow for pilot combat.

During the event, we specifically reviewed AI behaviors and the player experience. Having all the relevant directors in the review allowed us to quickly collect feedback and discuss what could have been done in the defined timeframe.

What didn’t go so well
The idea behind XenoThreat was to use a specific timeframe assigned to the mission to achieve very specific goals. The intention was to not overload the AI Team with requests and make smart decisions in starting development of capital ship combat. For example, focusing on improving target selection distribution across the multiple ship seat operators or implement the ability to use the railgun. Unfortunately, in the overall schedule, the amount of work required to iterate on making it fun meant addressing edge cases was not fully considered and it affected the workload of the team.

During development, we also got caught in situations where we assumed bugs were caused by AI code or performance. However, they turned out to be a combination of other things that could have been addressed quicker if brought properly to our attention. Better due diligence while testing the flow can prevent a lot of this confusion.

Trying to improve performance at the same time as bug fixing made the final part of development a bit hectic but we also expected that, so we were not so surprised.

Francesco Roccucci, AI Director

Vehicle Experience Team
What went well?
The Vehicle Experience Team’s work on XenoThreat was mostly focused on balancing the flight experience of capital ships with the weapons they used, but also the weapons that would be used against them by ships like the Eclipse and Retaliator.

The tuning of the flight balance with the Idris and Javelin came down to allowing the AI to position each ship correctly to attack and defend. We made some compromises along the way so when these ships get into the players’ hands they will feel slightly different. But at the time, this was the right decision to make until we have improved some of the turret behaviors so they don’t rely upon the movement of the ships as much as they do now.

The most impactful changes we made were to the speed and strength of the weapons. The entire event pushed us to add stronger weapons, including the introduction of the railgun to the PU, but we wanted to balance this with speed. The more powerful a weapon is, the slower it generally travels, apart from the railgun which we used to push the limits of speed in Star Citizen. We really wanted the railgun to be feared, using the visuals to build the anticipation of what was to come and give players the chance to avoid it if the Idris managed to line them up, and for the moment it fired to be special.

The highlight of XenoThreat for the Vehicle Experience Team was seeing the various teams come together to deliver what was a fantastic event to not only take part in but to watch. We worked closely with the audio and visual effect teams to make sure the balance we had implemented was accurately reflected in how it looked and sounded. The devs spent many evenings watching the various streams that the backers had put on, each morning sharing the best moments with the rest of the team.

What didn’t go so well?
From a balance perspective, the main difficulty we had with XenoThreat was getting consistent, repeatable results to evaluate the balance in a real-world battle as the event was so dependent upon the performance of the game. As development went on, this got easier as performance increased, but it wasn’t until the final couple of weeks before release that we got to a point where, if things ran smoothly, the event would play out from start to finish as expected.

But on the positive side, the performance problems we had during development highlighted specific issues with both the weapons and shields that we were able to drastically improve under lower-performance scenarios. It also further highlighted some of the holes that we currently have in the balance, which we will be rectifying over coming builds.

Rich Towler, Lead Game Designer

Performance
What went well?
We are constantly looking to improve performance, which is very hard as we constantly add more features to the game. And while there’s a lot to do to get framerates at the levels we are aiming for, I think the main performance positive for XenoThreat is that we got it to a level where we could deliver an overall enjoyable event for lots of players. When we’ve tried this kind of capital ship combat in past years, we’ve been so far off the mark performance-wise that we couldn’t deliver an event of this scale, so hopefully this is a good first foot on the ladder that we can now improve on.

What didn’t go so well?
Many players mentioned poor client framerate, especially during ship battles in both Phase 2 and Phase 3, with emphasis on the latter. We know that server performance needs to be better, as that will help improve things like the response times of AI.

What we’ll do differently in the future:
I think the main takeaway from XenoThreat and the last few releases is we need a better system for profiling and performance regression, so we have the tools for all teams to more easily profile and optimize for performance rather than being bottlenecked or dependent on a small handful of engineers. The Engine Team and I have already been looking into this and made some good first steps this year. We have a new stress test framework that’s easy for code to use, better imGUI debug profiling support, and better auto-capturing and analysis of performance data. Whilst it may be a little early to bear much fruit for Alpha 3.13, it should be a big help for us in the long run. There is also a mandate this year for all teams to assign more time to optimizations that will help in this effort. And with server meshing on the horizon along with several other longer-term planned optimizations, we potentially have some much more significant gains we can make when those come online.

Rob Johnson, Engineering Director - Persistent Universe Gameplay

--- Thu Mar 11 2021 01:00:00 GMT+0100	
+++ Thu Mar 11 2021 01:00:00 GMT+0100	
@@ -2 +2 @@
-03/10/2021 - 12:00 AM
+03/10/2021 - 1:00 AM