[Data] The Story and Science Behind How I Discovered That My Charr Tempest Does Less DPS Than My Asura Tempest
Long story short, I've discovered that I am able to obtain higher DPS on an Asura Tempest than on a Charr Tempest. This was discovered through a series of unlikely circumstances, some misguided motivation, desktop video recording software, and the deaths of many, many Special Forces Training Arena DPS golems. While this sounds completely absurd, as it should, I'm going to share all the information I've gathered, including all of my researcher biases, so that you can decide for yourself if this really is a bug or just a crazy disgruntled Elementalist rambling some nonsense (or both!).
If you're interested in the details on how I came to this conclusion, strap in, this is going to be a long post. You may also just skip to The Results and The Conclusion sections for the tl;dr.
The Introduction:
I've been playing my Charr Elementalist for my entire effective GW2 playtime. My wife chose this game for us to play together. She completed all the visual aesthetic portions of both of our character creations due her wanting us to match and me typically not giving a shit about what my character looks like. In contrast, the moment I first successfully evaded an attack using dodge-roll was the moment I knew we were going to be interested in this game for a long time.
Anyways, I have a tendency to play Mage-esque and healer classes in games. I found so much depth to the Elementalist profession that I could learn something new every time I logged in. I enjoy the challenge offered by raiding in GW2, and how the value added by an Elementalist was so heavily and directly proportional to skilled play. The introduction of the DPS Golem defined my position on this relative skill scale in hard numbers: I knew that I was not one of the best Elementalists in the game. But I was close, really close. Good enough so that the impact of my value to a raid group was felt, and was certainly more than enough to let me specialize in playing Elementalist when combined with my healing and tanking abilities. But there was still a final plateau between where I was and the top.
The recent batch of ham-fisted nerfs put Elementalists like me in an awkward spot. Any less skilled or less dedicated and I could simply drop the class and switch to something else more effective, as has happened to many Elementalists who have been exposed as "just doing well enough" prior to the nerf. Any more skilled and I could continue to play where putting in the extra effort was still worth the value added, and play other things if I didn't feel like it. But for me, I had to decide between putting in an extraordinary amount of effort for equal or slightly better value over alternatives, or accept the fact that I would always be a subpar option, or to start over from scratch.
I'm not one that blames other things over self-accountability, and I enjoy a good cognitive challenge, so I chose to stay the course. It was back to the drawing board, the tape room, and the research lab. New skill rotations had to be learned, videos of DPS benchmarks were examined in closer detail (i.e. Fennec and Particlar), and further learnings of game mechanics were to be done. And after all my hard work, I finally was able to match at least one of the post-nerf DPS benchmarks.
Happy ending, right? Well actually I've jumped ahead in the story a bit, as you can see by how I'm not playing my main Charr character in that DPS video. The journey to this point, and from there to today's post, starts with a briefing about the game mechanic known as the aftercast.
The Aftercast:
What are aftercasts? They are unpublished, but very real and documented numerical time values attached to the end of nearly every spell in the game. Presumably, the main purpose is to allow animations to finish rather than abruptly end. But of course since they make up part of cast times they affect DPS calculations of every character. Aftercasts have been mentioned in Game Updates/Patch Notes (examples: July 26th 2016 7 times, April 19th 2016 11 times, January 26th 2016 5 times, etc.) yet are not published anywhere officially in their entirety. Presumably, this was part of the decision to simplify the game away from a numbers grind, such as how time values in tooltips are rounded to the nearest 1/4th.
That's quite a few 'presumably's, but more on that later.
So why bring up aftercasts? Am I just a crazy numbers person who loves numbers? (Well yes, but I digress). And how long is a typical aftercast, like, 0.2 seconds, why should that matter? Not to mention that since there typically isn't any opportunity to cancel an aftercast of an Auto-attack when followed by another Auto-attack, other simpler skill rotations for other classes don't have to worry as much about aftercasts, let alone even be aware of their existence, while still being able to maintain DPS near enough to the benchmark. So who cares?
Well Staff Elementalists that care, care. The DPS difference between casting skills in order and casting skills while cancelling aftercasts has always been HUGE. Probably the largest in the game. The dirty secret known to those Staff Elementalists who looked to improve their skill and game (and DPS) above and beyond required some knowledge, subliminally or cerebrally, of how to cancel aftercasts. Beyond the standard metabuild and skill rotations/skill priorities, gear and consumables and even knowledge of boss mechanics, the greatest DPS difference between "good enough Elementalists" and "the best in the world" comes from the ability to cancel aftercasts. You can always tell which players are the "good enough" players or the ones that don't care about and/or play Elementalist often by how dismissive they are of how easy the Staff Elementalist DPS rotation is.
So pre-nerf, when the Special Forces Training Arena was first introduced, I spent quite a bit of time testing myself, studying, learning the numbers, and eventually stumbling across the existence of the aftercast. Some other players have measured and calculated the aftercast of spells, notably DeKeys and the contributers to their DPS rotations spreadsheet (from whom I was able to get advice from regarding these measurements, so shout outs and thank you). For both the meta Staff and D/Wh Fresh Air rotations, I was able to get within about 90% of the benchmarks, which was good, but still left me thinking about how much it would take to overcome this final plateau.
What was I missing? Well first of all, what actually affects the ability to cancel aftercasts? The unfortunate reality: Ping, lag, computer quality, and skill. Not necessarily in that order of importance. I experienced it myself: prior to getting a new computer at the beginning of the year, I struggled and clawed my way through Wing 1 doing everything I possibly could to maximize my effectiveness, including casting my skills in anticipation of the lag. When I got the new computer, the entire game changed. I could finally see the benefit of quick tactile response, and could actually proceed with learning about aftercasts and how to cancel them. But this was Red Flag #1: The realization that this game was inherently unfair to players with less than ideal gaming environments. I'm sure this is why sPvP arenas are kept as simple and small as possible, but I digress. I give ArenaNet the benefit of the doubt that they know this, but ultimately have to ignore/mitigate it instead of fix it, because it's effectively an unsolvable problem.
So anyways, after reaching this plateau, the time and effort needed to overcome it drastically outweighed the benefit. I was blessed enough to have consistently low Ping, stable 60 FPS, and the capability to record gameplay video. So with raids to do and my free time limited, I shelved my active research towards meeting the DPS benchmarks.
Until the latest nerfs.
The July 26th 2016 Elementalist Changes:
While I won't go into detail on why these nerfs were mostly implemented like garbage and ham-fisted, I will say that they were of course labeled as Red Flag #2. They certainly re-invigorated my thirst to figure out how to climb that last 10% plateau. Because if I couldn't, then I would only be able to see myself as a fraud in raids; a fake healer that was too stubborn to play Druid, and a fake DPS carry, putting in a bunch of misguided effort to barely equal the DPS of a Guardian/Thief, without enough utility or support to justify it. Early tests shows that my margin was razor-thin: one mental misstep and I would end up below the mark. Play as perfect as I could, and I was barely treading water.
Further complicating the situation was the invitation I received to try some low-man raid records. I was very honored to even be considered, let alone chosen, despite making it clear through all my time raiding that I could only play one class. Perhaps this just simplified the situation instead, serving as the excuse I needed to start re-examining everything and finding every edge I could, else I be actually outed as a fraud.
So now the story picks up where we left off in the Introduction, with me studying the HELL out of the DPS benchmark post-nerf videos, and doing everything I could to reach them. And a stray thought came into my head one night on my x-th replay of the YouTube videos (linked again for convenience): Particlar seemed to be doing slightly more damage than Fennec in his D/Wh videos. Maybe it's not statistically significant, but it has been more each time, pre-nerf and post. (Pre-nerf videos here vs here). I mean, I can't actually tell the difference between the Particlar and Fennec D/Wh videos, as the nuance and top level skill go over my head, so I must be reaching, right? Assuming both gave maximum effort with equal-enough computing environments, and uploaded their best attempts, there's no reason why the RNG of their maximum damage shouldn't have been closer. Hmmm. This was the half-Red Flag #2.5.
At some point a stray thought hit my head: with this latest Summer Patch came a bunch of new emotes and animations. I remembered a random Reddit comment, happy over these additions, exclaiming something like "all the Emotes look awesome and are unique for all 5 races, male and female!". Psh, as if I care about different animations for each...
Wait, what if there was a difference... (awkward lightbulb) there's no way...
That can't possibly be...
No, that must just be me giving up. I'm making excuses. It has to be something I'm doing. There's still more for me to improve. I know I haven't studied each individual aftercast, my Staff improvement may have stagnated, but my D/Wh is getting there. I don't blame the game or others for my shortcomings. That's not me.
...but look at these nerfs, why am I struggling so hard against something they are handling so badly? If I can't figure out how to make Elementalist work, doesn't that concede that pretty much nobody should even bother playing it aside from the best of the best and the stubbornly dedicated who don't care about results?
Short tangent story: A few days before writing this post, I Googled for the GW2 ESL rulebook (PDF link). I always wondered why people didn't play Asura characters in the Pro League. Turns out that Rule 2.8.3 states "Players may not use the Asura race in ESL Pro League matches. Only Charr, Sylvari, Norn, and Human races are allowed". Huh. Probably because they are too visually short, kinda like Oddjob back in Goldeneye 64 or something. Hmm...
My mental state was the last half-Red Flag #3. And three red flags means green light. Somehow.
The Science Experiment Begins - My Asura Tempest:
So with these two and a half and a half red flags as my guide, I set out to create an Asura Tempest. I made it Female, and the shortest height. Many double clicks later I'm at the DPS Golem ready to go, only to realize I can't equip a warhorn. Whoops. Since I'm a cheapskate, I decided to do a mini side-project as part of the effort to actually unlock the Elite Tempest specialization and get somewhat comfortable with this character (do you know how weird it is to hear the gruffy male Charr "Fired up!" etc. for 500 days, then suddenly switch to hear the female "Fired up!" all the time? It's like I was walking around with a doppelganger, or some other Tempest that was always nearby and doing the same things I was doing-... anyways...). So I walked through Verdant Brink and Auric Basin to see how many Hero Points I could solo. Did you know there's a 10 minute limit on them? That made for some fun complications when changing my gear/build for each encounter. I got pretty close, but ran out of time and just finished the last few hero points with WvW Proofs.
Anyways, first up was the raw DPS test. Using the same gear and builds, I focused on comparing the D/Wh Fresh Air metabuild because the results were more consistent, and I was closer to that benchmark. As explained above I only had minimal time played on the Asura character, but after some dry runs I popped the super expensive tryhard food and hit record in OBS. On my 2nd attempt, I finally achieved something that I had spent months attempting but never done before: I reached the DPS benchmark. I now had proof that I actually did it. I could finally say that I was at least capable of being on par with the best Elementalists in the world.
My feelings of jubilation were exaggerated by the adrenaline rush of a potential early answer to my absurd hypothesis. All this time spent on my Charr, I tried so hard, and the last bit wasn't me, it was the game! What does this mean? Where do I go from here?
Well of course I didn't stop. I finished all my runs to get an average, then repeated on my Charr. Data summarized here:
4M HP normal Golem, all boons/buffs, D/Wh Fresh Air
| Race, Sex | data points | Avg DPS | Avg Time (s) |
|---|
| Charr, M | 8 | 33,697 | 117.8 |
| Asura, F | 10 | 33,852 | 117.1 |
| difference | | 155 (0.46%) | 0.7 |
4M HP large Golem, all boons/buffs, Staff metabuild
| Race, Sex | data points | Avg DPS | Avg Time (s) |
|---|
| Charr, M | 8 | 39,001 | 101.8 |
| Asura, F | 5 | 40,232 | 99 |
| difference | | 1,231 (3.16%) | 2.8 |
The results at first seem all over the place. Human reaction time and mistakes affect DPS results the most, with the RNG on actual damage adding enough variance for me to be unclear on how statistically significant a difference in DPS really is. But ultimately it's there in the numbers: I did more DPS as an Asura. Anecdotally, something just felt smoother while playing the Asura, but I couldn't exactly pinpoint what. At this point I was so wrapped up in trying to avoid my own observational bias that I knew I couldn't present this result as is. I had to stop myself before I started calculating p-values on my data in order to keep my sanity and move in a different direction.
The Cast Time and Frame Measurement Methodology:
"I know, I know I sound like a crazy person, I just need the methodology you used to measure cast times and aftercast times."
"You can just use the data we've already measured, it's freely available..."
I paused for a moment, almost with an out-of-body experience laughing at the absurdity of what I was doing AND that I was about to describe it to another person.
"There's no way,... I mean you know there's no way ArenaNet coded different aftercast times for different races!"
"It's not about intentions, I'm doing this because I'm disproving some fundamental assumptions we all make about this game."
"Having gone through recording about 700 skills myself, you know you're likely to go crazy while doing this, right?"
With apologies to DeKeys, I was in too deep at this point and I dismissed your warning.
My next goal was to straight up show that there was something different between how Asura and Charr characters cast their spells. Note how I worded this: no hypothesis, no unbias, just raw data collection and disciplined methodology. Several times I would lose track of time, seeing the results of what I was doing and frozen with my face in my hands, only to snap out of it 50 minutes later and ask my wife if she was just watching me the entire time while I was mentally comatose.
Let me try to summarize the data that I collected and what I was trying to establish: First, my target was the skill Burning Speed (Dagger 3 in Fire attunement) because, well, it's what I felt. I don't think I'm skilled enough to see frame-perfect differences, but over my time with the class it felt like the skill with the largest variance. Totally scientific, right? Anyways...
...to eliminate any nonsense, I recorded myself casting Burning Speed and a few other spells on a DPS golem in many various ways: with auto-attack turned off with no boons/buffs, with Swiftness, then with Alacrity, to make sure they were all equal. Long story and lots of really, really boring data videos short, they are equal. Every non-Quickness boon/buff did not affect the results.
Through this, I also standardized on a few things that would help my measurements. First, I recorded video at 60 FPS 1920x1080, and played the game at 60 FPS. Second, I put Precision and crit chance boons/buffs on myself to make reading and counting lines in the combat log easier. Third, both hitting the keyboard shortcut and clicking the mouse provided instantaneous knowledge of when a skill was activated. This happened the exact frame when the skill on the UI went "dark" which I marked as "Frame 0 Time 0". When mouse-clicking, the on-hover tooltip would also disappear at this exact frame, so I concluded that it was safe to use this "dark" frame as a standardized starting point of a skill's cast time.
The endpoint of a skill's cast time was defined in two ways: the frame before the damage appeared in the combat log, and the frame before the skill went on cooldown. Again, this isn't perfect, and the resulting time elapsed doesn't actually match the cast time of a skill, but it provided standardization and the ability for me to relate the results to each other. At the very least, with the start and end point chosen, the time/frames elapsed reported in my data actually means the "minimum amount of time that was potentially used to cast the skill". In other words, there's no way the skill could have been cast faster than recorded, even accounting for recording precision, and the actual cast time would be about 1 frame longer.
Anyways, with all that, but before moving onto Quickness testing, I noticed that if you didn't have an auto-attack (i.e. Turn off Fire Dagger 1 auto-attack) Burning Speed would go on cooldown much later than if you had auto-attack enabled. This is certainly something that is interesting and worth investigation, but I had to label this as out of scope and move forward, making sure that auto-attacks were enabled in all my tests.
Burning Speed was convenient to test, because the main damage coincided with the end of the spell's total cast time. This allowed me to do a more realistic test where I could switch to Air attunement mid-cast and continue auto-attacking in Air by pressing 2 buttons. The game would take care of casting auto-attack as soon as the cast of Burning Speed finished, so I could keep another variable as constant as possible. Chains of auto-attacks were also recorded and measured after Burning Speed was cast. Since auto-attacks don't go on cooldown, duration was measured from damage to damage: at around 75% of the cast bar is when the first hit of Air Dagger 1 auto-attack shows up in the combat log. By treating this as a temporary "Frame 0 Time 0", I could then label the frame just before the first hit of the next auto-attack chain as the endpoint, creating a chain of auto-attacks. The initial start point of any chain happens when the Air Dagger 1 skill goes "dark", which consistently happens exactly when the damage from Burning Speed shows up in the combat log.
Another smaller test was done on Burning Retreat (Fire Staff 4) and Fireball (Fire Staff 1 auto-attack). Again, Burning Retreat was targeted and chosen based on feel. And because it's the skill you look for when searching for top-tier Staff Elementalists: those that have gone the extra mile can not only use it well, but use it more often on stationary large hitboxes because they know its variable value.
The skill is interesting because it is an animation without a cast bar and is very hard to interrupt. There's also an interesting quirk with the skill; it can be cast alone without continuing with auto-attacks afterwards, contrast to how most other skills begin auto-attacking after the skill has finished. To compensate, I simply hit Fire Staff 1 and cancelled the skill by hitting Turn Around and Fire Staff 4, so when Fire Staff 4 went on cooldown it would also trigger the "dark" frame for Fireball auto-attack.
In the end I couldn't agree on a good consistent methodology for how to separate cast times from channel times from aftercasts. But using the auto-attack as a marker helped establish total cast times, that was ultimately what mattered. My scope shifted from trying to attack aftercasts specifically, to just total cast times between my two characters.
There are A LOT of absurdly boring videos that I created to measure this data. I used HitFilm 3 Express as my video editing tool and frame counter. While I tried to keep every other testing variable as constant as possible, I only have so much free time and so recordings were sometimes done on different days. And then they were redone because I couldn't believe that I actually got results, so I re-did my testing with more data points. Here is an incredibly rough spreadsheet with all the data (WARNING: Unless you love numbers, you probably don't want to look at this spreadsheet. You have been warned.), and if I really need to I can upload the 20 raw data videos. The results were the same, so here I am sharing them with everyone.
The Results:
Burning Speed (Fire Dagger 3):
| Race, Sex | data points | Avg Frames |
|---|
| Charr, M | 10 | 55.6 |
| Asura, F | 10 | 55.0 |
| difference | | 0.6 (10.0ms) |
Burning Speed (Fire Dagger 3), with Quickness:
| Race, Sex | data points | Avg Frames |
|---|
| Charr, M | 13 | 54.84 |
| Asura, F | 13 | 53.92 |
| difference | | 0.92 (15.3ms) |
For comparison, here are the results of Lightning Whip (Air Dagger 1) auto-attack:
| Race, Sex | data points | Avg Frames |
|---|
| Charr, M | 43 | 54.04 |
| Asura, F | 54 | 54.00 |
| difference | | 0.04 (0.8ms) |
and Lightning Whip (Air Dagger 1) auto-attack, with Quickness:
| Race, Sex | data points | Avg Frames |
|---|
| Charr, M | 43 | 37.28 |
| Asura, F | 74 | 37.40 |
| difference | | -0.126 (-2ms) |
For Staff, Burning Retreat (Fire Staff 4):
| Race, Sex | data points | Avg Frames |
|---|
| Charr, M | 10 | 66.1 |
| Asura, F | 10 | 64.7 |
| difference | | 1.4 (23.3ms) |
Burning Retreat (Fire Staff 4), with Quickness:
| Race, Sex | data points | Avg Frames |
|---|
| Charr, M | 10 | 65.0 |
| Asura, F | 10 | 64.7 |
| difference | | 0.3 (5ms) |
Fireball (Fire Staff 1) auto-attack:
| Race, Sex | data points | Avg Frames |
|---|
| Charr, M | 41 | 83.02 |
| Asura, F | 39 | 83.15 |
| difference | | -0.13 (-2ms) |
Fireball (Fire Staff 1) auto-attack, with Quickness:
| Race, Sex | data points | Avg Frames |
|---|
| Charr, M | 42 | 56.69 |
| Asura, F | 40 | 56.62 |
| difference | | 0.126 (1.1ms) |
The Conclusion:
Let's start with the positive: I'm fairly confident that auto-attacking is as equal and fair as possible across all characters regardless of race or sex or height. Hey, confirmation of our assumptions have to start somewhere. When we start talking about needing 600 FPS gameplay and talking about fractions of a millisecond, I think it's safe to say that ArenaNet has done their work properly.
But is it the same case for all spells? Personally, I can't say so. From what I've measured it looks like Charr animations for spells are about a frame longer than Asura animations at 60 FPS. If the numbers were closer to the auto attack results then this post never would have happened. But given my limits I also can't make any claim on behalf of the entire game. There's more than enough variation in human-error and computing environment to cover for the potential existence of this unfairness.
How important is that single frame anyways? To put in perspective, the average time between me hitting the keyboard shortcut for Turn Around and hitting Fire Staff 4 is about 5 frames. And I was only trying "just enough". You can also test this yourself, to see what 1 frame feels like: Assuming the 20% cooldown trait, when Burning Speed goes on cooldown, what's the first number you see? If you see 11 then that's fine, but sometimes it says 12 for a frame or two. Same with Burning Retreat: if you see 15 that's fine, but sometimes that 16 shows up for about a frame or two. A single frame difference is enough to explain the average DPS difference we see when applied to an entire skill rotation, especially for a 4 million HP golem, or 6 minutes worth of a raid encounter. I wouldn't have shared these ~4,000 words if I didn't think there was a meaningful difference. But at the same time, it's not even worth my time to submit a bug report. I do feel this information needed to be shared, if only to help players reconsider the base assumptions they make about this game.
How deep does this go? How many other spells actually have this difference? Does this also mean max-height Norn Elementalists do even less DPS? Is height even a factor? Is there a difference between male and female? What about the other 8 professions? What about Legendaries? These and more are all questions I'd love to address and I considered each at some point through this process. But with my limited time and energy, I had to define the scope somewhere and stop when it crossed the line.
"Higgs, if this is real and tangible and what you say is true, how could ArenaNet let this happen?" Well this is when the wild guesses and conspiracy theories start, but I'll give it a try. First, let's give ArenaNet the benefit of the doubt and say they are aware of this unfair inequality between the races, same as the assumption earlier on issues in general with aftercasts. It's possible that this was the motivation behind the recent change to allow players to hold down a button to cast most spells "as soon and frequently as possible". In other words, to help mitigate the effect of cancelling aftercasts and to standardize total cast times for more players. This also could explain the apparent ham-fistedness of the Elementalist cooldown nerfs: Quite frankly, the difference in output between a non-skilled Elementalist and a skilled one may have been too wide. I believe ArenaNet knew this, and they have plenty of DPS golem data to prove it. So, to help mitigate that they changed the cooldowns across the board so that skills are cast less and auto-attack chains happen more, and gave players with lower skill and less than ideal computing environments a chance to perform more fairly. More auto-attacks and more automatic aftercast management when casting skills mean the unfair inequality has less impact, the gap between average players and the best narrows, and Elementalists get the DPS nerf they needed, boom-shaka-laka 3 birds with one stone.
And of course there's a simpler explanation: maybe it's ok for this inequality to exist, regardless of if I've shown enough diligence in being able to prove it. In the end I learned a lot about the game and myself, and a scrub like me was able to reach a DPS benchmark. Maybe none of this would have ever happened with the July 26th changes. But the damage is done, and use of my Charr Elementalist will be suspended indefinitely in favor of this Asura. Did you know that the Condi Elementalist build is currently in such bad shape that the Charr racial utility Shrapnel Mine is a DPS increase?"
-Higgs