29 August 2016

Dice Hate Part II

EDITED: Hi all - thanks for catching a mistake on the table. Looks like I made a typo and copied the wrong column over. As such the statistics have been updated. The change is the attack dice focus result on the FFG Dice App was not different than all other results, however, the attack dice blank result returned a similar trend as the evade dice but was only different from true probability and Nerd-X White attack dice. Please keep in mind that this is a low sample size, as such if you have beef with it (as many of you have stated) I would refer to this article that discusses power of sample sizes. To summarize for you in order to determine if a dice is fair or not you will only need to roll the dice 30x to reach 95% confidence in your outcomes based off a power of 0.80 (common power model). As we used statistical test based on this model we arrived at statistical probability. Does this mean it is 100% accurate, not in the least. This is one study with one low sample size. Take it for what it is worth or not, either way you are welcome to replicate the experiment and test it for yourself.

Welcome back nerds and nerdettes!

Nic here -

If you are just joining us, we have been discussing the use of third-party dice in X-Wing while simultaneously reviewing some high quality third-party dice from Nerd-X. Our most recent post (Dice Hate Part I) recently broke the internet and warped the minds of many X-Wingers. In short, the X-Wing community does feel that third-party dice should not be legal (so does FFG per the official ruling of "hell no") in tournament play.

As such, we (aka Jeb, Thomas, and I) fellow slaves of the faithful (aka TMB readers, X-Wing aficionados, haters, and lovers) sought out to determine if third-party dice roll differently than standard FFG dice. Moreover, we included the dreaded FFG Dice App for comparison as it has been brought up in the discussions since last week and adds an interesting layer to our science experiment.

Brace yourself for what you are about to learn and please refrain from throwing things at your computer screens and smart phones (or not, as I do not really care because the files are in the computer).

Three dice from each category (Nerd-X Clear, Nerd-X White, FFG, and FFG Dice App) were rolled 50 times for a total of 150 dice rolls per set. Each symbol was recorded for attack dice (hit, critical hit, focus, blank) and evade dice (evade, focus, blank) and further analyzed.

(we rolled a bunch of dice, looked at them, and tallied the symbols)

Data Analysis:
These raw symbol recordings were converted to percentages (raw symbol result/50 = calculated percentage of symbol rolled) for each dice. From the calculated percentages an average and standard deviation was conducted across each dice type and set (e.g. [Nerd-X Clear 1 + Nerd-X Clear 2 + Nerd-X Clear 3]/3).

(add up dice tally, use math, get percentage)

Click for Larger Image

Statistical Analysis:
The average calculated percentages of each dice were compared to one another using a Pearson's Chi-Squared test to determine if the proportion of one dice type is statistically different from another dice type (e.g. is Nerd-X Clear's average calculated percentage different than FFG's average calculated percentage?). In addition, each average calculated percentage was compared to the theoretical true dice probability (Table 1).

(more math, but are dice different or not = yes, FFG Dice App is different)

Results (brace yourself as this is where it gets real!!!):

Evade Dice Results
The probability of rolling an evade result on the FFG Dice App (52%) was significantly higher than true probability (37.5%) and all other dice percentage outcomes (Table 1). In addition, the probability of rolling a blank result on the FFG Dice App (26%) was significantly lower than true probability (37.5%) and all other dice percentage outcomes (Table 1). No significant difference was observed between true probability and all dice percentages for the focus dice result.

Attack Dice Results
The probability of rolling a critical result on the FFG Dice App (24%) was significantly higher than true probability (12.5%) and all other dice percentage outcomes (Table 1) (yes you heard me correctly - start panicking). In addition, the probability of rolling a blank result on the FFG Dice App (14%) was significantly lower than true probability (25%) and the Nerd-X White dice (28%) (Table 1). No significant difference was observed between true probability and all dice percentages for the blank dice result.

(seriously, just look at the table or better yet just go do something else)

The Nerd-X dice (clear and white) and FFG dice are not different than one another or true dice probabilities. However, the FFG Dice App was different on numerous results (higher probability for evade, and critical hit; lower probability of rolling blank results on both evade and attack dice) (See Figures 1-7) when compared to the Nerd-X dice (clear and white), FFG dice, and true dice probabilities.

(damn it man, just look at the bar charts)

(Jeb Here: Isn't it funny that the quote for the dice app ad is: "In my experience, there’s no such thing as luck." – Obi-Wan Kenobi. It's like they know something we don't.  See it here.)

Figure 1: Evade Dice Blank Results: Click for larger version

Figure 2: Evade Dice Focus Results: Click for larger version
Figure 2: Evade Dice Focus Results: Click for larger version

Figure 3: Evade Dice Evades Results: Hey! Did you know you can click for a larger version?
Figure 4: Attack Dice Hit Results: Not sure if you know this? But you can click for a larger version

Figure 5: Attack Dice Critical Hit Results: Use the mouse Luke and click for a larger version.

Figure 6: Attack Dice Focus Results: Larger version you like.  Click your mouse you will

Figure 7: Attack Dice Blank Results: Wow! Death by charts.  If you haven't figure it out by now, you can click here and the image gets bigger.

Overall, I have arrived at the following conclusion: the FFG Dice App is SkyNet.

But seriously, these results are a bit shocking. In fact, I was so shocked that I rolled the FFG Dice App for a second time on a second device. I ended up with the same results with some minor changes (slight reduction in evade rolls down to 51% but still rather high). With this knowledge, I looked back at all of those games that I have played people who used the FFG Dice App (cough cough Thomas). They always seemed rather lucky to roll 3 natural evades (no modifiers required to alter results) at range one of my Fat Han or blast me with 3 crits at range three of my Poe. Now it seems like the science suggests (never proves, as nothing is perfect in science) that the FFG Dice App might be a bit biased and off. Alas, this is only one experiment.

For you programmers, how might one make a Dice App biased? Shouldn't it roll exact probabilities? Am I missing something? (Jeb here: I want to see the source code of it, anyone have access?)

One positive finding of this little research study is that the Nerd-X dice are not different than true probability or standard FFG dice. In fact, they even seem closer to true probability than standard FFG dice. Props again for making high quality dice that are fancy and accurate.

To be fair, it is possible when rolling these dice that we happened to get lucky from time to time. However, this shouldn't sway the results this drastically. There could be human error involved such as different roll strategies or error from the rolling surface could influence the results. Yet we tested this assumption and rolled dice on the FFG mats and a standard wooden table and didn't see any noticeable changes.

In the end, it is clear that if my opponent pulls out the FFG Dice App I will violently shake my fists in rage, flip tables, and fart in their general direction. But seriously, I will either ask them to roll other dice or pull out my own FFG Dice App and rain down my hatred of 10,000 years on them in the form of Critical Hits and Evades for days.

Think we are wrong? Let us know! We welcome your observations and data on this. In fact, if you are interested in adding to our sample size please repeat our experiment and send us your results. I am very curious to see what everyone comes up with.

On a random side note - FFG if you are listening you had better give us a cut from the millions of dollars you will receive from X-Wingers who will download the FFG Dice App after this article breaks the internet.

With that, I say goodbye, goodbye, and good riddance to bad luck.