Originally posted on Variance Hammer
A few days ago, Bell of Lost Souls published an article with the following title: Anatomy of a 40K Disaster: USA vs Europe ETC Analysis.
My first though reading the title was “Oh my god, what happened to the poor Americans?” Actually reading the article, we find out the answer is: nothing, nothing at all. The fundamental assumption of the article is that the USA’s showing in the ETC (finishing 9th in a field of 28) is some sort of unmitigated fiasco that says all kinds of important things about the state of tournament play in the United States.
In truth, it doesn’t say anything of the sort. Nor is Team USA’s performance anything even close to a “disaster”.
This article, like many that are written about international contests like the Olympics, begins with the implied assertion that the USA is somehow favored to win. Perhaps it’s because I grew up watching World Cup Ski Racing in an era where this was absolutely not true, but it always irks me when “The USA didn’t dominate the field” is immediately recast as something having gone horribly, horribly wrong.
Let me get this out of my system right now: Even if their placement was entirely their fault and did say something about how American tournament players play 40K, being among the top 1/3rd of all competitive 40K players in the world is kind of a big deal.
The author goes on to make some really sweeping and badly supported assertions about how the the Continental vs. American tournament players “play”. Americans, apparently, love them some cheese, while the Europeans manage a sublime virtuousness of being tactically flexible, maintaining the background of the game, and being in all ways Just and Good.
As evidence of this, he points to a European list with three Dreadknights, Draigo, Tigurius, and a Centurion squad. That list is many things, but some sort of zen-perfect 40K list that an American would never ever come up with is not among them.
There is also the question of what happened to the 2/3rds of teams the Americans did beat. How did their Euro-Zen builds get defeated by our blind and inept fumbling? Some commenters claim that the difference is that the Americans rely on MathHammer and theory exercises while the the Europeans actually play games, and this is clearly the difference. I have three thoughts on that:
- [Citation Needed]
- Small N simulation (what the Europeans are allegedly doing via gameplay) vs. deterministic, “On average I should…” MathHammer are both the wrong way to look at 40K as a system (in my humble opinion), so arguing about which wrong approach your using gets a little silly. What you should be doing is Large N simulations informed by play experience instead, but this is admittedly a very biased perspective.
- Again, Team USA beat the majority of the field. Where are you played games now, most of Europe?
So – if this doesn’t actually suggest that Europeans are endowed with some sort of ancient list-building arts, presumably passed down to them before 1776, nor that they totally just play more games than those lazy Americans, what do these results say?
That random systems are random. Or, for the more technical: Stochastic systems are stochastic.
Team USA’s tournament trajectory went like this: A strong series of opening games, a key loss to an opponent in a highly influential game, and then a middling-good final placement. If we view this as but one of a single sample of an infinite number of potential ETC 2015 tournaments held in some sort of hellish wargamer version of Groundhog Day, we find that this isn’t a particularly unlikely result. Indeed, in a balanced tournament scene where most teams have a ~50% chance of winning each matchup, this is an extremely likely outcome. To have done otherwise is exceptional (well played Germany, btw). If I flip a coin 6 times, and don’t get all heads, this does not mean I’m terrible at flipping coins. Nor, if I get 6 heads, does this mean that I am some sort of Ninja Coin Flipping Master.
So lets actually simulate those other possible tournaments. I’m going to do this in a slightly odd way, because I don’t actually want to fully simulate the ETC. I’ve thought about how I would do it, and it falls under the heading of “Maybe someday when I have an abundance of free time”. Instead, we’re going to treat Team USA’s result as a single draw of the round-by-round tournament results, and then use a procedure known as bootstrapping to simulate other random results that could have happened. This is somewhat handy for a number of reasons:
- It uses only the data from the ETC, so inherently the simulation cannot produce impossible results.
- This data is inherently balanced. Rather than having to simulate winners and losers, and the margins by which they win and lose, we can simply pull these from the data.
- All the ETC team’s trajectories can be reproduced using this particular system. The simulation that grabs all of Germany’s results? That’s the Germans.
- It’s way, way faster to code.
This does have some drawbacks to it – you can’t simulate things outside what happened, so someone having a streak of games better than anyone actually did at the tournament is impossible in this particular simulation. It’s not ultra-high fidelity, so you can’t do things like put in particular players with individual win percentages, etc. Maybe someday – support our Patreon drive if you want things like that to occur.
Anyway, onto the simulation.
First, lets look at the data we’re drawing from:
There’s a clear bimodal-ness to the distribution, which is what’s to be expected – the winners are scoring ~100 points, while the losers for each round score ~60, with both extremely good rounds and extremely bad rounds as unusual but not impossible outcomes. Broken down by Wins (Purple) vs. Losses (Blue), the distribution looks like this:
Now, let’s sample from that distribution to get a bunch of simulated team outcomes:
The purple histogram is the distribution of those 10,000 draws – essentially, reaching into the round results for the ETC and picking six out of the hat for their results. The black line is the mean score of 478.79. The red line the American result, the yellow line the Swedish result (the team we played to determine who got 2nd) and the orange line the German result that won the tournament.
The conclusions we can draw from this? First, again, well done Germany. That performance really is impressive. Now whether or not its possible to do better than the Germans is again something this simulation can’t touch on because the truly profoundly good games that were better than anyone actually did don’t arise in this simulation, but it is possible to draw the best points result from each round, and even then, you’re not doing much better than the Germans did (the maximum result in the simulation was 722, only 21 points better than the Germans actually did).
The second is that Team USA both performed better than average assuming the tournament scene is balanced, and despite placing in the middle of the pack, when they went in against Sweden, they really were playing for 2nd place. The difference between Team USA and Team Sweden was 62 points – so if we swapped that single result around, the Americans and the Swedes essentially swap places.
The third is the importance of defensive play and going for all the points you can every game. Unlike a tournament where results are determined simply by “Team X wins and advances to the next round” ala March Madness or any bracket-based system, or scoring systems where a loss is catastrophic in terms of earning points, mitigating the impact of losing by avoiding those particular dire loss results is a potentially important strategy. While it won’t win you the tournament, making sure your losses are hard-fought, close matches is a good way for a team to scoop up a decent showing while the teams contending for first place hopefully savage each other. For example, lets compare Team Wales (which went 3-3 and was, entering round for, 2-2 compared to the undefeated Team USA) with Team USA.
Despite Team USA’s better record, the combination of an early, big win for Wales, combined with a series of less serious losses (an average loss score of 73 vs. 59) meant they were never really “behind” the USA by a meaningful amount, even when they lost to Team USA in the 4th Round, and were thus able to secure a 4th place finish overall.
Enjoy what you read? Enjoyed that it was ad free? Both of those things are courtesy of our generous Patreon supporters. If you’d like more quantitatively driven thoughts on 40K and miniatures wargaming, and a hand in deciding what we cover, please consider joining them.
This is a great article, and I love how much thought was put into it.
Excellent post, well played sir!
I love how math and science basically bludgeon sensationalism to death.
Your only mistake was reading an article written by BBF to begin with. 😉
This. Oh god, so much this. BBF is a terrible writer and an even worse analyst and barely understands the game system he plays. You can practically play Mental Error Bingo while reading through his stuff.
One of the greatest things I like most about Reece is that he tries his best to be unbiased, have an open mind, and give everything the old college try.
I would expect nothing less of him to respond to such an article with nothing but the utmost professionalism and sincerity, regardless of the author’s reputation.
Ha, it looks like I read the wrong author section when I noticed Reece’s upload tag.
Looks like I’ve found a new writer to follow though.
Though in fairness, Reece only posted it. I wrote it 😉
I’d like to think the comment holds for myself as well – the issue I took with what was written wasn’t based on the author (honestly, I rarely even bother to look at by-lines) but with the reasoning within.
I love bbf his tyranids tactics articles on hols are amazing. Its the comments that really make his articles great coz the majority are people insulting him for getting basic stuff wrong lol
Thanks for the kind comments all – make sure to check in at http://www.variancehammer.com if you’d like to read more.
And yes, my first mistake was reading the article in the first place.
“Now whether or not its possible to do better than the Germans…”
Of course that’s not possible 😉
the analysis is clear.
very interesting.
Man its been so long since I’ve been to BoLS and as always… nothing’s changed. Super charged hyperbole and next to no content… and no context either. He charges the team USA with all these gaming “crimes” and then fails to even give a single example of the type of list they were bringing or even their overall record of wins and losses. BoLS is one of the saddest hubs of 40k culture on the net… and the wellspring of so much discontent.
BoLS is basically the Fox News of wargaming.
Really? BoLS never occurred to me fair and balanced. 😉
Anyway, the worst thing you can do with that superficial and inconsistent BoLS article is to pay too much attention. The author purposefully came to twisted conclusions just to make people upset and comment.
Usually there are between 25 and 30 Teams at the ETC, 6 Rounds are played. Each of the 8 players can get a maximum of 20 game points per round. If a team makes 75 to 85 points, that round is considered to be a draw and both teams get 1 point. A team with more than 85 game points gets 2 round points. The American Team ended up with 7 out of 12 points with two really close results (85 points vs Italy and 73 Game Points vs Sweden, which is the third successful Team in the ETC 40k-record after Germany and Poland)
During the last few years, Team America used to have 7 or 8 points every year. With the exception of 2012, they are consistently above average and usually end up in the first 30% or so.
In 2010, first place had 10 points, 2nd and 3rd 9 points, 4th to 7th had 8 Points, 8th to 10th had 7 points.
Year 2011: 1st/2nd 10 points, 3rd 9 points (USA by the way), 4th to 7th 8 points, 8th to 11th 7 points.
Year 2012: 1st 11 points, 2nd 10 points, 3rd/4th 9 points, 5th/6th 8 points, 7th to 13th 7 points.
Year 2013: 1st and 2nd 10 points, 3rd 9 points, 4th/5th 8 points, 6th to 12th 7 points.
Year 2014: First 12 points, 2nd 10 points, 3rd 9 points, 4th to 7th 8 points, 9th to 12th 7 points.
Year 2015 1st 12 points, 2nd 9 points, 3rd to 7th 8 points, 8th to 11th 7 points.
What are his solutions for this sudden “disaster”? To exchange players and make them play lists in compliance with some obscure philosophy instead of those they figured out in a process of playtesting? As far as I know, almost half of the US players qualified based on their good results during the last ETC. There is no problem with individual skill. I have an opinion about what might help to get the USA into the Top 3. But there are just 8 players in this world who can really figure out how the missing 2 or 3 points could have been achieved this year.
The ETC would not be the same without teams from overseas, and the Americans where the only ones who held up the flag in 40k this year. It is amazing how much time, effort and money they spent to participate. And what do they get in return? They get a (sometimes) quite badly managed event. They play a strange international mix of mission design and rule interpretations. They probably have a lot of tense games due to language barriers combined with about 30 different understandings of the word “fairness”. Please, treat those nice guys a bit better at home. They deserve it. Congratulations to first place in the FOW-ETC by the way.
Really? BoLS never occurred to me fair and balanced. 😉
Anyway, the worst thing you can do with that superficial and inconsistent BoLS article is to pay too much attention. The author purposefully came to twisted conclusions just to make people upset and comment.
Usually there are between 25 and 30 Teams at the ETC, 6 Rounds are played. Each of the 8 players can get a maximum of 20 game points per round. If a team makes 75 to 85 points, that round is considered to be a draw and both teams get 1 point. A team with more than 85 game points gets 2 round points. The American Team ended up with 7 out of 12 points with two really close results (85 points vs Italy and 73 Game Points vs Sweden, which is the third successful Team in the ETC 40k-record after Germany and Poland)
During the last few years, Team America used to have 7 or 8 points every year. With the exception of 2012, they are consistently above average and usually end up in the first 30% or so.
In 2010, first place had 10 points, 2nd and 3rd 9 points, 4th to 7th had 8 Points, 8th to 10th had 7 points.
Year 2011: 1st/2nd 10 points, 3rd 9 points (USA by the way), 4th to 7th 8 points, 8th to 11th 7 points.
Year 2012: 1st 11 points, 2nd 10 points, 3rd/4th 9 points, 5th/6th 8 points, 7th to 13th 7 points.
Year 2013: 1st and 2nd 10 points, 3rd 9 points, 4th/5th 8 points, 6th to 12th 7 points.
Year 2014: First 12 points, 2nd 10 points, 3rd 9 points, 4th to 7th 8 points, 9th to 12th 7 points.
Year 2015 1st 12 points, 2nd 9 points, 3rd to 7th 8 points, 8th to 11th 7 points.
What are his solutions for this sudden “disaster”? To exchange players and make them play lists in compliance with some obscure philosophy instead of those they figured out in a process of playtesting? As far as I know, almost half of the US players qualified based on their good results during the last ETC. There is no problem with individual skill. I have an opinion about what might help to get the USA into the Top 3. But there are just 8 players in this world who can really figure out how the missing 2 or 3 points could have been achieved this year.
The ETC would not be the same without teams from overseas, and the Americans where the only ones who held up the flag in 40k this year. It is amazing how much time, effort and money they spent to participate. And what do they get in return? They get a (sometimes) quite badly managed event. They play a strange international mix of mission design and rule interpretations. They probably have a lot of tense games due to language barriers combined with about 30 different understandings of the word “fairness”. Please, treat those nice guys a bit better at home. They deserve it. Congratulations to first place in the FOW-ETC by the way.
Well said, sir.
So…. What was the conclusion? Because it just seems to end without having/drawing (either that or 5am is to early to be reading an article like this 😛
Very good article, thank you!
I was very, very annoyed when I read through the BoLS article, presenting the American performance at the ETC as awful, because they clearly performed well and only ended up relatively low due to how the system works, and the whole cultural speculations seems like total nonsense to me.
Furthermore, even if America had actually ended up really low, so what? My county (Norway) ended up in last place (but I think Greece lost some points and ended up below them for some reason), but the guys I’ve spoken to who went had a great time. They got to play their hobby, meet new people and visit a beautiful city, and the experience of those 8-9 people is what really matters in this event, not some clickbaiting blogger’s nonsensical speculations and attempts at sensationalism.
In regards to BBF et al, AbusePuppy already wrote a great article about how people look to 40k to accomplish different things. BBF wants to be competitive, but within a strict confines of what he views the game should look like. People like AbusePuppy don’t care about those arbitrary confines and play the rules of the game.
To me, it’s similar to sports like basketball where the analytics movement has changed pre-conceived notions. Not sure how much cross over I’ll see here between basketball fans and 40k, but the metaphor is strong if you care about such things. AP plays like the Houston Rockets; finding the most optimal shots and selecting only those to the near absence of unoptimal elements (3s and layups only, no midrange!). He gets side eyes from people who look at the game from a historic stand point and think his style will eventually fail because it lacks balance (like someone advocating more post ups or isos). What’s funny is that even coaches who succeed by using analytics dislike the aesthetics of that style (Coack Pop for instance hates 3 pointers, even though he’s built a team that runs on them).
Anyway, I’m rambling…
Great work, VH!
Another great article.
I’m playing heresy, where the games are nearly always team based, and I’m really starting to like the requirements of having players in your team that are defense specialists, to blunt your opponent’s unbeatable list!
What’s missing from this straight probability style analysis is the role of skill or ability. If you believe that 40k is a game of skill and chance then skill has an outcome on results. If you look at sports as analogy, countries, districts, etc with the largest pool of potential players always outperform smaller ones. E.g. there is no way Wales will compete in Olympic sports with the US. Not because the Welsh are physically weaker but because the population is lower.
Based on this analysis larger countries should always enjoy an advantage in games of skill or partial skill. Like for instance the olympics. Now investment in selection can mitigate population to a certain extent but never truly overcome it.
This is a long way of saying that if you assume any countries’ odds of success is equal, the 9th is a fine ranking, but given the USA’s larger affluent population then 9th is not such a great result after-all.
That said based on the above I think the only salient point is that the selection process for the team should be scrutinized. Second guessing list strategy or play style is silly.
I think the selection process- as well as the admission process- is the reason you see typically midrange performances from the USA. A larger population translates to to better overall results ONLY if the team is able to take advantage of the superior pool of available players. As the US has a haphazard (at best) system of recruiting and significant barriers to entry (such as the cost of the plane ticket and stay over), many of the better American players simply can’t or won’t join the team.
To place things in context: no European player has ever won NOVA, Adepticon, or any of our other “big” events (at least not to the best of my knowledge) for many of the same reasons. The self-selection process of the costs involved- time, money, and effort, not to mention learning a new format- place a much higher burden on any players traveling long distances to the event, hence the absence of high-level performers.
As was put on the BOLS Website, one of my main issues with his article was the fact that he calls the qualification system a “good ol boy system”. This is a downright lie, and unfortunately I think its propagating. The qualification system for the US team was completely redesigned over a year ago.
There is no longer:
-Veteran requal from the event
-Qualifications for winning single events
-Straight up captains choices (except in the case of drops)
For more information on what the actual US qualification system is, please read the articles that have been posted, some even here at frontline!
https://frontlinegaming.org/2015/04/07/team-america-qualification-system-updates-for-2015-2016-season/
http://www.torrentoffire.com/6397/team-america-40k-2015-qualification-circuit-summary
http://www.torrentoffire.com/6856/team-america-qualification-system-updates-for-2015-2016-season
http://www.torrentoffire.com/6092/team-america-qualification-circuit-update
Normally we have the documents posted on the Team America website as well, but unfortunately we’re having some issues with the page right now and it’s down. We’re working to get that back up
I’m aware there has been some significant overhauls to thinks- which I approve of, in most respects- but that still leaves a LOT of barriers for anyone wanting to join the team. And, as I already said, one of the largest of these is simply money- flying to Europe and staying there for a week isn’t cheap, even by 40K standards. This isn’t Team America’s fault in any way, it’s simply a reality of the physical location of the tournament, but it’s still something that should be acknowledged when talking about who can go to ETC and why.
My thoughts as well
One of the limitations to something like this is, indeed, that you can’t include slight biases in favor of one side or the other, like saying “Well, you have a 65% chance of winning each match”.
That being said, I’m not convinced that’s necessarily a thing. There’s a certain point, there’s no proportional increase of, in this case, 40K skill, for an increase in the population. To draw the parallel back to Olympic sports – the U.S., on a population basis alone, should dominate alpine ski racing. They don’t. Tiny little countries like Austria and Switzerland pose major competitive threats to the U.S., despite their smaller population size. Similarly, all you need is a double digit number of very good 40K players to compete. Just shoving more people into the pipeline doesn’t give you better results. To use another analogy, while moving from a town with 5,000 people to one with 10,000 people might increase the number of dates you go on, moving from a city of 5,000,000 to one with 10,000,000 might not, because as some point you just get to “I found someone I like”. The only time that that might not be true is if there are true 40k virtuosos – *and* that your selection system can find them.
Given how tightly the scores between 9th and 2nd were, I’d argue that there *is* evidence that the field is pretty even
In sport there is also the issue of genetic predisposition and overall country wide focus. It’s why Jamaica is able to excel at sprinting and several African nations excel at marathon running.
Genetics is of course a tricky subject to get into, but in some ways it’s obvious- Even with a population much larger than the US, China and India will have a lower pool of players to choose from in sports like basketball, where your genetics (primarily height) have a disproportionate impact.
Comparing that to 40K, obviously there is no genetic bias, but there can be a population bias. If a higher percentage of the overall country plays 40K (or are tournament players), the pool can be much bigger than what the US offers, or at least comparable.
I think the biggest problem for Team America lies in preparations. The players are very spread out and have a hard time getting together for training sessions. Moreover, most US tournament missions and scoring systems seems to differ quite a lot from the ones at ETC. It can take some time to get used to the difference between a win-loss-draw system and a 20-0 system.
I have been coach for the Swedish team three times and have watched team America play a lot. My observations tells me that there is absolutely no problem whatsoever with the player skill of the ‘murican boys. Instead, contradictory to the strange BOLS article, I think that your armies actually lack some “cheese” ;), i.e. they are a bit too balanced for the ETC system where single-purpose, hammer lists actually works pretty well.
My standpoint is that good preparation for the ETC has to be gained through specific training for the ETC, playing the actual missions, playing against lists you are supposed to face and so on. I guess this is more difficult to achieve for team US compared to most other European teams where players has it much easier to train together.
This is why I love Frontline Gaming! What a great, thoughtful, article. I’d be interested in taking the statistical analysis one level deeper and looking at the individual tendencies of the codecies, match ups and the players. I suspect that you will find a tendency for rock/paper/scissors in the match ups,and for for individuals to rise to the top, just like any sports team. But that’s just a hypothesis waiting to be tested. Thanks!
It’s a hypothesis I’ve been considering looking at, especially for tournaments as a whole rather than a single tournament, but I also have a day job :\
This article made my night 🙂 Bravo!
Hi
I fully agree with article and made the following response to the orginal article on Terminus Est (edited):
I was present at this year’s ETC. I think the American team was maybe the strongest
ever and maybe the strongest performance to date, considering who they meet. I
therefore would be hesitant to change the selection process (eg in Denmark we
base our selection on a team national playoff, which also have it’s drawbacks).
Regarding the American performance:
– Round 1: Won over Spain
– Round 2: Drew Italy (missed 1 BP for a win)
– Round 3: Won over England who came 6th at ETC 2015
– Round 4: Won over Wales who came 4th at ETC 2015
– Round 5: Lost to Germany the winner of ETC 2015
– Round 6: Just lost (by 2 BP) to Sweden who came second (if had won US would have been 2nd, if draw would have been 3rd).
Very tough opposition in all 6 rounds. Maybe only Germany faced as tough opponents during the tournament. Especialy England and Wales are two very good opponents to win over.
Overall I would say really good performance and only small margins prevented a higher placement. Also the team seemed perform really well together.
*I fully agree with the article on Variance Hammer:)
How 9th out of 28 is bad?is BFF assume the rest of the field stupid that team usa should be like god beating them? How ending up at top 1/3 not deserve a pat on the back and say job well done?