Evaluating FiveThirtyEight's Predictions

Before the World Cup started, FiveThirtyEight posted predictions of each team's chance to win the Cup, finish first or second, and even percentages to win each game. Some of their odds were reviewed in the comments as obvious, others seemed to be off. I knew from their NCAA tournament that they would update the odds as they went along, which is very helpful. And yet it erases their pre-tournament predictions, something that important for judging their model in the first place. 

The trick for evaluating their odds is obvious. If you say that Germany has a 69% chance of beating Ghana and a 20% chance of tying them, and then they tie...is your model correct and this was a 1 in 5 scenario...or did the model undervalue the chance of a draw? With small sample sizes, there really isn't a good answer. The best I can do is to measure how often they were accurate and make a subjective evaluation if that seems within a margin of error, considering the sample size. 

I captured screenshots of Group G (because Group of Death and USA) and Group D (because they predicted England would advance while the comments seemed certain that Italy would advance). 

Germany took first place. They were given a 69% chance of doing so.
USA took second place. They were given a 25% chance of doing so.
Portugal was eliminated. 55% chance of that.
Ghana was eliminated 70% chance of that.

This group is interesting in that the three non-Germany teams each had the most likely odds of not advancing. (This is always an interesting conundrum. See also: That the favorite to win the NCAA tournament only has a 14% chance of doing so.) Basically, whoever got 2nd place was going to be unlikely, and yet we know it had to be someone.

Let's look at the individual matches.

USA-Ghana. Highest odds were a USA win at 38%. Correct. (Again, the USA win should only happen, according to the model, less than 4 out of 10, but it was the model's most likely outcome.)
Germany-Portugal. Highest was Germany win at 63%. Correct.
USA-Portugal. Highest odds was a Portugal win at 39%. Incorrect. Draw was 29%.
Germany-Ghana. Highest odds was a Germany win at 39%. Incorrect. Draw was 20%.
USA-Germany. Highest odds was a Germany win at 67%. Correct.
Portugal-Ghana. Highest odds was a Portugal win at 41%. Correct.

4 out of 6 correct in this group.

Costa Rica took 1st place. 9% chance of that.
Uruguay took 2nd. 28% chance of that.
Italy was eliminated. 53% chance of that.
England was eliminated. 36% chance of that.

FiveThirtyEight was really down on Costa Rica. It seems like this was a genuine surprise run. And 30% chance of advancing seems to reflect a puncher's chance.

Without typing out all the matches, 538 was only assigned 2 of the 6 matches the highest odds correctly.

My net conclusion is that 538 has reasonable odds predictions, but aren't really any better than just a soccer fan's common sense. They claim to offer specific percentages but they don't really mean anything.