How does the Fantasy Judge score the sites that he covers?

Note: a more detailed description will be published in the future. In simple terms, the Judge awards points depending on how well the sites predicted the performances of each week's "case file." The goal, from a fantasy owner's perspective, is to find a site he or she can trust to give him advance notice of a player who is setting up to excel, or who is likely to disappoint.

What about statistically-based accuracy rankings?

According to the most prominent rating site, FantasyPros, their methodology involves scoring a minimum of 20 QBs, 40 RBs and 50 WRs every week. That doesn't match the experience of a fantasy owner who likely has a maximum of 4, maybe 5 RBs on his roster to choose between. If an expert, while ranking 50 WRs, views two of them as essentially equal, the difference between ranking them, say #14 & #15, or #15 & #14, is ultimately negligible when it comes to that week's overall accuracy score. But for a fantasy owner, one of those players will be in his/her lineup earning points while the other is on the bench contributing nothing. Virtually every fantasy owner has experienced the horrible feeling of watching a player he or she benched have his best game of the season, getting zero out of it, and losing the matchup. When an expert ranks 40 or 50 players, one mistake can't upset the whole week.

Another problem with existing methodologies is the accuracy gauge is linear, which again doesn't match that of the fantasy owner. For example, when choosing which QB to start, a ranking of 2nd or 7th is not really all that different, as both are still in the elite, must-start status. But there is a huge difference between a site ranking a specific QB 7th and another site ranking him 12th or 13th -- now he's very much outside "must-start" status and there is a good chance the fantasy owner will have a higher-ranked QB option. Between 2nd and 7th, not too likely it will change the start/sit decision. But 8th to 13th, now there's a big chace it will affect the owner's decision of who to start. This effect doesn't come into play at all in linear accuracy measures.

The week's case is irrelevant to me, the highlighted player isn't even on my team!

Well, if you're in a 10-team league, you own at most 10% of all the players available. The vast majority of fantasy content is going to be about players who aren't on your team or matchups that don't affect the players on your roster. Many weeks will be won or lost regardless of any one specific decision. The key to a championship, as opposed to just simply a good year, often comes down to one or two critical moments. From the "Immaculate Reception" to the "Tuck Rule" to David Tyree's catch or Malcolm Butler's interception, these single plays decided championships. In fantasy football, one or two incorrect decisions – a wasted draft pick, an incorrect start/sit, a missed waiver opportunity – can have the same effect, and the Judge's goal is to reward the experts who are most likely to steer you towards the correct side of critical, championship-winning decisions.

OK, but I still disagree with your choice of cases. There were more important players than the one you picked.

Most likely we'll have to agree to disagree. The NFL has a lot of story lines affecting almost every team, the Judge simply tries to pick one or two each week that had an outsized impact from a fantasy perspective. Keep in mind writing about the elite players is rarely the key story, it's not a surprise if Zeke Elliott or Patrick Mahomes or Julio Jones has a huge day. And it's very rare that an owner would question whether to start the players at this level, which means there isn't much of a case to be made. During the season, if one analyst ranks Alvin Kamara as his top RB and another ranks him 4th, these are both must-start rankings and there isn't much point penalizing the lower-ranked analyst.