The NFL Is Changing Faster Than Win Probability Models Can Understand

Somewhere along the line, when laughing at a team’s failures, it stopped being enough to merely estimate how embarrassing a loss had been. We had to know, precisely, how badly they’d screwed up. So, luckily, data-crunching nerds developed models to tell us exactly how likely teams are to win at any point in any game, much to the chagrin of Atlanta Falcons fans.

Win probability models are useful for a variety of reasons; NFL teams can consult them to determine whether or not to make certain decisions, like whether punting or going for it on fourth down is more likely to increase their chances of winning. Sportsbooks use them to set live betting odds. (NFL teams and sportsbooks won’t be sharing their internal win probability models with the public anytime soon.) But win probability charts are also useful for generating retweets when teams blow huge leads. Like when the Falcons blew a 28-3 lead in Super Bowl LI.

I think the 99-plus percent win probability for the Falcons in that game was fair. To date, no team, in any game, has ever held a lead that big, that late, and lost. But this year, the win probability meter has again haunted Atlanta. In Week 2, the Falcons supposedly had a 99.9 percent chance to beat the Cowboys before they stared at this onside kick:

And in Week 3, they supposedly had a 99.6 chance of beating the Bears before they allowed 20 points in the game’s final eight minutes. And in Week 7, they supposedly had a 98.7 percent probability of beating the Lions before Todd Gurley’s infamous accidental touchdown:

The probabilities above are ESPN’s numbers, but in case you were wondering if the Worldwide Leader’s formula just sucked, well, the Pro-Football-Reference win probability model gave the Falcons a 99.9 percent peak win probability against the Cowboys, and a 99.8 percent peak win probability against the Bears.

ESPN helpfully tweeted that the Falcons had been responsible for three of the NFL’s seven losses this season in which the losing team had a 98 percent win probability or higher. A little embarrassing for the Falcons, sure—but still revealing that four other teams had also lost games with a 98 percent win probability.

After seeing this tweet, I started to become skeptical of the win probability meter. A team that has a 98 percent chance to win should lose about 1-in-50 times. There are 256 NFL games per season, but only a relatively small portion of them allow for such a comeback, so this scenario shouldn’t occur often. Even if every game allowed for a 98-percenter, you would usually see only about five a year. But at the time of that ESPN tweet, it had already happened seven times. (There has since been an eighth 98-percenter—the Chargers had a 98.7 percent chance of beating the Broncos in Week 8.)

It seems clear that the win probability models are highly underestimating the likelihood of comebacks. But I don’t think it’s because the people who made them are bad at math. I think it’s because this year’s NFL is different than it’s ever been.

Everything about the NFL has gone off the rails this season. Teams are averaging 25.3 points per game, demolishing a 72-year-old record of 23.6 points per game set back in 1948. (I wanna grind some film to see if we can learn something from offenses from 1948; unfortunately, the All-22 back then sucked.) The average team is picking up 362.9 yards per game; that’s an all-time high. The average play is going for 5.6 yards; that’s an all-time high. The average team is committing 1.3 turnovers per game, that’s the lowest since 1932. The average drive is lasting 6.17 plays; that’s an all-time high. The average team is scoring on 41.4 percent of drives; that’s an all-time high. The average team is getting 2.1 first downs by penalty per game; that’s tied for the all-time high. The average quarterback is completing 65.8 percent of passes; that’s an all-time high. The average quarterback is throwing an interception on just 2.2 percent of passes; that’s an all-time low. The average team is fumbling just 1.1 times per game; that’s an all-time low. The average team is punting 3.5 times a game; that’s an all-time low.

And some of this season’s stats are not NFL records, but most are close. Kickers are making 84.6 percent of field goals, the third-highest rate of all time. Quarterbacks are getting sacked on just 5.9 percent of dropbacks, tied for the second-lowest rate of all time. And quarterbacks are averaging 7.4 yards per attempt, the fifth most of all time, but also the highest mark since the 1960s.

It is impossible to identify any one thing that has led to all this. It’s not just that passers are getting better, or that defenses are less effective, or that coaches are making smarter fourth-down decisions, or that referees are calling holding less often. It’s that passers are getting better while defenses are getting less effective and coaches are making smarter fourth-down decisions and referees are calling holding less often, plus a thousand other things. It’s like the person controlling the NFL simply turned the “POINTS” dial on their dashboard all the way up.

In 2017, The Ringer’s Kevin Clark talked to model makers about how changes in the league affected their probability models. ESPN’s Brian Burke pointed out that the average net yards per passing attempt had jumped from 6.17 in 2010 to 6.37 in 2016, which might seem insignificant but was actually “enormous” from a probability perspective. Well, now it’s at 6.54—the highest mark in NFL history.

The main variable in any win probability model is how likely teams are to score—and that likelihood has increased dramatically. This is screwing with win probabilities on both ends of the scale. When every possession is more likely to end in points, teams are more likely to build big leads, and more likely to blow leads previously considered safe. Big leads used to be reserved for dominant squads, but now that team up 17-0 might suck, and that team trailing 0-17 might actually be pretty good.

These win probability models actually worked pretty well for the old NFL. But now, unprecedented things are happening. Win probability models said that there was a .0007 percent chance of the Falcons blowing those leads in back-to-back weeks. But guess what! No team had ever done that before! In fact, no team had ever blown 15-point fourth-quarter leads twice in the same season!

When win probabilities tell you that something is impossible, it’s because they were crafted to reflect a league in which certain things never happened. But the NFL has gone haywire, and the points meter has been cranked to 11. It’s a different world, and the models aren’t quite set to deal with it.