Numlock Awards Supplement: How To Predict The Oscars

The Numlock Awards Supplement is your one-stop awards season update. You’ll get two editions per week, one from Michael Domanico breaking down an individual Oscar contender or campaigner and taking you behind the storylines, and the other from Walt Hickey looking at the numerical analysis of the Oscars and the quest to predict them. Look for it in your inbox on Saturday and Sunday mornings. Today’s edition comes from Walter.

I've spent the past five years of my career fascinated by the Academy Awards.  I think that the Oscars are important. I think that recognizing cultural achievement matters. I think that for its many, many flaws, the fact that the Oscar race compels large portions of the country to spend months debating the merits and faults of artistic work makes us better people. I think that we'd be far worse culturally without them, and that if commerce were the sole incentive to produce art, your culture would be determined by the lowest bidder. 

But mainly I like predicting them. I think it's hard, and it's fun, and it helps me understand models more every time. 

The Academy Awards present a unique set of problems when it comes to our ability to predict the outcome. 

If you want to forecast an election, you know for certain who is registered to vote there, and you can even call large groups of those people on the telephone and ask them who they plan to vote for. Not so for the Oscars: besides the annual announcement of who's been invited to join the Academy, we don't know who votes on the Oscars. 

If you want to forecast a hurricane, you can look at current patterns and compare them to detailed analysis of what happened in previous situations like this one, and you can rely on detailed and transparent records maintained for that very purpose. Not so with the Oscars: we'll never know vote counts, individual ballot rankings, which film was the runner-up, or how many shifted ballots it took to crown Best Picture. 

If you want to predict the Super Bowl, you have at least 18 previous competitions that you can analyze, 18 other simulations where you can ascertain the strengths and weaknesses of the competitors, and can do so on a play-by-play basis under essentially identical field conditions. When you talk about the Oscars, the precursor awards are all different constituencies or different nominee pools and different voters. It's like predicting the Super Bowl based on the outcome of a friendly rugby scrimmage. This will be the final sports analogy of this newsletter. 

Anyway this problem is hard. That makes predicting it quite fun. Here's how I did it when I was at FiveThirtyEight:

• You identify a number of precursor awards — Golden Globes, regional critic association prizes, guild awards — that also award prizes in the lead up to the Oscars.

• You go back and find out how often the winner of those awards went on to win the Academy Award.

• You turn that into a score, one that is higher for more consistent prizes and lower for less consistent ones. You arbitrarily double that score for precursor awards with voters who are represented in the Academy. If a prize was given out by actors, double its score since there are actors in the Academy. If a prize was given by critics, don't. 

• Movies and people get points for winning those precursor awards, and a fraction of those points for being nominated for a precursor award. The highest score is the favorite to win. 

It's a mathy way to articulate the implicit state of the race, and it's fairly good at it! We can't poll Oscar voters, so we look at people who are similar to Oscar voters and see who they vote for. We don't have historical data on how the votes shake out, but we can look at how historically good the precursor prizes are at predicting outcomes. We can't draw any enormous conclusions from any one precursor, but by following lots and lots of them we can improve our general understanding of the state of the race. 

But here's the thing: especially when it comes to Best Picture, I've got some doubts. 

This model was designed in 2008. Since then:

• The Academy has expanded the Best Picture field to more than five films. 

• The Academy has rolled out ranked-choice voting, a voting algorithm that gives broadly favored films an edge over movies with a robust, if smaller, fan base.

• The Academy has expanded significantly. Here's a line I wrote last year in the 2018 prediction post. It's the moment I realized that something would have to change:

[O]f the 7,258 eligible voters, a staggering number of them have been added to the academy in the past five years: Since 2013, the academy has invited 2,296 people to join its ranks, which, after accounting for deaths, declined invitations and loss of voting status, translated to a net increase of 1,402 voters. This means somewhere between 19 and 32 percent of the Academy voters have joined in the past five years. As such, the 25-year reliability of the guilds and critics groups we use to forecast this prize may not be as useful as it once was.

Emphasis mine. The Academy, to its credit, has grown and changed to look more like the industry it represents. Models aren't permanent, and if the world is changing, so must they.

This Oscar season, I'm going to roll out some experimental adaptations of the old models to recognize what must come in the new. The principles will remain the same: precursors are still good at predicting the Oscars, they nailed it last year, and conceivably could do so for a little while. But with a few substantive tweaks, I want to make something that can adapt to the changing times and be ready for when the ascendant Academy begins to exercise its power. 

Next week: how do critics awards matter?