Numlock Awards: The 2022 Mailbag

Numlock Awards is your one-stop awards season newsletter. We’re wrapping up this season, and wanted to thank you all for reading!

How did you think of the model performance?

Walter: Nobody asked this but I think it’s assumed. Yeah I was thrilled with the model this year. It nailed all of the acting and directing races, most of them were pretty clear but getting the Chastain win for Best Actress right was the really difficult one this year and it got that.

On Best Picture I was also pretty happy with it, because while some of the other forecasts gave CODA just incredibly remote odds to win, the model correctly called it was a pretty dead even race.

The model looks at the previous twenty years of precursor performance, but it weights recent performance stronger than historical performance: the past 10 years has double the weight and the past 5 years has triple weight. That is a decision that I made to ensure that we were keeping up with the Academy expansion.

But it’s one where I’ve lately wondered if I actually went far enough. Is triple enough? Is 20 years of data too much? These are basic model design questions, and I can test them because the model is, at the end of the day, a very nice spreadsheet. I was curious if that there was a different weighting system that would have put CODA on top going in to Oscar night.

I did a little under-the-hood analysis, and I looked into tweaking those weights to see if any of them put CODA as the leading contender. To explain this visually, here’s the weights assigned to each year under a few different possible model designs:

The reality is, all the little tweaks I made still had The Power of the Dog as the slight favorite going into Oscar night. You simply can’t design a model, at the current weighting of the precursor awards, that would say that a movie that wins at BAFTA, the DGA, Critics’ Choice and the Golden Globes is not the frontrunner.

That said, the model will change with the times: the PGA in particular got a big boost this year, but more on that below. We’ll keep getting better, but I’m pretty happy with Best Picture this year because if you treat it how I think it should be treated — as evidence to be interpreted, not as specific hard probabilities you can take to the bank — the model told us that we had an incredibly tight race this year and that CODA could not have been dismissed as readily as many had been writing it off.

Hey guys, how come you didn’t predict the chances of a live felony assault? You better incorporate that for next year. - Andrew C.

Walter: Unfortunately, the model as it’s designed can only forecast absolute robberies.

Long winded question but here’s something I’m noodling on: in the acting categories, it often seems like there’s an early frontrunner who either 1) locks in the award (Ariana DeBose, Will Smith this year) or almost falls from contention entirely (Kristen Stewart, KSM this year). I haven’t truly checked that trend with history, it’s just a feeling from following the awards for a while, but I’m curious if you get the same sense of polarity with the trajectory of early frontrunners, and if so, why you think that might be? - Geoff W.

Michael: I think a lot of this has to do with what Walter has pointed out before — those early awards in particular have virtually no overlap with the Academy voters (or BAFTA or SAG voters, for that matter). I think critics’ awards are valuable in getting some dark horse contenders more publicity than they would have otherwise received, like Isabelle Huppert in Elle or Drive My Car for Best Picture. 

But the time between the fall, when the critics have seen the entirety of the year’s film slate, to early winter, when Academy voters are finally catching up on even some of the big awards contenders, is long. So you’ll have people like Regina Hall (Support the Girls) or Ethan Hawke (First Reformed) who slowly fade from the conversation. 

By the time we get to SAG/BAFTA/Critics’ Choice and of course the Oscars, you’ll start to see a lot more convergence. All of the acting winners at the 94th Oscars won every single one of those awards except for Chastain missing out on BAFTA (though that might have had something to do with BAFTA’s revamped nomination rules). 

Perhaps this is a long winded way of answering the question but the tl;dr — those early frontrunners who perform well with critics can lose out on an Oscar nomination given the lack of overlap among voting bodies, but by the time you get to the more serious precursors that have Academy voters in their ranks, you’re going to see more coalescence. 

The model had it pretty much as a two-film race. My question is whether or not the model weighed which film would be more polarizing and get more 1st place votes, but also say more 4th place votes. I strongly believe that because CODA was so likable that it won because of 2nd place votes and such. - Irish Twilight

Walter: This is pretty much spot on I think. The conversation I had last year with Rob Richie really confirmed a lot of my thinking on this, personally, and I think it’s always worth a re-read when talking about Best Picture and the challenges in predicting it.

Here’s an important quote from Richie about this topic that I think gets at your question:

You have to get more than 50 percent in that instant runoff of head-to-head. You can't win by being, say, everyone's second choice. So, you do have to be a film that some people think is the best, and probably a pretty decent share. It is a system that both balances your need to have enough of the passion vote that you're in the mix of the top two or three, but of those ones, you're the one that's the most inclusive choice, the one that people are also willing to rank second and third, and so on.

That certainly sounds like CODA to me! It wouldn’t be shocking if The Power of the Dog won the first few rounds of voting but that there was enough overlap in the Belfast and the West Side Story and the King Richard first-ballot voters that CODA became a consensus pick. Indeed, that’s the very goal of ranked choice voting: at the end of the day, if you asked every Academy voter which film they prefer, CODA or The Power of the Dog, a majority would say CODA. Though it’s meandering, that’s what ranked choice actually measures.

How did the BAFTAs’ off-year change its weighting? - Katherine M.

Walter: Good question, we’ll see if the changes they make to the nominating process continue to have an effect, but outside of Best Actress and Best Picture they did tip off the winners.

Last year I made some charts about the overall changes in the precursor performance over time, this feels like the place to bring them up. These are weighted for recency like normal, but without the doubling for guilds yet. Here’s Best Picture:

Bad year for anything that isn’t the PGA, the WGA and SAG, clearly, but a few dynamics are emerging. The DGA’s slide is continuing, in 2015 they lost the top slot as the most-predictive precursor to the PGA, and they’re only getting worse. A few more bum years and they’ll be worse than SAG, which has, along with the PGA, mainly just held the line.

This is interesting to me, because while the acting branch has been shedding voteshare over the past decade as other branches begin to outpace their expansion, SAG is pretty much as reliable a predictor as it was at any point in the past dozen years. Indeed, SAG will beat out the Critics’ Choice for the first time ever in our raw weights next season.

In addition to the DGA, man has BAFTA fallen from grace. They had a great streak up until 2013, but it’s just been diminishing since.

Here’s Best Actress:

Again, a little bit of a stumble for BAFTA, and the SAG award isn’t actually getting a massive bump because of the way that previous successful years drop off, but things are still holding firm. Best Actress remains a perennially difficult category to call, but honestly all the other ones — the supporting categories, director, and Best Actor — their precursors are only getting better and more predictive year over year. This I attribute to the fact that those use first-past-the-post voting systems, while Best Picture votes using ranked choice.

Best Director is getting downright easy. You can set your watch to the DGA as always, but like the BAFTA used to have its own taste, lately they’re just herding along with everyone else.

Like just look at the improvement in the supporting categories, this is wild:

Anyway, I think that Best Picture is going to be continually difficult to call for the next three years or so, but that after that we’re going to get increasingly better at predicting it mainly because by then the Academy will likely be in a steady state in terms of membership and the precursor weightings will finally have the chance to “catch up” to where they truly should be. But again, that ranked-choice is a real killer, and so it’ll never be “predictable.”

If the Academy really wanted to up the ante and make the show much harder to predict, they’d start moving downballots to ranked choice as well.

Thanks so much for reading! We got one more post for you coming.