March Madness: Research shows analytics get the better of the selection committee

Ah, it’s Selection Sunday!

Tonight at 6 p.m. ET, the NCAA tournament field will be revealed, and while the selection committee generally does a solid job, there are occasionally a few egregious misses.

Like in any other sport, analytics are everywhere in college basketball, with the ratings site KenPom.com enjoying a particularly cult-like following among hoops diehards.

While the selection committee now uses KenPom’s ratings as one of its tools, it also uses a bunch of other vague, made-up garbage like NET rating and quadrants, something called “team sheets” and the ultimate fuzzy-math “quality wins and losses.”

(It would be a whole lot more acceptable in this author’s eyes if the committee simply said it was optimizing for TV ratings.)

But what happens when analytics and the NCAA tournament selection committee disagree?

Well, the committee gets its way, duh. It is their tournament, after all.

However, that doesn’t stop us from playing a little revisionist history and evaluating the outcome of those disputes. Using KenPom data, we went back and looked at all instances over the last 20 years where analytics and the selection committee disagreed on a team’s seeding.

Then we compared how many tournament games (in the round of 64 or better) each team won compared to what we would have expected them to win based based on the historical tournament performance of all teams with that seed.

Key takeaways:

• Teams that the analytics liked more than the selection committee outperformed historical expectation, with the return increasing as you move down the table.
• The 57 teams that were under-seeded by four or more by the NCAA selection committee combined to win 42 tournament games versus an expectation of 32.1 wins for a return on expectation (ROE) of 135%.
• This list includes three of the most memorable Cinderella runs of the last two decades — 2006 George Mason’s shocking run to the Final Four and Elite Eight trips by Antonio Gates’ 2002 Kent State Golden Flashes and Steph Curry’s 2008 Davidson Wildcats.
• Despite their mid-major status and underwhelming seeds, George Mason, Kent State and Davidson were each among the top 25 teams in the nation entering the NCAA tournament according to analytics, meaning their success should have been less of a surprise than it was.
• There have been 37 double-digit seeds to reach the Sweet Sixteen since 2002, and of those 37 teams, 26 of them were under-seeded while only 7 were over-seeded. On average, those 37 Cinderellas were under-seeded by 1.8 spots according to analytics. Their average seed was 11.3 despite KenPom reflecting that an average seed of 9.5 would have been more fitting.
• The most egregious under-seeding by any Sweet Sixteen team in the last 20 years belongs to 2014 Tennessee, which entered the tournament ranked No. 14 by KenPom but somehow earned a paltry 11-seed, meaning the Volunteers were under-seeded by a whopping 7 spots.

Key takeaways:

• The 407 teams that have been over-seeded by at least one spot won just 84% of the games they were expected to win based on the historical performance of that seed. And the degree to which teams underperformed grew as the gap between analytics and the selection committee grew.
• If you look at teams that have been under-seeded by just two spots, the list does include seven Final Four teams including one national champion — 2003 Syracuse led by freshman star Carmelo Anthony and long-range bomber Gerry McNamara.
• Oddly enough, the 2011 Butler team that went all the way to the national title game before losing to Duke was a touch over-seeded. The Bulldogs entered the tournament as an 8-seed despite the analytics thinking a 10-seed would have been more appropriate.
• However, your chances of success drop substantially if you’re over-seeded by three or more spots. Out of the 104 teams that have been over-seeded by at least three spots, not a single one has ever advanced to the Final Four.
• Of the 52 teams over-seeded by at least four spots, 22 of them flamed out in the round of 64 as a higher seed than their opponent, and only four of the 52 teams reached the Elite Eight, despite their generous seeding advantage.
• Vanderbilt is the worst offender on the list, twice being upset in the first round by a 13-seed. In 2008, the Commodores got blown out by Siena. In 2010, Vandy lost by a point to Murray State. In hindsight, it wasn’t much of a surprise as the ‘Dores should have been a double-digit seed both years.
• Another badly over-seeded team to get upended by a 13-seed was 2003 Dayton, which got drilled by Tulsa in the round of 64. The Flyers were awarded the 4-seed after winning the Atlantic-10 tournament, but in reality, Dayton should have been closer to a 10-seed.
• The list of teams over-seeded by four or more spots also features five No. 3 seeds that didn’t make it out of the second round: 2005 Gonzaga, 2007 Washington State, 2010 New Mexico, 2010 Pittsburgh and 2016 Utah.

In summary, when analytics and the selection committee have disagreed, analytics have enjoyed the advantage. In a single-elimination tournament, when so many of these games come down to the last few possessions and outcomes can be decided on a couple bounces here and there, it would be foolhardy to think you can pick your entire bracket using analytics.

However, as the disagreement between analytics and the selection committee grows in size, just know that analytics has, on average, demonstrated itself to have the upper hand over the last couple decades. So keep that in mind when a team such as Loyola-Chicago — yes, it’s Sister Jean again! — gets assigned an 8-seed despite the analytics placing the Ramblers among the top 10 overall teams in America.

And before anyone texts me, “But but but Loyola hasn’t played anyone!” just remember that we’re approaching a quarter-century since the last time the vaunted, never-been-deeper Big Ten had a team win the national title. So spare me your hasn’t-played-anyone BS.

Enjoy Selection Sunday!