Excellent analysis and support for the proposition that the NFL Draft is a highly watched manifestation of a roulette wheel. Your Draft "success" is largely determined by getting very high 1st round picks, extra picks and/or miraculously hitting on a key player - usually a QB.
Perhaps a larger question is why is this still true? You constantly hear how the NFL now has all of this new "data" with much being made of "complex film studies" allegedly deep into the night and "cloud computing" somehow sprinkling fairy dust on the bad data to make it all somehow work - yet, year after year there is no clear incremental material benefit in outcomes to what was happening years before.
It strikes me that much of the data being used today is junk data that pollutes decision making. The recent failures of the Wonderlic - and the more recently hyped S2 - to not remotely correlate to NFL success or even to identify the obvious stellar outliers like CJ Stroud - is but one cautionary tale of the over-reliance on data that is not empirically derived and demonstrably and statistically sound. Yet NFL front offices and media seem so vulnerable to any sales pitch that a new data source will win the day even when it pollutes your data pool even more.
In my experience, data segmentation - by that I mean reducing the size of the data cohort being considered, but increasing the relevancy of the use cases contained within that cohort - is the better approach. So, what might such a segmentation look like here and what does it say about why the Draft remains a roulette wheel? Here is but one possible sample:
*Start with for most collegiate players, there are approximately 25-28 games to evaluate over the later junior and senior seasons. Some will have less if they have switched positions.
*For most games, the average number of plays is around 70. However, since most players do not play both ways, only about 50% of game plays will involve offense of defense in total at all...or 35 total plays per game.
*Many plays do not directly involve the player being evaluated. (e.g. QBs handing off to RBs, WRs on runs to the opposite side etc.). Most players are only involved in some smaller percentage of plays on their side of the ball and even when they are, their involvement is often not material to the play or its outcome so as to inform analysis. When you take these plays out, the sample gets smaller still. But, it gets worse.
*How many of the plays occur when the game is out of reach or the half is approaching and the team is simply running the clock down? Garbage time or game clock management situations reduce the useful plays by another percentage.
*Only about half of games played by top teams – at most - involve teams that can be considered competitive by collegiate standards. For many games each year, the competition is not at a high level. This is acknowledged by pundits who note that the evaluations of SEC and Big 10 players are aided by the higher overall level of play in those conferences. Players in lesser conferences face far less elite competition to help evaluate their projection to the NFL. Those players carry more risk. Even for the big conferences, a lack of competition reduces the sample even more.
*Most players across the line interacting with the prospect – even on a particularly good or top opposing team – are unlikely to remotely approach the quality of play to be faced at the NFL level. Consider how few players are drafted or signed as FAs - even out of top programs. Our sample gets even smaller.
*Before considering injuries, that leaves probably perhaps less than 50 plays that are instructive for a grounded evaluation to try and project NFL performance. Even if you double that figure to be conservative, you are still looking at only about one game's worth of relevant plays (70 to 80 plays) over two years.
You wouldn’t evaluate a team on so few plays even in a single season…so, why is this data considered sufficient to grade a particular player over two?
In truth, it is incredibly difficult to accurately evaluate human performance with so few plays and games over such an extended period and in the presence of so many variables that exist in college football e.g. weather, competition, illness, coaching, home v. away, night v. day games, playing surface, schedule, injuries to other key players (e.g., QB) etc. Adding an S2 here or a hand diameter report there or even a long jump distance...won't solve it.
Looking at the sample of relevant plays provides clarity, and here it reveals an immovable object. College football players playing only half of the plays in a handful of games over a couple of seasons do not generate a sufficient sample of performance to make any accurate projections beyond obvious “bright line” ones. It’s why the NFL Draft quickly dilutes even in the first round to a point where players taken later are no less likely to be solid picks than earlier ones.
It doesn't mean you stop trying, but you must be smart about it. Just adding more small sample size data and hoping for the best is the wrong approach.
A follow-up that I wanted to share. I stumbled across a writer for The Athletic - Jourdan Rodrigue - who apparently (I am not a subscriber to The Athletic) wrote a two-part series entitled "Finding Rams" where she was embedded in the LA Rams' Draft process for a year.
I stumbled across her on Kevin Clark's "This is Football" podcast. That discussion was very illuminating. Interestingly, Les Snead and Sean McVay appear to agree with some of my views above on the Draft. For example, Snead makes it clear at the outset to the staff that they "are doomed" in their efforts to figure out the Draft and that luck is a part of the outcome. He reportedly says the goal of their process is to minimize mistakes - the self-inflicted wounds. As a scout don't get too focused on staring at a ceiling, because you always stand on the floor. Don't get too clever on the upside.
There was also a suggestion that the Rams look hard at what NOT to do in their process, versus endlessly searching for more data that pollutes their thinking like so many teams seem to be doing. There seemed to be a huge commitment to honing the efficiency and efficacy of their process - akin to a high-performance test and learn environment.
There were others. The Rams do not invest or value the Combine except for medical information. They do not appear to use cognitive testing - only an internal personality test that seemed to evaluate cultural fit. They do not participate in the short interviews at the Combine believing they are too rehearsed and worthless. Instead, they conduct targeted secret all day sessions with prospects at their school that they are interested in. For these, they appeared to use the same personnel to ensure consistency. There was virtually no mention of film studies as particularly insightful. They have a process for doing that, but it did not seem over weighted in the evaluation process the way it is in the media.
So, what do they value? It appears that the Senior Bowl is a key input for them. The reason is, I suspect, what I suggest above. That regular season games rarely involved competitive talent opposite the prospect for accurate evaluation and projection. The Senior Bowl fixes that problem to some degree by at least pitting them against top collegiate talent in a game and practices. Not surprisingly, the Rams select a large percentage of Senior Bowl talent.
All in all, very worthwhile and suggests that the Rams aren't afraid to go their own way - not recklessly as is often perceived, but in a very thoughtful and deliberate manner.
Excellent analysis and support for the proposition that the NFL Draft is a highly watched manifestation of a roulette wheel. Your Draft "success" is largely determined by getting very high 1st round picks, extra picks and/or miraculously hitting on a key player - usually a QB.
Perhaps a larger question is why is this still true? You constantly hear how the NFL now has all of this new "data" with much being made of "complex film studies" allegedly deep into the night and "cloud computing" somehow sprinkling fairy dust on the bad data to make it all somehow work - yet, year after year there is no clear incremental material benefit in outcomes to what was happening years before.
It strikes me that much of the data being used today is junk data that pollutes decision making. The recent failures of the Wonderlic - and the more recently hyped S2 - to not remotely correlate to NFL success or even to identify the obvious stellar outliers like CJ Stroud - is but one cautionary tale of the over-reliance on data that is not empirically derived and demonstrably and statistically sound. Yet NFL front offices and media seem so vulnerable to any sales pitch that a new data source will win the day even when it pollutes your data pool even more.
In my experience, data segmentation - by that I mean reducing the size of the data cohort being considered, but increasing the relevancy of the use cases contained within that cohort - is the better approach. So, what might such a segmentation look like here and what does it say about why the Draft remains a roulette wheel? Here is but one possible sample:
*Start with for most collegiate players, there are approximately 25-28 games to evaluate over the later junior and senior seasons. Some will have less if they have switched positions.
*For most games, the average number of plays is around 70. However, since most players do not play both ways, only about 50% of game plays will involve offense of defense in total at all...or 35 total plays per game.
*Many plays do not directly involve the player being evaluated. (e.g. QBs handing off to RBs, WRs on runs to the opposite side etc.). Most players are only involved in some smaller percentage of plays on their side of the ball and even when they are, their involvement is often not material to the play or its outcome so as to inform analysis. When you take these plays out, the sample gets smaller still. But, it gets worse.
*How many of the plays occur when the game is out of reach or the half is approaching and the team is simply running the clock down? Garbage time or game clock management situations reduce the useful plays by another percentage.
*Only about half of games played by top teams – at most - involve teams that can be considered competitive by collegiate standards. For many games each year, the competition is not at a high level. This is acknowledged by pundits who note that the evaluations of SEC and Big 10 players are aided by the higher overall level of play in those conferences. Players in lesser conferences face far less elite competition to help evaluate their projection to the NFL. Those players carry more risk. Even for the big conferences, a lack of competition reduces the sample even more.
*Most players across the line interacting with the prospect – even on a particularly good or top opposing team – are unlikely to remotely approach the quality of play to be faced at the NFL level. Consider how few players are drafted or signed as FAs - even out of top programs. Our sample gets even smaller.
*Before considering injuries, that leaves probably perhaps less than 50 plays that are instructive for a grounded evaluation to try and project NFL performance. Even if you double that figure to be conservative, you are still looking at only about one game's worth of relevant plays (70 to 80 plays) over two years.
You wouldn’t evaluate a team on so few plays even in a single season…so, why is this data considered sufficient to grade a particular player over two?
In truth, it is incredibly difficult to accurately evaluate human performance with so few plays and games over such an extended period and in the presence of so many variables that exist in college football e.g. weather, competition, illness, coaching, home v. away, night v. day games, playing surface, schedule, injuries to other key players (e.g., QB) etc. Adding an S2 here or a hand diameter report there or even a long jump distance...won't solve it.
Looking at the sample of relevant plays provides clarity, and here it reveals an immovable object. College football players playing only half of the plays in a handful of games over a couple of seasons do not generate a sufficient sample of performance to make any accurate projections beyond obvious “bright line” ones. It’s why the NFL Draft quickly dilutes even in the first round to a point where players taken later are no less likely to be solid picks than earlier ones.
It doesn't mean you stop trying, but you must be smart about it. Just adding more small sample size data and hoping for the best is the wrong approach.
Rinse repeat next year at this time...
A follow-up that I wanted to share. I stumbled across a writer for The Athletic - Jourdan Rodrigue - who apparently (I am not a subscriber to The Athletic) wrote a two-part series entitled "Finding Rams" where she was embedded in the LA Rams' Draft process for a year.
I stumbled across her on Kevin Clark's "This is Football" podcast. That discussion was very illuminating. Interestingly, Les Snead and Sean McVay appear to agree with some of my views above on the Draft. For example, Snead makes it clear at the outset to the staff that they "are doomed" in their efforts to figure out the Draft and that luck is a part of the outcome. He reportedly says the goal of their process is to minimize mistakes - the self-inflicted wounds. As a scout don't get too focused on staring at a ceiling, because you always stand on the floor. Don't get too clever on the upside.
There was also a suggestion that the Rams look hard at what NOT to do in their process, versus endlessly searching for more data that pollutes their thinking like so many teams seem to be doing. There seemed to be a huge commitment to honing the efficiency and efficacy of their process - akin to a high-performance test and learn environment.
There were others. The Rams do not invest or value the Combine except for medical information. They do not appear to use cognitive testing - only an internal personality test that seemed to evaluate cultural fit. They do not participate in the short interviews at the Combine believing they are too rehearsed and worthless. Instead, they conduct targeted secret all day sessions with prospects at their school that they are interested in. For these, they appeared to use the same personnel to ensure consistency. There was virtually no mention of film studies as particularly insightful. They have a process for doing that, but it did not seem over weighted in the evaluation process the way it is in the media.
So, what do they value? It appears that the Senior Bowl is a key input for them. The reason is, I suspect, what I suggest above. That regular season games rarely involved competitive talent opposite the prospect for accurate evaluation and projection. The Senior Bowl fixes that problem to some degree by at least pitting them against top collegiate talent in a game and practices. Not surprisingly, the Rams select a large percentage of Senior Bowl talent.
All in all, very worthwhile and suggests that the Rams aren't afraid to go their own way - not recklessly as is often perceived, but in a very thoughtful and deliberate manner.
Thanks for this -- I gotta find that podcast and listen!