I think the data can definitely lie. Especially something like this which is in it's infancy.
The S2 is at such an early point like this with low amounts of actual data.
They've advertised their product in a very clever way but there's several obvious holes in it.
1. It's a new product. It may have been around for 7 years but it was not testing every single eligible QB over that period. I don't even think they managed to test every single one for 2023 - just the top ones, which I actually think was a first for them. So saying a QB who tests poorly hasn't made it makes sense when there's QBs who have made it but who could have tested poorly, but weren't actually tested.
2. With that established the veracity of the claim, "no one who tested poorly has made it" comes into question and actually creates several questions like: Who has been tested? What constitutes "made it"? What constitutes a poor score? All these questions become very important because we need to know whose been tested out of the pool of "made it" QBs and if making it takes multiple years, which let's be honest for QBs it does in most people's eyes then 7 years is a really short amount of time. If for example you need 3 years of a QBs production to say they made it then only the first 4 years of data from S2 even matter and then how many QBs in those years were actually tested? How many QBs have "made it" in the NFL over a 7 year period? Probably not many in all truth. How many of those that "made it" took the S2? Again, probably not that many in truth.
I think if the S2 made a list of QBs who "made it" and which of those QBs took the test pre-draft I think you'd see it doesn't line up 1 for 1 because they likely don't have enough that actually took the test.
We also need to see the list of people who scored poorly. Which again is information they withold. What if that list is like 6 people long? Pretty easy to say "no one who scored poorly has made it" when it's a list of 6 guys and some like CJ are rookies or second year players who can't constitute "made it" by default.
3. From an early look it appears most score well on the test so the QBs who do not create a very small sample size. Most QBs don't make it and if so few score poorly and so few actually make it you've created a statistical probability that it's likely that someone who scores poorly also won't make it not because of the veracity of your own test but simply because that's the largest statistically likely outcome in the first place.
My final word on this matter also simply has to do with tests, especially ones that use multiple functional simulation tools like this one. Some people don't test well but they produce well. This is why in any field and in any industry there will never, ever be a one size fits all test. We all probably went to school with and work with people who produce well and test poorly. That's just a fact of life.
I realize this looks somewhat like word soup so I decided to break it down in a more digestible way to get across my point.
I will list every starting quarterback across the league. We will then judge whether they took the test and whether they've "made it" and then see what's left.
Jets - Rodgers - DNP (Did Not Participate) Too Old
Dolphins - Tua - Did he test? Does he count as "made it"? I could go either way on this one but for sake of argument let's say yes and yes
Patriots - Jones - Very similar to Tua except I'm saying he has not made it
Bills - Allen - This is the first of a few where I seriously question whether he tested pre-draft. At this point the S2 would be a year or two old. Is a projected first rounder taking some one or two year old test and are NFL teams even using it at this point? My intuition says no and no.
Raiders - Garoppolo - DNP too old
Chiefs - Mahomes - Exact same as Allen except the test is even younger. I'd be shocked if he took the test pre-draft.
Chargers - Herbert - This is likely the area where I could see the test start being used so I'll just say yes for sake of argument.
Broncos - Wilson - DNP too old
Browns - Watson - Same as Mahomes
Ravens - Jackson - Same as Mahomes, Allen, etc. test is too new at this point
Steelers - Pickett - Doesn't constitute as made it
Bengals - Burrows - Just like Herbert I'll say he took it and he's definitely made it
Jaguars - Lawrence - Same as Burrows
Titans - Tannehill - DNP too old
Texans - Stroud - Too early
Colts - Richardson - Too early
Cowboys - Prescott - DNP too old
Commanders - Howell - Too early
Eagles - Hurts - I'll say he took it and made it
Giants - Jones - Tough call because I don't want to say he's made it. I wouldn't want him as my QB, would you?
Rams - Stafford - DNP too old
49ers - Purdy - Doesn't qualify as having made it. Early signs are good but we've seen QBs fail at this point many times.
Cardinals - Dobbs - Hasn't made it
Seahawks - Smith - DNP too old
Packers - Love - Hasn't made it
Lions - Goff - DNP too old
Vikings - Cousins - DNP too old
Bears - Fields - Hasn't made it
Falcons - Ridder - Hasn't made it
Buccaneers - Mayfield - Hasn't made it
Saints - Carr - DNP too old
Panthers - Young - Too early
So from a very early breakdown this is exactly what I'm talking about. Connecting QBs who have made it and who took the test is incredibly difficult because there's not many who have made it and who I'd believe even took the test within the tests 7 year window of existence. The window for the S2s validity sandwiches itself into a small frame between old QBs who didn't take the test and young ones who haven't made it yet.
The statement, "no one who failed has made it" can both be true and can be incredibly misleading when it only looks like I'd say Tua, Herbert, Burrows, Lawrence, and Hurts took the test and made it as QBs. That's also giving the benefit of the doubt to S2 and even saying they all took that test. In all likelihood 1 or 2 didn't. So that's five guys. Five. That's such a small sample size where the more accurate statement is Tua, Herbert, Burrows, Lawrence, and Hurts passed the test and not those who fail never make it. Both statements are true but only one contains the more pertinent and accurate information. Failure doesn't preclude someone from actually making it, it's just the data is so new and entry points are so few that it hasn't been done yet. Key point there is "yet".
For example lets say I use the statement "every girl I've ever slept with says I'm the best in bed". That could be a true statement even if it's one that makes you roll your eyes. It could just be true because I slept with two who were virgins so obviously I'm the best they've had to that point and two were liars so they said it but they didn't mean it, but my statement only cares whether they said it or not. If I actually slept with more women as my statement implies then it probably fails to remain true but I've conveniently left out the relevant information, much like S2 has done with its claim that those who fail never make it. It's far too early to see whether that's true and I have full confidence CJ will break that claim for what it is.