Saturday, 15 October 2022 14:17

Recruit Rating Methodology

Class of 2022 Ratings

Class of 2023 Ratings

About the Recruit Ratings

Our recruit ratings debuted in 2022.  Years ago, we maintained a women's "Recruitable Database", with scores, skills, videos and club information.  We shut that system down due to the effort required to maintain it and to focus solely on our core mission of covering NCAA Gymnastics.  At the time, we did not attempt to rank or rate recruits.  Scores and video were not as widely available as they are today, and therefore fair comparisons were not feasible.  Today, thanks to Instagram and YouTube, recruits from all over the world are able to publish training and competition videos on a regular basis.  Online coverage of competitions are available from around the world, with both video and results available.  The widespread availability of this information has made modern recruit rankings possible.  

Our goal in creating this system was to provide ratings similar to what is available in other sports, but to base these ratings as much as possible on a non-subjective methodology, using transparent criteria and publicly available data whenever possible.  Although gymnastics and the recruiting process itself is subjective, we wanted to create a system that depends less on a rater's subjective perception and instead on measurable and observable facts and data.  We started from taking the same criteria we have been using to assess each team's Recruiting Class in our annual ranking.  However, for the first time, we have formally encoded the criteria in a set of rules, with the intent to minimize subjectivity.      

The coaching staffs at different schools have different weights they apply various criteria which they use to assess potential student-athletes.  A single rating system can't possibly reflect all of these different priorities.  Instead, we focus here on results from competitions, scores achieved, skill level, execution, injury/competition history, consistency and recent and current performance trend.  There are gymnastics criteria that all coaches tend to look to some degree in their potential recruits.  However, additional important criteria such as grades, motivation, leadership, and team fit are too specialized or impossible to fully assess in an observational rating system.      

There are several key principles that we focused on in developing our rating system:

  • Recent is Better:  We focus on data from the last two years, with exceptions (and then, with penalty) for gaps in competition history.  Results, scores and routines from four or five years ago are simply not as relevant as those from the current year.    
  • Competition, not Practice:  We prioritize competition results and skills competed, not what is shown during training.  Training video can be highly selective and not fully reflective of what is achievable in a full routine.   
  • Placements First, then Scores:   Gymnastics is fortunate when it comes to developing a recruit rating system, versus other sports like football and volleyball.  While statistics exist, a rating has to depend on evaluations of each athlete's relative competitiveness and skill.  We have the luxury of hundreds of independent raters --- they are called judges.  However, as we all know, scoring variation exists in the club system, just like the NCAA.  Judging may be more strict in certain situations or areas than others.  The meet environment may range from large arenas to small meets in home gyms, with non-regulation landing surfaces (in early season meets).  It is only when gymnasts finally meet head to head in National-level meets, that scoring becomes more consistent and the environment and pressure is more like what the athlete will seek in college.  As a result, we look first at placements in National-level meets, and then at top scores achieved, second.  
  • Accommodate Score Variation:  A club athlete's scores throughout a typical club gymnastics season often will vary more than a college athlete's.  Unlike college competition, averages are not used to qualify for "post season" competition and single qualifying marks form a relatively low hurdle to post-season qualification.  Thus, many gymnasts and their club coaches take more risks during a season as they try out new skills and combinations, often at lower stake invitational meets.  These high risk elements and combinations are often edited out from the competition sets in the "post-season" meets.  Therefore, using averages won't accommodate differences in training and meet philosophy for an entire club season.  It is better to use a set of the athlete's highest scores, enough to dilute any anomalous result but not too many to run afoul of the shorter club seasons.  We also recognize of course, that consistency is important and a different method is needed to identify any issues there. 
  • Quantifiable not Subjective:  Our criteria relies on things we can measure or observe without a subjective assessment by the rater.  In fact, only one criteria, worth just a half a star, is dependent on a partially subjective evaluation of gymanst's technique, execution and form.  

The recruits are rated from NR to 5 stars, in half star increments.  However, only 3 star and above are listed in our ratings.  A 3 star athlete is still highly accomplished and talented and will have a future impact in Division I competition.  There are also athletes in the ratings that are not currently shown because they have too few meets to their credit or will move up as they continue their improvement and development. 

The recruits are not in ranked order with each rating group, as our methodology's focus on reducing subjectivity doesn't afford such fine gradations in ratings.      

Evaluations will be conducted each summer, after the National meets are complete.  Ratings will be finalized at the start of the athlete's first year of competition.  However, corrections or updates may be made in the interim by contacting the site directly. 

 

The Methodology

In our methodology, and base star rating is assigned based on finishes and/or scores and then half stars are added or subtracted based on additional criteria like high level skills, high level execution, consistency or weak events, and gaps in competition history, typically due to injury. 

Base Rating:  The Start

A gymnast is initially assigned a star rating based competition results and/or scores received.  There is a minimum threshold to qualify and a number of potential rated athletes with limited competitions (typically due to injury) are unable to qualify at this time.  From the base rating, a gymnast can increase her rating by up to one star.    

US Level 10s:  For US Level 10s, an athlete's base rating is based on her placements at the last two Level 10 National meets or her her top four AA or three event scores in the last two years, or a combination of the two.  If a gymnast has a major injury and misses the season, an additional year from the past is added but a penalty of 0.5 stars is subtracted from her base rating.  The following chart explains the methodology:

5 Stars

Top 4 at Women's Development Program (WDP) Nationals with a second Top 10 Placement

4.5 Stars

Top 10 twice at WDP Nationals OR once with 4 L10 AA scores over 38.5

4 Stars

Top 15 twice at  WDP Nationals OR once with 4 L10 AA scores over 38.25

3.5 Stars

Top 25 twice at WDP Nationals OR Top 10 once with 2nd WDP National qualification OR Top 25 once with 4 L10 AA scores over 38 OR 4 L10 AA scores over 38.25

3 Stars

 Top 30 twice at WDP Nationals OR Top 15 once with 2nd Nationals qualification OR Top 30 once with 4 L10 AA scores over 37.75 OR four L10 AA scores over 38.0 OR four 3-event L10 scores over 28.8

2.5 Stars

 Top 20 once at WDP National OR 4 AA scores or four 3-event scores over 28.5.

The chart is rather complex as it allows a gymnast multiple ways to earn stars and provides flexibility for gymnasts who have missed a National meet or is in a more competitive Region.  In our methodology, there is a minimum threshold to qualify but for this reason the entry level at 2.5 stars was made intentionally broad.  The vast majority of future college gymnasts come from the US L10 system, and so the criteria has to be flexible to accommodate multiple situations while ensuring a focus on high placements and achievement.  Two or one event specialists are also excluded but can qualify via the three event rules.  Four scores are used to ensure some measure of repeatability, and a minimum threshold versus an average is used due to score variation and to guard against one-off meets with unusual situations (like small club invitationals and "NCAA" rule meets).    

Elites:  For US elites, stars are assigned based on elite scores from the last two years, and for the highest star levels, qualification to the US Championships.  Training camp marks are excluded.  For international elites, scores from all AA competitions are considered.  Different star ratings are assigned based on ranges of scores achieved.  Different ranges are used for 2022+ than from the prior quadrennium, due to changes in the scoring system.  These levels largely mirror existing score levels used to qualify US elites to various levels of competition, with slightly lower marks for US gymnasts at the 4+ star level due to the need to qualify through to Championships. 

5 Stars

Elite National Team member with a minimum 55 (2017-2021) OR 53.5 (2022+) AA score

4.5 Stars

US Championship qualifier with minimum 53.5 (2017-2020) OR 52 (2022+) AA score
Any international elite with a minimum 54 (2017-2020) OR 52.5 AA (2022+) AA score

4 Stars US Championship qualifier with a minimum 52 (2017-2021) OR 50.5 (2022+) AA OR 39.75 (2017-2021)/38.85 (2022+) 3-event score
Any international elite with a minimum 52.5 (2017-2021) OR 51.0 (2022+) AA score
3.5 Stars

Any elite with a 51.0 (2017-2021) OR 49.5 (2022+) AA score OR US elite with a 39 (2017-2021)/38.1 (22+) 3-event score

3 Stars Any elite with a 49.5 AA (2017-2021) score OR 48 (2022+) AA score OR US elite with a 38 (2017-2021)/37.1 (22+) 3-event score
2.5 Stars

Any elite with a 48 AA (2017-2021) score OR 46.5 (2022+) AA score

Canadian Level 10:  For Canadian level 10s, scores are assigned based on AA scores achieved at the national or provincial championships.   A different scoring system than the US and a low number of available meet scores makes using the US criteria impossible.  These marks are:  38 for 4 stars, 37.5 for 3.5 stars, 37 for 3 stars and 36.5 for 2.5 stars.  

Bonus:  Added to the Base

Gymnasts have two opportunities to add a half a star to their base rating, through outstanding difficulty and through very good form and execution.  These are assessed solely from video of competition routines, not workouts.  

The difficulty bonus focuses on the leg events.  A half star bonus is awarded to a gymnast that has successfully competed a NCAA 10.0 Start Value vault and a floor exercise with a FIG E skill OR FIG D plus D bonus combination.  This bonus rewards gymnasts with a high skill level on events where it can make a difference at the NCAA level.  This criteria could arguably applied to the vault alone.  However, top vaulters are typically also strong floor workers and this can increase their overall appeal as a recruit, as big tumbling often is a key to unlocking scores of 9.9+.  However, to avoid overemphasizing difficulty, this bonus can not be used to elevate a gymnast from 4.5 to 5 stars.    

The execution bonus focuses on execution and form, but on deductions other than those from steps, hops, falls and infrequent or random errors.  This criteria assesses execution and form errors like insufficient split, clubbed or crossed feet, incorrect body position, bent arms, lack of amplitude and other deficits in technique that can produce deductions at the NCAA level.  We ultimately are looking for gymnasts that have a clear path to 0.1 or less of these type of so-called "built-in deductions" on each of UB, BB and FX.  These gymnasts were awarded 0.5 stars in bonus.  This is a high hurdle and was rarely granted in our initial ratings.  In was, as you might suspect, most common with gymnast that already held a high rating.  Some latitude was granted to gymnasts with excess difficulty, especially those in the elite ranks, where problematic skills can simply be dropped in a future NCAA set.  This is the primary criteria in our system where some level of subjectivity from the rater is allowed.  Our base criteria is weighted on placement and scores, and therefore execution already largely drives the gymnast's base rating.  The intention of the bonus here was to reward gymnasts that have the capability for high scores at the NCAA level once consistency is improved and/or routines are edited.  

We intentionally avoided any criteria that relied on highly subjective assessments of artistry or exceptional style or on problematic criteria related to body type or shape.         

Penalties:  Deducted from the Base 

Up To The Level/Start Value:  A gymnast with an event that does not meet the NCAA "Up to the Level" requirement was docked a half a star for each event.  On UB, BB and FX, the gymnast needed to start from a 10 start value and on vault, from a 9.9 or higher. This assessment is based on competed routines and could change with the athlete's next assessment.  On BB and FX, some latitude was given to gymnasts that were short of the required start value.  It was assumed a gymnast could easily add a missing 0.1 through a dance or dance-acro combination.   On vault, this deduction was commonly applied to gymnasts who were competing vaults such as a layout Yurchenko or handspring tuck front.  On UB, gymnasts that were missing a second D release or major release were automatically penalized.  Some latitude was also given for non-compliant dismounts, due to the rule differences between FIG/L10 and the NCAA.  It is possible to score well in the FIG/L10 system with routine construction that would cause a lower start value in the NCAA.  Some latitude is given in certain situations, which is where some level of subjectivity enters the picture.         

Major Injury/Missed Season:  A gymnast missing a season of competition (three meets or less, unless the post-season was included) or with a major injury (where known) had 0.5 stars deducted from her rating.  An additional year of competition history was added during her base rating assessment.   The gymnast was given the benefit of the doubt in all situations and any resulting rating changes were to the benefit of the athlete.  For example, an additional year was not added to the athlete's base assessment if she was already able to qualify without it.    

Inconsistent/Weak Event:  This criteria was applied to US Development Program Level 10 athletes only.  A half star was deducted from the gymnast's rating if 50% of her competition results scored below a 9.3.  This was intended to compensate for a gymnast's inconsistency or weaker event.  This deduction was NOT taken where UTL/Start Value deductions had been already taken, or where a three-event score was used for the base rating assessment.   

Methodology Downsides and Gaps

No methodology is perfect, and that is most certainly in the case here.  We've tried to create a method that relies on data and facts, while reducing subjectivity introduced by the raters.  We've tried to rate a recruit's potential as a future college star by focusing on the most common criteria used to evaluate athletes.  At the same time, we've tried to work around inherent problems like variation in scoring while also accommodating unusual cases.  However, there are still some gaps.  Here are a few:

  • Our ratings make it impossible for a L10 gymnast to be rated if she has four or less competitive events to her credit (with three to four events each) over a two year period.  Any effort to add a gymnast with less competitive experience would have to rely on subjective and non-competition criteria and potentially a vastly different set of measurements.  That would be unfair the 90%+ of the other athletes, who were evaluated against the same criteria.  This also means a current Level 9 or below are naturally excluded until she makes the step up to L10.  However, these prospective candidates, while still strong athletes, are not typically not considered to be in the upper echelon of recruits, until they have demonstrated greater achievement.    
  • Our ratings treat all injuries or missed competition years equally.  However, each situation is unique and not all of the reasons for missing a season are equal in severity.   However, since this type of information is private and typically not available, we cannot consider it here.  At the same time, we do want to acknowledge that missed seasons due to injury or other reasons may be considered a negative, depending on the exact situation, by recruiters. 
  • Our ratings can't fully evaluate trend and potential.  By focusing on the last two seasons of competition, a good portion of a gymnast's performance trend is captured.  However, potential upside is more difficult to assess objectively.  The bonus point system for inherent form and execution on skills is an attempt to elevate gymnasts who have not yet hit their stride, but have strong fundamentals.  
  • Our ratings are limited to gymnastics skills and achievements.  Other considerations like leadership, work ethic, academic achievement, fit and community involvement are impossible to evaluate from afar and often are valued differently by different recruiters.