Saturday, 15 October 2022 14:17

Recruit Rating Methodology

Class of 2023 Ratings

About the Recruit Ratings

Our recruit ratings debuted in 2022.  Years ago, we maintained a women's "Recruitable Database", with scores, skills, videos and club information.  We shut that system down due to the effort required to maintain it and to focus solely on our core mission of covering NCAA Gymnastics.  At the time, we did not attempt to rank or rate recruits.  Scores and video were not as widely available as they are today, and therefore fair evaluations were not feasible.  Today, thanks to Instagram and YouTube, recruits from all over the world are able to publish training and competition videos on a regular basis.  Online coverage of competitions are available from around the world, with both video and results available.  The widespread availability of this information has made recruit ratings possible.  

Our goal in creating this system was to provide ratings similar to what is available in other sports, but to base these ratings as much as possible on a non-subjective methodology, using transparent criteria and publicly available data, whenever possible.  Although gymnastics and the recruiting process itself is subjective, we wanted to create a system that depends less on an individual rater's subjective perception and instead on measurable and observable facts and data.  We started from taking the same criteria we have been using to assess each team's Recruiting Class in our annual ranking.  However, for the first time, we have formally encoded the criteria in a set of rules, with the intent to minimize subjectivity.      

The coaching staffs at different schools have different weights they apply to the various criteria which they use to assess potential student-athletes.  A single rating system can't possibly reflect all of these different priorities.  Instead, we focus here on results from competitions, scores achieved, skill level, execution, injury/competition history, consistency and her recent performance trend.  There are gymnastics criteria that all coaches tend to look to some degree in their potential recruits.  However, additional important criteria such as grades, motivation, leadership, and team fit are too specialized or impossible to fully assess in an observational rating system.      

There are several key principles that we focused on in developing our rating system:

  • Recent is Better:  We focus on data from the last two years, with exceptions (and then, with penalty) for gaps in competition history.  Results, scores and routines from four or five years ago are simply not as relevant as those from the current year.    
  • Competition, not Practice:  We prioritize competition results and skills competed, not what is shown during training.  Training video can be highly selective and not fully reflective of what is achievable in a full routine, on a consistent basis.   
  • Placements First, then Scores:   Gymnastics is fortunate when it comes to developing a recruit rating system, versus other sports like football and volleyball.  While statistics exist, a rating has to depend on evaluations of each athlete's relative competitiveness and skill.  We have the luxury of hundreds of independent, highly trained raters deployed across the world --- they are called judges.  However, as we all know, scoring variation exists in the club system, just like in the NCAA.  Judging may be more strict in certain situations or locations than others.  The meet environment may range from large arenas to small meets in home gyms, and at times, even non-regulation landing surfaces are used (in early season meets).  It is only when gymnasts finally meet head-to-head in National-level meets that scoring becomes more consistent and the environment and pressure is more like what the athlete will seek in college.  As a result, we look first at placements in National-level meets, and then second, at top scores achieved over the course of the season.  
  • Accommodate Score Variation:  A club athlete's scores throughout a typical club gymnastics season often will vary more than a college athlete's.  Unlike college competition, averages are not used to qualify for "post season" competition and a single qualifying mark forms a relatively low hurdle to post-season qualification (typically a 35.0 AA at Level 10).  Thus, many gymnasts and their club coaches take more risks during a season as they try out new skills and combinations, often at lower stake invitational meets.  These high risk elements and combinations are often edited out from the competition sets in the "post-season" meets.  Therefore, using averages won't accommodate differences in training and meet philosophy for an entire club season.  It is better to use a set of the athlete's highest scores to evaluate scoring potential.  This set of scores should be numerous enough to dilute any anomalous result but not too many to run afoul of the relatively short club seasons.  We also recognize of course, that consistency is important and a different method is needed to identify any issues there. 
  • Quantifiable not Subjective:  Our criteria relies on things we can measure or observe without a subjective assessment by the rater.  In fact, only one criteria in our system, worth just a half a star, is dependent on a partially subjective evaluation of gymanst's technique, execution and form.  

The recruits are rated from NR to 5 stars, in half star increments.  However, only 3 star and above are listed in our ratings.  A 3 star athlete is still highly accomplished and talented and will have a future impact in Division I competition.  There are also athletes in the ratings that are not currently shown because they have too few meets to their credit or will move up as they continue their development. 

The recruits are not in ranked order with each rating group, as our methodology's focus on reducing subjectivity doesn't afford such fine gradations in ratings.      

Evaluations will be conducted each summer, after the National meets are complete.  Ratings will be finalized at the start of the athlete's first year of competition (typically by September).  However, corrections or updates may be made in the interim by contacting the site directly. 

 

The Methodology

In our methodology, a base star rating is assigned based on finishes and/or scores and then half stars are added or subtracted based on additional criteria like high level skills, high level execution, consistency or weak events, and gaps in competition history, typically due to injury. 

Base Rating:  The Start

A gymnast is initially assigned a star rating based competition results and/or scores received.  There is a minimum threshold to qualify and a number of athletes with limited competitions (typically due to injury) are unable to qualify for a further rating with a base score.  From the base rating, a gymnast can increase her rating by up to one star.    

US Level 10s:  For US Level 10s, an athlete's base rating is based on her placements at the last two Level 10 National meets or her her top four AA or three-event scores in the last two years, or a combination of the two.  If a gymnast has a major injury and misses the season, an additional year from the prior year(s) is added but a penalty of 0.5 stars is subtracted from her base rating.  The following chart explains the methodology:

5 Stars

Top 4 at Women's Development Program (WDP) Nationals with a second Top 10 Placement

4.5 Stars

Top 10 twice at WDP Nationals OR once with 4 L10 AA scores over 38.5

4 Stars

Top 15 twice at  WDP Nationals OR once with 4 L10 AA scores over 38.25

3.5 Stars

Top 25 twice at WDP Nationals OR Top 10 once with 2nd WDP National qualification OR Top 25 once with 4 L10 AA scores over 38 OR 4 L10 AA scores over 38.25

3 Stars

 Top 30 twice at WDP Nationals OR Top 15 once with 2nd Nationals qualification OR Top 30 once with 4 L10 AA scores over 37.75 OR four L10 AA scores over 38.0 OR four 3-event L10 scores over 28.8

2.5 Stars

 Top 20 once at WDP National OR 4 AA scores or four 3-event scores over 28.5.

The chart is rather complex as it allows a gymnast multiple ways to earn stars and provides flexibility for gymnasts who have missed a National meet or is in a more competitive Region.  In our methodology, there is a minimum threshold to qualify but for this reason the entry level at 2.5 stars was made intentionally broad.  The vast majority of future college gymnasts come from the US L10 system, and so the criteria has to be flexible to accommodate multiple situations while ensuring a focus on high placements and achievement.  Future two or one event specialists are also excluded but can qualify via the three event rules if they compete up to three events.  Four scores are used to ensure some measure of repeatability.  A minimum threshold score is used versus an average due to scoring variation and to guard against one-off meets with unusual situations (like small club invitationals and "NCAA" rule meets).    

Elites:  For US elites, stars are assigned based on elite scores from the last two years, and for the highest star levels, qualification to the US Championships.  Training camp marks are excluded.  For international elites, scores from all AA international or national level competitions are considered.  Different star ratings are assigned based on ranges of scores achieved.  Different ranges are used for 2022+ than from the prior quadrennium, due to changes in the scoring system.  These levels largely mirror the existing score levels used to qualify US elites to various levels of competition, with slightly lower marks for US gymnasts at the 4+ star level as qualification through to Championships is also required. 

5 Stars

Elite National Team member with a minimum 55 (2017-2021) OR 53.5 (2022+) AA score

4.5 Stars

US Championship qualifier with minimum 53.5 (2017-2020) OR 52 (2022+) AA score
Any international elite with a minimum 54 (2017-2020) OR 52.5 AA (2022+) AA score

4 Stars US Championship qualifier with a minimum 52 (2017-2021) OR 50.5 (2022+) AA OR 39.75 (2017-2021)/38.85 (2022+) 3-event score
Any international elite with a minimum 52.5 (2017-2021) OR 51.0 (2022+) AA score
3.5 Stars

Any elite with a 51.0 (2017-2021) OR 49.5 (2022+) AA score OR US elite with a 39 (2017-2021)/38.1 (22+) 3-event score

3 Stars Any elite with a 49.5 AA (2017-2021) score OR 48 (2022+) AA score OR US elite with a 38 (2017-2021)/37.1 (22+) 3-event score
2.5 Stars

Any elite with a 48 AA (2017-2021) score OR 46.5 (2022+) AA score

Canadian Level 10:  For Canadian level 10s (CCP10), scores are assigned based on AA scores achieved at the national (Canadian or Canada Games), regional (i.e, Western or Eastern) or provincial championships.   A different scoring system than the US and a low number of available meet scores makes using the US criteria impossible.  These marks are:  38 for 4 stars, 37.5 for 3.5 stars, 37 for 3 stars and 36.5 for 2.5 stars.  For 4*, only scores at the National Championships are considered.  

Bonus:  Added to the Base

Gymnasts have two opportunities to add a half a star to their base rating, through outstanding difficulty or through very good form and execution.  These are assessed solely from video of competition routines, not workouts.  

The difficulty bonus focuses on the leg events.  A half star bonus is awarded to a gymnast that has successfully competed a NCAA 10.0 Start Value vault and a floor exercise with a FIG E acro skill OR FIG D plus D bonus acro combination.  This bonus rewards gymnasts with a high skill level on events where it can arguably make a difference at the NCAA level.  This criteria could have been applied to the vault alone.  However, top vaulters are typically also strong floor workers and strength on these two events can increase their overall appeal as a recruit, as big tumbling often is a key to unlocking scores of 9.9+.  However, to avoid overemphasizing difficulty, this bonus cannot be used to elevate a gymnast from 4.5 to 5 stars.    

The execution bonus focuses on execution and form, but on deductions other than those from steps, hops, falls and other infrequent or random errors.  This criteria assesses execution and form errors like insufficient split, clubbed or crossed feet, incorrect body position, bent arms, lack of amplitude and other deficits in technique that can reliably garner deductions at the NCAA level.  We ultimately are looking for gymnasts that have a clear path to 0.1 or less of these type of so-called "built-in deductions" on the UB, BB and FX.  These gymnasts are awarded 0.5 stars in bonus.  This is a high hurdle and is rarely granted in our ratings.  It is applicable, as you might suspect, most commonly with gymnasts that already hold a high base star rating.  Some latitude is granted to gymnasts with excess difficulty, especially those in the elite ranks, where problematic skills (those with systemic execution faults) can simply be dropped in a future NCAA set.  This is the only criteria in our system where some level of subjectivity from the rater is allowed.  Our base criteria is weighted on placement, skills and scores, and therefore execution already largely drives the gymnast's base rating.  The intention of the bonus here was to reward gymnasts that have the capability for high scores at the NCAA level once consistency is improved and/or routines are edited.  

To provide for flexibility around the more substantial and step function nature of the Level 10 criteria, in 2023 we added an exception criteria for gymnasts with a consistent level of high success.  A gymnast who, after a review of her last two seasons of performances, is short by ONE TENTH in TOTAL scores OR ONE placement from achieving a higher category of base rankings, is allowed a third year of competition results to factor into her base evaluation.  This third year can raise the gymnast's rating by a maximum of one half star.  For example, a gymnast who placed 5th AA at WDP Nationals in one year and 8th AA the next, would normally be rated 4.5*.  However, if the third prior year was considered, when she placed 2nd at WDP Nationals, she would be elevated from 4 1/2 to five stars.  However, the gymnast would also be evaluated for consistency in that third year (as well as the across the initial two years). 

We intentionally avoided any criteria that relied on highly subjective assessments of artistry or exceptional style, or on problematic criteria related to body type or shape.         

Penalties:  Deducted from the Base 

Up To The Level/Start Value:  A gymnast with an event that does not meet the NCAA "Up to the Level" requirement was docked a half a star for each event.  On UB, BB and FX, the gymnast needed to start from a 10 start value and on vault, from a 9.95 or higher. This assessment is based on competed routines and could change with the athlete's next assessment, as she adds difficulty.  On vault, this deduction was commonly applied to gymnasts who were competing vaults such as a layout Yurchenko or handspring pike front.  On BB and FX, some latitude was given to gymnasts that were short of the required start value by one tenth (i.e., with 0.5 in bonus).  It is assumed a gymnast of this level could easily add a missing 0.1 through a dance skill or dance-acro combination.  Gymnasts missing a required dance series were given full latitude due to differences in rules.  On UB, gymnasts that are missing a second D release or major release were automatically penalized.  Some latitude was also given for non-compliant dismounts, due to the rule differences between FIG/L10 and the NCAA.  A non-connected C dismount was penalized unless a clear path to an eligible combination was shown or an eligible dismount was used in a prior year.  It is possible to score well in the FIG/L10 system with routine construction that would cause a lower start value in the NCAA.  This was typically only applied on vault or on UB due to the lack of eligible releases.          

Major Injury/Missed Season:  A gymnast missing a season of competition (three meets or less, unless the post-season was included) or with a major injury (defined as involving more than 4 months of recovery time like a torn ACL or Achilles) had 0.5 stars deducted from her rating.  An additional year of competition history was added during her base rating assessment.   The gymnast was given the benefit of the doubt in all situations and any resulting rating changes were to the benefit of the athlete.  For example, an additional year was not added to the athlete's base assessment if she was already able to qualify without it.  This criteria is added because major injuries do impact an athlete's recruiting potential.     

Inconsistent/Weak Event:  This criteria was applied to US Development Program Level 10 athletes only.  A half star was deducted from the gymnast's rating if 50% or more of her competition results were scored at 9.3 or below.  This was intended to compensate for a gymnast's inconsistency or weaker event.  This deduction was NOT taken where UTL/Start Value deductions had been already taken for the event, or where a three-event score was used for the base rating assessment.   

Methodology Downsides and Gaps

No methodology is perfect, and that is most certainly in the case here.  We've tried to create a method that relies on data and facts, while reducing subjectivity introduced by the raters.  We've tried to rate a recruit's potential as a future college star by focusing on the most common criteria used to evaluate athletes.  At the same time, we've tried to work around inherent problems like variation in scoring while also accommodating unusual cases.  However, there are still some gaps.  Here are a few:

  • Our ratings make it impossible for a L10 gymnast to be rated if she has four or less competitive events to her credit (with three to four events each) over a two year period.  Any effort to add a gymnast with less competitive experience would have to rely on subjective criteria and potentially a vastly different set of non-quantitative and subjective measurements.  That would be unfair the 90%+ of the other athletes, who were evaluated against the same criteria.  This also means a current Level 9 or below are naturally excluded until she makes the step up to L10.  However, these prospective candidates, while still strong athletes, are not typically considered to be in the upper echelon of recruits, until they have demonstrated greater achievement.    
  • Our ratings treat all injuries or missed competition years equally.  However, each situation is unique and not all of the reasons for missing a season are equal in severity.   However, since this type of information is private and typically not available, we cannot consider the nature of the injury here.  At the same time, we do want to acknowledge that missed seasons due to injury or other reasons may be considered a negative by recruiters. 
  • Our ratings can't fully evaluate trend and potential.  By focusing on the last two seasons of competition, a good portion of a gymnast's performance trend is captured.  However, potential upside is more difficult to assess objectively.  The bonus point system for inherent form and execution on skills is an attempt to elevate gymnasts who have not yet hit their stride with adequate consistency, but have strong fundamentals and potential to score well in the future.  
  • Our ratings are limited to gymnastics skills, scores and achievements.  Other considerations like leadership, work ethic, academic achievement, fit and community involvement are impossible to evaluate from afar and often are valued differently by different recruiters.  

Last updated:  7/7/23