Advertisement

Characterizing Standardized Letters of Recommendation in Urology Residency Applications

Open AccessPublished:September 18, 2021DOI:https://doi.org/10.1016/j.urology.2021.06.051

      Abstract

      Objective

      To identify the current formats of standardized letters of recommendation (SLORs) and evaluate their characteristics, the distribution of applicants’ ratings, correlation between SLOR domain ratings and conventional application metrics, and potential biases.

      Methods

      We evaluated all applications submitted to our residency program for the 2020-2021 urology match. Two main formats of SLOR were identified. We extracted application characteristics and SLOR domain ratings.

      Results

      Ninety SLORs from 82 applicants were reviewed. Applicants were highly rated among top tiers in both formats. Some correlations were observed between domain ratings and application metrics such as Step 1 and Step 2 Clinical Knowledge scores, and percentage of Honors in core clinical clerkships. No statistically significant differences were found between female and male applicants in terms of domain ratings. Alpha Omega Alpha members received higher ratings in “urology resident potential,” “academic urologist potential,” and “performance as a sub-intern” domains. Applicants from top 40 US medical schools performed better as sub-interns, and were more likely to be ranked higher. Letters from home institutions were associated with higher ratings in several domains. In-person vs virtual interactions received similar ratings except for “communication”.

      Conclusion

      While it is promising to observe such number of SLORs submitted for the first time in urology, the current formats could benefit from further refinement in their structures and domains to distinguish between highly qualified urology applicants more efficiently. Given the transition in Step 1 score reporting to pass/fail outcome, the need for a reliable urology-specific SLOR will be critical.
      The United States urology residency match continues to be highly competitive. In the 2021 match, only 357 vacancies were listed for 528 applicants.

      2020 Urology Residency Match Statistics. Available online:https://www.auanet.org/documents/education/specialty-match/2020-Urology-Residency-Match-Statistics.pdf (Accessed January 25, 2021).

      Each year, residency programs evaluate the applicants on a number of objective metrics including United States Medical Licensing Examination (USMLE) Step 1 score, grades, scholarly production, and awards to screen who to interview and how to rank applicants for the match.
      • Huang MM
      • Clifton MM.
      Evaluating urology residency applications: What matters most and what comes next?.
      Another important component of applications are letters of recommendation (LORs). They provide valuable subjective information regarding an applicant's skillsets, qualifications, and potential weaknesses beyond available metrics.
      • Huang MM
      • Clifton MM.
      Evaluating urology residency applications: What matters most and what comes next?.
      A recent survey of urology program directors showed that LORs are the most important factor for interviewing and matching into urology.
      • Weissbart SJ
      • Stock JA
      • Wein AJ.
      Program directors' criteria for selection into urology residency.
      Historically in the field of urology, letter writers tend to use Narrative Letters of Recommendation (NLORs), without a uniform structure, to introduce the applicant. Previous studies have found the commonly used NLORs to be highly flattering and ambiguous, contain gender bias, and to have very low interobserver reliability in their interpretations.
      • Fortune JB.
      The content and value of letters of recommendation in the resident candidate evaluative process.
      • Keim SM
      • Rein JA
      • Chisholm C
      • et al.
      A standardized letter of recommendation for residency application.
      • Hu AC
      • Gu JT
      • Wong BJF.
      Objective measures and the standardized letter of recommendation in the otolaryngology residency match.
      • Dirschl DR
      • Adams GL.
      Reliability in evaluating letters of recommendation.
      • Filippou P
      • Mahajan S
      • Deal A
      • et al.
      The presence of gender bias in letters of recommendations written for urology residency applicants.
      Additionally, objective metrics available in Electronic Residency Application Service (ERAS) applications (such as transcripts, USMLE scores, etc.) may not be consistent with superlatives used in NLORs.
      • Fortune JB.
      The content and value of letters of recommendation in the resident candidate evaluative process.
      To overcome variability and bias in NLORs, Standardized Letters of Recommendation (SLORs) have been introduced for residency applications in several specialties including emergency medicine, otolaryngology, and orthopedic surgery.
      • Love JN
      • Doty CI
      • Smith JL
      • et al.
      The emergency medicine group standardized letter of evaluation as a workplace-based assessment: the validity is in the detail.
      • Tang AL
      • Howard JJM
      • Singh E
      • et al.
      Are standardized letters of recommendation in residency applications correlated with objective data?.
      • Inclan PM
      • Cooperstein AA
      • Powers A
      • Dy CJ
      • Klein SE.
      When (Almost) everyone is above average: a critical analysis of american orthopaedic association committee of residency directors standardized letters of recommendation.
      SLORs usually contain several domains pertaining to different aspects of applicant's capabilities and characteristics as well as a global assessment of the applicant. They have the potential to differentiate applicants more objectively and effectively, reduce gender bias, and offer higher inter-reviewer reliability compared to NLORs.
      • Friedman R
      • Fang CH
      • Hasbun J
      • et al.
      Use of standardized letters of recommendation for otolaryngology head and neck surgery residency and the impact of gender.
      ,
      • Girzadas Jr., DV
      • Harwood RC
      • Dearie J
      • Garrett S
      A comparison of standardized and narrative letters of recommendation.
      SLORs are also more efficient to write and read than NLORs.
      Although the AUA has not officially endorsed a uniform SLOR for candidates seeking urology residency, we observed a substantial increase in the usage of SLORs submitted to our residency program in the 2020-2021 match cycle. We aimed to identify the formats of SLOR used in urology and characterize their domain ratings. We also evaluated the correlation between SLOR domain ratings with several components of applicants’ background and the presence of biases in the current formats of SLOR. We hypothesize that a significant ceiling effect is present among SLOR domains, meaning most applicants are rated in the top tiers by letter writers.

      METHODS

      Study Design

      We retrospectively evaluated all LORs submitted to the corresponding author's residency program during the 2020-2021 match cycle. Two main types of SLOR were identified (Fig. 1A,B). Format 1 was originally proposed by Dr. David Penson and format 2 was distributed by the Society of Academic Urologists. Those with either a separate SLOR or a SLOR integrated into an additional narrative letter (hybrid SLOR) were included. This study was reviewed and approved by our institutional review board.
      Figure 1
      Figure 1Domains of the 2 SLORs used in urology residency applications (A and B). SLORs, standardized letters of recommendation.

      Variables

      For each SLOR, type and domain ratings were identified and entered into REDCap. Data regarding the type of rotation (home vs away, and in-person vs virtual) were also extracted. The sex and highest academic ranks of letter writers were obtained from the SLORs themselves or letter writers’ institutional websites.
      Associated ERAS applications were reviewed to gather information on age, gender, race/ethnicity, USMLE Step 1 and Step 2 Clinical Knowledge (CK) scores, applicant type (ie, MD, DO, international medical graduate [IMG]), and honor society memberships (eg, Alpha Omega Alpha [AOA], Gold Humanism Honor Society [GHHS], etc.). Medical school transcripts and medical student performance evaluations were reviewed to calculate the percentage of Honors in core clinical clerkships. Core clerkships varied slightly among medical schools but mostly included internal medicine, general surgery, pediatrics, obstetrics/gynecology, neurology, psychiatry, and family medicine. Grades were excluded if a pass/fail reporting system was used. Grades affected by the coronavirus disease 2019 (COVID-19) pandemic (ie, grading system switched to pass/fail for rotations held during the pandemic) were excluded as well and percentages were calculated based on the unaffected grades. Medical school rank was obtained from the 2021 US News and World Report according to the research category.

      2021 Best Medical Schools: Research. Available online:https://www.usnews.com/best-graduate-schools/top-medical-schools/research-rankings (Accessed January 5, 2021).

      The number of peer-reviewed publications, poster presentations, oral presentations, and book chapters were recorded for each applicant.

      Data Analysis

      Descriptive statistics were used to report applicant and SLOR characteristics as well as SLOR domain ratings. Domain rating scales were converted into ordinal numbers for statistical tests. The normal distribution of data was verified using Shapiro–Wilk test.
      • Shapiro SS
      • Wilk MB.
      An analysis of variance test for normality (complete samples).
      The Pearson's chi-squared or Fisher's exact test was used for categorical variables. The two-sample t-test, Kruskal–Wallis by ranks, or Wilcoxon rank-sum test was used to compare continuous variables, as appropriate. The statistical significance was set at an alpha level of 0.05. Kendall's coefficient of rank correlation tau-b (τb) was calculated to assess the level of correlation between SLOR domain ratings and application metrics (eg, percentage of Honors grades in core clinical clerkships, USMLE scores, number of publications, etc.). We tested for equality of Step 1 scores when comparing domain ratings between subgroups of applicants. All statistical tests were performed using STATA, version 14.1. The STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) statement was followed for design and reporting of this study.
      • von Elm E
      • Altman DG
      • Egger M
      • Pocock SJ
      • Gøtzsche PC
      • Vandenbroucke JP.
      The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.

      RESULTS

      Applicant Characteristics

      A total of 311 applicants applied to our urology residency program for the 2020-2021 match cycle. Among those, 90 SLORs from 82 urology applicants were identified and included for further review. We excluded two other applicants with SLORs in format 2 in which a 5-point scale was used to rate the domains (instead of the 4-point scale). Forty-nine (59.8%) applicants were male and the mean age was 27 (standard deviation [SD] 1.9) years (Table 1). The average Step 1 score was 242.5 (SD 13.7). Of 82 applicants, 57 (69.5%) reported a Step 2 CK score with an average of 251.8 (SD 11.8). Seventy-six (92.7%) applicants attended an allopathic school of medicine in the US. The mean percentage of Honors in core clinical clerkship was 53.8 (SD 34.0%) based on transcripts and medical student performance evaluation of 56 applicants. In 19/56 (33.9%) applicants, the grading scale for clerkships performed during the COVID-19 pandemic was switched to pass/fail grading and those clerkships were not included in calculation of their Honors percentages. Twenty-one (25.6%) applicants were members of AOA whereas 10 (12.2%) were members of GHHS. The median (interquartile range [IQR]) for the number of peer-reviewed publications, poster presentations, and oral presentations were 2 (1-3), 3.5 (2-6), and 1 (0-4), respectively. Table 1 demonstrates applicant characteristics defined by the formats of SLOR.
      Table 1Urology residency applicant (n = 82) and letter (n = 90) characteristics defined by format of the SLOR
      Format 1Format 2TotalP value
      Applicants, n (%)60 (73.2)22 (26.8)82 (100.0)
      Gender, n (%).15
       Female27 (45.0)6 (27.3)33 (40.2)
       Male33 (55.0)16 (72.7)49 (59.8)
      Age, mean (SD)27.1 (2.0)26.8 (1.5)27.0 (1.9).97
      Race, n (%).47
       White31 (51.7)10 (45.5)41 (50.0)
       Asian12 (20.0)4 (18.2)16 (19.5)
       Black6 (10.0)4 (18.2)10 (12.2)
       Hispanic9 (15.0)1 (4.6)10 (12.2)
       Other1 (1.7)1 (4.6)2 (2.4)
       Unknown1 (1.7)2 (9.1)3 (3.7)
      USMLE Step 1 score, mean (SD)243.3 (13.9)240.3 (13.2)242.5 (13.7).39
      USMLE Step 2 CK score, mean (SD)251.7 (12.6)252.1 (10.1)251.8 (11.8).90
      Applicant type, n (%)1.00
       MD55 (91.7)21 (95.5)76 (92.7)
       DO3 (5.0)1 (4.6)4 (4.9)
       IMG2 (3.3)0 (0)2 (2.4)
      Percentage of Honors in core clinical clerkships, mean (SD)56.0 (33.5)48.4 (35.7)53.8 (34.0).46
      Report of grades affected by COVID-19 pandemic, n (% in those with reported grades)16 (40.0)3 (18.8)19 (33.9).13
      Honor society membership(s), n (%)
       Alpha Omega Alpha17 (28.3)4 (18.2)21 (25.6).41
       Gold Humanism Honor Society7 (11.7)3 (13.6)10 (12.2)1.00
       Other12 (20.0)3 (13.6)15 (18.3).75
      Medical school rank, n (%).04
       Top 4032 (53.3)6 (27.3)38 (46.3)
       Non-top 4028 (46.7)16 (72.7)44 (53.7)
      Publications, median (IQR)
       Peer-reviewed article2 (1-3.5)2 (1-3)2 (1-3).97
       Poster presentation4 (3-8)3 (2-4)3.5 (2-6).11
       Oral presentation2 (0-4)1 (1-2)1 (0-4).53
       Book chapter0 (0-0)0 (0-0)0 (0-0).50
      SLORs, n (%)62 (68.9)28 (31.1)90 (100.0)
      Type of letter, n (%)<.001
       Hybrid SLOR59 (95.2)13 (46.4)72 (80.0)
       SLOR alone3 (4.8)15 (53.6)18 (20.0)
      SLOR is from, n (%)<.001
       Home institution53 (85.5)11 (39.3)64 (71.1)
       Away institution9 (14.5)17 (60.7)26 (28.9)
      SLOR is the result of a/an, n (%)<.001
       In-person interaction62 (100.0)19 (67.9)81 (90.0)
       Virtual interaction0 (0)9 (32.1)9 (10.0)
      CK, clinical knowledge; IMG, international medical graduate; IQR, interquartile range; SD, standard deviation; SLOR, standardized letter of recommendation; USMLE, United States medical licensing examination.
      Statistically significant P values are shown in bold typefaces.

      Letter Characteristics

      Of 90 SLORs, 62 (68.9%) letters were in format 1 while 28 (31.1) were in format 2. Most letters (59, 95.2%) in format 1 were hybrid SLORs whereas about a half (15, 53.6%) of letters in format 2 were SLORs alone (P <.001). Fifty-three (85.5%) and 11 (39.3%) format 1 and format 2 letters were from home rotations, respectively (P <.001). All SLORs in format 1 were the result of in-person rotations while 9 (32.1%) SLORs in format 2 were obtained from virtual rotations performed during the COVID-19 pandemic (P <.001). The vast majority of applicants in both formats fell in the highest tiers for almost all domains (5/6 in format 1, and 9/9 in format 2), except for “likely rank position at letter writer's institution” domain of format 1 which has a median equal to the middle tier. Figure 2 demonstrates the distribution of domain ratings box plot for both SLOR formats.
      Figure 2
      Figure 2Box and whisker plots showing the distribution of SLOR domain ratings for (A) format 1 and (B) format 2. Box lines represent the 25th, 50th, and 75th percentiles. Whisker lines represent the 5th to 95th percentile range, and dots represent outliers outside of this range. SLOR, standardized letters of recommendation. (Color version available online.)

      Correlations Between Letter Domains and Application Metrics

      In correlations between domains of format 1 and components of ERAS applications, the academic potential was significantly correlated with the number of peer-reviewed papers (τb = 0.32, P =<.01) and poster presentations (τb = 0.27, P = .01). Among format 2 domains, knowledge base vs Step 1 score (τb = 0.32, P = .04) and percentage of Honors (τb = 0.37, P = .04) were significantly correlated. Additionally, the correlation between technical aptitude and the percentage of Honors (τb = 0.59, P =<.01) was statistically significant. Other tested correlations are presented in Supplementary Table 1.

      Impact of Applicant and Letter Characteristics on Domain Ratings

      Female applicants had a higher mean in all format 1 domains compared to male applicants except for the “performance relative to other sub-interns” domain; however, such differences were not statistically significant. In format 2 domains, although male applicants received higher ratings in more domains compared to their female counterparts, these differences were not statistically significant (Table 2). Applicants self-identifying as white received higher ratings than all other applicants in the “potential as a urology resident” (4.5 vs 3.9, P = .01), and “performance as a sub-intern” (4.4 vs 3.7, P = .01) domains. Of note, white applicants with format 1 SLOR had significantly higher Step 1 scores as compared to applicants with other races in this subgroup analysis. The observed differences in all format 2 domains were not statistically significant (Table 2).
      Table 2Comparison of domain ratings based on gender and race of applicants
      DomainFemaleMaleP valueWhiteOtherP value
      Format 1
       Potential as a urology resident4.34.1.694.53.9.01
       Potential as an academic urology attending3.93.7.684.03.4.09
       Performance as a sub-intern4.24.0.464.43.7.01
       Urologic knowledge base2.62.2.162.52.3.35
       Performance relative to other sub-interns3.33.4.643.63.1.13
       Likely rank position3.42.9.213.33.1.58
      Format 2
       Communication3.43.2.313.33.2.68
       Professionalism3.23.5.463.43.4.86
       Team player3.13.3.503.33.2.76
       Teachability/response to feedback3.23.3.893.53.1.21
       Technical aptitude2.92.9.873.12.8.27
       Leadership potential3.02.9.683.22.8.26
       Knowledge base2.83.2.313.42.8.11
       Other stakeholder assessments3.03.01.003.12.8.41
       Overall rank of candidate3.32.9.233.22.9.35
      Values are mean of domain ratings. Of note, white applicants in format 1 had significantly higher Step 1 score compared to applicants with other races in format 1.
      Statistically significant P values are shown in bold typefaces.
      The difference between AOA members and nonmembers were significant in urology resident potential (4.7 vs 4.0, P <.01), academic urologist potential (4.3 vs 3.6, P = .04), and performance as a sub-intern (4.5 vs 3.9, P = .03) among format 1 domains. Members of GHHS were only ranked significantly higher in the “overall rank of candidate” domain (3.8 vs 2.9, P = .02). Applicants from the top 40 US medical schools performed better as sub-interns (4.3 vs 3.8, P = .04) and were more likely to be ranked higher at writer's institution (3.6 vs 2.8, P = .02).
      Letters from applicants’ home institution were rated higher than letters from away institution in the “potential as an academic urology attending” (3.9 vs 2.9, P = .02), “performance as a sub-intern” (4.2 vs 3.4, P = .01), “performance relative to other sub-interns” (3.7 vs 2.1, P <.01), and “likely rank position” (3.3 vs 2.0, P = .03) domains of format 1 (Supplementary Table 2). No significant differences were observed among domains of format 2. Additionally, applicants experiencing an in-person interaction with letter writers received significantly higher ratings in the communication domain (3.4 vs 2.9, P = .04).
      Domain ratings in both formats remained stable with increasing depth of interaction between the applicant and letter writer (Supplementary Table 3). The academic rank of letter writers was not associated with domain ratings where the number of observations was sufficient to run statistical tests (Supplementary Table 4). Letters from female writers had higher means in all domains of format 1 and 6 out of 9 domains of format 2 SLORs; however, these differences were not statistically significant (Supplementary Table 4).

      DISCUSSION

      This is the first study that investigates the utility of the SLORs in urology residency applications. While there was a substantial increase in SLORs use this match cycle, in our cohort 74% had NLORs. We also found that there is a marked ceiling effect where most applicants are rated among top tiers in both formats of urology SLORs (Fig. 2). The tight distribution of domain ratings may make it challenging for interviewers to rely on SLORs to distinguish between applicants. Similar outcomes were observed in SLORs in emergency medicine, orthopedic surgery, and otolaryngology suggesting the ceiling effect could be a pervasive drawback to SLORs (similar to NLORs).
      • Inclan PM
      • Cooperstein AA
      • Powers A
      • Dy CJ
      • Klein SE.
      When (Almost) everyone is above average: a critical analysis of american orthopaedic association committee of residency directors standardized letters of recommendation.
      ,
      • Love JN
      • Smith J
      • Weizberg M
      • et al.
      Council of Emergency Medicine Residency Directors' standardized letter of recommendation: the program director's perspective.
      ,
      • Kominsky AH
      • Bryson PC
      • Benninger MS
      • Tierney WS.
      Variability of ratings in the otolaryngology standardized letter of recommendation.
      This phenomenon may have several underpinnings. First, given the competitiveness of matching into urology, applicants are highly self-selected and typically outstanding candidates amongst their peers. Second, the ceiling effect could be due to the fact that medical students tend to get LORs from attendings that are known for writing strong letters or who will review their candidacy favorably. Furthermore, using Likert-like scales to rate the applicants, which is the case for the current formats of urology SLORs, may decrease the observed variability of ratings compared to wider scales (eg, 0-100 quantitative scales). Lastly, the distinguishing feature of urology applicants may not be captured by the current domains of SLOR or may be difficult to be ranked by a Likert-like scale.
      • Inclan PM
      • Cooperstein AA
      • Powers A
      • Dy CJ
      • Klein SE.
      When (Almost) everyone is above average: a critical analysis of american orthopaedic association committee of residency directors standardized letters of recommendation.
      Therefore, urology SLORs could be improved to contain more domains pertaining to different aspects of an applicant. Commitment to urology, initiative and drive, and research capabilities are examples of domains that could enhance existing urology SLOR templates. Moreover, we suggest the use of quantitative scales with wider range to increase the variability in future urology SLORs. Also, the letter writers should also be instructed that the ratings they give on SLORs should be based on a comparison with the urology applicants pool, not the total population of fourth-year medical students.
      This study indicated few meaningful correlations between ERAS application metrics and corresponding SLOR domain ratings. Those mainly included correlations with USMLE scores, percentage of Honors grades in core clinical clerkships, and number of publications/poster presentations (Supplementary Table 1). Although some correlations approached our predefined significance level, we found no correlations in some expected areas (eg, urologic knowledge base vs Step 1/Step 2 CK scores; performance as a sub-intern vs % of Honors; overall rank vs Step 1 score; etc.). The latter finding could be due to the relatively low number of observations in some tested groups of our study. It should be noted that we observed the grading system for core clinical clerkships was affected by the COVID-19 pandemic in about a third of this year's transcripts and was switched to a pass/fail system. Additionally, some medical schools routinely use a pass/fail system to report grades. These might justify the relatively low level of correlations between Honors grades and domain ratings in our study.
      A recent study on linguistic analysis of urology NLORs demonstrated that letters written for match-successful applicants had more power words which was also the case for the NLORs written for male urology applicants. The authors found an implicit gender bias in urology NLORs.
      • Filippou P
      • Mahajan S
      • Deal A
      • et al.
      The presence of gender bias in letters of recommendations written for urology residency applicants.
      Using predefined scales to rate specific characteristics of applicants, the SLORs have been found in prior studies in other specialties to reduce gender biases.
      • Friedman R
      • Fang CH
      • Hasbun J
      • et al.
      Use of standardized letters of recommendation for otolaryngology head and neck surgery residency and the impact of gender.
      ,
      • Kominsky AH
      • Bryson PC
      • Benninger MS
      • Tierney WS.
      Variability of ratings in the otolaryngology standardized letter of recommendation.
      This is of utmost importance especially in a field like urology where gender imbalance is a real concern. Interestingly, our findings were congruent with those studies indicating that gender bias did not exist in either domain of both SLORs, with the mean of most domain ratings being slightly higher for female applicants compared to their male peers. We also looked for other potential biases in the report of current formats of SLOR. No significant differences were found in domain ratings with regards to applicants’ race, depth of interaction between letter writers and applicants, and gender of letter writers. To our best knowledge, the presence of racial bias has not been evaluated in urology NLORs before. Therefore, at this point we are unable to determine if SLORs were able to reduce such an important bias among urology applicants. We also found SLORs in format 1 from home institutions had significantly higher ratings. With an increasing number of virtual sub-internships this year, there was a growing concern about whether that type of interaction would affect the letter writers’ perception about the applicants. Based on our data, no statistically significant differences were found in domain ratings between in-person and virtual interactions, except for the “communication” domain, which is understandable in the virtual context.
      In February 2020, the USMLE parent organizations announced a transition in Step 1 score reporting from a 3-digit numeric score to pass/fail.

      Change to pass/fail score reporting for Step 1. Available online: https://www.usmle.org/incus/#decision (Accessed January 25, 2021).

      This decision was made after several months of debates and will take place no sooner than January 2022. Previous studies have emphasized the role of Step 1 score in urology match. More than 80% of urology programs used the Step 1 score as a cutoff point to screen applicants for offering interviews.
      • Weissbart SJ
      • Stock JA
      • Wein AJ.
      Challenges facing program directors in the urology match.
      It was also one of the predictors of a successful match into urology to the extent that the odds of matching multiplied by 1.5 for every 10-point increase in the Step 1 score.
      • Pagano MJ
      • Cooper KL
      • McKiernan JM
      • Badalato GM.
      Outcome analysis of factors impacting the urology residency match.
      The Step 2 CK score could serve as a surrogate considering the more clinical nature of this examination. However, a decent number of applicants have not taken their Step 2 CK examination when filing their residency applications, especially this portion will be higher among urology applicants given that urology is an early match. In our study, only 69.5% of applicants reported their Step 2 CK scores. With the upcoming transition of the Step 1 examination to pass/fail reporting, there is an emerging call for another equitable and effective screening tool. A recent study indicated that most urology program directors (84.6%) believe this change will make it more difficult to objectively compare applicants.
      • Chisholm LP
      • Drolet BC.
      usmle step 1 scoring changes and the urology residency application process: program directors' perspectives.
      All in all, these further highlight the importance of introduction and widespread usage of other quantitative measures such as SLORs in urology residency applications. A cultural shift toward a more reliable, honest, and fair evaluation of applicants is crucial as well to ensure the success of this new assessment tool.
      • Huang MM
      • Clifton MM.
      Evaluating urology residency applications: What matters most and what comes next?.

      LIMITATIONS

      We acknowledge there are several limitations to our study. First, due to the limited use of SLORs by urology attendings, there were a relatively low number of observations in some subgroups of our study. Our findings could pave the road for additional research and development of more effective urology-specific SLORs, and could promote the widespread use of such letters by urology attendings.
      Moreover, we derived the data on the nature of rotations (in-person vs virtual) based on the mention of that by letter writers. Additionally, no domain was specified for this aspect of the rotations in the current formats. Therefore, there might be some SLORs resulting from virtual rotations that were inadvertently classified as in-person. We compared honor society members to nonmembers. This section of results might have been more meaningful if we could exclude applicants from medical schools with no honor society chapters. However, we found that some honor society elections are held during the senior year, making exclusion of medical schools more complicated. Finally, the SLORs of applicants to our program may not be generalizable to other programs.

      CONCLUSION

      The use of SLOR in urology residency applications is still limited compared to other specialties. There is a lack of variability among domain ratings of the current SLOR formats which makes it hard to differentiate between highly qualified urology applicants. This study noted some correlations between conventional application metrics with SLOR domain ratings, mostly in format 1 SLORs. Interestingly, discrepancies between female and male applicants were not found. However, a bias existed between format 1 SLORs from home institution compared to away institutions. Given the transition in USMLE Step 1 score reporting to pass/fail outcome, the need for a reliable urology-specific SLOR will be critical for the field to select the most qualified applicants to enter this competitive surgical subspecialty.

      Appendix. SUPPLEMENTARY MATERIALS

      References

      1. 2020 Urology Residency Match Statistics. Available online:https://www.auanet.org/documents/education/specialty-match/2020-Urology-Residency-Match-Statistics.pdf (Accessed January 25, 2021).

        • Huang MM
        • Clifton MM.
        Evaluating urology residency applications: What matters most and what comes next?.
        Curr Urol Rep. 2020; 21: 37
        • Weissbart SJ
        • Stock JA
        • Wein AJ.
        Program directors' criteria for selection into urology residency.
        Urology. 2015; 85: 731-736
        • Fortune JB.
        The content and value of letters of recommendation in the resident candidate evaluative process.
        Curr Surg. 2002; 59: 79-83
        • Keim SM
        • Rein JA
        • Chisholm C
        • et al.
        A standardized letter of recommendation for residency application.
        Acad Emerg Med. 1999; 6: 1141-1146
        • Hu AC
        • Gu JT
        • Wong BJF.
        Objective measures and the standardized letter of recommendation in the otolaryngology residency match.
        Laryngoscope. 2020; 130: 603-608
        • Dirschl DR
        • Adams GL.
        Reliability in evaluating letters of recommendation.
        Acad Med. 2000; 75: 1029
        • Filippou P
        • Mahajan S
        • Deal A
        • et al.
        The presence of gender bias in letters of recommendations written for urology residency applicants.
        Urology. 2019; 134: 56-61
        • Love JN
        • Doty CI
        • Smith JL
        • et al.
        The emergency medicine group standardized letter of evaluation as a workplace-based assessment: the validity is in the detail.
        West J Emerg Med. 2020; 21: 600-609
        • Tang AL
        • Howard JJM
        • Singh E
        • et al.
        Are standardized letters of recommendation in residency applications correlated with objective data?.
        J Med Educ Curric Dev. 2019; 62382120519893976
        • Inclan PM
        • Cooperstein AA
        • Powers A
        • Dy CJ
        • Klein SE.
        When (Almost) everyone is above average: a critical analysis of american orthopaedic association committee of residency directors standardized letters of recommendation.
        JB JS Open Access. 2020; 5: e20.00013
        • Friedman R
        • Fang CH
        • Hasbun J
        • et al.
        Use of standardized letters of recommendation for otolaryngology head and neck surgery residency and the impact of gender.
        Laryngoscope. 2017; 127: 2738-2745
        • Girzadas Jr., DV
        • Harwood RC
        • Dearie J
        • Garrett S
        A comparison of standardized and narrative letters of recommendation.
        Acad Emerg Med. 1998; 5: 1101-1104
      2. 2021 Best Medical Schools: Research. Available online:https://www.usnews.com/best-graduate-schools/top-medical-schools/research-rankings (Accessed January 5, 2021).

        • Shapiro SS
        • Wilk MB.
        An analysis of variance test for normality (complete samples).
        Biometrika. 1965; 52: 591-611
        • von Elm E
        • Altman DG
        • Egger M
        • Pocock SJ
        • Gøtzsche PC
        • Vandenbroucke JP.
        The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies.
        Epidemiology. 2007; 18: 800-804
        • Love JN
        • Smith J
        • Weizberg M
        • et al.
        Council of Emergency Medicine Residency Directors' standardized letter of recommendation: the program director's perspective.
        Acad Emerg Med. 2014; 21: 680-687
        • Kominsky AH
        • Bryson PC
        • Benninger MS
        • Tierney WS.
        Variability of ratings in the otolaryngology standardized letter of recommendation.
        Otolaryngol Head Neck Surg. 2016; 154: 287-293
      3. Change to pass/fail score reporting for Step 1. Available online: https://www.usmle.org/incus/#decision (Accessed January 25, 2021).

        • Weissbart SJ
        • Stock JA
        • Wein AJ.
        Challenges facing program directors in the urology match.
        Urology Practice. 2016; 3: 486-492
        • Pagano MJ
        • Cooper KL
        • McKiernan JM
        • Badalato GM.
        Outcome analysis of factors impacting the urology residency match.
        Urology Practice. 2016; 3: 296-301
        • Chisholm LP
        • Drolet BC.
        usmle step 1 scoring changes and the urology residency application process: program directors' perspectives.
        Urology. 2020; 145: 79-82