Advertisement

Adapting to the Educational Challenges of a Pandemic: Development of a Novel Virtual Urology Subinternship During the Time of COVID-19

  • Author Footnotes
    1 Financial support: D.L. was supported by grant number K12HS026372 from the Agency for Healthcare Research and Quality and by the American Cancer Society. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.
    Cheyenne Williams
    Footnotes
    1 Financial support: D.L. was supported by grant number K12HS026372 from the Agency for Healthcare Research and Quality and by the American Cancer Society. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.
    Affiliations
    Perelman School of Medicine University of Pennsylvania, Philadelphia, PA
    Search for articles by this author
  • Author Footnotes
    1 Financial support: D.L. was supported by grant number K12HS026372 from the Agency for Healthcare Research and Quality and by the American Cancer Society. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.
    Olivia O Familusi
    Footnotes
    1 Financial support: D.L. was supported by grant number K12HS026372 from the Agency for Healthcare Research and Quality and by the American Cancer Society. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.
    Affiliations
    Perelman School of Medicine University of Pennsylvania, Philadelphia, PA
    Search for articles by this author
  • Justin Ziemba
    Affiliations
    Division of Urology, Department of Surgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA
    Search for articles by this author
  • Author Footnotes
    1 Financial support: D.L. was supported by grant number K12HS026372 from the Agency for Healthcare Research and Quality and by the American Cancer Society. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.
    Daniel Lee
    Footnotes
    1 Financial support: D.L. was supported by grant number K12HS026372 from the Agency for Healthcare Research and Quality and by the American Cancer Society. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.
    Affiliations
    Division of Urology, Department of Surgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA

    Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, PA
    Search for articles by this author
  • Sameer Mittal
    Affiliations
    Division of Urology, Children's Hospital of Philadelphia, Philadelphia, PA
    Search for articles by this author
  • Phillip Mucksavage
    Affiliations
    Division of Urology, Department of Surgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA
    Search for articles by this author
  • Ariana Smith
    Affiliations
    Division of Urology, Department of Surgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA
    Search for articles by this author
  • Robert Caleb Kovell
    Correspondence
    Address correspondence to: Robert Caleb Kovell, M.D., Division of Urology, Department of Surgery, Perelman School of Medicine University of Pennsylvania, 3 West Perelman Center for Advanced Medicine, 3400 Civic Center Boulevard, Philadelphia, PA 19104.
    Affiliations
    Division of Urology, Department of Surgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA
    Search for articles by this author
  • Author Footnotes
    1 Financial support: D.L. was supported by grant number K12HS026372 from the Agency for Healthcare Research and Quality and by the American Cancer Society. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality.
Published:October 09, 2020DOI:https://doi.org/10.1016/j.urology.2020.08.071

      Objective

      To design, implement, and evaluate learner attitudes of a virtual urologic surgery clinical rotation for medical students.

      Methods

      Ten senior medical students at the Perelman School of Medicine at the University of Pennsylvania were enrolled. Students were administered a precourse test on their perceived confidence of their urologic knowledge, confidence in identifying urologic conditions, comfort with performing urologic evaluations, and confidence placing consults for urologic issues. Students participated in a 2-week curriculum that included both asynchronous and synchronous content. Asynchronous content included prerecorded lectures, self-paced problem-based learning modules, directed reading and video content, and an online discussion board. Synchronous content included real-time videoconferences covering case discussions, simulated patient presentations, and critical literature reviews. At the conclusion of the course, students were administered the postcourse survey evaluating changes in their ability to identify and understand urologic conditions.

      Results

      The postcourse survey demonstrated this course significantly increases students’ scores in: self-perceived urologic knowledge, confidence in naming urologic conditions, comfort with performing urologic evaluations, and confidence placing consults for urologic conditions (P <.05).

      Conclusion

      Virtual medical student rotations are scalable and effective at delivering surgical material and can approximate the interpersonal teaching found in clinical learning environments. They may be a useful tool to supplement or augment clinical learning in select situations.
      The COVID-19 pandemic has affected every aspect of the healthcare system including medical education. In accordance with social distancing guidelines, in March 2020, medical students were removed from all clinical care at institutions across the United States.
      • Newman NA
      • Lattouf OM
      Coalition for medical education-A call to action: a proposition to adapt clinical medical education to meet the needs of students and other healthcare learners during COVID-19.
      ,
      • Sam P
      • Heermans JT
      • Schmidt CM
      • Deibert CM
      Current state of urologic medical school education: a scoping review.
      This major shift had a significant effect on surgical upper-level students whose educations heavily rely on direct patient care in clinical spaces. In fact, a similar response was also observed for Urology resident learners, drastically reducing their direct patient care, and thus curtailing their education.
      • Porpiglia F
      • Checcucci E
      • Amparore D
      • et al.
      Slowdown of urology residents’ learning curve during the COVID-19 emergency.
      In response, multi-institutional coalitions like the UCSF Collaborative Online Virtual Didactics (COViD) and New York Educational Multi-institutional Program for Instructing Residents (EMPIRE) lecture series swiftly pivoted to delivering virtual didactics to residents and medical students. Although filling some educational gaps, these didactics failed to recapitulate the benefit of networking, mentorship, and personable feedback that come from medical students interacting in-person with residents and faculty.
      AAMC
      Effective Use of Educational Technology in Medical Education: Colloquium on Educational Technology: Recommendations and Guidelines for Medical Educators.
      We hypothesized that this may be approximated by an online subinternship that combines flexible, web-based, modular content with real-time interactions between medical students, residents, and faculty. Recognizing the urgent call to action, we describe our experience developing this virtual surgical subinternship and present early programmatic evaluation outcomes.
      Following the decision to suspend all student clinical activity, Urology faculty and 2 senior medical students at the University of Pennsylvania Perelman School of Medicine developed a novel virtual urologic curriculum. All upper-level, postclerkship medical students were eligible to register for this 2-week virtual Urology subinternship over the course of 1 month. The course was advertised as a fundamental-level course with the only perquisite requirement being the completion of the Surgery core clerkship.
      The primary goal of this curriculum was to expose students to Urology while advancing their foundational knowledge of urologic evaluation and management. The secondary goal was to innovate methods for implementing a virtual curriculum that is scalable, sustainable, and adaptable to the needs of learners and instructors across surgical disciplines.
      By the end of the course, learners were to achieve the following learning objectives: (1) Describe the steps to performing a urologic evaluation; (2) deliver urologic history, exam, assessment and plan in an oral case presentation; (3) list and describe the common conditions that urologists manage; (4) name key anatomical structures relevant to urological pathologies and urinary drainage; and (5) demonstrate ability to appraise urologic literature that intersects with other medical and surgical specialties.

      Methods

      Curriculum Format

      The curriculum used both asynchronous and synchronous content (Table 1). The online learning management system, Canvas (Instructure, Salt Lake City, UT) hosted asynchronous readings, lectures, and discussion boards. BlueJeans (BlueJeans Network, San Jose, CA) videoconferencing was utilized for synchronous discussions and case presentations.
      Table 1Breakdown of synchronous and asynchronous materials
      Overview of Penn Medicine Virtual Urology Curriculum
      Type of ActivityLearner Hours Per CourseAdditional Educator Hours Per CourseDescription
      Asynchronous

      Material
      Recorded lectures202Faculty recorded 30 min lectures in benign, pediatrics, and oncology
      Online cases1001-2 problem-based learning cases accompanied each lecture
      Additional readings150Supplemental reading from AUA medical student curriculum, Cambell-Walsh-Wein, Penn Urology Handbook, and landmark articles
      Discussion board post150.5Students posted one question, comment, or article a day.

      Students replied to at least one of their peers

      Faculty and resident provided comments on board
      Cumulative602.5
      Synchronous

      material
      Live discussion sessions101.5Faculty and resident facilitated on rotating schedule

      Review of online cases with students
      Live lecture- urology department grand rounds

      (once per week)
      30Students were invited to attend weekly virtual grand round meetings
      Live student presentations

      (once)
      22Students delivered final evidence-based medicine presentations to department
      Cumulative153.5
      Cumulative course total

      (2-wk)
      756
      Asynchronous material accommodated the uncertainty of faculty clinical schedules. Faculty members prerecorded, introductory lectures on core topics for on-demand viewing by learners. Topics included medical and surgical management of benign, oncologic, and pediatric Urology and aligned with the American Urological Association Medical Students Curriculum.

      American Urological Association. AUA Medical Students Curriculum. American Urological Association. Accessed June 10, 2020.https://www.auanet.org/education/auauniversity/education-and-career-resources/for-medical-students

      The majority of lectures were repurposed from pre-existing educational content created by each faculty member. Supplemental material included surgical videos, directed readings, and illustrations referenced from the AUA video library, Urology by Campbell's-Walsh-Wein, and the Penn Clinical Manual of Urology (Fig. 1) . Following completion of the lecture and supplemental materials, a discussion board opened to facilitate asynchronous communication and feedback about lecture contents among students, residents, and faculty. Learners were required to post one comment per day, with their activity identified by a distinct username. This served to increase interaction and participation with personal feedback on individual topics of student interest (Fig. 2).
      Figure 1
      Figure 1Two weeks schedule of virtual urology subinternship. (Color version available online.)
      Figure 2
      Figure 2Example of discussion board thread.
      Medical students were instructed to post a brief comment on a topic of interest from each day's lectures. Canvas discussion board threads allowed for asynchronous interactions among medical students, residents, and faculty.
      Synchronous material included daily virtual discussion sessions focused on the day's lecture topic. Similar to an in-person rotation, facilitators of these sessions rotated daily to provide flexibility for schedules and expose students to multiple faculty and residents. During these sessions, participants discussed surgical decision-making through problem-based learning (PBL) cases. PBLs were either written de novo or, when available, adapted from the AUA medical student case slides.

      American Urological Association. AUA Medical Students Curriculum. American Urological Association. Accessed June 10, 2020.https://www.auanet.org/education/auauniversity/education-and-career-resources/for-medical-students

      Facilitators used Socratic teaching methods to approximate what is observed in our clinical environment during a surgical rotation. Learners also prepared a surgical SOAP note from PBL cases and presented a simulated patient to the group for feedback by faculty and fellow students. At the conclusion of the course, learners gave an evidence-based medicine capstone presentation on a urologic topic of interest to the entire Urology division in a Grand-Rounds-style presentation.
      In accordance with our institution's grading guidelines for online courses, students were graded Pass/Fail based on completion of course requirements as outlined in Table 1.

      Programmatic Evaluation

      The Kirkpatrick model for programmatic evaluation was used.
      • Yardley S
      • Dornan T
      Kirkpatrick's levels and education “evidence.”.
      We focused our outcomes on Kirkpatrick Level 1 (participation) and Level 2a (attitudinal). To evaluate Level 2a outcomes, we developed a pre- and post-course questionnaire assessing participants perceptions of Urology using a 5-point Likert scale (1 = poor/strongly disagree and 5 = excellent/strongly agree) (Supplemental 1). Areas for evaluation of student attitudes included self-perceived urologic knowledge, confidence in naming urologic conditions, comfort with performing urologic evaluations, and confidence placing consults for urologic conditions. Students also provided end-of-course open-ended comments.
      Median scores with interquartile ranges of anonymized pre- and postcourse surveys were calculated. A paired Mann-Whitney U test compared pre- to postcourse responses for each item using RStudio software, v. 1.2.5033 (RStudio Inc, Boston, MA). A P-value <.05 was considered statistically significant.
      Qualitative content analysis on open-ended answers was performed using conventional content analysis methodology.
      • Hsieh H-F
      • Shannon SE
      Three approaches to qualitative content analysis.
      Thirty comments of written feedback were compiled, read in entirety, and analyzed inductively. Codes were then derived from prevailing themes captured among responses. Each comment was read word for word and assigned to its appropriate theme. Coding was performed separately by 2 investigators, C.W. and O.F., to minimize subjectivity. Disagreement among investigators regarding theme titles was resolved by discussion between the 2 coders. This study was considered quality improvement by the University of Pennsylvania Institutional Review Board, and informed consent was waived.

      Results

      Kirkpatrick Level 1 outcomes (participation) were a total of 10 senior medical students over 2 separate 2-week subinternship iterations occurring during May 2020. Participant's specialty of interest was recorded at the start of class and included Urology (3), General Surgery (1), OB/GYN (3), Emergency Medicine (1), Orthopedic Surgery (1), and Plastic Surgery (1). Twenty of 26 (77%) division faculty participated in lectures, discussion boards and/or discussion groups. Faculty reported that the creation of lecture material and participation in the discussion board on average was a cumulative 2.5 hours (Table 1). Faculty participation in synchronous learning activities, including discussion sessions and final capstone presentations, was a total of 3.5 hours.
      Kirkpatrick Level 2a outcomes include learner attitudes. Prior to taking the course, median student scores for all items ranged from 2 to 3 (Table 2). By the end of the curriculum, median scores significantly improved to greater than 3 in every topic area (P <.05). At the completion of the course, 2 students reported a shift of specialty commitment to Urology.
      Table 2Median scores in learner attitudes
      TopicMedian Baseline Score (IQR)Median Post-course Score (IQR)P
      Overall knowledge of urology3 (2-3)4 (4-4)0.01
      Naming urologic conditions confidence3 (2-3.75)4.5 (4-5)0.01
      Urologic evaluation confidence2 (2-2.75)3.5 (3-4)0.02
      Urology consult confidence3 (2.25-4)5 (4.25-5)0.01
      5-point scale: 1 poor, 5 excellent.
      n = 10.
      In qualitative content analysis of 30 end-of course feedback comments, themes included that the course had high perceived educational value, and that the course increased appreciation of Urology among students who were not previously considering the field. A representative comment stated that “[the] surgical patient presentations and capstone project were fun! It allowed me to think critically about a patient and get real time feedback on my surgical decision making.” Student feedback also highlighted the theme that discussion sessions that were didactic in nature were less favored than sessions that took a question and answer format. An additional theme was the convenience and appreciation of pre-recorded lectures and varied interaction with faculty members. One student remarked, “The flexibility of the course was great! The prerecorded lectures were succinct but with all the high yield facts I needed. I really liked that I could watch them on my own time.” Another student reported, “the daily discussion groups were my favorite part of the course. It was nice interacting with all of the Urology team.”
      Feedback volunteered by facilitators revealed that faculty and residents enjoyed the opportunity to continue teaching medical students during social distancing, and they appreciated the practicality of now having pre-recorded lectures for other teaching opportunities.

      Comments

      We provide a framework that allows for resuming urologic education for medical students while social distancing. Although there is no replacement for in-person, direct patient care for senior medical students, a virtual surgical subinternship is an effective and feasible alternative that has been recently explored in other surgical specialties.
      • Ehrlich H
      • McKenney M
      • Elkbuli A
      We asked the experts: virtual learning in surgical education during the COVID-19 pandemic—shaping the future of surgical education and training.
      ,
      • Chick RC
      • Clifton GT
      • Peace KM
      • et al.
      Using technology to maintain the education of residents during the COVID-19 pandemic.
      Our approach is unique in recreating an online version of the interpersonal aspects of clinical learning. These initial programmatic outcomes demonstrated an improved perception of urologic knowledge during the rotation. Although this was a temporizing innovation during the COVID-19 pandemic, this curriculum has the potential for lasting improvement of the future of urologic undergraduate medical education. Considering the AAMC recommendation to suspend visiting student rotations, the Society of Academic Urologists has encouraged Urology programs nationally to develop online virtual rotations. Unfortunately, no best practices yet exist. Our curriculum can model this adaptation for other Urology and surgery residency programs.
      Beyond the pandemic period, virtual course materials may augment or replace traditional curricula. An organized virtual platform is well-suited to expose “visiting” students to more programs by providing flexible scheduling and reducing the cost of visiting rotations. For students uncertain about their interest in a specialty, a virtual format with synchronous and asynchronous learning methods may allow them to learn about the specialty and its culture before deciding on a traditional rotation.
      • Hughes BD
      • Perone JA
      • Cummins CB
      • et al.
      Personality testing may identify applicants who will become successful in general surgery residency.
      Additionally, with the incorporation of students into outpatient telemedicine visits, there is potential for virtual subinternships to include patient care. The use of live video streaming of cases from the operating room may allow students to experience surgical decision making and operative techniques in real-time.
      • Chao TN
      • Frost AS
      • Brody RM
      • et al.
      Creation of an interactive virtual surgical rotation for undergraduate medical education during the COVID-19 pandemic.
      Finally, this format could correct the pervasive lack of Urology (and surgical subspecialties) within preclinical curriculums.
      • Casilla-Lennon M
      • Motamedinia P.
      Urology in undergraduate medical education.
      Prerecorded lectures and case modules created for virtual subinternships and stored on open-access Canvas sites are easily scalable and adaptable for a broader preclinical audience with minimal additional educator time. Furthermore, the learner-driven discussion board can become a portal to activate wider learning by empowering students to ask specific questions and learn from each other.
      We acknowledge limitations with this reported experience. Our programmatic evaluation is based on 2 course-sessions with a relatively small number of participants. Additionally, 2 students involved with curriculum design participated in one iteration of this study, introducing potential bias of the evaluations. Areas for improving of the course include more interaction with residents and more exposure to actual patients. In addition to accruing more participants, future directions include a postcurriculum exam assessing knowledge acquisition of urological principles (Kirkpatrick level 2b) and longitudinal follow-up to determine student's choice to enter Urology residency (Kirkpatrick level 4).

      Conclusion

      Urology has always been on the forefront of healthcare innovation, and here we apply this same spirit to medical education. By integrating didactic lectures, problem-based cases, and discussion sessions, we provide a model for recapitulating the interpersonal relationships of a subinternship experience in the virtual environment. With continued healthcare and educational innovation, we have an opportunity to embrace online learning with virtual subinternships and continue to push our collective mission of educating the next generation of Urologists during these unprecedented times.

      Appendix. SUPPLEMENTARY MATERIALS

      References

        • Newman NA
        • Lattouf OM
        Coalition for medical education-A call to action: a proposition to adapt clinical medical education to meet the needs of students and other healthcare learners during COVID-19.
        J Card Surg. 2020; 35: 1174-1175
        • Sam P
        • Heermans JT
        • Schmidt CM
        • Deibert CM
        Current state of urologic medical school education: a scoping review.
        Urology. 2019; 123: 59-63https://doi.org/10.1016/j.urology.2018.07.044
        • Porpiglia F
        • Checcucci E
        • Amparore D
        • et al.
        Slowdown of urology residents’ learning curve during the COVID-19 emergency.
        BJU Int. 2020; (n/a(n/a))https://doi.org/10.1111/bju.15076
        • AAMC
        Effective Use of Educational Technology in Medical Education: Colloquium on Educational Technology: Recommendations and Guidelines for Medical Educators.
        AAMC Institute for Improving Medical Education, 2007
      1. American Urological Association. AUA Medical Students Curriculum. American Urological Association. Accessed June 10, 2020.https://www.auanet.org/education/auauniversity/education-and-career-resources/for-medical-students

        • Yardley S
        • Dornan T
        Kirkpatrick's levels and education “evidence.”.
        Med Educ. 2012; 46: 97-106https://doi.org/10.1111/j.1365-2923.2011.04076.x
        • Hsieh H-F
        • Shannon SE
        Three approaches to qualitative content analysis.
        Qual Health Res. 2005; 15: 1277-1288https://doi.org/10.1177/1049732305276687
        • Ehrlich H
        • McKenney M
        • Elkbuli A
        We asked the experts: virtual learning in surgical education during the COVID-19 pandemic—shaping the future of surgical education and training.
        World J Surg. 2020; 44: 2053-2055https://doi.org/10.1007/s00268-020-05574-3
        • Chick RC
        • Clifton GT
        • Peace KM
        • et al.
        Using technology to maintain the education of residents during the COVID-19 pandemic.
        J Surg Educ. 2020; 77: 729-732https://doi.org/10.1016/j.jsurg.2020.03.018
        • Greene K
        Society of Academic Urologists Letter to VSAS.
        2020 (Published online May 14)
        • Hughes BD
        • Perone JA
        • Cummins CB
        • et al.
        Personality testing may identify applicants who will become successful in general surgery residency.
        J Surg Res. 2019; 233: 240-248https://doi.org/10.1016/j.jss.2018.08.003
        • Chao TN
        • Frost AS
        • Brody RM
        • et al.
        Creation of an interactive virtual surgical rotation for undergraduate medical education during the COVID-19 pandemic.
        J Surg Educ. 2020; (Published online July 1)https://doi.org/10.1016/j.jsurg.2020.06.039
        • Casilla-Lennon M
        • Motamedinia P.
        Urology in undergraduate medical education.
        Curr Urol Rep. 2019; 20: 69https://doi.org/10.1007/s11934-019-0937-x