Lessons learned from first faculty development session on Milestones

I did my first faculty development session on milestones/EPAs (see prior post).  It was an hour long session.  The first half of the session included faculty members, residents and medical students and the second half was just faculty. 

Key Insights:

1) The assigned pre-work was helpful.  The participants did the pre-work.  It gave people background on the project and terminology and allowed us to start the session with a discussion of what surprised people when they rated themselves on the milestones.  It also allowed people who couldn't attend the session to get some background on the topic.

2) Both faculty and residents thought that the milestones used dense "edu-speak" and that it wasn't obvious what was meant by terminology such as "semantic qualifiers" and "illness scripts".  It was suggested that a glossary of terminology would be helpful. 

3) Many faculty assumed that they would be at level 5 (expert) that was not the case when they rated themselves on some of the milestones.   It will be a "culture shift" for faculty and residents to realize that it is ok to not be marked at the "top" of the scale.  It also brought up the question of the "expert" descriptions and if they represent ideal rather than reality.  It would be interesting to have  multiple experienced faculty rate themselves on the milestones and to see if there are certain milestones with a lower percentage of faculty rating themselves as expert.

4) It can be difficult to decide where a resident should be on a milestone.  There was an acknowledgment that a resident may sometimes be at an advanced beginner level and sometimes at a competent level on a milestone such as history taking depending on their familiarity with the presenting chief complaint of the patient.

5) Some of the milestone descriptions are not observable behaviors but rather require "getting inside the head" of the residents to discern their thought process and those milestones will be much more difficult to assess.

6) Residents discussed that that faculty will need to be observing them more and discussing cases in a different way in order to judge where they were in the milestones in a meaningful way.  With multiple rotating faculty, residents wondered how such assessment will take place.

7) Feedback was also discussed by the residents.  Most did not feel that they got much day to day feedback regarding their  skills except after a faculty had done a direct observation (structured clinical observation).  They appreciated some of the inpatient services who did feedback on a weekly basis but said that when the person giving the feedback had only worked with them for a day or two, the feedback was less valuable.

8) It was helpful to have residents and students at the session to understand their experience with assessment and as a way to introduce the new assessment system to them.  It was also helpful to have some time with just the faculty to discuss the logistics of the new system.

9) "Snapshots" of a learner at different levels (novice through expert) for an entrustable professional activity was helpful but will need more discussion and time than half an hour.  After discussing a single snapshot, faculty did not agree when actually evaluating a resident.

10) In the supervision guidelines for entrustable professional activities, "direct supervision" of advanced beginners is needed.  Our faculty did not agree on what "direct supervision" meant.  All agreed that it included repeating a physical exam and key pieces of history.  Some faculty wondered if it meant doing direct observation of the entire encounter including information sharing prior to discharge.  Since this is a recommendation coming from a national organization (Academic Pediatric Association), it would be helpful to have clarification on what is meant by direct supervision and reactive supervision.

11) When we reviewed the entrustable professional activities for our rotations, it was clear that some occur multiple times in a two week rotation (such as provide well child care) and others are less frequent (develop an evaluation and management plan for behavioral, developmental or mental health concerns).  It may be helpful to have a resident "passport" to record feedback on some of the EPAs.

12) All faculty agreed that in order to implement the milestones/EPAs we will need to re-assess how we currently structure the role of the precepting attendings.   Do our current staffing patterns meet the increasing intensity of educational assessment? Probably not.  For example, at Stanford, some faculty are being given 10% time in order to observe residents over time and in different settings on the milestones.  It is not known how much additional faculty time will be needed to effectively implement the new evaluation strategy and who will pay for that time.  It would be helpful if on a national level different programs shared models of faculty assessment strategies.

Assessing Residents in the Era of Entrustable Professional Activities and Milestones

The Accreditation Council for Graduate Medical Education (ACGME) is implementing the "Next Accreditation System"(NAS) for medical and surgical post-graduate training programs in the United States.  As a part of NAS, programs will be reporting the progress of their residents on milestones of clinical competence that have been developed by each specialty.  The milestones for specialties that will begin reporting this year can be viewed at http://www.jgme.org/toc/jgme/5/1s1.

With the continuing shift to outcomes-based assessment, there is a need for ongoing education of both faculty and residents about this evolution in assessment.  Pediatric residency programs will begin reporting on 21(of the 48) milestones in June 2014. 

Below is a proposed method for education about the Milestones that I am going to pilot for my department (ambulatory pediatrics).  Challenges for education in our department include faculty working in different locations/ different shifts and variability in familiarity with the milestones.

After reviewing the proposed faculty development plan, please feel free to comment and offer suggestions.

Proposed Faculty Development Plan regarding new resident evaluations.

Independent Learning (estimated time 30-45 minutes)
  
1) Review the 15 minute narrated PowerPoint- overview of the milestones and entrustable professional activities (EPAs). Video uses Windows Media player.

2) Do a self- assessment using selected milestones to become familiar with content and method of assessment.  Please complete PC-1,4,5, MK-1, PBL-4, ICS-1, PROF-1, SBP-1,3 of Pediatric Milestones http://www.acgme-nas.org/assets/pdf/Milestones/PediatricsMilestones.pdf

3) Additional readings (optional)

a) CL Carraccio, JB Benson, LJ Nixon, PL Derstine "From the educational bench to the clinical bedside: translating the Dreyfus developmental model to the learning of clinical skills" Acad Med 2008;83:761-767
http://journals.lww.com/academicmedicine/Fulltext/2008/08000/From_the_Educational_Bench_to_the_Clinical.17.aspx

b) PK Hicks et all "The Pediatrics Milestones: Conceptual Framework, Guiding Principles, and Approach to Development" Journal of Graduate Medical Education, Sept 2010; 410-418
www.jgme.org/doi/pdf/10.4300/JGME-D-10-00157.1

c) JR Frank et all "Competency based medical education: theory to practice" Medical Teacher 2010; 32:638-645.
http://groups.medbiq.org/medbiq/download/attachments/15991574/Frank+et+al.+-+Competency-based+medical+education+theory+to+practice.pdf?version=1&modificationDate=1327513526000


Group Learning Session (60- 90 minutes.  Independent learning above done as "pre-work")

1) Guided discussion of the evaluations for our rotation and review of "Snapshots" to provide examples of what novice, advanced beginner, competent, proficient and expert learners would look like for different EPAs.

2) Practice as a group evaluating current or recent residents using the new evaluation system.  Discuss areas that are difficult to evaluate or where there is disagreement.

(For faculty that are community-based we will need to explore additional opportunities for group discussion and learning via webinars or other on-line opportunities)

On-going Application ("post-work" 60 -90 minutes).  You may pick either option. 

1)Attendance at group evaluation sessions of residents (in person or call in)

2) Implement a Plan Do Study Act cycle for an assessment method for the milestones (examples- resident self-assessment on a milestone, chart stimulated recall).   Discuss outcome and lessons learned at monthly provider meeting (or alternate way to share learning for community based physicians)

Faculty would document their work on resident assessment as a part of their annual departmental goals and objectives and would receive institutional faculty development credit for their work.