I did my first faculty development session on milestones/EPAs (see prior post). It was an hour long session. The first half of the session included faculty members, residents and medical students and the second half was just faculty.
1) The assigned pre-work was helpful. The participants did the pre-work. It gave people background on the project and terminology and allowed us to start the session with a discussion of what surprised people when they rated themselves on the milestones. It also allowed people who couldn't attend the session to get some background on the topic.
2) Both faculty and residents thought that the milestones used dense "edu-speak" and that it wasn't obvious what was meant by terminology such as "semantic qualifiers" and "illness scripts". It was suggested that a glossary of terminology would be helpful.
3) Many faculty assumed that they would be at level 5 (expert) that was not the case when they rated themselves on some of the milestones. It will be a "culture shift" for faculty and residents to realize that it is ok to not be marked at the "top" of the scale. It also brought up the question of the "expert" descriptions and if they represent ideal rather than reality. It would be interesting to have multiple experienced faculty rate themselves on the milestones and to see if there are certain milestones with a lower percentage of faculty rating themselves as expert.
4) It can be difficult to decide where a resident should be on a milestone. There was an acknowledgment that a resident may sometimes be at an advanced beginner level and sometimes at a competent level on a milestone such as history taking depending on their familiarity with the presenting chief complaint of the patient.
5) Some of the milestone descriptions are not observable behaviors but rather require "getting inside the head" of the residents to discern their thought process and those milestones will be much more difficult to assess.
6) Residents discussed that that faculty will need to be observing them more and discussing cases in a different way in order to judge where they were in the milestones in a meaningful way. With multiple rotating faculty, residents wondered how such assessment will take place.
7) Feedback was also discussed by the residents. Most did not feel that they got much day to day feedback regarding their skills except after a faculty had done a direct observation (structured clinical observation). They appreciated some of the inpatient services who did feedback on a weekly basis but said that when the person giving the feedback had only worked with them for a day or two, the feedback was less valuable.
8) It was helpful to have residents and students at the session to understand their experience with assessment and as a way to introduce the new assessment system to them. It was also helpful to have some time with just the faculty to discuss the logistics of the new system.
9) "Snapshots" of a learner at different levels (novice through expert) for an entrustable professional activity was helpful but will need more discussion and time than half an hour. After discussing a single snapshot, faculty did not agree when actually evaluating a resident.
10) In the supervision guidelines for entrustable professional activities, "direct supervision" of advanced beginners is needed. Our faculty did not agree on what "direct supervision" meant. All agreed that it included repeating a physical exam and key pieces of history. Some faculty wondered if it meant doing direct observation of the entire encounter including information sharing prior to discharge. Since this is a recommendation coming from a national organization (Academic Pediatric Association), it would be helpful to have clarification on what is meant by direct supervision and reactive supervision.
11) When we reviewed the entrustable professional activities for our rotations, it was clear that some occur multiple times in a two week rotation (such as provide well child care) and others are less frequent (develop an evaluation and management plan for behavioral, developmental or mental health concerns). It may be helpful to have a resident "passport" to record feedback on some of the EPAs.
12) All faculty agreed that in order to implement the milestones/EPAs we will need to re-assess how we currently structure the role of the precepting attendings. Do our current staffing patterns meet the increasing intensity of educational assessment? Probably not. For example, at Stanford, some faculty are being given 10% time in order to observe residents over time and in different settings on the milestones. It is not known how much additional faculty time will be needed to effectively implement the new evaluation strategy and who will pay for that time. It would be helpful if on a national level different programs shared models of faculty assessment strategies.