Lessons learned from first faculty development session on Milestones

I did my first faculty development session on milestones/EPAs (see prior post).  It was an hour long session.  The first half of the session included faculty members, residents and medical students and the second half was just faculty. 

Key Insights:

1) The assigned pre-work was helpful.  The participants did the pre-work.  It gave people background on the project and terminology and allowed us to start the session with a discussion of what surprised people when they rated themselves on the milestones.  It also allowed people who couldn't attend the session to get some background on the topic.

2) Both faculty and residents thought that the milestones used dense "edu-speak" and that it wasn't obvious what was meant by terminology such as "semantic qualifiers" and "illness scripts".  It was suggested that a glossary of terminology would be helpful. 

3) Many faculty assumed that they would be at level 5 (expert) that was not the case when they rated themselves on some of the milestones.   It will be a "culture shift" for faculty and residents to realize that it is ok to not be marked at the "top" of the scale.  It also brought up the question of the "expert" descriptions and if they represent ideal rather than reality.  It would be interesting to have  multiple experienced faculty rate themselves on the milestones and to see if there are certain milestones with a lower percentage of faculty rating themselves as expert.

4) It can be difficult to decide where a resident should be on a milestone.  There was an acknowledgment that a resident may sometimes be at an advanced beginner level and sometimes at a competent level on a milestone such as history taking depending on their familiarity with the presenting chief complaint of the patient.

5) Some of the milestone descriptions are not observable behaviors but rather require "getting inside the head" of the residents to discern their thought process and those milestones will be much more difficult to assess.

6) Residents discussed that that faculty will need to be observing them more and discussing cases in a different way in order to judge where they were in the milestones in a meaningful way.  With multiple rotating faculty, residents wondered how such assessment will take place.

7) Feedback was also discussed by the residents.  Most did not feel that they got much day to day feedback regarding their  skills except after a faculty had done a direct observation (structured clinical observation).  They appreciated some of the inpatient services who did feedback on a weekly basis but said that when the person giving the feedback had only worked with them for a day or two, the feedback was less valuable.

8) It was helpful to have residents and students at the session to understand their experience with assessment and as a way to introduce the new assessment system to them.  It was also helpful to have some time with just the faculty to discuss the logistics of the new system.

9) "Snapshots" of a learner at different levels (novice through expert) for an entrustable professional activity was helpful but will need more discussion and time than half an hour.  After discussing a single snapshot, faculty did not agree when actually evaluating a resident.

10) In the supervision guidelines for entrustable professional activities, "direct supervision" of advanced beginners is needed.  Our faculty did not agree on what "direct supervision" meant.  All agreed that it included repeating a physical exam and key pieces of history.  Some faculty wondered if it meant doing direct observation of the entire encounter including information sharing prior to discharge.  Since this is a recommendation coming from a national organization (Academic Pediatric Association), it would be helpful to have clarification on what is meant by direct supervision and reactive supervision.

11) When we reviewed the entrustable professional activities for our rotations, it was clear that some occur multiple times in a two week rotation (such as provide well child care) and others are less frequent (develop an evaluation and management plan for behavioral, developmental or mental health concerns).  It may be helpful to have a resident "passport" to record feedback on some of the EPAs.

12) All faculty agreed that in order to implement the milestones/EPAs we will need to re-assess how we currently structure the role of the precepting attendings.   Do our current staffing patterns meet the increasing intensity of educational assessment? Probably not.  For example, at Stanford, some faculty are being given 10% time in order to observe residents over time and in different settings on the milestones.  It is not known how much additional faculty time will be needed to effectively implement the new evaluation strategy and who will pay for that time.  It would be helpful if on a national level different programs shared models of faculty assessment strategies.

Assessing Residents in the Era of Entrustable Professional Activities and Milestones

The Accreditation Council for Graduate Medical Education (ACGME) is implementing the "Next Accreditation System"(NAS) for medical and surgical post-graduate training programs in the United States.  As a part of NAS, programs will be reporting the progress of their residents on milestones of clinical competence that have been developed by each specialty.  The milestones for specialties that will begin reporting this year can be viewed at http://www.jgme.org/toc/jgme/5/1s1.

With the continuing shift to outcomes-based assessment, there is a need for ongoing education of both faculty and residents about this evolution in assessment.  Pediatric residency programs will begin reporting on 21(of the 48) milestones in June 2014. 

Below is a proposed method for education about the Milestones that I am going to pilot for my department (ambulatory pediatrics).  Challenges for education in our department include faculty working in different locations/ different shifts and variability in familiarity with the milestones.

After reviewing the proposed faculty development plan, please feel free to comment and offer suggestions.

Proposed Faculty Development Plan regarding new resident evaluations.

Independent Learning (estimated time 30-45 minutes)
  
1) Review the 15 minute narrated PowerPoint- overview of the milestones and entrustable professional activities (EPAs). Video uses Windows Media player.

2) Do a self- assessment using selected milestones to become familiar with content and method of assessment.  Please complete PC-1,4,5, MK-1, PBL-4, ICS-1, PROF-1, SBP-1,3 of Pediatric Milestones http://www.acgme-nas.org/assets/pdf/Milestones/PediatricsMilestones.pdf

3) Additional readings (optional)

a) CL Carraccio, JB Benson, LJ Nixon, PL Derstine "From the educational bench to the clinical bedside: translating the Dreyfus developmental model to the learning of clinical skills" Acad Med 2008;83:761-767
http://journals.lww.com/academicmedicine/Fulltext/2008/08000/From_the_Educational_Bench_to_the_Clinical.17.aspx

b) PK Hicks et all "The Pediatrics Milestones: Conceptual Framework, Guiding Principles, and Approach to Development" Journal of Graduate Medical Education, Sept 2010; 410-418
www.jgme.org/doi/pdf/10.4300/JGME-D-10-00157.1

c) JR Frank et all "Competency based medical education: theory to practice" Medical Teacher 2010; 32:638-645.
http://groups.medbiq.org/medbiq/download/attachments/15991574/Frank+et+al.+-+Competency-based+medical+education+theory+to+practice.pdf?version=1&modificationDate=1327513526000


Group Learning Session (60- 90 minutes.  Independent learning above done as "pre-work")

1) Guided discussion of the evaluations for our rotation and review of "Snapshots" to provide examples of what novice, advanced beginner, competent, proficient and expert learners would look like for different EPAs.

2) Practice as a group evaluating current or recent residents using the new evaluation system.  Discuss areas that are difficult to evaluate or where there is disagreement.

(For faculty that are community-based we will need to explore additional opportunities for group discussion and learning via webinars or other on-line opportunities)

On-going Application ("post-work" 60 -90 minutes).  You may pick either option. 

1)Attendance at group evaluation sessions of residents (in person or call in)

2) Implement a Plan Do Study Act cycle for an assessment method for the milestones (examples- resident self-assessment on a milestone, chart stimulated recall).   Discuss outcome and lessons learned at monthly provider meeting (or alternate way to share learning for community based physicians)

Faculty would document their work on resident assessment as a part of their annual departmental goals and objectives and would receive institutional faculty development credit for their work. 

Reflection on Open Learning

This is my final reflection for the Open Learning course (and my first attempt at a narrated video).  I've learned a lot and will continue to think about and refer to the resources in this course.  It has shifted my perspective, especially in considering what roles learners can have in open education. Thank you to Martin Weller and his team for the course and for having it continue as an OER.  And thanks to all those in the Google+ community that I learn from and learn with.

Three Key Issues for Open Educational Resources #H817

What is meant by Open Educational Resources (OER)?

“Open provision of educational resources, enabled by information and communication technologies, for consultation, use and adaptation by a community of users for non-commercial purposes”- UNESCO

“Open Education Resources (OER) are teaching and learning materials freely available for everyone to use, whether you are a teacher or a learner. This includes full courses, modules, syllabi, lectures, homework assignments, quizzes, lab activities, pedagogical materials, games, simulations, and many more resources contained in digital media collections from around the world.” –OER commons

The free sharing of teaching and learning materials that is now possible because of advances in digital technology also has a number of challenges.  I am looking at these challenges through the lens of medical education.   

1)     Accessibility of Open Learning Resources

Hatakka described a number of barriers for the reuse of OER in developing countries including finding suitable material and having difficulty accessing material because of lack of technical resources (computers, bandwidth, internet access) (1)  The multitude of resources in many different places can make it difficult to know where to look for resources in developed countries as well.  Increasingly there are repositories of OER for a particular discipline.  For example in medicine, one could go to Med Ed portal or to African Health OER to look for teaching materials.  The lack of technical resources is more challenging as it relates to resources and poverty.  Technology however continues to advance and mobile phone technology may offer additional options for accessing information.

2)     Adaptation of Open Learning Resources

Learning resources often need to be adapted because of language, cultural or situational context, difficulty level or context.  Questions that arise include:

Can the learning resource be adapted?  This question includes whether the creator given permission for adaptation through a mechanism such as Creative Commons licensing.  Creative Commons is used in Med Ed Portal to allow for free adaptation of resources while acknowledging the original creator.   This question also refers to whether the technology and the expertise are available to modify a resource.  For example it may be difficult to modify a video while it would be easy to modify a Power Point presentation.

What is the learning value adapting a learning resource?  Learning resources are often adapted by teachers to better meet the needs of their students.  Adapting a resource may also be a learning experience for the teacher or the student. As Leinonen et al described in the Wikiversity paper, focusing on building a wiki, adapting and changing content is itself a learning experience.   Students learn when they create or modify learning resources.

3)     Effectiveness of Open Learning Resources

Much still needs to be learned about what makes a learning resource effective.  McGill et al (page 28) provided a nice mindmap from the National Symposium of Learning Resources Repositories to begin to think about measuring effectiveness beyond “number of downloads”.  These include measures such as satisfaction, quality, benchmarking, cultural change as well downloads, referencing and stories.  As with many things that are worth evaluating, it won’t be easy.

References:

1)     Albright, P. (2005) UNESCO (IIEP): Final forum report. 2008-09-01 http://learn.creativecommons.org/wp-content/uploads/2008/03/oerforumfinalreport.pdf

2)     Hatakka, M. (2009), ‘Build it and they will come? – Inhibiting factors for reuse of open content in developing countries’, in EJISDC - The Electronic Journal of Information Systems in Developing Countries, Vol. 37, n. 5, pp. 1-16 http://www.ejisdc.org/ojs2/index.php/ejisdc/article/view/545/279

3)     Leinonen, T., Vadén, T. and Suoranta, J. (2009) 'Learning in and with an open wiki project: Wikiversity’s potential in global capacity building' First Monday, Volume 14, Number 2 - 2 February 2009 http://firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/viewArticle/2252/2093

4)     McGill, L, Currier, S, Duncan, C and Douglas, P (2008)  Good Intentions: Improving the Evidence Base in Support of Sharing Learning Materials. http://repository.jisc.ac.uk/265/

Relevant  Websites:

http://www.oercommons.org/

http://www.jisc.ac.uk/publications/programmerelated/2013/Openeducationalresources.aspx

http://www.oerafrica.org/healthoer/Home/ConnectwithColleagues/tabid/1868/Default.aspx

https://www.mededportal.org/

A MOOC Comparison #h817Open

#H817Open Week 4- Activity 14

A MOOC Comparison:

 Digital Storytelling (DS106) and E-learning & Digital Culture (EDCMOOC)

In the interest of full disclosure, I was a participant in E-learning and Digital Culture and it was from that course’s twitter feed that I learned about H817 Open Education.  I was interested in doing the comparison because my experience in EDCMOOC transformed the way I viewed e-learning.  I have not been a participant in DS106, at least not yet, but I did go to the DS106 site, reviewed some of the webpages and listened to the professors discuss the creation and results of the first year of the course.

This activity is designed to compare the technology, pedagogy and general approach and philosophy between a  cMOOC (connectivist MOOC) and an xMOOC (cognitivist MOOC) but as with most things, labels don’t capture the wonder and messiness of it all.   These courses were interesting to compare because they both have as their subject “Who we are as Humans in a Digital World?”. 

DS 106 had defined learning objectives: 1)  Develop skills in using technology as a tool for networking, sharing, narrating, and creative self-expression 2) Frame a digital identity wherein you become both a practitioner in and interrogator of various new modes of networking 3)Critically examine the digital landscape of communication technologies as emergent narrative forms and genres.  While the learning goals were set, the content of the course changed and evolved based on the suggestions and skills of the participants.  For example, participants were encouraged to submit their own learning assignments.  As they explored different media for self-expression, one participant developed a DS106 radio station, which became an important part of the course.  The emphasis of the course was on experiential learning through creation using multiple types of media and learning from each other was the expectation.  Indeed the community did create the curriculum as one might expect in a connectivist MOOC.

EDCMOOC suggested that learners develop their own learning goals that might include 1) gaining new perspectives on e-learning; 2) experiencing a MOOC; 3)networking with some of the fascinating people from all over the world who are signed up; 4) experimenting with digital and visual ways of representing academic knowledge; and 5)exploring the connections between education, learning and digital cultures.  The content however was set and learners were encouraged to share reflections through blog posts or social media.  Creation of a digital artifact at the end of the course was the learning assessment to receive a certificate for the course.  Learning from each other was an explicit expectation of EDCMOOC as well through use of the blog aggregator in inclusion of a twitter feed.  In addition other social media such as Google+ and Facebook were encouraged.  A number of students developed personal learning networks, quad-blogged, and used Google hangouts.   Weekly twitter chats were organized by students in which students would mentor others to be facilitators.   There was structure for the content (though no video lectures as is characteristic of Coursera MOOCs) and yet the instructors did create an environment that encouraged collaboration and learning from each other.  Because the structure was different from most Coursera courses, some participants were unsure of expectations and what their role was and the role of the instructors.

These courses differed  in duration and in philosophy of assessment.  DS106 seems to have evolved into an on-going community as well as a course where students are welcome to drop in at any time and contribute.  “First of all, in ds106, there are multiple levels of participation- but most importantly, it is designed so you can pick and choose the when and where. We expect NO APOLOGIES for not being able to participate when other parts of life intrude. There is no concept in ds106 of “dropping out” c.f. Groom, Jim (2010-present), “ds106 is #4life”.  EDCMOOC was a five week course though the social media sites as well as the course archive remain open. 

DS106 did not do any assessment of its open students with the philosophy that what you learn is the reward.  “More than just the cliché sense, ds106 is a community that is made better from the ideas and contributions of the people who come inside that door.  We do not give out badges or certifications, the creations you do, the connections you make with other, and just the experience of challenging yourself to tell stories is its own reward.”  One of the questions for DS106 is what will the evolution of this learning community be. 

EDCMOOC culminated in peer assessment of a digital artifact.  This was initially designed to be an exercise where you would receive a certificate of completion if you submitted an artifact and assessed three other people, it evolved into the possibility of “completion with distinction” if you were scored high enough by your peers.  The certificate of completion and the assessment process were viewed as important components by both the instructors and by many of the participants.  One of the questions for EDCMOOC and for xMOOCs in general is that of assessment : What should one measure and how? Is there a role for peer assessment? For automated assessment?   Assessment is linked to certification especially as some certificates become eligible for college or continuing education credit.

Both courses have some similar questions.  What should the role of the instructor be in a course where the expertise and knowledge of students help to shape or in fact create the course?  Who are the MOOC students and what digital literacy do they bring? 

While not a connectivist MOOC because the learners did not create the curriculum, EDCMOOC did foster community for many, despite using an xMOOC platform (Coursera).  Though some learners were confused by the different format and expectations, many others embraced the experience of learning from their peers.  

	DS 106	Coursera- EDCMOOC
Technology	Web-platform- Independent website with enrolled university students having to do assignments on the University Canvas LMS Multi-media potential with weekly assignments	Web-platform- Coursera framework with add-ons for twitter feed, blog aggregator Discussion forums were set up as standard Coursera framework
◦pedagogy     Assessment	Connectivist On-going community- Rejection of need for badges/recognition. “ We do not give out badges or certifications, the creations you do, the connections you make with other, and just the experience of challenging yourself to tell stories is its own reward”	Connectivist/Cognitive 5 week course though the twitter feed and social media communities are still available as is the course archive Credit based on completion of a digital artifact.  Assessment done by peer review, with distinction a certain grade receiving “with distinction”
◦general approach /philosophy.	Enrolled students and open participants – no distinction initially in course or participation involvement except open participants were not graded. Course changed based on input from community- Expectation of high participation, creation of assignments, student leadership Focus on digital story-telling and establishment of digital identity Set learning objectives with emerging, changing content	Enrolled students and open participants- with enrolled students having to a small degree a “teaching assistant role” and providing examples of digital artifacts Openess re: process from Hangouts with professor and their blogs Encouraged alternate ways of building community via social networking. Students reflected on various digital representations and were grounded in history and theory of digital culture and then were asked to create a digital artifact. Goals to be set by the learner  with stable content
Questions being asked	Interest in analytics of who participates Role of the instructor? What will be the evolution of the community?	Interest in analytics of who participates Role of the instructor? Peer assessment process?

Triple A priorities for Open Education - Activity 4 #h817open

My initial thoughts for activity and research in the area of Open Education

Activity 4, Week 1:  Imagine you are advising a funding organisation that wishes to promote activity and research in the area of open education.  Set out the three main priorities they should address, explaining each one and providing a justification for your list.

Thinking in broad terms, I would advocate research in the areas of Accessibility, Assessment and Accountability.

• Priority One- Accessibility
• Accessibility research would involve assessing who has access to Open Education and who doesn't and for what reasons.   The term "accessibility" often focuses on making web information available to people who have various physical disabilities (for example hearing or visual impairments).  As described by the Web Accessibility Initiative (http://www.w3.org/WAI/users/Overview.html): "Inclusive design, design for all, digital inclusion, universal usability, and similar efforts address a broad range of issues in making technology available to and usable by all people whatever their abilities, age, economic situation, education, geographic location, language".   Barriers to accessibility includes lack of access to necessary technology and/or lack of access to digital information because of low income, geographic or political barriers. 
• Web technology has the potential to assist in overcoming many barriers and to make education more open but also faces barriers.  For example - for distance learning, the Internet makes information available in one's home but only if you live in a place where you can obtain Internet access.  
• Is open education more inclusive of different learners who may not be accessing information through a traditional academic pathway?  Does open education allow for collaboration between people that is different than what occurs in more traditional education? 
• Priority Two- Assessment
• Assessment of learners- What are different ways to assess learners?  What is the role of peer assessment and how effective is it? What are authentic learning assessments in an open education environment?
• Assessment of open education courses/materials- How should open education courses by assessed for quality?  How do you measure effectiveness of open education courses beyond looking at completion rates and satisfaction rates?
• Priority Three- Accountability
• Accountability includes looking at who is responsible for the quality of open education courses.  Is there transparency regarding the process in open education and intellectual property rights/copyrights?  Is there transparency regarding how open education is funded and potential conflicts of interest?

Joining Open Education and developing a learning plan #817open

Somewhat to my surprise, I have decided to take another MOOC (Massive Open On-line Course).  This one is from the Open University in United Kingdom and is on Open Education (http://www.open.edu/openlearn/education/open-education/content-section-0).  I recently completed the E-learning and Digital Culture (EDC MOOC)and I learned about this course from the EDC Google+ group and on Twitter.  I

I am a pediatrician and I am involved in medical education during the clinical years (medical students and pediatric residents).  I learned a tremendous amount through the E-learning and Digital Culture MOOC particularly about the process of learning from peers in social media groups and about the many tools available for on-line education- for content delivery and for collaborative learning and creation.  It was a bit overwhelming and time-intensive, though I really enjoyed it and it "energized" me in my work.   I did wonder if I would apply what I learned in my teaching. 

Open Education #817 is an opportunity for me to think e-learning in a more systematic way and to think about what aspects might be applicable to medical education.  I include in medical education: trainee education, family and patient education, and collaboration/knowledge creation across disciplines and groups.

My Personal Learning Plan for this course includes:

1) Develop a strategy to organize what I am learning.
    - I am still using paper files (or the digital equivalent of Dropbox for Word documents).  I do not have a system for organizing what I learn, what I want to read in the future etc.  I plan to explore the use of Pintrest, Diigo, Evernote and Pearltree, both for my own organization and as teaching tools.  First stop is http://pinterest.com/twahlert/dropbox-evernote-diigo-in-education/

2) Utilize social media to learn from peers
   - I joined Google + and Twitter six weeks ago as a part of the EDC MOOC.  I found it to be a wonderful community and way to learn.  Who knew I would like twitter chats (thank goodness for tweetchat during those frentic chats).  I haven't yet figured out circles, following etc but I will participate in the 817open google + group and have more opportunity to learn how I want to use social media for learning.

3) Use a variety of tools in creating learning objects
   - My technology skills have been for the most part limited to Word and PowerPoint.  I plan to explore a bit more of the many different tools that I was introduced to during the EDCMOOC and to think about when to use various tools (if at all).  The starting list of tools from EDC MOOC is below and I plan to continue to look at the various digital artifacts created-
http://padlet.com/wall/edcmooc_artefact

1. Facebook Interaction Tracker-http://www.zeebly.com/social_me
2. Timeline-http://timeline.verite.co/#
3. Scoopit-http://www.scoop.it/ (Laurie Niestrath)
4. Tiki-Toki Timeline - http://www.tiki-toki.com/ (HB Hessler)
5. Diigo - www.diigo.com (Rick Bartlett) (Laaurie Niestrath)
6. Pinterest - http://pinterest.com (Ary Aranguiz)
7. Glogster - http://glogster.com
8. Youtube- http://youtube.com
9. Ustream - http://ustream.com
10. Infogr.am- http://infogr.am11.Mixbook - http://mixbook.com
11. Storify- http://storify.com
12. New Hive -http://newhive.com
13. Slideshare- http://slideshare.net
14. WebDoc- http://webdoc.com
15. BlogTalkRadio- http://blogtalkradio.com
16. Knovio-http://knovio.com
17. Google Hangouts - record your hangouts
18. Prezi -http://prezi.com (Laurie Niestrath)
19. Voicethread- http://voicethread.com (Ary Aranguiz)
20. Photostory- http://microsoft-photo-story.en.softonic.com/ (Laurie Niestrath)
21. Thinglink - http://www.thinglink.com/ (Kay Oddone)
22. Animoto - http://animoto.com
23. Piktochart - http://piktochart.com
24. Wix - http://www.wix.com (Jono Purdy)
25. Popplet - http://www.popplet.com (Jono Purdy)
26. Animaps - http://www.animaps.com (Jono Purdy)
27. Museum Box - http://museumbox.e2bn.org (Jono Purdy)
28. Sqworl - http://www.sqworl.com (Jono Purdy)
29. Popcorn Maker - http://popcorn.webmaker.org (Jono Purdy)
30. Ipiccy - http://ipiccy.com/ (Anne Robertson)
31. Sketchguru - free android app (Anne Robertson)
32. Picmonkey - http://www.picmonkey.com/ (Marina Shemesh)
33. Wordle - http://www.wordle.net/ (create word clouds) (Marina Shemesh)
34. Adobe Captivate - http://www.adobe.com/products/captivate.html (authoring tool) (Madhura Pradhan)
35. Articulate Suite - http://www.articulate.com/ (authoring tool) (Madhura Pradhan)
36. Storybird - http://www.storybird.com (Cristina Silva)
37. ImageChef - http://www.imagechef.com/ (Cristina Silva)
38. Dipity - http://www.dipity.com/ (Cristina Silva)
39. Livebinders http://www.livebinders.com/ (Eileen Lawlor)
40. Videoscribe: http://www.sparkol.com/videoscribe.php (Angela Towndrow)
41. PearlTrees: www.pearltrees.com (Cathleen Nardi http://bit.ly/14iwRkM)
42. SlideRocket- http://www.sliderocket.com/ (Annie Oosterwyk)
43. Meograph- http://www.meograph.com / (Annie Oosterwyk)
44. Wallwisher: http://www.wallwisher.com (Ora Baumgarten)
45. Trello.com: https://trello.com/ Organize anything, together ! ( gianni buspo)
46. Mahara https://mahara.org/ (Linda Pospisilova)
47. Jing www.techsmith.com/jing.html (Kay Oddone) - Screen capture and screen casting tool - great for creating tutorials!