|
The authors reported no conflicts for disclosure. Correspondence to: James Leiphart, [email protected] Abbreviations: ACGME, Accreditation Council for Graduate Medical Education; CLASS Center, Clinical Learning and Simulation Skills Center; GW, George Washington University; SP, standardized patient Received: June 24, 2007 Accepted: Jan. 2, 2008 Key Words: graduate medical education, academic practice, ethics, medical error, training |
The ACGME has mandated that resident competency in six areas be assessed before graduation. These areas are: (1) patient care; (2) medical knowledge; (3) practice-based learning and improvement; (4) interpersonal and communication skills; (5) professionalism; and (6) systems-based practice. However, the ACGME has not provided specific details on how to train or assess residents in these areas. Each U.S. residency program is to develop its own training program and assessment tools to address each competency.
The GW Department of Neurosurgery, in conjunction with the CLASS Center, created and conducted a preliminary test of a novel program to assess residents in one of these competencies, interpersonal and communication skills, in a way that would capture the nature of neurosurgical practice and disease processes. This paper will describe the program and report the preliminary test findings.
The program that was initially developed utilized a methodology for internal review, 360-degree feedback (1). A set of questionnaires was developed to evaluate the residents by all the different groups of individuals with which they interact in the course of their clinical responsibilities, including faculty, patients, and nursing staff. In turn, the residents would have an opportunity to review the faculty, the clinical service, and their own training. These reviews proved effective in generating data for the evaluation of the residents, but there was still no standardized training for the competency other than the training received in the course of clinical practice. In its review of the residency program, the ACGME commented that the 360-degree feedback was too subjective and asked that a more objective assessment tools be developed.
| Abstract The ACGME has instituted a set of six core competencies for residents and is requiring residency programs to provide evidence of effective standardized training through objective positive outcome measures for these competencies. The GW Department of Neurosurgery has implemented a pilot program to meet these requirements for interpersonal and communication skills. Ten neurosurgery residents participated. In three workshop sessions, each resident interacted with a standardized patient; there was a didactic session, and then the residents practiced new communication skills on two additional standardized patients. The standardized patients simulated clinical neurosurgical scenarios and provided feedback to the residents on their communication. In a final session,the residents interacted with standardized patients in the same clinical scenarios as before. Analysis of standardized patient feedback showed that the average feedback scores increased for all residents between the first and second exposures to the same clinical scenario (46 to 50), but this was not statistically significant (Z=0.53, p > 10). The present study demonstrates a pilot program which provided neurosurgery residents with standardized instruction and assessment of interpersonal and communication skills in accordance with the mandate of the ACGME. This program will require further refinement and assessment with a larger number of subjects for further development and implementation. |
The CLASS Center’s resources had previously been utilized by the anesthesia and medicine departments, which had taken their residents through a similar program designed for those specialties. Therefore, many of the fundamental components like the didactic group presentation videos and the general format of the workshops were already established, and the neurosurgery department’s program was able to utilize many of these established resources. The SP cases and the group discussions were altered to be more specific to neurosurgery.
Methods
Ten neurosurgery residents from program years two to seven participated in the program. The participants constituted a convenience sample. Because of scheduling conflicts, not every resident was available for every workshop. The residents attended an introduction and three communication skills workshops given one day a week.
The workshops, Informed Consent, Bad Result, and Cross-Cultural Communication, were selected from a preestablished GW program called CREATE (Cross-Residency Exercises for ACGME Training and Evaluation), which was designed to teach residents interpersonal skills (Table 1). The CREATE workshops last two hours and are divided into a DVD-guided didactic section (Appendix 1) and a practice section with SPs. The didactic DVDs present a challenging communications case for discussion, the skills to address the challenge, and an example of an experienced clinician modeling the skills while interviewing a patient. The skills taught in each workshop are based on expert consensus from the literature. The patient cases were adapted for neurosurgery by the lead author, J.L.
After the DVD presentation, the material was openly discussed by the residents with two facilitating faculty from the CLASS Center and one junior faculty from the Department of Neurosurgery who was able to provide additional neurosurgical context. The subsequent practice section, using cases customized for neurosurgery, allowed residents to try out the skills and receive feedback from SPs and faculty.
The residents were assessed before and after the workshops by SPs who were carefully standardized and trained to evaluate the residents using checklists. Workshop-specific pretests were conducted at the beginning of each workshop and thus were given serially. Before each workshop, residents interviewed the SP who was to present a communications challenge specific to that workshop. The posttest, consisting of three cases corresponding to each of the three workshops (and identical to the pretest cases), was given as a single event three weeks after the last workshop. The checklists for all cases assessed the same set of general communication skills specific to each of the three workshops. The SP scores from the three clinical scenarios in the posttest session were compared to the SP scores from the pretest. The data were “de-identified” prior to the analysis. Institutional review board approval for retrospective analysis was obtained prior to the analysis. The Wilcoxon signed-rank test was used to compare each resident’s average post-workshop score to his or her average pre-workshop score.
Results
Analysis of the residents’ scores averaged across the three SP scenarios (Figure 1) demonstrated a slight improvement in average score from pretest to posttest that was not statistically significant (Z=0.53, p > .10). Analysis of the subset of residents that obtained less than 50 percent on their initial SP feedback revealed a statistically significant improvement from pretest to posttest (Z=2.02, p < .05). In the informed consent and bad result scenarios the majority of the residents received higher scores in the posttest, but in the cross-cultural workshop, such was not the case. None of these differences analyzed by specific workshop was statistically significant.
 |
| Click here to enlarge |
Discussion
These workshops, originally designed for internal medicine and surgery, were adapted to be relevant for neurosurgical residents. The workshops allow residents to reflect on the communications challenge, provide them with literature-based approaches, and demonstrate to them an experienced clinician using these approaches. Highly interactive, the workshops encourage discussion of residents’ experiences and reactions to give the topic personal relevance in accord with adult learning principles. In this process, controversies arise, and residents are encouraged to explore them fully in a safe learning climate. For example, one scenario involved providing compensation to a patient who had experienced a bad outcome (2). This prospect was met with skepticism and resistance from many of the residents because of the assumed potential negative implications in our highly litigious society. In addition to promoting discussion and teaching communications approaches, the workshops provide extensive opportunities for residents to practice skills with SPs and receive feedback. This is a learning-in-action method recommended by the ACGME as one of the most effective for skills acquisition.
 |
| Click here to enlarge |
There are obvious limitations to a small-scale pilot study such as this. Thesmall number of residents involved in this preliminary trial precludes firm, meaningful conclusions from being drawn. Though the overall posttest scores trended toward improvement and the low scorers did demonstrate statistically significant improvement, the null hypothesis that the program was not effective cannot be ruled out. Some residents did not improve their scores, and the performance of some worsened. Also, improvement in the performance of initial low scorers may be an artifact of the statistics explained by regression to the mean. This program needs further evaluation with large enough numbers to power meaningful conclusions. Other limitations include lack of generalizability because the trial was performed with so few residents at a single institution.
We are eager to proceed with further development and study of this program. Review of resident videos suggests that we need to achieve better SP standardization and create better checklist items, with more attention paid to evaluation of nonverbal interaction. Also, we should move toward criterion-referenced scoring, setting the bar at a minimum pass level. Instead of evaluating the residents in a comparative manner, residents must reach a threshold value before advancing, similar to the thresholds that residency programs mandate for resident scores on the board exam. Residents who demonstrate proficiency by achieving this threshold in the pretest will be excused from further training; those who do not will proceed with the workshops and the posttest. We also will explore ways to increase the numbers of residents in the program in order to better evaluate it. Possibilities include presenting the program to neurosurgery residents over a number of years and analyzing the data in aggregate, partnering with neurosurgery programs at other institutions, or combining data across specialties.
With some creative adaptations, this model could be used for teaching and evaluating other ACGME competencies. The communication workshop addresses factors within the professionalism competency, such as compassion; integrity; respect for others; responsiveness to patient needs that supersedes self-interest; respect for patient privacy and autonomy; accountability to patients; and sensitivity and responsiveness to a diverse patient population, including diversity in gender, age, culture, race, religion, disabilities, and sexual orientation. Developing a scale that recognizes these factors would address the professionalism competency. Standardized colleagues rather than SPs could be used to teach and assess the professionalism competency components of responsibility to the profession and society. Standardized colleagues also would make possible the instruction and assessment of the systems-based practice competency. Coordination of patient care, advocating for quality patient care, working in interprofessional teams and implementing potential systems solutions are all components of the systems-based practice competency that could be assessed through a program like this.
Conclusions
The GW pilot program of standardized instruction and assessment of interpersonal and communication skills, in accordance with the mandate of the ACGME, incorporates learner-centered training and evaluation methods that may be of interest to other institutions. These methods include interactive DVD-based workshops, challenging neurosurgery communications cases, and the use of SPs for training and evaluation.
Preliminary program evaluation suggests that some residents, especially low scorers on the pretest, may improve as a result of the program. However, the difference between resident pretest and posttest scores was not statistically significant. Definite conclusions are not possible because of the small number of participants. Based on video review and resident feedback, the program will be refinedand reassessed.
References
1. Eichinger RW, Lombardo MM: Patterns of rater accuracy in 360-degree feedback. Human Resource Planning 27(4): 23–25, 2004
2. Wu AW, Cavanaugh TA, McPhee SJ, Lo B, Micco GP: To tell the truth: ethical and practical issues in disclosing medical mistakes to patients. J Gen Intern Med 12: 770–775, 1997
Special to the Online Edition