Aim: To prepare family practice residents for an international rotation, and simultaneously to address recent mandates for objective standardized evaluation of residents, the authors have initiated a series of standardized patients (SPs) specifically designed to teach and evaluate cross-cultural issues. These cases supplement the authors' existing cross-cultural curriculum.
Methods: Two SP cases, based on actual presentations of patients from the Marshall Islands who presented to the authors' outpatient clinic, are used to provide an ethnographic exploration of other cultural models of illness and to examine the narrative foundations of the illness experience. These SPs were piloted on the intern class of 2002-03.
Results: Increased cross-cultural sensitivity and improved residents' communication skills.
Conclusion: The authors have found that using standardized patients specifically designed to teach cross-cultural issues, is a very powerful addition to existing evaluation strategies. Standardized patients can be designed not only as evaluation tools, but also as teaching tools and group discussion points for sensitive patient care issues.

Introduction

The Family Practice Residency Program at the University of Hawaii requires its residents to undertake two international rotations in the Republic of the Marshall Islands (RMI) during postgraduate years 2 and 3 (PGY-2 and PGY-3). To prepare residents for this international rotation and simultaneously to address recent mandates from the American Association of Medical Colleges (AAMC)¹ and the Accreditation Council for Graduate Medical Education (ACGME)² (see Table 1) we have piloted a series of objective standardized learning experiences (OSLEs) that emphasize cross-cultural sensitivity and are intended to supplement our existing cross-cultural curriculum.

In doing this we are seeking to supplement findings that standardized patients (SPs, defined as people who are trained to portray a clinical scenario for teaching or research purposes)^3,4 are an effective means to assess components of interviewing, physical examination, communication skills and certain clinical tasks.^5,6 We urge other programs to use OSLEs to address the following important issues:

• cultural appropriateness and effectiveness of interventions;
  

• culturally relevant values and practices involved in end-of-life decision-making, including attitudes toward suffering and death, understanding of advance directives and their relevance, and the possibility of unaddressed spiritual concerns of patients and their families;⁷
  

• the accuracy and efficacy of patient-physician communication;
  

• ways to avoid medical errors that may involve cross-cultural misunderstandings;
  

• the possibilities of complementary and alternative medicine use by patients in specific diagnostic categories (including chronic pain disorders and anxiety disorders);
  

• the quality and equity of care delivered to underserved populations.

Despite nationwide expertise and experience with standardized patients, there is a paucity of literature in the use of standardized patients to teach and assess cultural competency. Therefore, we believe that our efforts will prove useful to other departments struggling to develop similar programs.

Methods

To measure more accurately residents' skills in cultural sensitivity, we needed to create a safe, standardized learning environment and use a teaching and evaluation method that more accurately assesses desired skills and qualities for family physicians.

In order to do this in our pilot program, we created standardized patient cases, based on real patient scenarios commonly encountered in our outpatient clinic, seeking to provide cross-cultural training within a clinical framework in the following three areas:
ethnographic exploration of other cultural models of illness;^8,9
qualitative analysis of illness narratives;^10,11
micro-analytic techniques of discourse analysis.^12,13

The evaluation framework

Our assessment instruments are based on the Toronto Consensus Statement¹⁴ and the Kalamazoo Consensus Statement,¹⁵ emphasizing rapport-building, active listening, agenda-setting, information management, appreciating the patient's perspective, and reaching common ground.¹⁶

Our program simulates an entire 15-minute patient encounter and the standardized patients are trained to react to the physician as a 'real' patient might. For example, if a physician is rushed and appears uninterested in the patient, then the patient may not answer the physician's questions very well, because no rapport or trust has been established. In our program, each case and the relevant expectations are adjusted to evaluate what a first-year family practice resident (or whomever is being tested) should be able to perform in a 15-minute office visit. Residents are expected to complete the entire office visit (focused history, physical examination, patient counselling) in 15 minutes, then discuss with a faculty member (who has been watching the encounter on video monitor) for 2 minutes in front of the patient. After the faculty member has taught one teaching point, the resident then has 5 minutes to 'wrap-up' the visit. Depending on the teaching point, this 5-minute 'wrap-up' session may include gathering more information, redoing a portion of a physical exam, or negotiating a more patient-centered and culturally appropriate treatment plan with the patient. After this approximately 25-minute session, the resident then has 15 minutes to complete their progress note and clinic encounter form (on which they are graded). During this time, the SPs also have 15 minutes to complete their evaluation of the resident. After the resident has completed both cases, they also complete a self-assessment form. The final competency scores ('grades') are comprised of scores and written feedback given on the evaluation forms completed by the faculty and the SPs, accurate completion of the progress note and clinic encounter form (billing and coding). The competency scores are then compared with the resident's self-assessment scores. All scores are tracked longitudinally.

The SP program is obviously different from a traditional objective structured clinical exam (OSCE) which comprises multiple testing stations: the student is asked to complete a specific task (obtain a focused history, complete an expanded abdominal exam, interpret an EKG [electrocardiogram], etc.) at each station.

The scores from the SP experiences, in conjunction with the competency-based 360-degree evaluation system (see Addendum) are used to monitor the resident's progress through the six core competencies, over time.

Details of the cases

The two cases used were performed by actual ethnic Marshallese healthcare workers who are familiar with healthcare in the RMI as well as the conditions at our clinic (to which many of the Marshallese patients living in Hawaii come for their care). (see Boxes 1 and 2) They were explicitly instructed to respond to the residents' approach and questioning in traditional Marshallese ways. If they did not feel comfortable with the doctor, they would not share information or would answer 'yes' and 'no' and not elaborate. If the resident used medical jargon, the SPs were told to prompt the resident only once to speak in simpler terms ('what does that mean?'). The SPs evaluated the residents not only on the core competencies, but also in their ability to translate medical jargon into words and phrases appropriate for patients for whom English is not their primary language.

Results

Competency scores were based on the resident's interaction and performance during all parts of the OSLE - interview, physical examination, patient counselling/interaction, presentation to faculty, receptivity to preceptor feedback, accuracy and legibility of progress notes and encounter forms (billing sheet). Scores were derived as follows.

• Upon completion of each session, SPs completed the first side of the evaluation form, circling descriptions that best reflected the resident's performance.
  

• Faculty evaluators had a 'checklist' which was used to generate a feeling of how well the resident did on each portion for immediate management, short-term management and long-term management (if appropriate), case presentation and progress note/encounter form documentation. This checklist was not used to generate 'percent completion', but was helpful to compare how well each resident did in comparison to each other. After reviewing the progress notes and encounter forms, the faculty member completed the evaluation form.
  

• Depending on how many descriptors were circled in each column, the faculty member and SP evaluators then assigned a competency score for each of the six core competencies.
  

• In addition to the competency rating, faculty members and SPs were asked if they would refer a family member or return to this physician for their care. 'No' answers needed to be explained in the comment section.
  

• Standardized patients were also asked to evaluate the resident on their English/non-medical language skills on a scale of 1-9 (9 being the highest).
  

• The resident's self-assessment form is essentially the same as the faculty evaluation form. Currently, residents complete one self-assessment form for both SP cases.

Scores and written comments were compiled for each resident, which was then reviewed, along with the videotapes, by both the resident and faculty advisor. Composite averaged scores (scores by faculty and SPs) are compared with the resident self-assessment score for all faculty members to analyze and to facilitate comparison between residents. General trends are also noted on the summary scores and averages sheet (Table 2).

For these two cases, residents had more difficulty in communicating with these patients. Both SPs commented that the residents often incorrectly assumed that the patient understood more English than they actually did and did not clarify with the patient often enough to assure understanding and proper communication. SP1 (irritable bowel case) was very straightforward, but many residents were quick to assume that a 'yes' answer actually meant, 'Yes, that symptom is there' versus 'Yes, I don't really understand the question'.

Residents who were able to speak without medical jargon and who took time to explain their questions were better able to make a correct assessment of the situation. For SP2 (advanced cervical cancer), she was able to detect which residents felt very uncomfortable and she rated them lower because she felt they were communicating ineffectively. Residents that took the time to explain female anatomy, where the cancer was, how it might be contributing to her symptoms, what the next steps would be and who checked to see how the patient was reacting to the news, were rated very highly by the patient.

Discussion

We found our cross-cultural standardized patient program to be effective in the following ways:
allows for early observation of resident performance in a standardized setting;
allows for targeted interventions to assist the resident in improving their skills in one or more areas;
allows all faculty members to observe resident performance and determine group recommendations for improvement;
helps to identify areas in the residency curriculum which need improvement;
allows for large group-review of different cultural models of illness and wellness; and
allows for 'safe' patient feedback to the resident physicians.

This is in keeping with the use of patient-centered communication skills which may lead to improved patient and physician satisfaction, and better clinical outcomes.^12,16

An unexpected finding was the ability to detect which resident physicians had a tendency to over-rate or under-rate themselves compared to the evaluators' assessment (refer to bold-type results in Table 2). When this was discussed and reflected upon by all of the faculty, this did in fact, correlate with faculty observation of the residents' presentations and self-confidence in the actual clinical setting.

Our 'pilot' cases were intentionally difficult, with multiple psychosocial issues. More advanced and skilled residents should, in theory, be able to elicit more history from SPs who trusted them. While none of our interns were able to elicit everything on the checklist, it was very clear to all observers which residents had better communication skills and rapport; the SPs rated those physicians higher who tended to discuss at least two hidden issues with the patient. Interns were overwhelmed, not by the complexity of the cases, but by the 15-minute time restriction, though they were aware that the rationale for this was to mimic future practice. We did allow 15 minutes after the patient visit for SPs to complete the evaluations and for residents to complete their progress notes. This does not allow for true measure of residents' performances in a busy clinical setting, but the faculty members felt that it was more important for the SPs to have time to adequately assess the residents.

For the Marshallese cases, the core faculty member met with one of the SPs ahead of time to ensure accurate portrayal and issues in the standardized cases. The Marshallese healthcare workers were very pleased with the cases and felt they accurately reflected common medical and social issues for Marshallese, either in the RMI or in Hawaii. They also thought this an excellent way to help better prepare our residents for their cultural immersion/mandatory rural rotations in the RMI in the PGY-2 and PGY-3 years.

We have only had two completed OSLEs (four cases), but in that short time (3 months) have noticed overall improvement in the performance of four of the interns. Their documentation on the progress notes and encounter forms, in particular, were more thorough and complete. We feel that this is probably due to more consistent clinic supervision and feedback by the faculty member, as well as direct feedback on the OSLE progress notes. More of the residents were able to pick up patient cues and address hidden messages, although they had a more difficult time because of the communication style of Marshallese patients.

Conclusion

We feel that our SP program has been successful in meeting our need to evaluate effectively our cross-cultural curriculum, as well as evaluating residents' progress toward achieving the ACGME core competencies. The group discussion, based on case scenarios (problem-based learning) tends to be a more effective curricular method for teaching cross-cultural issues to our residents and faculty members, as well as for teaching other medical topics.

We have found that using SPs to teach and evaluate cross-cultural issues are a very powerful addition to existing evaluation strategies. Standardized patients can be designed not only as evaluation tools, but also as teaching tools and group discussion points for sensitive patient-care issues.

Acknowledgments

The development of these cases was funded in part by a Health resources and Services Administration, Bureau of Health Professions Academic Administrative Units grant award to the University of Hawai'i John A. Burns School of Medicine Department of Family Practice and Community Health.

Addendum

One of the most commonly used methods for evaluating a resident's performance in a given area is through a 360-degree evaluation. 360-degree evaluations consist of measurement tools completed by multiple people who come into contact with the subject, such as supervisors, peers, nurses, clinic staff, and patients and families. Most 360-degree evaluation processes use a survey or questionnaire to gather information about an individual's performance on several topics, including teamwork, communication, management skills and decision-making, and include rating scales to assess how frequently a behavior is performed (e.g. a scale of 1-5, with 5 meaning 'all the time' and 1 meaning 'never'). The ratings are summarized for all evaluators, by both topic and overall.

References

Table 1 Programs that train resident physicians for any specialty are now required by the Accreditation Council for Graduate Medical Education to document educational outcomes and graduate physicians competent in six core areas.2

Table 2 Objective standardized learning experiences summary scores†

back to text

†SP1 and SP2, Marshallese women with irritable bowel syndrome and new diagnosis of advanced cervical cancer. ‡The evaluators' average is listed in bold and is compared to the resident self-rating, in parentheses. R, resident. Rating: 1-4 = below; 5-7 = meets; 8-9 = exceeds expectations. §R1 and R6 excused from the second OSLE. Their scores reflect only the first OSLE cases (Filipino female with poorly controlled diabetes mellitus, hypertension, no drug coverage and Catholic, married, 'hidden' victim of domestic violence). Bold-type results: resident physicians with a tendency to over-rate or under-rate themselves compared to the evaluators' assessment.

Box 1

Box2

back to text