Lectura crítica: How to set the bar in competency-based medical education: standard setting after an Objective Structured Clinical Examination (OSCE)
Dwyer, T., Wright, S., Kulasegaram, K. M., Theodoropoulos, J., Chahal, J., Wasserstein, D., … & Ogilvie-Harris, D. (2016). How to set the bar in competency-based medical education: standard setting after an Objective Structured Clinical Examination (OSCE). BMC medical education, 16(1), 1.
- ESPECIALIDAD Educación médica, Docencia universitaria, Evaluación por competencias
- RELEVANCIA 6/7
- NOVEDAD 6/7
Comentario crítico
Este es uno de los papers sobre docencia más técnicos entre los que he escrito un comentario crítico. Versa sobre una cuestión clave para la docencia y el aprendizaje de nuestra profesión clínica: cuál es el nivel competencial que debería exigirse a un estudiante, a un fisioterapeuta novel y a un experto ante un mismo problema clínico dado.
Específicamente, en este estudio se intenta establecer el nivel que un clínico he de demostrar ante un ECOE/OSCE, una de las pruebas habituales para evaluar competencias en Medicina; pero su metodología y conclusiones se pueden extender al mini-CEX y otros tipos de evaluación
Methods should also be supported by a body of published research, be transparent, easy to implement and easy to explain – such methods justify the final result
La conclusión más relevante para mí es la siguiente:
- El diseño, el nivel y la reproductibilidad de las pruebas de evaluación por competencias se puede ver estadísticamente refrendado, siempre que se siga un modelo de trabajo adecuado
Otras implicaciones que sugiere el estudio y que pueden ser interesantes para el diseño de este tipo de pruebas:
- No existe un solo método de calcular el nivel deseado, aunque parece que el objeto de investigación de este paper era determinar la capacidad del método de Angoff modificado
- Para lanzarse a realizar un estudio similar los participantes deben estar bastante familiarizados con el tipo de prueba elegida (OSCE, mini-CEX…)
Puntos fuertes:
- El artículo nos ofrece un método relativamente sencillo de trabajo para realizar esta tarea docente
The advantages of the modified Angoff method include its relative simplicity, as well as the fact that the pass/fail standard can be set before the OSCE is undertaken
Puntos débiles:
- No queda suficientemente claro si el método es útil para discriminar niveles competenciales bajos
Futuras investigaciones:
- Realizarlo con otros Profesionales Clínicos de la Salud
A continuación tienes el Abstract del artículo, pero si pinchas arriba en el logo del distribuidor podrás acceder a la lectura completa. Espero que disfrutes de la lectura del mismo y que estos comentarios ayuden a tu pensamiento crítico.
Abstract
Background
The goal of the Objective Structured Clinical Examination (OSCE) in Competency-based Medical Education (CBME) is to establish a minimal level of competence. The purpose of this study was to 1) to determine the credibility and acceptability of the modified Angoff method of standard setting in the setting of CBME, using the Borderline Group (BG) method and the Borderline Regression (BLR) method as a reference standard; 2) to determine if it is feasible to set different standards for junior and senior residents, and 3) to determine the desired characteristics of the judges applying the modified Angoff method.
Methods
The results of a previous OSCE study (21 junior residents, 18 senior residents, and six fellows) were used. Three groups of judges performed the modified Angoff method for both junior and senior residents: 1) sports medicine surgeons, 2) non-sports medicine orthopedic surgeons, and 3) sports fellows. Judges defined a borderline resident as a resident performing at a level between competent and a novice at each station. For each checklist item, the judges answered yes or no for “will the borderline/advanced beginner examinee respond correctly to this item?” The pass mark was calculated by averaging the scores. This pass mark was compared to that created using both the BG and the BLR methods.
Results
A paired t-test showed that all examiner groups expected senior residents to get significantly higher percentage of checklist items correct compared to junior residents (all stations p < 0.001). There were no significant differences due to judge type. For senior residents, there were no significant differences between the cut scores determined by the modified Angoff method and the BG/BLR method. For junior residents, the cut scores determined by the modified Angoff method were lower than the cut scores determined by the BG/BLR Method (all p < 0.01).
Conclusion
The results of this study show that the modified Angoff method is an acceptable method of setting different pass marks for senior and junior residents. The use of this method enables both senior and junior residents to sit the same OSCE, preferable in the regular assessment environment of CBME.
Keywords
Competency-based medical education, OSCE, Angoff method
SlowPT 