Evaluation of a workplace assessment method designed to improve self-assessment in operative dentistry: a quasi … – BMC Medical Education

Ethical approval was acquired from the ethical committee at Damascus University Faculty of Dental Medicine on January 15, 2022 (no. 98,735).

Study design

This is a quasi-experimental study conducted at Damascus University Faculty of Dental Medicine during the second semester of the academic year 2021/2022, which began in March 2022 and ended in June 2022. Participation in this study was voluntary and confidential, and the reported data does not compromise this confidentiality. Participants provided their consent to participate in the recruiting survey. The assessment method used in this study was piloted in 2021 [11].

Participants and settings

The study was conducted on fifth-year (last year) dental students during the clinical operative dentistry training module in which they had to treat patients in authentic work settings. The required total sample size of participants was calculated using G*Power 3.1 [12] based on the findings of the pilot study [11] for an effect size of 0.64, a power of 90%, and an alpha value of 0.05; the calculated sample size was 28. Participants were recruited in the study via an online survey in which they were also requested to self-evaluate their overall performance in clinical operative dentistry on a 4-point scale similar to that used in the study.

To ensure fairness between participants and non-participants, it was explicitly stated that partaking in the study would not affect their grades in the module which would be assigned based on the traditional assessment criteria of the Faculty as their non-participating peers.

Assessment method

Participants were assessed using the DOPS (Direct Observation of Procedural Skills) method and were instructed to perform self DOPS. In comparison to the pilot study [11], the grading scale was reduced from a 5-point scale to a 4-point scale (1 = clear fail, 2 = borderline fail, 3 = borderline pass, 4 = clear pass) because participants rarely hit the excellent point; second, two assessment criteria namely tooth preparation and restoration were subdivided into 4 and 7 sub-criteria respectively (supplementary file I). The English version is available in supplementary file I. The DOPS forms for clinical teachers and participants were identical except for the addition of item coded III (supplementary file I) to the clinical supervisor’s form which was designed to assess participants’ abilities to pinpoint areas of improvement and excellence. All forms designed and used in the study were in Arabic to overcome the language barrier. A grading rubric was also shared with both participants and supervisors; the rubric provided a detailed description of each intersection between a scale point and a criterion (supplementary file II). All documents went through a double-translation process to ensure the accuracy of the language.

Participants underwent five DOPS assessment encounters in which they assessed their own performance and were also assessed by a clinical supervisor simultaneously. Between each encounter, there was a one-week interval. Participants independently completed the form directly after they finished their procedure (retrospectively), whereas supervisors completed the form directly as participants were conducting the procedure. Participants did not receive feedback from supervisors until they have completed the self-assessment form.

There were three calibrated clinical assessors. Each student was assessed at least once by each supervisor. This was done to increase assessment inter-reliability [13] and mitigate possible bias that might result from participants becoming calibrated to a certain supervisor.

Five assessment encounters were decided to be the minimum number of encounters required by each participant; a reliability study of DOPS found that five encounters achieved a generalizability coefficient of 0.87 (95% CI: 0.59) [14] despite the fact that assessors were not trained. The previous pilot study [11] conducted four encounters and found significant findings; hence, five encounters were set to be sufficient, especially since the assessors were calibrated; further assessment encounters were difficult to achieve due to pragmatic reasons and limited human resources. Data from those who completed less than 5 encounters was excluded from the analysis.

Calibration of clinical supervisors

Three qualified GP dentists were chosen as clinical supervisors. These clinical supervisors had experience in conducting DOPS assessments as they participated as assessors in the pilot study. Moreover, the grading rubric was discussed in detail between the three supervisors to make sure that all agreed upon the meaning of each criterion.

Prior to commencing the study, the inter-rater reliability of the three clinical supervisors was evaluated on three DOPS occasions in which they assessed two different participants conducting three different classical operative procedures on real patients; one participant had done one procedure and the other had done two procedures. This evaluation procedure assessed inter-rater reliability across different cases.

As for the intra-rater reliability, a simulated trainee-scenario paper-based exam was designed. The rationale behind using the simulated scenario is that it can be repeated exactly the same, whereas a participant’s clinical performance can be inconsistent on different occasions. Supervisors had to evaluate the simulated trainee’s scenario using the DOPS form twice with a one-week interval. Thereafter, the intra-rater reliability was calculated for each supervisor.

During the actual DOPS encounters, the clinical supervisors did not intervene unless the participant was totally incapable of completing a certain step of the procedure, and in this case, only, the supervisor assigned the performing participant the lowest grade on that certain skill before taking over the participant and completing the step.

Self-assessment training

A full detailed description of self DOPS assessment protocol was sent to participants along with the DOPS form and grading rubric; all sent documents were in Arabic. An instructional video that explains the assessment process and criteria was filmed and sent to participants; the instructional video demonstrated correct application of the grading rubric and assessment form on scenario cases. These scenario cases covered cases that were clear pass, clear fail as well as borderline fail/pass. Before the clinical assessment encounters and to ensure that participants read and understood the assessment instructions, an electronic quiz was made in which a virtual case was put forward to participants to assess using the DOPS form and grading rubric. After submitting their answers to the quiz, participants could assess the discrepancy in their evaluations compared to the actual evaluations assigned by the clinical supervisors. Further, in line with training students to self-assess, the grading criteria, assessment approach, and basic assessment skills were illustrated to participants face-to-face before the study commenced.

Before each self DOPS encounter, clinical supervisors held a 5-minute feedforward session with participants to discuss the pitfalls they noticed participants making in assessing themselves according to the observations of the previous session [15]. After each Self DOPS, a 5-minute feedback session took place during which clinical supervisors compared their scores with that of the self-assigned ones and thereafter discussed the differences in scoring with participants so that the reason and/or the specific observations that merited a certain score were clear to participants. Further, areas of improvement and excellence were highlighted and an action plan to improve clinical performance was agreed upon. This model of starting a feedback discussion with self-assessment has been widely supported [16,17,18], and it is based on multiple pedagogical theories suggesting that feedback providers should engage in a dialogue with feedback receivers; self-assessment being a good conversation starter [19]. This sequence is especially useful in providing negative feedback as it is easier for faculty to ask participants about what they did wrong and then elaborate in comparison to directly criticizing their performance; the former approach is less pejorative [20]. Further, utilizing a standard assessment form for both participants and supervisors helps in making the content of the feedback provided by both sides closely related; an aspect that previous studies were limited by [20].

Quantifying self-assessment accuracy

Three main methods were used to quantify self-assessment accuracy. First, the mean arithmetic difference between SA and TA scores indicated how much participants overestimated or underestimated their performance at each item. Second, the sum of deviations (absolute differences) between SA and TA scores of each of the 22 items indicated how far SA was from that of TA. The deviation was calculated at the domain level as well. These two methods of quantifying self-assessment accuracy were inspired by a previous study [6].

The third method of quantifying self-assessment accuracy is novel to our study and was illustrated in the pilot study [11]; briefly, on the back page of the DOPS form (supplementary file I), participants and supervisors were asked to separately identify three areas of improvement and three areas of excellence. Thereafter, the clinical supervisors compared their points to that of participants and assessed how many points identified by participants matched theirs on a 4-point scale (0 = no matching points, 1 = one matching point, 2 = two matching points, 3 = three matching points). This variable assessed participants’ ability to identify the most serious areas of improvement and the most prominent areas of excellence in their own performance as perceived by the more experienced and calibrated clinical supervisors.

To examine changes in participants’ performance, the sum of points given by the supervisors for each participant (at each of the 22 items) was calculated and the mean was used as the main indicator of participants’ performance.

Participant’s attitudes towards the assessment method

Participants’ attitudes toward the self-assessment method were assessed after each encounter on the DOPS assessment form in two items (a- and b-, supplementary file I, pg.4).

Data analysis

The skewness of data and histograms were used to examine data normality. Minor violations of normality were disregarded as the sample size is over 15 (sample size = 32) [21]. The intraclass correlation coefficient was used to measure inter-rater and intra-rater reliability. Paired t-test was used to measure the mean difference between SA and TA at each item across different encounters. Deviation (absolute difference) between SA and TA scores was calculated for each domain at each encounter. Repeated-measures ANOVA was used to measure the difference between deviation mean scores of the five encounters; a P-value < 0.10 was considered significant and the confidence level was set at 90% in all conducted statistical tests as per previous recommendations [22]. The significant test was made less stringent considering the human factor as well as the limited assessment encounters. Statistics conducted did not focus merely on hypothesis testing but also estimation; therefore, effect size and observed power were reported whenever possible. Mixed ANOVA was used to test for the effect of sex, percentage grade (PG), case complexity and restoration type on the deviation mean scores in the five encounters. Friedman’s two-way analysis was conducted to measure the difference in the number of matching points between participants and supervisors in regard to areas of improvement and areas of excellence.

Data processing and analysis were conducted using Microsoft Excel (2019) [23] and IBM SPSS Statistics for Windows, version 26 (IBM Corp., Armonk, N.Y., USA). G*Power 3.1 [24] was used to calculate the sample size. Google Forms [25] was used to conduct the screening survey.