Digest 22. 360 degree feedback: What makes it more reliable?
Image by akiragiulia from Pixabay
Organizational growth is related, in part, to employees’ development and to help employees grow, evaluations are necessary to find the gaps that indicate which areas should be improved. Ideally, this requires evaluations to be accurate and as objective as possible. Accuracy can be achieved via the use of reliable measures. However, even when using such reliable scales, raters are prone to subjectivity as any evaluation is unavoidably subjective. To make the most of this subjectivity, 360 degree feedback/appraisal, also known as multi-source feedback, can be adopted.
In 360 degree feedback, instead of solely relying on a single source for evaluations, multiple sources are involved. Sometimes this implies having individuals self-assess themselves in addition to receive evaluations from their supervisor. A study conducted by van der Heijden and Nijhof (2004) shows that self-ratings are on average higher than supervisors’ ratings. While one reason of such discrepancies could be due to the fact that normally individuals rate themselves favorably, van der Heijden and Nijhof (2004) argue that these differences in evaluations could be due to cognition biases or errors. Supervisors/raters could be victims of the halo or horn effect, when they allow one trait, either good (halo) or bad (horn), to extend to other traits and behaviors and affect the overall assessment of the employee/ratee. For example, a Harvard graduate could be considered as perfect regardless of his/her performance, simply due to the prestigious institution from which she/he graduated. Also, an under-sampling effect can occur when supervisors do not have enough information about their employees and use stereotypes instead to complete an assessment. For instance, not having other information, assuming that since one employee is old, he/she underperforms (check out Digest 8 for more insights on how to give feedback to different age groups).
This study clearly shows the need for having more visual angles over an individual’s performance, which can bring more or different information. In a nutshell, to have more reliable evaluations, 360 degree feedback/appraisal can be used to collect performance evaluations from various sources such as supervisors/managers, subordinates/employees, peers/coworkers, and clients/customers. This should ensure a more holistic view of performance and qualities of individuals (see Digest 19 for impact of coworker feedback). Reliability of feedback is especially crucial when evaluations are used with developmental purposes. Strong negative emotions will be experienced by individuals when they receive unreliable feedback on their personal qualities, and even more negative in the context of social comparisons (see Digest 14 for more information on social comparisons). Now the crucial question that may arise is: what’s the magic number of raters?
How many raters are needed to achieve reliable evaluations?
To answer this question, Hensel, Meijers, van der Leeden, and Kessels (2010) conducted a study collecting data from a strategic human resource development (SHRD) training program. There were 22 trainers/supervisors each leading one training group that included12 people (managers and professionals) from multiple Dutch organizations. The training program consisted of 12 days and 8 evenings over a six-month period.
Each participant was required to set developmental goals for themselves (e.g., develop supportive leadership style, increasing cognitive flexibility to manage conflicts). Then, the trainer/supervisor and other 11 participants/peers had to rate the individual and provide feedback on his/her capacity to bring about the development of competences related to the goals and on his/her motivation to develop those competences. To try to reduce biases, participants were taught about the cognition errors mentioned earlier.
Hensel and colleagues (2010) found that having more raters results in higher reliability of peer ratings and higher levels of agreement between supervisors and peers. Moreover, they found that in order to have a high reliability level (which is conventionally set at between 0.7 and 1) for ratings of individuals’ capacity to develop competences, ten raters are needed; while for the same level of reliability for ratings on individuals’ motivation to develop the competences, six raters are needed.
Infographic created by REAL PAL - Reliable evaluations in 360 degree feedback
Organizational implications
When using 360 degree evaluation systems, the following recommendations may be beneficial to HR practitioners if they are after more reliable ratings:
To make the most of the 360 degree appraisals, formal goals need to be set so that employees can be more focused on improvement areas.
To have more reliable ratings, make sure to include more than 6 peer raters.
Even with six raters, there would be differences in ratings. Employees can benefit a lot from dialogues concerning these discrepancies.
The use of 360 degree feedback with a high number of raters is particularly useful in developmental contexts (i.e., during training programs) where individuals are working towards improving their personal competences and qualities.
Training programs for supervisors and employees on how they can help each other develop their competencies and provide accurate evaluations can also be useful in increasing the reliability of ratings (check out Digest 16 for more information on FOR training and provision of accurate ratings).
——
References:
Hensel, R., Meijers, F., van der Leeden, R., & Kessels, J. 2010. 360 degree feedback: How many raters are needed for reliable ratings on the capacity to develop competences, with personal qualities as developmental goals? International Journal of Human Resource Management, 21(15): 2813–2830. https://doi.org/10.1080/09585192.2010.528664
van der Heijden, B. I. J. M., & Nijhof, A. H. J. 2004. The value of subjectivity: Problems and prospects for 360-degree appraisal systems. International Journal of Human Resource Management, 15(3): 493–511. https://doi.org/10.1080/0958519042000181223