A CAT with caveats: is the Consensual Assessment Technique a reliable measure of graphic design creativity?

Jeffries, Karl K. orcid iconORCID: 0000-0002-8936-1800 (2015) A CAT with caveats: is the Consensual Assessment Technique a reliable measure of graphic design creativity? International Journal of Design Creativity and Innovation, 5 (1-2). pp. 16-28. ISSN 2165-0349

[thumbnail of VOR]
Preview
PDF (VOR) - Published Version
Available under License Creative Commons Attribution Non-commercial No Derivatives.

1MB

Official URL: https://doi.org/10.1080/21650349.2015.1084893

Abstract

The Consensual Assessment Technique (CAT) is considered one of the gold standards for creativity assessment, and graphic design, arguably, is the most ubiquitous domain within the creative industries. For the first time, this study tests two tasks to measure graphic design creativity, and by extension, the reliability of the CAT as a measure of graphic design creativity. Initial research suggested the level of consensus amongst judges (often referred to as inter-rater reliability) was too low to be reliable, and may be unduly influenced by a judge’s preference for technical execution. In this study, 16 professional graphic designers were randomly assigned instructions to discount technical execution from creativity ratings, or given instruction that gave no stipulation, for 60 artworks. Inter-rater reliability scores were acceptable for each task and experimental condition, but were higher for judges that received instructions to discount technical execution. These and other results are discussed, and the argument presented that, for future CAT studies in this domain, specific instructions to discount technical execution offers a more reliable measure of graphic design creativity.


Repository Staff Only: item control page