Examining refusal and acceptance sequences employing a data-driven binary rating approach

Abstract

This paper presents the development and preliminary validation of a data-driven analytic binary rating rubric for assessing roleplay-elicited pragmatic performance of learners. The roleplay task consists of two items—one lunch invitation to elicit a refusal and one gift offer to elicit an acceptance in Chinese, both of which involve initiating proposals to return an earlier favor. Responses to return-to-favor invitations or offers in Chinese are challenging for L2 learners as they involve complicated discursive features. To create a practical, reliable tool for evaluating L2 performance, the researchers conducted a discourse analysis of L1 Chinese performance data (N = 22) to identify key discursive features in two domains— Intention (i.e., refusing the return-for-favor intention) and Decision (i.e., refusing/accepting the offer/invitation). A roleplay can receive up to 4 points in both domains, including one point each for orientation/directness, position, modification, and justification, for 8 total possible points. Scores assigned to both L1 and L2 performance according to the rubric reflected high inter-rater reliability. The effect of proficiency was significant on learner scores and interacted with rating domains and scenarios. Lower-level learners scored lower with addressing the interlocutor's return-for-favor intention than with the invitation or offer itself; they also scored lower with refusing than accepting an offer. By focusing on key discursive features rather than exhaustive interactional details, the rubric streamlines the evaluation process while maintaining accuracy and reliability. Future studies may extend the use of the rubric to other speech act sequences and languages and apply it to automated grading systems.