Behind the Wizard’s Curtain: Designing and Developing Intelligent Systems for use in Educational Contexts

Abstract

This thesis concerns the design and development of intelligent systems for use in educational contexts. The work presented took part either side of the Covid-19 lockdowns of 2020 and 2021 which profoundly affected its direction. The earlier chapters, two to four, describe research conducted prior to 2021 and consider system design from the perspective of the system stakeholders and how interface choices may impact on stakeholders’ perceptions of a system’s capabilities.

The latter part of the thesis, chapter six onwards, presents work conducted after the Covid-19 hiatus and is motivated largely by personal experience teaching remotely using video platforms such as MS Teams or Zoom. Stakeholders’ trust and acceptance of the outputs from intelligent systems are a common theme throughout the work. Chapter 5 reviews the literature between 2019 and 2022 spanning either side of the Covid period.

Chapter 1 provides an introduction to the thesis outlining the approach taken, the research aims and objectives and the research contribution.

Chapter 2 provides background to the main concepts presented and discusses the field of Child-Computer Interaction (CCI) with a focus on its development as a discrete research discipline distinct from Human-Computer Interaction (HCI). The chapter also highlights some of the challenges faced when conducting research with children including ethical considerations.

It then presents an overview of Artificial Intelligence and some of its applications, followed by a brief history. It discusses Machine Learning based approaches that serve as support for the supervised learning implementations described in chapters 6, 7, and 8. Some of the building blocks of artificial neural networks, including feed-forward networks, backward propagation, and activation functions are also introduced. These ideas are further developed in Chapter 8 which describes an implementation that develops these concepts.

The chapter concludes by looking at similar work currently being conducted in the field and notes that while other researchers are working on systems to automatically recognise engagement. The work described in the next chapters differs in its scope, intended target audience and methodology.

Chapter 3 considers the deployment of an intelligent system in an educational context that monitors children’s behaviour during interaction with a computer or other digital technology and potentially makes an intervention if it identifies activity that may not be in the child’s best interest. A model is proposed to inform the design of such a system based on the relationship between trust and acceptance. The Trust Acceptance Mapping Model (TAMM) is presented as a tool to indicate the likely success of the intelligent system design.

Chapter 4 explores how design choices regarding an IS’s interface may affect both acceptance of its outputs and perceptions of its capabilities. Two studies are presented both of which introduce children to a Poppy Humanoid Robot. The first study examines how anthropomorphising the system may impact children’s acceptance of its outputs. The children participating in the study perceived that a robot is able to learn while a computer is a rule based technology designed to perform well defined tasks.

In the second study the researcher introduces the Poppy robot in either “humanised” or “robot” form. In humanised form, the robot is referred to as she or Poppy and the children are asked to suggest things Poppy can learn to do. In robotised form, the robot is referred to as it or the robot and the children are asked to identify tasks it can be programmed to complete. The study finds that when the robot was introduced in humanised form, the children were more likely to attribute actions requiring learning or intelligence to it. When the robot was introduced in robot form, the children are more likely to attribute physical activities to it.

Chapter 5 presents a semi-systematic mapping review of the literature on HCI and CCI research related to AI. The terms HCI-AI and CCI-AI are used to describe the intersection between the disciplines. The AI taxonomy developed by AI Watch, the European Union’s service “to monitor the development, uptake and impact of Artificial Intelligence”, is used to classify and map the literature (Samoili et al., 2020).

In reviewing the literature, three approaches are adopted. Natural Language Processing (NLP) is used to perform semantic labelling of the research. The papers are classified by the researcher using the AI domain and subdomains described in the taxonomy. Finally, the research methods employed to produce the research are classified using the same AI taxonomy.

Chapter 6 presents PDLS, a peer observation approach to generate a labelled data set suitable for use in CCI research. The system is evaluated against the usability metrics of, effectiveness, efficiency, and satisfaction and is judged to be both efficient and satisfactory. Validation of its effectiveness is presented in Chapter 7. The CCI principle of Child Participation is central to the PDLS process, which generates labelled data in both a time and cost effective manner. Pupils were surveyed for their feelings on the accuracy of both their own and their peers’ judgments on engagement status after completing the task and expressed their confidence in both these aspects.

It concludes by offering some thoughts that are intended to be helpful to other researchers who may wish to carry out similar studies and proposes the development of a data set that can be used as a resource for members of the CCI community who wish to undertake CCI research on emotion recognition or the application of computer vision to research with children.

Chapter 7 uses two methods to evaluate the accuracy or effectiveness of the PDLS. The first method uses the iMotions software to retrospectively analyse the video data generated. The second method employs expert reviewers to watch the videos captured by the pupils in the PDLS study and record engagement statuses independently of the original decisions.

Where there is an agreement between one or both of the reviewers and the observers original judgment, then the pupil observer’s label is considered accurate. Where there is disagreement, then this is reviewed by the author with the goal of establishing the reasons for the inconsistency. The chapter concludes by discussing the strengths and weaknesses of the system and makes recommendations for its development and improvement.

Chapter 8 provides an overview of a Machine Learning based approach to implementing an engagement classifier for use with children in an educational context. The model described is a variant of a Recurrent Neural Network (RNN) called the Long Short-Term Memory (LSTM) Model and is selected for its ability to process sequences or cycles in the data. The output from the model is a binary classification which characterises the engagement level of the pupil completing the task as either engaged (1) or disengaged (0) and writes the classification to a video output.

In presenting the model the author acknowledges its limitations and it does not represent a production model but rather demonstrates the feasibility of the approach. Although the implementation displays the engagement classification to the video, this is not intended as a preference over the other potential interfaces considered in Chapter 4. As such, the ML model which provides the engine for the implementation of this IS could support multiple embodiments of the system.