International Journal of Child-Computer Interaction

We present Tick Box Design, a rapid co-design method for research and industry that allows users to gather many design ideas from large numbers of participants in a limited time whilst adhering to ethical principles around users understanding their contributions. The method is based on a design workshop model and can be packaged for delivery by remote teams making it well suited for distributed PD work. In this paper we describe an instance of the method in which 198 teenagers in one country, remotely contributed design ideas for a team in another country, across four rapid 60-minute workshops. In a systematic evaluation of the workshop, we take the needs of both sides into account, the teen participants, and the design team. We explore the participants’ ability to contribute ideas and the usefulness of these ideas to the design team. We show that the teenagers successfully participated in the activities and that the process delivered ideas that were useful to the design team. We discuss our evaluation in the context of ethical and useful participation of minors in HCI research and conclude that Tick Box Design is an efficient method that can be packaged for remote use and delivers value for designers and participants. ©


Introduction
A core tenet of User-Centred Design (UCD) is that a focus on users will result in an improved user experience (UX) of a final product (Gulliksen et al., 2003).Within the Child Computer Interaction (CCI) and HCI communities, researchers and designers are constantly seeking new ways to engage with users.Whilst few would argue that rich participatory design is an optimal way to really understand user needs, there are many instances where design teams and researchers want to get rapid contextual information from users to confirm or expose design ideas (Lockton & Lallemand, 2020;Séguin, Scharff, & Pedersen, 2019;Swearngin, Wang, Oleson, Fogarty, & Ko, 2020).Software companies and researchers invest considerable time into designing methods and techniques to ensure maximum value from design workshops (Larusdottir, Roto, Stage, Lucero, & Šmorgun, 2019).These pressures require methods and flexible configurations to ensure that the user voices can be heard.With children and teenagers, there has been a concerted effort over the last 15-20 years to explore how they can participate in design.Different configurations of participation result in different outcomes and are based on different philosophical and practical positions.At one extreme are extended co-design events, similar to those practiced by Yip et al. (2013) and Fails, Druin, and Guha (2010), where a small group of children cooperatively develop a single idea over several iterations working closely with software developers and other adult partners.At the other extreme a design team may work with a group (often a school class) of children for a short length of time with each child contributing his or her own ideas.This latter approach aligns well with traditional design workshops which have long been used in industry to gain valuable userrelated insights (Nousiainen, 2009;Ravenscroft, Schmidt, Cook, & Bradley, 2012;Righi & James, 2010;Roussou et al., 2015).Both approaches, small group co-design and larger group workshops, have been used successfully in a range of contexts to support the design and development of products for children; e.g.Wood et al. (2019) and Dylan et al. (2020).In our paper we focus on this second approach where a group of individuals attend only once to a design session in a design workshop format.
Many designers want to access a wide range of users, who classically would be hard to reach using traditional PD approaches.An example is when looking to develop global products where culturally appropriate input is needed (Merritt & Stolterman, 2012).In these instances there are pressures of time (where a commercial product may need to be rapidly designed (Kensing & Blomberg, 1999;Pilemalm, 2018)), distance (where travel to user groups may be expensive, climate unfriendly, time consuming or limited for health concerns (like COVID-19), (Gumm, Janneck, & Finck, 2006;Mastrianni, Kulp, & Sarcevic, 2021)), and access, for example where one or more of the team may find direct access to children difficult or where children may speak in different languages (Hirom et al., 2017;Robertson & Wagner, 2012).When running courses for researchers in HCI and CCI common questions that are asked are: 'How can I work with children?How can I get into a school?'One solution to this dilemma is for a design workshop to be delivered by a third party to a group of children 'on behalf' of the company or organization.The method we describe can be organized by local stakeholders or actors on behalf of a design team (Constantin et al., 2021).
In all design activities, but especially when working with children and young people, the value of running design workshops must be considered.Workshops need to balance the requirements of the recipient of the design ideas, and the participants.There is a need for ethical symmetry (Christensen & Prout, 2002) that raises fundamental questions when evaluating design workshops with children: What does the design team gain?What do the participants gain?In a rapid event, as is common in a design workshop, the value of an individual contribution can be challenging to appreciate, yet, in the right circumstances, many 'small' contributions might have a useful impact.We explore the value of short rapid multi people events in this paper.
We present a design workshop method (Tick Box Design), which is well specified, bounded and controlled that can provide useful insights for design teams.Tick Box Design is based on the authors' experiences of running similar fast and furious design workshops (Anonymous) with young people, but it extends earlier work by both exploring value and by showing how such a technique can be used 'at a distance'.Through our work with 198 teenage girls, we show that Tick Box Design offered value to participants and designers and contributed useful design ideas.The method, and associated evaluation, is a valuable addition to the toolbox of design methods for use with young people.

Participation of children in design
In HCI research and design, participation of users is generally considered valuable and worthwhile.This participation has traditionally been positioned with a focus on democratization and end user involvement in the design processes in line with the Scandinavian traditions of participatory design (Bødker, Kensing, , & Simonsen, 2004;Ehn, 2017;Frauenberger, Good, & Keay-Bright, 2011;Islind & Lundh Snis, 2018;Iversen & Smith, 2012;Moffatt, McGrenere, Purves, & Klawe, 2004;Van Mechelen, Zaman, Bleumers, & Mariën, 2019).Participation of users in design is noted across contexts, like ubicomp (Hornecker et al., 2006) and games design (Chomutare, Johansen, Hartvigsen, & Årsand, 2016), and with different populations, including the elderly (Lindsay, Jackson, Schofield, & Olivier, 2012) and the displaced (Fisher, Yefimova, & Yafi, 2016).Ehn outlined two values that motivate HCI researchers to include users in design -one being the social and rational idea of democracy, the other being the importance of bringing the participants' 'tacit knowledge' into the design process (Ehn, 2008).
The practice of involving children in design is relatively new with its roots in the early work of Scaife, Rogers, Aldrich, and Davies (1997) and Druin, Stewart, Proft, Bedersen, and Hollan.Research has since focused primarily on describing new methods and techniques for use with children that take account of the technology context and the ages and abilities of the children (Fails, Guha, & Druin, 2013).Walsh, Foss, Yip, and Druin (2013) describe a design method as a ''collection of techniques used in conjunction with a larger design philosophy'', and a design technique as an activity that a design team participates in.There are many techniques used in design with children; there are considerably fewer methods.
Methods that are clearly delineated as such (and there is considerable debate) vary across a range of axes and are described in different frameworks that apply a design philosophy to the understanding of different approaches.The FACIT Framework from Walsh et al. (2013) is inherently practical considering aspects like design goal and the children's ability as a way to position different methods and techniques.The Early Design Framework from Sluis-Thiescheffer, Bekker, Eggen, Vermeeren, and de Ridder (2011) studied outcomes as a way to think about design, and the Content, Context and Engagement Framework from Mazzone, Iivari, Tikkanen, Read, and Beale (2010) used broad dimensions to position a range of methods, some specific to children, others more universal.Each of these frameworks takes a different approach towards positioning design methods.The extent to which adults co-engage with the children is common to all of them, this has been simplified in the informant balanced facilitated scale as proposed by Read et al. (2002).
Co-operative inquiry (often simply referred to as co-design in CCI) is a well-known method in CCI that relies on there being adult facilitators who act as co-designers alongside children.This method is very commonly used in the US where it has been developed over time to include mobile (McNally et al., 2018) and marginalized groups (Walsh, Donahue, & Pease, 2016).Many codesign projects use teams of children recruited into out of school groups (often referred to as KidsTeams (Walsh & Foss, 2015)), who work with the researchers over several different projects and thus gain their own expertise in the design techniques and philosophy (Yip et al., 2013).Variations have included Distributed Co-Design (Walsh & Foss, 2015) which is described alongside a novel technique of layered elaboration (Walsh et al., 2010).In this case adults are very active in their involvement.
The BRIDGE Method described by Iversen and Brodersen (2008) is less adult facilitated than co-design and is typical of many workshop methods described in the literature.BRIDGE describes a process where children contribute as a community of practice at their own level.Three techniques are described that suit the method; the KidReporter technique from Bekker, Beusmans, Keyson, and Lloyd (2002) is given as a useful example of a technique that has been especially designed for children, another is Mission from Dindler, Eriksson, Iverson, Lykke-Olesen, and Ludvigen (2005), a child friendly activity where children describe a design to a Martian.
The Bluebells method, described by (Kelly, Mazzone, & Read, 2006) is a method that is positioned firmly towards the child as informant and is focused on technology build.Bluebells intersperses researcher activity with children's activity in three stages with four techniques mentioned.In this method the adults step away from the children as they contribute designs.
For large groups of children, the latter two of these approaches are the easiest to manage but as a design session becomes more workshop based, shorter, and with more participants than codesign, there are questions to be asked about how to refine methods to make the involvement of the children valuable and keep motivation high (Dodero, Gennari, Melonio, & Torello, 2015;Read, Fitton, & Horton, 2021).

Distributed design
Whilst participation of users is often considered ideal, there are many challenges to this practice especially when the users are children.Access to children typically requires work in either homes or schools which may not be accessible to designers with physical disabilities (see Andrews (2005) who writes 'whilst undertaking the fieldwork, in that as a wheelchair user I was constantly plagued by an often hostile, inaccessible physical environment').The adult team may need to go through training, to gather police checks and IRB clearances which may be problematic for students who have criminal convictions (Pager, 2008) or live only in temporary residences (Robertson, 2011), and the children, who are going to be the recipients of the designed artefact, may be located many miles away from the design team.This latter situation is quite common in HCI work where many researchers seek to design artefacts for populations across the globe.The same difficulty can apply in the commercial world where designers want to create universally appealing products.In many cases the researchers and developers have individually gone to the locations and worked with the local children (Fisher et al., 2016;Ramachandran, Kam, Chiu, & Frankel, 2007); another solution is to do all or some design work online (Campos, Blikstein, & Azhar, 2017).Distributed Participatory Design (DPD) is a term that has been used since 1999 to capture the remote process of gathering in design ideas (Danielsson, Naghsh, Gumm, & Warr, 2008;Farshchian & Divitini, 1999;Gumm et al., 2006).More recently Distributed PD has been considered from a Child Centred perspective in which different models for distribution were highlighted (Constantin et al., 2021).
In Fig. 1, three different models of DPD are illustrated with two of the three having a human facilitator who works with children on behalf of the researcher or designer.In the literature, these 'removed' options with children include work with 'in-country' research partners (Shahid, Mubin, Al Mahmud, Iftikhar, & Arshad, 2021) and packaging the design experience for a third party to deliver, for example with school teachers (Minoi et al., 2019;Yasir, Abid, & Shahzada, 2019).The difficulties with 'outsourcing' design in this way are generally associated with complexities in ensuring facilitators do not take on too much of the design work whilst understanding what is needed (Carroll, Chin, Rosson, & Neale, 2000), management issues around gathering together the research/ design data, and keeping motivation high (Frauenberger et al., 2011;Gray, 2021).

Evaluating design workshops
When evaluating design workshops and design methods there are several lenses that can be applied.To assist in critical evaluation and bring some consistency to the domain of evaluation of participatory design approaches, Frauenberger, Good, Fitzpatrick, and Iversen (2015) defined a ''tool-to-think-with'' composed of four lenses (epistemology, values, stakeholders, and outcomes).These lenses provoke consideration of the knowledge gained from an event, of conflicts and dilemmas and of benefits to stakeholders which are themes that we apply to our own work.From a more pragmatic position, methods can be evaluated in terms of the extent they can be effective in supporting participants to do design work, Mazzone, Tikkanen, Read, Iivari, and Beale (2012) referred to this approach as being one of considering 'suitability' and 'capability' and suggested this could be measured in terms of how easy an approach was for participants to use and how easily it delivered outcomes.
Ease of use for participants can be measured using self-report or from observations and can also be computed in terms of the percentage of completion of task (Turner, Merle, & Diochon, 2011).Outcomes from design sessions can be evaluated for their appropriateness in terms of how they support the design effort.The effectiveness of a method towards design outcomes is usually judged subjectively through a critique of the ideas generated or an assessment of the end product (Cibrian et al., 2020;Kyfonidis & Lennon, 2019).Most evaluations take a this sort of qualitative approach when discussing the quality of designs (Perttula, Krause, & Sipilä, 2006), but Shah, Smith, and Vargas-Hernandez (2003) proposed four quantitative metrics for the effectiveness of an ideation process from a designer's perspective, covering quantity, quality, variety and novelty of ideas.In Read, Fitton, and Horton (2014) outcomes were empirically evaluated in a consideration of how designs from children were used.This research proposed a method called IDEAS that broke up the design contributions from children into 'ideas' and then later accounted for the ideas in the analysis of the designs to ensure representation.

Balancing benefits
As highlighted by Frauenberger et al. (2015) and many others, in any design activity there is a clear tradeoff between what matters to the participants and what matters to the design team.Hansen et al. (2019) acknowledge that the value in a design activity is concerned with all changes related to project outcomes and impact, including an assessment of the participants' personal gains.Brown, Weilenmann, McMillan, and Lampinen (2016) refer to a value equation where the benefit to the participants is countered against the benefits to the research team.The same authors argue that the more vulnerable a group is the stronger the case must be made for their participation in research.This balancing of benefits and power in a joint endeavour is often referred to as 'ethical symmetry' (Constantin et al., 2021).
In 2010, Bossen, Dindler, and Iversen (2010) interviewed participants three years after they had contributed to a design project to explore what they felt, with retrospection, they had gained from their participation.The study showed that different participants gained in different ways.Expected or anticipated benefits were explored in a study with adults that sought to explore the participants' experience and gains (Garde & Van Der Voort, 2014).In this study questionnaires were used before, during and after the design events to determine changes in the participants' creativity and self-efficacy.A central theme in CCI in terms of evaluating design work with children has been to discover how children feel about participation.Malinverni et al. (2014) used observations to study creativity with children with special needs in a series of workshops, Van Mechelen, Schut, Gielen, and Klapwijk (2018) looked at empathy and collaboration and in a later study, the same authors used survey tools to measure creativity, empathy and collaboration in a co-design session with 16 children (Van Mechelen, Schut, Gielen, & Södergren, 2019).It is important to highlight that participants can value design workshops for the experience and opportunity; experiences include being able to collaborate with a design team in a creative dialogue (Joshi & Bratteteig, 2016;Sanders & Stappers, 2008), having feelings of empowerment (Harrington, Erete, & Piper, 2019;Iivari & Kinnula, 2018), or having the opportunity to collaborate on a project (Van Mechelen, Schut et al., 2019).The designers' counter to the participants' gain is to consider value in terms of design insights gained (Sim, Cassidy, & Read, 2018).Diver and Higgins (2014) suggest that value to participants can, and should be, designed into methods.They position this belief in terms of 'dynamic reciprocity' which is to say that both before and during participation, value should be an area of focus.It is known that, for participants, value is closely associated with the belief that outcomes will be used and so reinforcing that in a design session will add value to participants.Value related to outcomes is hierarchical beginning with the value to the individual, to the small group that that person identifies with, then to a small group the user does not have an affinity with, and lastly in terms of value to the whole community (Rashid et al., 2006).Thus, in a design session in which the users are contributing designs for a product only they will ever use, then the value to those individuals will be high -providing they perceive that their designs will be used.In CCI this scenario is unusual in so far as it is rare for a group of children or teens to be involved in the design of a product that only they will use; more commonly they will perceive the value of their participation in design against the lens of value to a community that they might represent.The RAId protocol (Read, Fitton, Sim, & Horton, 2016), is an analysis method that tracks ideas from children's designs in such a way that they can be 'attributed' to children.This is one way in which children can be made better aware of the value of their contributions.
To consider value in the context of participation in design workshops we apply three evaluation protocols to our design workshop.The first is to determine if the teenagers were able to contribute from an effectiveness/ suitability position, for which we will use the IDEAS method, the second is to establish if the designs from the teenagers could be useful to a design team, which we evaluate with the RAId protocol and the third is to survey participants to gather self-report of their involvement.This will enable the triangulation of results to understand the value of our proposed method.

Tick Box Design
The Tick Box Design method aims to provide a design team with a bounded and structured method that can be used with groups of young people when there may only be a short time (an hour or so) but where there may be a limited number of adult facilitators.It relies on there being a team who carefully plan the event and a team who then run the event.The team running the event may be different from those who have planned the event -this is why we talk about the method being 'packaged'.This packaging ensures that the method is well suited to distributed design where a third party can deliver the event on behalf of a design team.Tick Box Design follows a defined linear process with essential detail being wrapped around a supported design activity (hence the notion of boundedness).This bounding ensures that design ideas are gathered, but also that the young people who participate feel informed and empowered.Tick Box Design is highly suitable for STEM events, distributed design and for novice researchers.
There are five essential stages (the tick boxes) for a Tick Box Design workshop which are as follows: 1. Welcome and Why: Participants are introduced the project, the people involved and told why their participation is valuable.This ensures that value is designed in Diver and Higgins (2014) and that participants can understand what the workshop is for, where the ideas will go and other detail that will enable them to make an informed decision about participation.2. Confirm Confidence: Participants are told that they do not have to be experts in design or drawing for their contribution to be valuable.Their ideas for the design as users are what gives value.The aim here is to actively build confidence (Van Mechelen, Schut et al., 2019), which will overcome uncertainty and illustrate that you do not have to be a great artist to participate.3. Layer Landscape: With a short time to design, it is important that the design team clearly lay out the vision for the system of interest.This helps the participants understand what the context is (Mazzone et al., 2010).4. Drive Design: In the design phase, we use the metaphor of 'driving' to show that the design activity is itself bounded, the participants are placed on a route and there may signs during the design activity that participants are slowing down or coming 'off the route' so the adults need to keep an eye on the activity so they can interject as needed.
For example, midway through design some concepts of design could be explained to the participants, to give them some additional inspiration; this will facilitate success (Van Mechelen, Schut et al., 2019).A mid-session re-focus can be left to the team running the event or can be pre-planned by the design team who have planned the event simply by putting in a specific focus.5. Pack-up and Promise: The participants are thanked and the design group promises to attend to all their ideas and build on them; at this stage feedback mechanisms will also be described and participants get to decide what to hand in Diver and Higgins (2014).
In use, it would be expected that the design specific activities of Tick Box Design (shown in Fig. 2 in yellow) would take up most of the time.Use of the method is described in the next section in terms of a single design 'event' that was 'run' in a distributed way by individuals (local team) at the locality on the design team's behalf.In the event described below a collection of techniques are described that can be adopted or adapted for others using the method.
The event described in the next sections shows how the method is rapid, and it can be delivered remotely on behalf of a design team thus allowing savings of cost (carbon and monetary) and can therefore facilitate access to groups who might otherwise be difficult to work with.From the interpretation of distributed PD from Constantin et al. (2021) -this event is positioned with facilitated large groups as shown within the red central rectangle in Fig. 1.

'Game of Stones' workshop
'Game of Stones' is a project which is seeking to design a mobile game for children and teens that takes them outdoors and can be played anywhere in the world.The project has been ongoing and began with a six week ideation activity in the UK which resulted in a basic game being specified (Read et al., 2021).The workshop event described here is one of three (others have been held but not evaluated; one with children aged 7 and 8 and another with children aged 10 and 11) that contributes to a second version of that game which is richer and more culturally diverse.The game design concept uses small rocks/stones with

Table 1
Terms used.

Term Description
Design Team Researchers from (ANON1) who did not attend the workshop but who were developing the game.This team developed all the materials (planned the event) and later designed, and took part in, the analysis.

Organising Team
The people in (ANON2) who organized the STEM Day at a meta level Local Team Researchers from (ANON2) who ran the workshops on the day and did all the translation of materials.This team later also took part in the analysis.

Design
A contribution from a teenager in the form of a sketch/ sketches of concepts towards the game Idea An identifiable element in a design -either as a thing or as a game concept that could be described and used in another design.These were the things that were measured for the evaluation.
markers on them as tangible component of physical gameplay used in conjunction with an app on a mobile phone.
In the following narrative we use the following terms (see Table 1):

Participants and context
The Game of Stones (GoS) Workshop was included as an event in a STEM Girls and Technology Day, organised by the local university, and held in May 2019 in (ANON2).During the day girls took part in a variety of events that introduced different IT aspects e.g., programming, web design languages, computer games, and 3D printing.On the day, 198 girls (57% were 14 years old and the other 43% were 15) from four different schools attended the GoS Workshop.Each school had its own instance of the workshop (numbered here as 1 -4), so girls were participating in quasi friendship groups although all the work was done individually.Each workshop had a maximum of 50 girls depending on the number from the school.The organizing team choose which schools would attend to each activity, so there was no selection of the participants in the workshops.The four identically organized workshops took place on a single day at the University and each session lasted for 60 min and was held one after the other in the same room with the same two adult facilitators (the local team) who had breaks between the workshops for drinks and lunch.All the content described in Section 4.2, supplied by the design team, was translated into the local language before use.The participants did not get any information about the workshops in advance.
Before the event took place, the design team created a set of resources for the local team.These resources mapped to the Tick Box Design stages and also included materials for evaluation.PowerPoint materials were used for stages 1, 2, 3 and 5 and wireframes, enclosed in a booklet, were used for stage 4.

Individual design and evaluation booklets
To collect design ideas and data for evaluation, and to ensure that the local team could facilitate the event effectively, a Design and Evaluation booklet was constructed in ANON1 which was made up of eight pages and given out as an A4 stapled pack.Section 4 of this booklet mapped to the Drive Design stage of Tick Box Design; the other sections were used to evaluate the workshop.Five of the pages (there were two more wireframe pages and a front page -not shown) are shown in Fig. 3.This booklet both structured and bounded the workshop event.The front and back (Sections 2, 3A, 3B and 5) were used for the evaluation of the event.The centre of the booklet (Section 4) had space for wire frame designs and could be easily separated from the survey data so that participants could hand in one, or both, of the different components.Participants wrote onto, and drew in, the booklet and handed their contributions in at the end of the session if they wanted to.

Workshop procedure
The schedule for the workshop is summarized in Table 2. PowerPoint slides were used in steps 1 and 2 to talk about the research project, clarify what research was, and ensure that participants were clear about what they were about to do (stages 1 and 2 of Tick Box Design).The teens completed Expectations and Skills surveys in step 3 before being given the specifics of the design activity (Tick Box Design stage 3) in step 4.This was followed by another skill rating activity (step 5), and then design (steps 6, 7 and 8 which was stage 4 in Tick Box Design).This was followed by an end evaluation (step 9), that gathered post event impressions (Garde & Van Der Voort, 2014), and a wrap up using PowerPoint (step 10 -stage 5 in Tick Box Design)).
In Step 4 the participants were introduced to the game concept (Layer Landscape).In this case, an initial game was described to the participants as being one in which you aim to be the first to build a character by finding a selection of stones, and scanning them, so you have a body, head legs arms etc. Fig. 4 shows the Design Intro Slides explaining this initial game concept.These same slides were presented in each instance of the workshop, so all participants had something to work from.They also enabled the design team to gauge the novelty of later ideas as all participants had seen the same thing.
Partway through the drawing activity (Drive Design), participants were stopped in their design work for a short explanation about game design based on a single PowerPoint slide which is shown in Fig. 5.
The intention of this interruption was to keep the teenagers on track and give extra impetus.After this interruption they went back to design work before being asked to complete the last sections of the booklet.They then individually decided whether to hand anything in.They could hand in nothing, just the survey data, just the designs, or both survey data and designs.

Evaluation of the method
We evaluated the four instances of the workshop to better understand how to optimise value for child participants and adult designers in sessions such as these.We used two techniques to explore the design contributions and used the survey data from the booklets to explore the value to the participants.

The IDEAS method
Given that there were many teens participating we used the IDEAS method to rapidly determine whether designs had been gathered and if so, in what quantity/ quality.This use of IDEAS at this stage spoke to the concepts of capability and suitability (Mazzone et al., 2010).Evidence of a good number of ideas from a good percentage of the teenagers would confirm that the method was suitable, and the teenagers were capable.The IDEAS analysis that we did was really only marginally focussed on the design-value of ideas as the authors of the technique reported in their paper that this method was not ideal for that purpose (Read et al., 2014).Design-value was considered in a follow-on analysis (Section 5.1.2) Four evaluators (two of whom had been at the design workshops) analysed all the submitted design booklets using the IDEAS method which involved separating the designs collected into  small groups of five or six individual design booklets, identifying three or four (if possible) specific ideas evident in each design (without judging these to be necessarily good or bad or novel).At this point some designs were considered N/A-viz.nothing could be found in the design of any value (see the top design analysis of the middle example (5524) in Fig. 6).Having described ideas in narrative the 'best' idea from each design booklet was chosen (see crosses in the columns marked 'individual' in Fig. 6).The evaluator was then required to highlight as many ideas across a group as the size of the group (so five ideas from a set of five booklets, six from a set of six booklets).The philosophy behind this being that if every participant was contributing with equal imagination and equal design skill one might imagine almost one idea per participant coming through to this stage).In the event, at this point, as seen in the righthand most sheet, a design might end up with nothing to contribute, with one idea to contribute or with more than one idea.
Having completed individual sheets each of the four evaluators made a tally of designs that were N/A, designs that contributed no ideas and designs that contributed 1, 2 or 3 ideas.The team discussed a subset of their individual ratings and on finding the variation of the ratio of contributing design ideas to be similar across all four evaluators, we calculated an average of the ratings.This allowed us to determine what proportion of the designs yielded useful ideas.

The RAId method
Design-value is difficult to determine and requires interpretation and design knowledge.In this case we chose to use the RAId protocol (Read et al., 2016) which enables examination of a set of designs in such a way that ensures all the designs are considered.The way the method works is that several evaluators each look at the same designs, but each evaluator looks at each design in terms of only two from a set of predefined constructs.In this way boredom is limited whilst designs are examined using different lenses.The constructs used in our RAId analysis were taken from the initial game design constructs and were: • Fun: the extent to which a game appeared fun • Do-able; the potential for a game to be made • Outdoor: the potential for the game to promote outdoor play Designs were considered in batches of six or seven and data sheets were prepared for each evaluator with a list of designs, space for comments and space for two ratings -one for each of the two constructs being considered for that specific set of designs.Each set of six or seven designs was considered against different constructs and the sheets were balanced so that different evaluators applied different constructs to designs whilst ensuring all designs were evaluated for each construct.For example, design 5524 might have been rated for Fun and Do-able by evaluator 1, for Environment and Engaging by evaluator 2, and for Do-able and Environment by evaluator 3 etc.In a rapid process, each evaluator rated each design with a number between 0 and 5 (with codes agreed beforehand) and then also selected winners from the batches of six or seven designs, before 'designing' a candidate game based on their own insights and inspired by the designs they had seen.A strength of RAId is that candidate designs, once produced, can be analysed against the data sheets that were created during the RAId activity to identify whether ideas in the candidate designs were unique, were commonly reported or were ideas that the evaluator put in that had not been seen.The ideas in the candidate designs can be coded to the following six categories: • INITIAL: Ideas that came from the initial concept (in our case as specified in the Design Intro Slides -Fig.4) • CORE : New ideas, beyond the initial concept that many teenagers suggested, and more than one evaluator used • ADD-ONS : New ideas, beyond the initial concept, that many teenagers suggested but only one evaluator used • NOVEL: New ideas, beyond the initial concept, suggested by only one teenager that one or more evaluators used • EXTERNAL: Ideas that came from outside the design space These categories for ideas are used when a participant in RAId is presenting their candidate design to others in the design team and reflecting on how that idea connects to ideas they have seen previously.A design method would demonstrate high 'designvalue' if the RAId analysis resulted in a good proportion of ideas being coded to CORE, ADD-ON and NOVEL categories and a relatively low number to the INITIAL and EXTERNAL categories.

Self-report instruments to determine value to participants
The survey instruments used in the booklet were based on research studies into empowerment (Martin & Calvert, 2018) and self-esteem (Rosenberg, 1965).We hypothesized that children and teenagers with high self-esteem might value the session, and their contribution, more highly.Self-efficacy is a perception of how well one can complete a task (Bandura & Adams, 1977) and is not based on any measurable skill (Fiske & Taylor, 2013) so questions relating to perceived ability and contribution needed to be designed into the survey.Within our work we wished to incorporate established practices used within the CCI Community such as the Rosenberg Scale (Rosenberg, 1965).As such, and in common with other work (Garde & Van Der Voort, 2014), the survey had different sections with the one section repeated at the end to determine the effect that the design workshop had on the participants.

Expectations and views survey
Section 2 of the survey (Section 1 simply asked for age and gender) was a single page with eight questions based around three themes: ability (questions 1 and 3), value of the design session (questions 4, 6, 8) and feelings of empowerment (questions 2, 5 and 7).Each was answered by ticking a five point scale using the Smileyometer (Read, 2008) which was annotated from one (No) through three (Maybe) to five (Yes).These eight questions were repeated at the end of the study in Section 5 where the only change was in the tenses used in the sentences.The eight questions in Section 2 were: 1.I can contribute to the design of games 2. I am able to change how games are made for children 3. I feel that I have the skills to develop games 4. I feel that this activity is worthwhile 5. What I do today will help companies improve children's games 6.I feel that activities like this are good 7. What I do today will improve the lives of children around the world 8.I believe that the game I am helping to design will be built Data was analysed using SPSS.For each participant, scores were recorded for each question.As the questions were designed to measure three constructs; 'Ability' (q1, q3), 'Empowerment' (q2, q5, q7) and 'Value' (q4, q6, q8), total scores for each of these were also calculated.High scores against any of these questions would indicate that the teenagers were valuing participation.Differences across scores from Expectations to Views could indicate that the experience was different than expected or that the perceived value of participation changed.

General and specific skills survey
Questions in Section 3 used a mixture of the thumbs up scale (Kano, Horton, & Read, 2010) which is a 5 point scale (used in questions 1 and 4 below), the Smileyometer (Read, 2008) (used in questions 3 and 6) with the same scale as in Section 2, and a four point scale also from Kano et al. (2010) used in questions 2 and 5.The following questions were used, with questions 1-4 being section 3A and 5-6 being section 3B: SECTION 3A (general skills) 1. How good are you at drawing (very good, good, okay, not very good, poor) 2. How often do you draw pictures?(loads, a lot, a bit, never) 3. How much fun do you think this activity will be?(awful, not very good, good, really good, brilliant) 4. How good are you at designing games?(very good, good, okay, not very good, poor) SECTION 3B (specific skills) 5. How much do you think you will contribute to the design of the game (loads, a lot, a bit, never) 6.I think the current game is (awful, not very good, good, really good, brilliant) Section 3 captured the teens' general skills relating to drawing and designing.A total score for the questions in Section 3A was calculated; this had a maximum value of 19 and minimum of 4, whilst 3B (specific skills) was scored between 9 and 2. Questions presented in the negative form were reversed for coding purposes so that all positive answers had the maximum value within the scale.Whilst arithmetic means and standard deviations were used for the Smileyometer rankings, the data was treated as non-parametric for analysis.

Results
The results are described according to how much value these bring to the participants and to the design.Value was measured according to (a) value that design ideas contributed for further design, and (b) value to the workshop participants.

Did we get design ideas?
All the teenagers were able to complete the survey and the design activity without difficulty.On the day, we collected in 140 design parts of booklets as itemized in Table 2. Examples of images from the designs are seen in Fig. 7.

Using IDEAS analysis to determine overall design contribution
The time it took the four evaluators to do the IDEAS analysis of the design booklets ranged from 132 min to 275 min.In Table 3 we show the results from the IDEAS analysis on the 140 design booklets that were handed in.Around 65% of the individual designs were considered to have noteworthy ideas embedded within them.
The ideas selected in this way from the IDEAS analysis were not catalogued, the analysis was done to understand both how easy the method was for the participants but also to obtain an overall rating of the efficiency of the method.Given the low numbers of N/A designs we can say that most of the teenagers were able to understand and complete the design task.With two thirds of the booklets yielding at least one idea which, competitively, was considered worthwhile, we can also say that in the main, ideas that had some value were incorporated in the designs.

Were the ideas valuable to designers?
One observation from the rapid IDEAS analysis was that there was quite a lot of repetition of ideas across the set of designs.To estimate the value of the design ideas to eventual design, the team decided to examine a subset of ten design booklets from each of the four workshops (ten from each school -i.e., 40 in total) and apply the RAId protocol.Ten were selected randomly from each workshop and the RAId analysis of the 40 designs took the five evaluators (four of whom had also done the IDEAS analysis) between 1.5 and 3 h each.Each evaluator looked at all the 40 designs but with different constructs as lenses.Each time a design was analysed it was scored 0-5 against two constructs, which resulted in each design accruing 10 scores (two from each of five evaluators).When the set of analysis sheets were examined, it was found that most of the designs had accrued a set of ten relatively high scores: 16.25% of the 400 ratings were five which represented lots of evidence towards the construct, 28.75% were four, representing a fair amount, more than average, 24% were three, which represented reasonable or average and 15.75% were two, meaning that there were some ideas, but not many, towards the construct.Only 15.25% of the scores given were one and below, suggesting the evaluator found almost nothing at all in the design that related to the construct being used in evaluation.These latter two percentages align well with the estimates of poor contribution from the IDEAS analysis suggesting that both the sampling and the techniques were predicting similar things.It is possible, for comparison purposes, to calculate an average rating for each design indicating a rough measure of quality.This average is an average of the ten scores that each design was given during the RAId process.The frequency of each average, as shown in Table 4, demonstrates that overall, the designs had high value in the eyes of the evaluators and that the constructs that were used to guide the designs were being addressed.This can be interpreted to say the designs were relevant.
The 40 designs were batched in sixes and sevens to be analysed, and each evaluator had to select a winning design from each batch.Therefore, each evaluator selected four winning designs and so, with five evaluators, there was a potential to identify Evaluator Descriptor

E1
This game began with a screen where the choice was teacher (set up) or pupil (play).Set up was described in so far as the teacher had to place stones that had randomized actions associated with them including playing music, giving riddles and requiring exercise.Other stones (called secondary stones) gave hints on recycling and first aid or told the player to keep out.The teacher then set out the map space allowing free explore or follow a route.Some of the stones were inside mittens -these had mini games associated with them.
When the player played the game there was a countdown timer which decided whether the drowning man made it out alive or not.

E2
Stones in this game had different actions associated with them including things to do, hints and risks.The risk stone either let you earn stones or lose stones from ones you had collected.The organizer set up the stones as above and decided what the player had to achieve to succeed.The player in this game was aiming to help Rocky -a homeless child -by gaining enough stones to buy him/ her a house.When meeting a risk stone, the player could choose to take the risk or skip.A bar at the top of the game showed progress towards gaining the number of stones that had been set up by the organizer of the game.In this game the player had to take photographs of the stones when found and had to take photos to show that he/ she had completed an activity.

E3
This game had a first screen where a teacher could set up a new game and a student could choose to play an existing game.The game used scanning of stones with a map that showed a restricted area where the player was not allowed to go.The game mechanics included the building of a stone man (getting parts for successful play) and the options to customize the man with clothes and different colours.
30 discrete 'winning' designs.In the event, seven designs were selected as winners by two or more evaluators and another 14 were selected by just one meaning that, from the 40 designs looked at, over 50% were marked as winners by one or more evaluators.Of the winning seven designs (selected as winners by two or more), the first two were from workshop 2, the second from workshop 1, the following three were from workshop 4 and the last from workshop 3. Thus, each school group was represented.
The main output from a RAId design is a set of candidate designs that are put forward from the evaluators.There were five such designs in this instance and three of these are described in Table 5.These designs were each completed individually with no conferring and took between 10 and 30 min to create.The ideas were drawn out on paper with no constraints as to how much detail was needed.Each design was then shown, and verbally explained to the other evaluators.The candidate designs used hints, exercise, levels, clues, customization, maps, accessories, a stone man, risk, and questions from the winning ideas but also brought in ideas from other designs they had looked at.
Once the designs had been captured, the five evaluators looked at them to see where ideas came from.Ideas that came from the initial concept (INITIAL), included scanning stones, being outside and the game mechanic of scanning stones to build a character.New (CORE) ideas included the concept of a restricted area, the use of a map, the inclusion of riddles and quizzes and hints and the potential to customize an avatar with clothing.Physical exercising was also a CORE concept as was the idea of having the game environmentally friendly.It could be argued that the concept of educative/ socially responsible gaming was also CORE as it came from several teens and was used by more than one evaluator.Fig. 8 shows how one part of the idea from E3 was brought in from designs from the teens.ADD-ONS, suggested by many but only used by one evaluator, included being able to select a playing character at the start and the inclusion of music and the use of the phone camera to capture detail.NOVEL ideas that made it into the candidate designs included the notion of stones in mittens and the drowning man as an element in the game.Ideas that were not seen in the teenagers designs but were included in the evaluator designs (EXTERNAL) included the idea to save a homeless person, the placing of 'recycling and first aid' tips on some stones and the idea of exercise as a punishment for having failed at riddles.
Note that there were quite a few ideas from the teenagers that were not used despite being relatively unusual in so far as few teenagers suggested them, These included game ideas that could not be considered in scope (e.g. a racing game and a drinking water game), ideas that could realistically not be made/ would be problematic to include (e.g.incorporating a chat function and giving store vouchers when certain stones were found) and some that, while interesting, seemed to not get chosen (e.g.having players move stones, putting spells on stones so others in the group got a different experience and including different interfaces for summer and winter).The RAId process which results in the concentration of ideas into a set of candidate coherent designs will naturally result in several interesting, novel or valuable ideas not making it into the final designs.In the follow on from this process, where designs were then brought into the new version of the game there was more loss of teenagers' ideas.This is one of the tensions when involving many individuals in the design of a single game and one we come back to in the discussion.
However, from the RAId analysis it appears that the inclusion of teenagers in the design process was valuable given the large numbers of CORE, ADD ON and NOVEL ideas that were seen in the candidate designs and the fact that the team were able to use designs to inform the newer version of the game.

The value to the participants
On the day, 152 teens handed in the questionnaire data and of these, 131 submissions had all parts completed and so these 131 were analysed.Table 6 below shows the mean scores and standard deviations for the Expectations and Views questions in Section 2 and Section 5 of the survey.
Responses were generally positive with Q7, 'What I do today will improve children's lives around the world', being the only question to consistently score low.A Wilcoxon test revealed that there was no significant difference between the total scores for the eight questions before and after the event (Z = −1.635,p = 0.102).Table 7, below, shows the mean scores and standard deviations for the aggregated ratings for each of the three constructs of Ability, Value and Empowerment.Note that for Ability (being measured by only two questions) the maximum possible value was 10, whilst the other two constructs had a maximum value of 15.
For the constructs 'Empowerment' and 'Value' a Wilcoxon test revealed no significant difference between the scores before the design activity and after but there was a significant difference for the scores relating to 'Ability' (Z = −2.775,p = .006).There was also a strong correlation between the girls' pre-Ability scores and their post-Empowerment scores (r(129) = .316,p < .001).
While there was no significant change across the two surveys for Empowerment or Value, the data does indicate that participation in the design activity increased teenagers' perception that they had the ability to contribute (see Table 8).
A Spearman's correlation was performed on the scores for Ability from Section 2 (Expectations) and Specific Skills from Section 3B and there was a strong correlation (r(129) = .371,p < .001).There was also a correlation between Ability from Section 2 (Expectations) and General Skills (from Section 3A) (r(129) = .471,p < 0.001).This latter finding suggests that perceived drawing ability influenced attitude towards level of contribution and perceptions of value.The specific skills reported in section 3B also strongly correlated with the scores for Value from Section 2 (Expectations) (r(129) = .529,p < 0.05), suggesting there may be some relationship between teens' ability and how they value the design session.
There was also a correlation (r(129) = 3.17, p < 0.001) between the girls' like of drawing and designing, Section 3A questions 1 and 4, and their post-Empowerment scores (Section 5 (Views)).This may suggest that feelings of empowerment for teenagers may be influenced by attitudes to art and design.
Overall, the self-report data painted a positive picture giving us confidence that the experience had been enjoyable and worthwhile.

Value in rapid design workshops with teenagers
In proposing a method that can be used with large groups in a short time, with scaffolded activities, we need to reflect on whether or not this delivers value as an activity.In our introduction we asked the questions: What does the design team gain? and What do the participants gain?
The design team gains were in terms of ideas that were eventually included in the product design.A CORE idea that was new to the design team was the use of a restricted area and a map which was incorporated, ADD-ONS that also were incorporated included the opportunity to select a character at the start and the notion of stones in mittons (NOVEL) -which eventually became stones in packets also ended up in the eventual game.It is actually very unusual for design papers to examine where ideas from teens and children go, and the use of the RAId analysis allowed this to be done but it does show that there was a lot of effort needed by the design team in order to gather these ideas.This is a richer way of thinking than simply reflecting based on the outcomes and insights gained (Sim et al., 2018).Similar to All, Van Looy, and Castellar (2013), and many others, the ideas from the teens can be considered useful but not extensive and the design team had to find ways to take ideas from several teens and bring them into a functional design.
The participants were at an event where they had to be doing something!All 4 groups of participants were able to contribute ideas.Based on the IDEAS analysis, that looked at all the designs submitted, 8% of teenagers did not submit anything that could be used and a further 27% submitted designs with little to commend them.It is important that these statistics are reported as they highlight that even with a bounded structured activity, there may be individuals who cannot effectively participate.Nevertheless.the scores from the Expectation and Views surveys show that most of the teenagers who submitted their surveys considered the event valuable across a range of axes which fits with a holistic view of value as being beyond simply design idea generation (Frauenberger et al., 2015;Hansen et al., 2019;Iversen, Smith, & Dindler, 2017).From the survey data we also were able to establish that the teens own views of their ability to contribute and participate were closely associated with their feelings of confidence which suggests that to raise value for participants it is important to raise self-esteem before and during such activities (Rosenberg, 1965).One aim in doing design with teenagers is to ensure that we constantly strive to bring more value to the participants and the results from the surveys used in this instance of Tick Box Design can contribute both specific and methodological insights.
Considering the findings from the survey data and from an analysis of the designs we can see that teenage girls can benefit from, and contribute to, design sessions of this type.From the designer's perspective, over 65% of the ideas generated by the participants were judged to be of value or interest based on the IDEAS analysis.Taking a subset of the booklets and analysing them using the RAId process also demonstrated that outcomes from the workshop could be translated into designs to enhance or compliment an existing game.Overall based on the evidence from the evaluations it is apparent that all stakeholders benefited from the workshop experience.

Tick Box Design -Distributed
The GoS Workshop was carried out as a distributed PD event which used a local team to organize the session on behalf of the design team who were in a different country.The design team set the design goal for the session, chose the techniques that would be used, and designed all the materials.The materials had to packaged up for the local team who then had to make sense of them, translate them into the local language, and deliver them.The choice to do the workshop this way, instead of delivering it entirely online helped keep the cognitive load on the teenagers low (it is known that constantly attending to a screen is an effort which may have been less pleasant for the teenagers, (Christensen, Oestergaard, Dieckmann, & Watterson, 2018)) and also ensured social presence in the design environment which is conducive to design thinking (Carlisle, Carlisle, Ricks, & Mylroie, 2018).
There had to be a benefit though for the local team; in this case it was that their inclusion in the analysis of the research associated with the design activity and their inclusion in any subsequent publications.If the design activity had purely been to gather design insights, other inducements might have been needed as once the workshop was over, the local team had to collect in all the designs and survey responses and scan and package them for the design team.Further down the line, the local team were also required to translate written content that the local teenagers had included in their design sketches.Because we were also researching, the local team also participated in the analysis of the designs and in the RAId activity.All in all, this was a considerable amount of work.
The attention to the artefacts and tight packaging of materials meant that the local team delivering the booklets and the sessions on behalf of the design team were able to easily translate content into the local language and felt comfortable running the sessions.They reported no problems with the management of the workshops but did note that this was partly due to the design team having spent time before the workshops to ensure there was clarity about execution.

Use of the method
The version of the Tick Box Design method we have described used pre-designed PowerPoint content to ensure that the teenagers in the workshops understood what they were doing, who their engagement was for, and how their ideas might be used whilst ensuring they felt confident to participate ('Welcome and Why' and 'Confirming Confidence').A Design and Evaluation booklet, Design Intro Slides and an Interrupt activity were used to ensure that the design process was bounded and understood ('Layering Landscape') and then energized and focused ('Driven Design').We also ensured, again using PowerPoint, that participants were clear about what they could hand in, and that they knew what happened next ('Pack up and Promise').
The content shown here, and the use of PowerPoint and the Design parts of the booklet, were specific to this particular design activity and we would not argue that these are the only methods that should be used.We would say that a team planning a Tick Box Design event should ask its own questions as to how best to facilitate each of the five stages for their own situation.For example, Confirming Confidence (stage 2) could involve a short practical activity with immediate feedback; video, theatre, and demonstrations are all alternatives that could be used in stage 3; what we would say is that are all five stages, as described in Fig. 2, are necessary as the essence of the approach is that a workshop is well planned, understandable to the participants, constructed so adult involvement is minimal, and easy collection of design ideas is facilitated.
For others seeking to use the Tick Box Design workshop approach we suggest the following process: (a) Carefully think about the ethics of including these children and decide how participants ideas will be used.Decide on an appropriate mechanism to convey this (Welcome and Why) (b) Consider how the event will be framed to ensure that children feel confident, valued, and informed.Pay particular attention to this if the event will be delivered remotely.If a workshop is being delivered remotely -the basic elements are the same, but more care has to be taken to ensure that a local delivery team fully understands both the aims and the protocols around the activity.Using booklets and PowerPoints etc. with help in this as it gives structure.An instruction sheet will also be needed for the local team and the delivery of all this material may need to go back and forth several times to ensure it makes sense to all participants.We would say that Tick Box Design is best suited to situations where designers can be confident that child participants coming to the session can rapidly and easily be instructed both as to what they need to do and also within which bounds.It can be carried out on behalf of a design company or for a third party so long as everything is clearly described; and in these cases, as shown here, it can result in useful insights.It is probably poorly suited to open ended design or to designs that are outside 2D realities.We would not recommend it for small groups, believing that with smaller groups more involvement from adults is beneficial, more sharing of ideas is beneficial, and more interaction is beneficial.We would also not recommend it for groups where children will need a lot of additional support as the 'hands-off' nature of the method could leave those children feeling stressed and anxious.

Limitations of the research
We have presented a method and shown a single case study as an example of how it can be instantiated.We have run similar workshops with younger children in different venues -these have not been evaluated in the way that this case study has been -but we are reasonably confident that Tick Box Design can be used with younger children to gather design ideas -we cannot say that the findings in this paper -on value etc. would necessarily port to younger children.In evaluating the above case study, we were limited to methods that could fit into the activity and that relied on us not taking personal information from the pupils and so the survey results could not be aligned with the design outputs.This limited our ability to comment on how selfefficacy and confidence directly manifested into designs.Despite that, the large numbers in the study did allow us to make some general observations on the value that participants gained from the workshop.
In terms of design, the teenagers had less than an hour to make sense of an idea, think of ideas and then articulate those ideas using a pencil and a piece of paper; their contributions were limited by many factors including the extent to which an idea can be drawn, the extent to which an idea can even be captured as a thing, and the extent to which they could even think creatively in such a short length of time.Many of the children's drawings included text that explained things that could not easily be drawn and many of the drawings showed things that could not be interpreted.We mitigated against the need for time to think of brand-new things by giving the teens a game idea to build from; this helped polarize ideas but does limit the generalizability of findings.

Conclusion
Our contributions in this paper are two-fold -we have presented Tick Box Design, a rapid co-design method that is encased in detail that promotes informed and empowered participation that can fit into industrial and academic contexts, and which can be distributed and performed with large groups.In this instance the design team were in ANON1 and the workshop, with over 150 teenagers, was facilitated in ANON2.Despite the geographical, cultural and language differences the method yielded useful ideas to extend game for the UK designers.Our second contribution is the robust evaluation of ideas from the workshops in which, using a combination of the IDEAS method, the RAId method and self-report surveys, we have been able to quantify participants' contributions and experience.Analysis of ideas from the participants showed that more than 65% of the teenagers were contributing ideas that were aligned to the design problem.Our research has indicated that teenagers who have more confidence in their own ability will consider their participation in a more positive light than the others and so we do consider this to be an important area for design teams to attend to in the future.
In our further work we will be exploring the impact of different techniques on teenagers' and children's understanding of participation in the context of Tick Box Design.We will also be looking to evaluate the costs and benefits of using the Tick Box Design method with industrial partners and with other user groups to better explore its specific limitations and benefits.

Selection and Participation of Children
Teens from Iceland participated in this work as part of a STEM activity day that was run by the local University.Selection was initially from the schools who would have determined which girls attended the event and then girls selected workshops from a menu of events and so those selecting our workshop attended on the day.Girls were clearly told about the purpose of the event and that we were both looking for game ideas but also looking to evaluate the event.General consent was gathered from all girls to attend the event and for our workshops we explained the idea of Assent in so far as we gave girls an option to hand in none, some, or all of the work they had done.

Declaration of competing interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Fig. 4 .
Fig. 4. The Design Intro Slides showing the initial game concept that the participants were asked to build from.

Fig. 5 .
Fig. 5. Refocus Activity intended to assist in keeping momentum and motivation.
• Environment: the extent to which the game promoted a positive view of environmental issues • Engaging: the potential for the game to keep engagement over time • Portable: the potential for the game to transfer to other contexts (countries)

Fig. 8 .
Fig. 8. Figure showing how design inspirations came from Teenage Designs into a Candidate Design.

Table 2
Workshop schedule.

Table 3
Design ideas from IDEAS analysis.

Table 4
How Good were the designs.

Table 5
Candidate designs from the RAId analysis.

Table 6
Mean scores from expectations and views survey.
Choose techniques and materials for the children that enable easy collection both of designs and of any survey results -allow children to submit either or both if at all possible.Consider a booklet form if multiple pages might be needed.(g) Plan how the participants will hear back about their contribution.Decide on how best to convey this to the children (Pack up and Promise)