SET-bullying: presentation of a collaborative project and discussion of its internal and external validity

Any kind of historical effect, which concomitantly to the SBABIs, could have modified pupils’ conceptualization of perception of bullying or could have influenced directly pupils’ behaviour, feelings, perceptions with respect to bullying [7, 14]. Although this kind of bias cannot be excluded in none of the studies, there is no evidence that such an event may have introduced it.

“program or participant maturation” [14]. Developmental changes in childhood and puberty may affect bullying related behaviours and its reporting independently of the implementation of a SBABI [7]. In parallel, the long-term implementation of a SBABI is expected to affect the cognitions of those directly or indirectly involved in it (i.e. pupils, teachers, school management and other key stakeholders) [7]. This could potentially lead to changes in the way that a SBABI is implemented in the school environment [7]. For both studies, the implementation of SBABI and the data collections expanded over several years.

Any vulnerability on the conclusions due to this dual maturation effect is also impacting the results of SET-bullying. Statistical analysis is planned to account for changes in pupils’ age in the results. However, the observed pattern in the relationship between time and BROs may be confounded by such an effect. Nevertheless, stability over time regarding the SBABI or its participants, would be unrealistic in real-life contexts of bullying research.

Not honest or “socially… desirable” [14] responses. For pupils to admit in a questionnaire item that they bully others or that they are the victims may be rather challenging. Eslea and Smith [8] discuss this issue as a potential explanation of the apparent differences between the number of girls reporting bullying others and being bullied.

In a study comparing the validity of self-reports versus peer nominations of bullying and victimization, Lee and Cornell [16] found greater disagreement between the two forms of reports of bullying as opposed to those of victimization. They suggest that perhaps “it is easier for a student to recognize that he or she is being bullied; a bully may not recognize that his or her aggressive behaviour constitutes bullying” [16]. On the other hand, Eslea and Smith [8] discuss a similar argument, especially for indirect bullying, but at the same time they question if the differences in the results between boys and girls could be due either “lack of awareness, or perhaps honesty”.

Regarding any differences in pupil responses based on the use of anonymous versus non-anonymous questionnaires, Lee and Cornell [16] discuss the results from other studies (Chan et al. 2005; O’Malley et al. 2000; van de Looij-Jansen et al. 2006) that have shown no evidence of differences in reporting of several types of behaviours, including bullying.

Reporting having bullied others appears to be the most difficult or challenging behaviour for a pupil to report. This may be less of an issue for a pupil to report been bullied. In any case, for both outcomes, any difficulty in reporting constitutes a threat to the internal validity. None of the studies has sound evidence for such a bias was introduced. However, none of them can be considered as immune from this bias. This vulnerability is also carried forward to the collaborative project.

A trend of pupil reports towards “socially and programmatically desirable” [14] responses cannot be excluded, especially for the measurements after the initiation of the SBABI. This can confound the relationship between time and effectiveness since the post-SBABI measurement may not fully reflect the SBABI effectiveness. This vulnerability can impact both the study and project results.

In addition to voluntarily alteration of pupils’ reports, it is likely that pupils may modify their report due to the multiple measurements. Any sensitization effect, due to the repeated measurements, may be considered negligible since the shortest time distance between two consecutive measurements was 1 year. Therefore, it seems unlikely that pupils may have remembered their responses and therefore replicated them.

“Instrumentation” [14] and changes in the perception of questionnaire items by the pupils [7]. Eslea and Smith [8] discuss the impact on increased awareness of pupils in bullying incidence and suggest that this may resulted in increased reporting of minor bullying incidences that would not have been reported otherwise. This kind of bias cannot be excluded for RESPEKT as well.

This type of bias influences the later study measurements and not the baseline measurements. Therefore, it may confound the relationship of effectiveness of SBABI with time and constitute a threat to the internal validity of SET-Bullying results.

In parallel, pupils’ self-reports may vary depending on the period of the school year that bullying is measured. It is reasonable to assume that the level in the classroom or the pupil reported severity and frequency of bullying may be different if a measurement takes place in the middle of the school year or towards its start or its end, closer to the school vacation periods. By study design, DFE-SHEFFIELD and RESPEKT ensured that all measures take place at the same period in the school year, in order to minimize such a seasonality effect. Therefore, any such vulnerability is less likely to also affect the results of SET-Bullying.

An additional point, related to the use of questionnaires in the assessment of the effectiveness of SBABI, is the difference in the recall time between studies. In each of the included studies, pupils were asked to report bullying incidents in the past. The recall time frame varied from five working days to last term [8] and last school year [9]. This variation between studies may explain some of the observed heterogeneity in their results.

In relation to the instrumentation, Ryan and Smith [17], in their recommendations, suggest the collection of information through “multiple methods and multiple informants” as well as the collection of “qualitative data” in order to be able to assess the consistency in SBABI results and “to contextualize implementation and outcome data”. Despite the fact both studies collected data in accordance with these recommendations, the collaborative project considers for analysis only pupils’ self-reports on frequencies of being bullied and bullying others. Given the afore mentioned limitations that are linked to pupil self-reports, the fact that SET-Bullying used only pupil self-reports of bullying others or been bullied is a threat to the internal validity of its results. Any attempt for contextualization of the results of SET-Bullying will be based on the input by the research teams owning the original databases.

The statistical methodology used in the analysis. Since the original data have been provided for analysis, any limitation due to the statistical analyses in the original studies does not impact the project results.

However, it should be noted that RESPEKT was designed as an “adjacent cohort design” study [9]. This principle has guided the analytical methodology of the original study report [9]. In SET-Bullying, the same data will be analysed as repeated measurements using mixed effect models. Therefore, the use of a different analytical methodology than the one for which the study was designed, may constitute a threat to the internal validity of the results.

Similar threats may be the lack of control group in both studies, which is in contrast to the recommendation from Ryan and Smith [17], and the lack of information on the degree of implementation of SBABI by each study group/cluster. Nevertheless, the mixed effect model analysis methodology will consider a separate pattern of the relationship between effectiveness and time for each study group/cluster. This may partially compensate for the two previous issues, since study groups/clusters with less effectiveness will be allowed to follow a different pattern as opposed to those with more effectiveness.

Another threat to the internal validity may be due to the attempt to harmonize the available data between the two studies. The analysis methodology for doing so may impose different assumptions and may result in loss of some information.

Two additional challenges derive from in one hand the distribution of anonymous questionnaires to pupils and on the other hand the hierarchical structure of school data (i.e. pupils nested within classrooms, within grades and within schools). The latter will be taken into account using mixed models effects statistical methodology for nested data. This approach is also consistent with the recommendation from Ryan and Smith [17]. Regarding the use of anonymous questionnaire data, the unit of analysis will no longer be at the pupil level but at the study group/cluster level.

Finally, a potential threat to the internal validity may be introduced by the use of some statistical methodologies. The statistical methodologies planned to be used in the secondary analysis are based on several assumptions about the underlying distributions of the data. Deviations from these assumptions may be a threat for the internal validity of the results of SET-Bullying. In order to address such a vulnerability, where appropriate, the assumptions of the statistical analysis methodologies will be kept to a minimum and any observed deviation from them will be discussed while reporting the results.

“Selection” [14] of control or intervention study groups [7]. Since none of the studies included a comparison group, this type of vulnerability due to selection and assignment of study groups to intervention or comparison groups is considered minimal. However, for DFE-SHEFFIELD only four primary schools decided to continue implementing the intervention and performed the third measurement [8]. The selection of these schools was based on their management willingness [8]. In RESPEKT, not all study groups had the chance to participate to all study measurements [9].

For this reason, all data from all study groups in both studies will be considered for analysis, as they can provide information that can inform the statistical models regarding the shape and magnitude of the relationship between time and effectiveness. Therefore, although selection bias will not be avoided, it will be taken into consideration for the collaborative project.

Changes in the study groups due to “participant [changes or] attrition” [14]. As mentioned above not all study groups participated to all measurements. Within each study group the number of pupils participating to each study measurement varied slightly over time. Although there is no reported mechanism of missing data, this kind of bias cannot be excluded. Having used anonymous questionnaires, a common practice in bullying research [7], it is not feasible to identify if the same pupils have responded in all measurements. Therefore, it cannot be excluded that some pupils involved in bullying may have been more prone to absenteeism and therefore missing systematically some study measurements [7]. This threat to internal validity is impacting also the project results.

Any “interactive effect” [14] of the above mentioned criteria. Any dynamic effect of a combination of the above sources of vulnerability to the internal validity of the results of SET-Bullying cannot be excluded.

The “reach and representativeness” [15] of the original studies. The studies that have contributed data were designed and implemented in specific contexts, time periods, geographical regions and on populations. For instance, RESPEKT was a pilot study implemented on a smaller population. DFE-SHEFFIELD was a much larger project, which was implemented in 24 schools, but only 4 of them participated in the follow-up assessment. The external validity of the collaborative project has vulnerabilities originating from the original studies that have contributed data.

Despite the SBABI differences, the same analysis will be performed for both studies. The statistical modelling approach will be rather generic and independent of the specific characteristics of the SBABIs. Therefore, it would not be unreasonable to assume that the form of the relationship between time and effectiveness of SBABI may hold for all kinds of similar interventions, although its magnitude may vary depending on the characteristics and the effectiveness of each SBABI.

The analysis will be based on pupils’ self-reported frequencies of been bullied and bullying others. As Chalamandaris and Piette [7] describe, such self-reported frequencies are commonly used in the evaluation of the effectiveness of SBABIs. However, other types of BROs (such as intentions, attitudes, feelings, perceptions) or reports from different sources (such as teachers, parents, other stakeholders, observations, archival records) have also used in other studies, as BROs in concluding on the effectiveness of SBABIs [7]. Therefore, this is a vulnerability of SET-Bullying since the under investigation relationship will not be evaluated for other types of BROs or information collected from other stakeholders of the school community. Thus, any extrapolation is not considered feasible.

Additionally, SET-Bullying included only two studies out of a pool of 22 studies. This pool was based on a literature review [7] which included articles published “prior to the end of January 2008”. Given the variability in terms of SBABI characteristics and study design characteristics among these 22 studies, the two participating studies are not representative of the pool of the 22 studies. Therefore, only a small portion of the available SBABI literature is included in SET-Bullying. This constitutes a threat to the external validity of its results.

The “implementation” and “adaptability” [15] of the statistical modelling process. The degree of “implementation” of the original SBABI or their “adaptability” in a different context should not bring any vulnerability to SET-Bullying. The reason is that the statistical modelling suggested by SET-Bullying will be rather independent of specific characteristics of the SBABI that have contributed data. It is also assumed that the relationship between time and effectiveness of SBABI may stand under various scenarios of effectiveness (i.e. from minimally to rather effective SBABIs). The numerical magnitude of that relationship is expected to differ between these scenarios.

Additionally, the ability to implement the same statistical models to any other set of effectiveness data from SBABI should be consider possible. In all disseminations of the statistical modelling of SET-Bullying the statistical methodology will be clearly stated and described giving the possibility of replication in other datasets. For this end, the selection of variables to be included will be rather basic including the time of measurement and two demographic characteristics, i.e. pupils’ age and gender.

The types of “outcomes” [15] and their potential future use. The outcome of the original studies was to conclude on the effectiveness of SBABI. SET-Bullying has a different aim which is to describe and predict the relationship between time and effectiveness of SBABI. The analysis methodology is expected to achieve this aim independently of whether the SBABI has been found to be effective or its effectiveness has limited external validity. Therefore, any vulnerability to external validity of the original studies, regarding their outcomes is not directly impacting the external validity of the collaborative project.

This is mostly threatened by the fact that it is a secondary analysis of data from studies that have been designed and implemented for a different aim. It is likely, that if a study was to be designed for the aims of SET-Bullying, its design and implementation might have been different than the original studies, aiming to better address its aims. Therefore, any outcome from SET-Bullying can be considered as hypothesis generating and cannot be conclusive for all current or future SBABIs. In order to confirm such a hypothesis, a specific study needs to be conducted for this aim.

The further and future efforts for “replication” [15] of the results of SET-Bullying, which Green and Glasgow [15] refer to as “maintenance and institutionalization”. It is very important to implement in the future the same statistical modelling methodology in other SBABIs that have already been concluded in order to asses whether the results from this collaborative project would have been replicated. The ideal scenario would have been a prospective study dedicated and sufficiently powered to address these aims.