The documents below were reviewed for Quality of Research. The research point of contact can provide information regarding the studies reviewed and the availability of additional materials, including those from more recent studies that may have been conducted.
Cornell, D., Sheras, P., Gregory, A., & Fan, X. (2009). A retrospective study of school safety conditions in high schools using the Virginia Threat Assessment Guidelines versus alternative approaches. School Psychology Quarterly, 24(2), 119-129.
Cornell, D. G., Gregory, A., & Fan, X. (2011). Reductions in long-term suspensions following adoption of the Virginia Student Threat Assessment Guidelines. NASSP Bulletin, 95(3), 175-194.
Cornell, D. G., Allen, K., & Fan, X. (2012). A randomized controlled study of the Virginia Student Threat Assessment Guidelines in kindergarten through grade 12. School Psychology Review, 41(1), 100-115. (NOTE: At the time of the NREPP review, the manuscript of this article had been submitted for publication but not yet accepted.)
Allen, K., Cornell, D., Lorek, E., & Sheras, P. (2008). Response of school personnel to student threat assessment training. School Effectiveness and School Improvement, 19(3), 319-332.
Bandyopadhyay, S., Cornell, D. G., & Konold, T. R. (2009). Validity of three school climate scales to assess bullying, aggressive attitudes, and help seeking. School Psychology Review, 38(3), 338-355.
Cornell, D. (2011). School Climate Bullying Survey: Descriptive report. Charlottesville: University of Virginia.
Cornell, D., & Gregory, A. (2008). Virginia High School Safety Study: Descriptive report of survey results from ninth grade students and teachers. Charlottesville: University of Virginia.
Cornell, D. G., Sheras, P. L., Kaplan, S., McConville, D., Douglass, J., Elkon, A., et al. (2004). Guidelines for student threat assessment: Field-test findings. School Psychology Review, 33(4), 527-546.
Gottfredson, G. D., Gottfredson, D. C., Payne, A. A., & Gottfredson, N. C. (2005). School climate predictors of school disorder: Results from a national study of delinquency prevention in schools. Journal of Research in Crime and Delinquency, 42, 412-444.
Threat Assessment and Response Protocol (adapted from the Guidelines for Responding to Student Threats of Violence, by Cornell and Sheras, 2006)
| Description of Measures | Long-term school suspensions and expulsions were measured using the following:
|
| Key Findings | A retrospective cross-sectional study was conducted during the 2006-07 school year with 9th-grade students in 95 public high schools that used V-STAG, 131 public high schools that used a locally developed threat assessment program, and 54 public high schools that had no threat assessment program. The average number of long-term school suspensions (i.e., denial of school attendance for more than 5 days) and expulsions (i.e., denial of school attendance for at least 1 year) per school was lower for all students in high schools that used V-STAG than it was in high schools that used a locally developed threat assessment program (10.50 vs. 15.71; p = .05) and in high schools that had no threat assessment program (10.50 vs. 15.28; p = .05), after controlling for total student enrollment, proportion of minority students, proportion of students eligible for reduced-price meals, annual number of neighborhood violent crimes, number of school resource officers employed at the school, and student perceptions of school security. Small effect sizes were associated with the difference between high schools that used V-STAG and high schools that used a locally developed threat assessment program (Cohen's d = 0.30) and the difference between high schools that used V-STAG and high schools that had no threat assessment program (Cohen's d = 0.30). A 3-year study was conducted with 23 public high schools that used V-STAG and 26 public high schools that had no threat assessment program. Data were retrieved from the VDOE archival database for the school years before (2006-07) and after (2008-09) V-STAG training and implementation in the intervention schools. The total number of long-term suspensions (i.e., denial of school attendance for more than 10 days) and expulsions (i.e., denial of school attendance for at least 1 year) by high school for the 2006-07 (baseline) and 2008-09 (follow-up) school years was converted to a rate on the basis of the fall student enrollment for the corresponding school year. Relative to baseline, the long-term school suspension and expulsion rate at follow-up in high schools that used V-STAG was lower than that in high schools that had no threat assessment program (8.2 at baseline and 3.9 at follow-up per 1,000 students in high schools that used V-STAG vs. 10.9 at baseline and 10.9 at follow-up per 1,000 students in high schools that had no threat assessment program; p < > A randomized controlled study was conducted during the 2008-09 school year with students in 40 public schools (26 elementary, 8 middle, and 6 high schools in a school division) that were assigned to the V-STAG condition or the control condition (i.e., a 1-year delay before receiving V-STAG). Of the students who made threats of violence at school during the study, those in schools assigned to the control condition were almost 3 times as likely as students in schools assigned to the V-STAG condition were to receive a long-term suspension (i.e., denial of school attendance for more than 10 days) or expulsion (i.e., denial of school attendance for at least 1 year) (p = .01). This group difference was associated with a medium effect size (odds ratio = 2.86). |
| Studies Measuring Outcome | Study 1, Study 2, Study 3 |
| Study Designs | Experimental, Quasi-experimental |
| Quality of Research Rating | 3.1 (0.0-4.0 scale) |
| Description of Measures | Alternative school placement was measured using information retrieved from principal-completed school records of student threat incidents. The records indicate whether or not students identified as having made a threat of violence were placed in an alternative school setting. The record of each student threat incident also includes a description of the incident and the steps taken by the school in response to the threat. |
| Key Findings | A randomized controlled study was conducted during the 2008-09 school year with students in 40 public schools (26 elementary, 8 middle, and 6 high schools in a school division) that were assigned to the V-STAG condition or the control condition (i.e., a 1-year delay before receiving V-STAG). Of the students who made threats of violence at school during the study, those in schools assigned to the control condition were more than 7 times as likely as students in schools assigned to the V-STAG condition were to receive an alternative school placement (p = .01). This group difference was associated with a large effect size (odds ratio = 7.69). |
| Studies Measuring Outcome | Study 3 |
| Study Designs | Experimental |
| Quality of Research Rating | 2.8 (0.0-4.0 scale) |
| Description of Measures | Bullying infractions in school were measured using the following:
|
| Key Findings | A retrospective cross-sectional study was conducted during the 2006-07 school year with 9th-grade students in 95 public high schools that used V-STAG, 131 public high schools that used a locally developed threat assessment program, and 54 public high schools that had no threat assessment program. Findings from this study included the following:
|
| Studies Measuring Outcome | Study 1, Study 2 |
| Study Designs | Quasi-experimental |
| Quality of Research Rating | 2.9 (0.0-4.0 scale) |
| Description of Measures | Supportive school climate was measured using the following:
|
| Key Findings | A retrospective cross-sectional study was conducted during the 2006-07 school year with 9th-grade students in 95 public high schools that used V-STAG, 131 public high schools that used a locally developed threat assessment program, and 54 public high schools that had no threat assessment program. Findings from this study included the following:
|
| Studies Measuring Outcome | Study 1 |
| Study Designs | Quasi-experimental |
| Quality of Research Rating | 2.9 (0.0-4.0 scale) |
| Description of Measures | School counseling support and parental involvement were measured using threat assessment team documentation forms and information retrieved from principal-completed school records of student threat incidents. The records indicate whether or not students identified as having made a threat of violence were provided with some form of mental health counseling service appropriate to their needs and whether or not a school conference was held with a student's parents. The record of each student threat incident also includes a description of the incident and the steps taken by the school in response to the threat. |
| Key Findings | A randomized controlled study was conducted during the 2008-09 school year with students in 40 public schools (26 elementary, 8 middle, and 6 high schools in one school division) that were assigned to the V-STAG condition or the control condition (i.e., a 1-year delay before receiving V-STAG training and implementation). Of the students who made threats of violence at school during the study, those in schools assigned to the V-STAG condition were almost 4 times as likely as students in schools assigned to the control condition were to receive mental health counseling services (p = .01) and approximately 2.5 times as likely as students in schools assigned to the control condition were to have a school-parent conference (p = .01). These group differences were associated with medium (odds ratio = 3.98) and small (odds ratio = 2.57) effect sizes, respectively. |
| Studies Measuring Outcome | Study 3 |
| Study Designs | Experimental |
| Quality of Research Rating | 2.5 (0.0-4.0 scale) |
The following populations were identified in the studies reviewed for Quality of Research.
| Study 1 | 13-17 (Adolescent) | 51% Male49% Female | 63% White23% Black or African American5% Hispanic or Latino5% Race/ethnicity unspecified3% Asian1% American Indian or Alaska Native |
| Study 2 | 13-17 (Adolescent)18-25 (Young adult) | 50% Female50% Male | 53% White19% Black or African American15% Hispanic or Latino11% Asian2% Race/ethnicity unspecified |
| Study 3 | 6-12 (Childhood)13-17 (Adolescent) | 73% Male27% Female | 73% Black or African American24% White3% Hispanic or Latino |
External reviewers independently evaluate the Quality of Research for an intervention's reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
| 1: Long-term school suspensions and expulsions | 3.5 | 3.2 | 1.7 | 4.0 | 2.7 | 3.5 | 3.1 |
| 2: Alternative school placement | 1.5 | 3.0 | 2.0 | 4.0 | 2.5 | 3.5 | 2.8 |
| 3: Bullying infractions in school | 3.1 | 3.3 | 1.5 | 3.5 | 2.5 | 3.5 | 2.9 |
| 4: Supportive school climate | 3.3 | 3.5 | 1.8 | 3.3 | 2.8 | 3.0 | 2.9 |
| 5: School counseling support and parental involvement | 0.5 | 2.5 | 2.0 | 4.0 | 2.5 | 3.5 | 2.5 |
Sample reliability was provided for the student self-report scales on bullying, help-seeking behavior, and the learning environment, all of which are known in the field and have good psychometric properties established by independent investigators. In considering test-retest reliability, there was an assessment of the stability of the number (or proportion) of long-term school suspensions issued in each school across multiple years, resulting in a report of high intraclass correlation. Criterion validity was high for the long-term school suspension and expulsion data, which came from either the VDOE public archival database or the standard school records completed by school principals. In all three studies, a detailed manual was used to train school threat assessment teams in the use of V-STAG. In one study, a compliance score for intervention fidelity was constructed from measurable items based on the manual. One study used random assignment by school type to control for many potential confounding variables as well as sophisticated hierarchical linear modeling to statistically control for the clustering of students within schools. All three studies included effect sizes for the group differences in outcome findings.
There was a lack of reliability information on principals' and school staffs' use of similar criteria for measuring alternative school placement outcomes. There was concern about interrater reliability across school staff for the school counseling support and parental involvement outcome measure, which relied on threat documentation forms that had been completed by the threat assessment teams of all participating schools; some teams did not complete these forms. Many bullying incidents may not have come to the attention of school staff and, consequently, may not have been reported. Intervention fidelity was not measured in the two quasi-experimental studies. Two studies lacked random assignment, and one of these studies (the retrospective study) did not offer information on potential preimplementation baseline differences between conditions.