Word Generation-related Studies

The original Word Generation program was first studied as a quasi-experiment in partnership with Boston Public Schools from 2006-2009. At this time, only the original Word Generation units existed for the middle grades (6th-8th), and assessment of the program primarily focused on vocabulary acquisition. Results of this study indicated that students in schools implementing Word Generation outperformed students in the comparison schools on study-designed vocabulary tests; descriptive results imply that this difference is approximately equivalent to the difference between 8th and 6th graders on the pre-test (in other words, two years of incidental learning) (Snow & Lawrence, 2011; Snow, Lawrence, and White, 2009). Improvements on this study-designed test also predicted performance on the Massachusetts Comprehensive Assessment System (MCAS) ELA test (R2 = .486, p<.001) (Snow, Lawrence, and White, 2009). In addition, this study found that language minority students in Word Generation schools acquired academic vocabulary faster than their English-only classmates or than language minority students in control schools (Snow, Lawrence, and White, 2009). Furthermore, their advantage persisted in a follow-up assessment a year after instruction had ended (Lawrence, Capotosto, Branum-Martin, White, & Snow, 2012). On the other hand, another analysis (Lawrence, Rolland, Branum-Martin, Snow, 2014) found that students who were on the whole better readers improved the most from program participation.

Following the initial Boston Public Schools study, the original Word Generation program was tested in a randomized trial across three large urban districts from 2009-2012. Generally referred to as the Word Generation efficacy trial, this study moved beyond vocabulary acquisition, with analyses also focused on classroom discussion quality. Results found that Word Generation classes had “dramatically” higher average discussion scores4 than control classes across all subject areas, particularly in math (Cohen’s d = 0.38-1.13) (Lawrence, Cross, Paré-Blagoev, Snow, 2015). The study measured little to no effect on vocabulary knowledge independent of the impact on discussion. The finding that improved discussion mediates treatment effects on student vocabulary learning is, to our knowledge, the first of its kind (Lawrence, Cross, Paré-Blagoev, Snow, 2015).5  Further analysis by Lawrence, Francis, Paré-Blagoev, and Snow (2016) found that, although there was no main treatment effect on the standardized tests, students in treatment schools who had low baseline vocabulary scores showed greater growth on standardized assessments than their peers.

Data6 from the efficacy trial were also used in an analysis by Hwang, Lawrence, Mo, and Snow (2014) to explore differential effects of Word Generation for language minority students. Results reported a vocabulary learning advantage for newly reclassified English learners over English-only classmates; that advantage declined but remained significant for up to four years after reclassification, an effect that also persisted for reading comprehension (Hwang, Lawrence, Mo, and Snow, 2014). Analyses were also conducted on surveys targeting civic engagement and communicative self-efficacy (confidence to participate in discussions).7  Word Generation students reported both higher communicative self-efficacy on recently covered topics (Lin, Lawrence, Snow, & Taylor, 2016) and higher civic engagement than their peers (Lin, Lawrence, & Snow, 2015).

1 Vocabulary was a measure of knowledge of word meaning from a sample of target words from the Word Generation curriculum. Perspective articulation and perspective positioning were measured using the SPTAM, which asks questions based on social scenarios commonly occurring in middle school. Academic language skills were measured using the Core Academic Language Skills Instrument (CALS-I), which focuses on linguistic features common in school content discourse but not necessarily common in casual conversation. Deep reading comprehension was measured using the GISA, an ETS scenario-based assessment that asks students to evaluate, integrate, and synthesize source materials for, as an example, a fictional class discussion.

2 The level of implementation in Word Generation classrooms was calculated in two ways. Jones, et al. (under revision) did so by calculating levels of workbook exposure, defined as the total workbook activities with work done by any student in the class, and found exposure levels of 57% for fourth- and fifth-grade classes and 46% for sixth- and seventh-grade classes. LaRusso, Donovan, and Snow (2016), calculated classroom implementation levels using the average number of activities completed across all students in each class, finding average program engagement levels of 40.2% for fourth- and fifth-grade classes and 31.2% for sixth- and seventh-grade classes.

3 These dimensions were evaluated using the Classroom Assessment Scoring System, Upper Elementary, which is an observational instrument used to assess classroom interactions (CLASS; Pianta, Hamre, & Mintz, 2012).

4 Discussion quality was measured using a composite discussion quality rating from classroom observations.

5 Lawrence, Crosson, Paré-Blagoev, and Snow (2015) only use data from year one of implementation, which only included two districts. Analyses in Lawrence, Francis, Paré-Blagoev, and Snow (2016) use data from year two of implementation, during which Word Generation was implemented in an additional (third) district.

6 The authors only used data from one participating district in order to obtain a sample with enough English learners.

7 The authors only administered surveys in one participating district to conduct these analyses (due to geographic proximity).

Jones, S. M., LaRusso, M., Kim, J., Kim, H. Y., Selman, R., Uccelli, P., Barnes, S., Donovan, M. S., Snow, C. (under revision). Experimental effects of Word Generation on vocabulary, academic, language, perspective taking, and reading comprehension in high poverty schools.

Kim, H. Y., LaRusso, M., Jones, S. Donovan, S., & Snow, C. (2017). Reducing the Academic Inequalities for English Language Learners: Variation in Experimental Effects of Word Generation in High Poverty Middle Schools. Presented at Society for Research on Educational Effectiveness, Washington, D.C.

LaRusso, M., Donovan, S., & Snow, C. (2016). Implementation challenges for Tier One and Tier Two school-based programs for early adolescents. In B. Foorman (Ed.), Challenges to implementing effective reading intervention in schools. New Directions for Child and Adolescent Development, 154, 11–30.

LaRusso, M., Jones, S. M., Kim, H. Y., Kim, J., Barnes, S., Donovan, S., & Snow, C. (2016). Impacts of a discussion-based academic language program on classroom interactions in 4th through 7th grades. Presented at the Society for Research on Educational Effectiveness, Washington, D.C.

LaRusso, M., Kim, H. Y., Selman, R., Uccelli, P., Dawson, T., Jones, S., Donovan, M. S., & Snow, C. (2016). Contributions of academic language, perspective taking, and complex reasoning to deep reading comprehension. Journal of Research on Educational Effectiveness, 9(2), 201-222.

Lawrence, J. F., Capotosto, L., Branum-Martin, L., White, C., & Snow, C. E. (2012). Language proficiency, home-language status, and English vocabulary development: A longitudinal follow-up of the Word Generation program. Bilingualism: Language and Cognition, 15(03), 437-451.

Lawrence, J. F., Crosson, A. C., Paré-Blagoev, E. J., & Snow, C. E. (2015). Word Generation randomized trial: Discussion mediates the impact of program treatment on academic word learning. American Educational Research Journal, 52(4), 750-786.

Lawrence, J. F., Francis, D., Paré-Blagoev, E. J., & Snow, C. E. (2016). The poor get richer: Heterogeneity in the efficacy of a school-level intervention for academic language. Journal of Research on Educational Effectiveness. http://dx.doi.org/10.1080/19345747.2016.1237596

Lawrence, J. F., Rolland, R. G., Branum-Martin, L, & Snow, C. E. (2014). Generating vocabulary knowledge for at-risk middle school readers: Contrasing program effects and growth trajectories. Journal of Education for Students Placed at Risk (JESPAR) 19(2), 76-97.

Lin, A. R., Lawrence, J. F., & Snow, C. E. (2015). Teaching urban youth about controversial issues: Pathways to becoming active and informed citizens. Citizenship, Social and Economics Education, 14(2), 103-119.

Lin, A. R., Lawrence, J. F., Snow, C. E., & Taylor, K. S. (2016). Assessing adolescents’ communicative self-efficacy to discuss controversial issues: Findings from a randomized study of the Word Generation program. Theory & Research in Social Education, 44(3), 316-343.

Snow, C. E., & Lawrence, J. F. (2011). Word Generation in Boston Public Schools: Natural history of a literacy intervention. The Senior Urban Education Research Fellowship Series. Volume III. Council of the Great City Schools.

Snow, C. E., Lawrence, J. F., & White, C. (2009). Generating knowledge of academic language among urban middle school students. Journal of Research on Educational Effectiveness, 2(4), 325-344.

Uccelli, P., Galloway, E. P., Barr, C. D., Meneses, A., & Dobbs, C. L. (2015). Beyond vocabulary: Exploring cross‐disciplinary academic‐language proficiency and its association with reading comprehension. Reading Research Quarterly, 50(3), 337-356.