Our work with the Bay State Reading Institute (BSRI) has come full circle.
In 2010, BSRI was one of the first group of recipients of the federal Investing in Innovation (i3) Grant. SchoolWorks supported the development of the highly competitive grant by collaborating with BSRI on its original proposal and then serving as the program evaluator. Although SchoolWorks had been conducting formative reviews of BSRI’s elementary literacy model since 2006, the grant required that we identify an appropriate research model to measure fidelity of implementation as well as impact on student performance.
From the beginning, SchoolWorks worked closely with BSRI to develop a theory of action for its literacy coaching model. We helped define BSRI’s logic model’s inputs, outputs, and outcomes, which would become the key measures of fidelity of implementation. These elements also drove the fidelity of implementation study and analysis, which examined 46 public elementary schools across Massachusetts. In addition, the impact study, a time-interrupted series, provided a quantitative comparison of the performance of each of the 18 BSRI i3 schools and a set of statistically matched cohorts.
Now, as that work draws to a close, SchoolWorks believes that our analysis provided our client with valuable insights that led to adjustments to its original model. The breadth of analysis provided BSRI with the information needed to examine which parts of their model provide the highest impact. We worked with BSRI to ask key questions about its model—for example, did the schools with less leadership coaching perform as well as those who received more hours?
Using the results from our evaluations as evidence, BSRI adjusted its model for its literacy coaching and other services. BSRI principal coaches began tailoring the time spent with principals based on the data uncovered in the study.
The final five-year summative evaluation for the i3 grant is now complete. Although the sample size was very small, the impact study suggests some promise of impact on student performance. In short, when delivered with a level of fidelity, BSRI’s coaching model may impact student outcomes.
Why does this evaluation work matter? It obviously matters to the grantor, in this case the U.S. Department of Education. But the clarity that came from developing the logic model also made it much easier for BSRI to talk with its stakeholders about what it was doing. Most importantly, it helped BSRI tailor its work in each of its schools in ways that led them to make meaningful changes. Participating schools adopted suggested curriculum and expanded instructional practices, including adding extended reading blocks and additional time for writing, as well as increasing the number and variety of literacy interventions with students. And while much remains to be done, we now know that students’ literacy skills can improve when exposed to this model, which is the ultimate goal of BSRI’s model and all of our work.