Skip to content

Impact by Design

Working with our partners to define the unique landscape for change at their sites.

Your district has invested in a two-year effort to rewrite curriculum in alignment with new state standards. You’ve organized a string of professional learning sessions to get it into classrooms. You and your staff are consistently using your “look-for” tools in schools to see if teachers are implementing the strategies from the initiatives central to your improvement planning. You regularly do step-backs to monitor progress. You have a well-established mentoring program for your beginning teachers, and you are ramping up your coaching program around your belief that every teacher deserves a coach. And yet, you still aren’t seeing anywhere close to the results you were hoping for. What isn’t working? Are teachers not implementing the curriculum? Do they need more professional development? How can coaches help? Or, are teachers and coaches doing what you asked them to do, but it’s just not the right approach for their kids, and teachers don’t have the time and the space to figure out how to adjust?

At NTC, we work closely with our partners to examine and overcome impact challenges like these.

“One of the things we’re learning,” said NTC’s Senior Director of Impact Lisa Schmitt, “is that sometimes assumptions about what the issue is can get in the way. There are so many moving parts, so many inputs, that it’s hard to zero in on where the disconnects are. That makes it hard to design solutions that will have real impact.” In response, we’ve been working with partners to define the unique landscape for change at their sites as a starting place for designing programs — What is the current state of instruction? How would students characterize the learning environment? Where are the strengths? What are the school- and system-level enabling conditions that have to be in place to support success?

Landscape Analysis

To help partners answer some of these questions, NTC’s program staff uses a suite of customizable data collection tools and processes to guide “current state” analyses. The goal is to target optimal starting points for designing and implementing improvements. These tools can be used for equity audits, instructional reviews, or strategic planning conversations to help identify trends, gaps, and opportunities to design appropriate interventions. The toolset may include:

  • teacher and student focus group protocols
  • student surveys
  • curriculum review rubrics
  • student work sample analysis rubrics using NTC’s mentoring/coaching tools
  • walk-throughs and observations using customized instructional practice guides and other rubrics aligned with specific teacher practice and/or student experience goals
  • consultative support focused on family/community representation and inclusion in audit and review processes

“We also encourage partners to extend invitations for ‘learning walks’ to site leaders, administrators, and others to help get a much wider group than might be typical on the ‘same page’ about the instructional landscape,” Lisa said. Our NTC team then supports partners in strategic planning conversations first to analyze the data and then to articulate very targeted findings and trends as the launching point for designing or modifying an improvement strategy or coaching priority. “In addition to observations, we really encourage partners to gather student and teacher perspectives from focus groups and surveys and analyze student work samples to better inform their planning,” she said.

“We encourage partners to extend invitations for ‘learning walks’ to site leaders, administrators, and others to help get a much wider group than might be typical on the ‘same page’ about the instructional landscape.”

— Lisa Schmitt, Senior Director of Impact

Common 5+1

Another “impact by design” approach we’ve been piloting is based on program evaluation research that integrates Guskey’s five levels of evaluation of professional learning. This framework differentiates data collection and indicators of success according to stakeholder input, program type, and depth of engagement. We call it: Common 5+1.

Participants value the experience → Participants learn → Organizations support, or change to support, implementation → Participants transfer learning to practice → Students learn and thrive + Improvements are sustainable/Ecosystems become healthier

The Common 5+1 approach is aligned with our theory of change — professional support of educators leads to student learning through a logical sequence of events and supporting conditions. Lisa continued: “We especially like this quote from Guskey about the broader system supports — ‘Organizational dimensions … can sometimes hinder or prevent success, even when the individual aspects of professional development are done right.’ That’s one shift we are trying to encourage by expanding the data we collect and by getting partners to articulate what the supporting conditions need to be. This drives a greater level of collective responsibility and accountability for the success of a new approach. It also helps us codify enabling conditions.”

Impact by Design

Both of these approaches to NTC impact measurement focus on more immediate, student-centered, and site-specific measures in the unique contexts of our partner sites. “So often we will take a research-based initiative, and our goal is to have an impact on student performance on state tests because that’s how the field tends to measure success. But we don’t take into account all the existing variables in the context for implementing the strategy, including starting places and potentially limiting conditions on the ground that you need to design and adapt for. Half the time, we don’t even know what those hidden challenges are,” Lisa said.

So, instead of prioritizing statistical models and test data over all else, we are elevating student-reported experience data, classroom observations, and other qualitative measures. We are working with our partners to gather site-specific baseline data and then support them to co-design solutions that reflect community and culturally responsive goals and methods of measuring impact. In all of this work, including expanded groups of stakeholders is key to designing solutions that honor and value partner, community, and student realities, needs, and priorities. Our ultimate goal is to build capacity at our partner sites to identify gaps for data collection and system supports, engage in goal-setting and action-planning around very targeted and meaningful outcomes for teachers and students, and effectively use authentic data to design solutions that are organically easier to monitor internally.

“We are learning from our partnerships all over the country about how to do this better,” Lisa said. In Florida and Houston, our partners are using student feedback from surveys to design professional development. In Massachusetts, parents and community partners are providing input to help shape social studies curriculum. Our Tennessee Math partners used our data collection tools and Common 5+1 structure to design site-specific strategies around a shared goal to prioritize conceptual understanding in math instruction. In Osceola, our partners are using a landscape analysis to align professional development and coaching strategies to support curriculum implementation.

These are just a few of the reports we’re hearing from our program staff.

— “This has just been a really good opportunity for us to think, with our partner’s end goal in mind, about what needs to happen at each level of the Common 5+1 plan to make sure that we get there. This work is super adaptive. We are meeting folks where they are … and helping them think through some of those system shifts and structures that need to be in place to be successful.”

— “We’re doing a set of focus groups with teachers, coaches, mentors, and administrators. We are also doing some with students and parents. We’ve been thinking – why are we trying to solve problems for students and parents without asking students and parents what they need?”

The aim of NTC’s new impact agenda is to be better able to co-design impact plans with our partners that measure outcomes that are meaningful and aligned with district needs. Our ultimate goal is to build capacity at our partner sites to engage in sustainable, evidence-based, systems-level continuous improvement that makes a real difference in the learning experiences of those students it is our mission to serve.