Design and Initial Test of the Instrument

Progress and Future Work on Internal Accountability Instrument

R. Elmore

We have worked through the initial design stage of the teacher survey portion of the instrument for assessing and diagnosing internal accountability in SERP schools. We have also field-tested it in one school-the Woodrow Wilson Middle School in Boston-where we got 26 out of 27 possible responses. There are two other phases of the initial assessment that have been scheduled, but not yet completed. One phase is face-to-face interviews with a small sample of teachers in the school and with the principal. The purpose of this phase is to test some of the initial diagnoses derived from the teacher survey. The second phase is a round of classroom observations, based on a broad protocol, derived from our work with the Connecticut Superintendents' Network and the Cambridge Leadership Network. The purpose of this phase is to generate base-line data on what instructional practice looks like in a set of classrooms. These interviews and observations will be conducted by Lasse and Richard, and by graduate research assistants, all of whom have been through training in observation and analysis of instructional practice in Richard's course last spring.

The instrument is derived from several sources: the Consortium for Chicago School Research (Tony Bryk) surveys, the School and Staffing Survey (connecting to a national sample of schools), the CPRE project on accountability, and Lasse Skogvold-Isaksen's study of low-performing schools in Norway. In the SERP group's conversations about this part of the project, we struggled making trade-offs between the predictive functions of the instrument and the diagnostic functions. If we took a largely predictive view of the instrument, we would say that the purpose of the instrument is to predict the readiness of the school to accept an instructional intervention, and to put the school in some classification scheme organized around readiness to implement. In this view, the more parsimonious the instrument the better. If we took a largely diagnostic view, we would say that the purpose of the instrument is to provide information about the internal functioning of the school as an organization that would allow us to (a) predict how it would respond to an instructional innovation, and (b) frame an intervention strategy to support the implementation of the innovation. We chose the latter purpose-diagnostic-with the constraint that the instrument would have to be short enough for a teacher to complete in one-half an hour or less. As noted below, the instrument is designed, to the degree possible, to track with a set of intervention activities that we might conduct with the school to speak directly to the evidence that the instrument reveals.

Definition of Internal Accountability

Internal accountability is, simply put, the ability of the organization to respond to external pressure or support in a way that improves its measured performance. The premise here, drawn from our case student work, is that in order for a school to "improve" against any set of external expectations, it has to have the internal capacity to bring those expectations into its daily operations in a coherent way. Pushing on an incoherent, atomized organization with a strong external force (testing and sanctions, for example), we found, frequently produces a more incoherent organization. Pushing on a relatively coherent organization, we found, usually produces a more coherent, higher-performing organization.

There are three elements to our definition of coherence, as a condition for internal accountability: (1) the individuals' sense of responsibility for, and agency toward, student learning and measured performance; (2) the shared norms, values, and expectations among individuals in the organization, again, toward student learning; and (3) the capacity of the organization to direct, manage, and support instructional practice-including the processes by which people account for what they do, the principals' practice around supervision and support of instruction, and professional development organized to support instructional improvement. To the degree that these elements are present and working in alignment with each other, we speak of the school as having relatively high internal accountability. Internal accountability, in our view, precedes and shapes the ability of schools to respond to external pressure and support, but schools cannot "learn" internal accountability without actually working collectively on problems of instructional practice. This last principle is the source of Tom Hatch's by now famous (I hope) statement that "you have to have capacity to build capacity." The Content of the Instrument

The instrument taps teachers' responses in five basic domains of organizational life: (1) the principal's role as an instructional leader; (2) the orientation of teachers toward external accountability measures-tests and standards; (3) teachers' assessments of their colleagues attitudes and expectations for student learning (collective expectations); (4) teachers' assessments of their own influence over students' learning (responsibility, agency or efficacy); and (5) the school's capacity to support teachers' instructional practice.

The theory underlying these domains is as follows: Schools operate as organizations-as opposed to collections of atomized teachers-to the degree that (1) the leadership of the school is focused on improvement of practice in the classroom; (2) teachers share expectations for student learning; (3) those expectations focus on the effect of instruction on student learning, rather than factors outside the control of the organization; and (4) there are resources and structures in the school that teachers actively use to improve their practice. The theory also directs our attention toward possible components of an intervention strategy that might be used to address organizational weaknesses surfaced by the instrument, principal interview, and classroom observations. Components of an Intervention Strategy

The basic premise of this strategy is that people learn how to do the work by doing the work in a structured, reflective, and supportive learning environment, rather than by talking about how to do the work or being told how to do the work. Another basic assumption is that internal accountability increases as the connective tissue among members of the organization increases around tasks that are central to the organization's performance. So the central purpose of the intervention piece of the project should be aimed at working on those dimensions of a school's internal accountability that need development, but doing so around activities that contribute directly to the school's performance or, in our case, to the school's ability to use an instructional intervention. These components are stated as if we were working in Boston-they would differ slightly depending on what the district within which the school is located specifies as standard school practices. And these components are adapted directly from our work with superintendents, system-level administrations and principles in our professional networks.

Work on Leadership Practice. This work occurs with the principal and the instructional leadership team of the school. It involves structured observations of classrooms, using a simple protocol. The purpose of the work is to engage in routine, continuous observation of teaching practice and to develop a common language for describing and analyzing practice. The work is not initially focused on the specific features of the instructional intervention, but it should be structured around a problem of performance that the school is working on that is related to the intervention. Every round of observations should conclude with a set of commitments for the next level of work, and every succeeding round should begin with a review of what has happened since the last round. This work should occur with support from an outside consultant and it should be done no less frequently than once every two weeks for one-half day. This work is designed to (a) develop individuals' strengths around observing, describing, and analyzing instructional practice in a subject matter domain; (b) develop and strengthen common expectations among the leadership team for what good instruction looks like; and (c) create a culture of lateral commitment among leaders within the school around influencing instructional practice.

Work on Instructional Practice. This work occurs primarily with teachers, although participation of administrators in some number of sessions should be mandatory. It involves a basic review of the intervention, the theory of how it is supposed to work and an introduction to the key practices. Teachers should be asked to observe video lessons, to develop their own protocols for assessing practice based on overview sessions, and to make suggestions about what the next level of work should be for the teachers in the video lessons. Teachers should then be put in a regular rotation for some period of time, probably three weeks, in which they work in groups of four observing each other-three teachers as observers, one in practice-followed by an analysis and debriefing after each observation. The question at the end of each observation should be "what is the next level of work?" And each teacher should be observed at least twice. These observations should be organized and run by the teachers themselves, facilitated by a consultant with subject matter knowledge. This work is designed to: (a) develop strengths in observation of practice; (b) develop common norms and expectations for practice, grounded in a specific model of instruction; and (c) develop lateral accountability for practice.

Analysis of Student Work. This work occurs in mixed groups of teachers and administrators. It involves drawing samples of student work from lessons designed around the intervention and using them to develop (a) a language for analyzing what students are doing; (b) a language for connecting what students are doing to the expected effects of the intervention on student performance; and (c) a set of ideas for what the next level of work should be for specific students. The goal here is to introduce the idea of diagnosis at the individual student level, and to use analysis of student work to surface issues regarding for whom the intervention seems to be working and for whom it is not. This work is designed to (a) strengthen norms of teacher efficacy and responsibility by creating a cause and effect link between instructional practice and student learning; and (b) create norms of practice that focus on the individual student as the unit of analysis, rather than broad groups of students. This work will require some initial adaptation of protocols for looking at student work to the specifics of the intervention, and the facilitation of someone knowledgeable in the intervention.

Download PDF of Internal Coherence Survey used in Boston Public Schools