Building Coherence Within Schools

About the Internal Coherence Assessment Protocol

The Internal Coherence Assessment Protocol (ICAP) was developed by Richard Elmore, Gregory Anrig Professor of Educational Leadership, and Michelle Forman, doctoral student and Internal Coherence Project Director, at the Harvard Graduate School of Education / SERP Institute, in collaboration with senior administrators in the Boston Public Schools. The protocol is designed to provide school leadership teams and their system-level supervisors with a structured body of information to use in tailoring the system’s broad improvement strategy to the particular demands of a specific school.  The protocol provides a profile of a school’s capacity to engage in deliberate improvements in instructional practice and student learning across classrooms. The profiles are then used as a point of departure for a series of professional development events and ongoing supports, for school and district leaders, along these three dimensions.

The purpose of the ICAP is to design a diagnostic that places schools on a developmental spectrum of factors related to their capacity for improving instructional practice and student learning. The work focuses on three common patterns of organizational features the school improvement literature indentifies as associated with schools’ capacity to improve: leadership focused on the support for instructional practice, individual and collective efficacy of teachers and administrators related to instructional practice and student learning, and organizational structures and processes that support improved instruction and student achievement. The Internal Coherence framework includes both the ICAP diagnostic process and guidance for school and district leaders on how to use the ICAP data to focus practice in areas that will increase the capacity of the organization to work as a collective in pursuit of common instructional goals.   

The ICAP consists of a teachers’ survey, a series of protocols for interviews and focus groups with teachers and principals, and a brief protocol for classroom observations. Data collected from the diagnostic process results in the development of a school-level profile. Originally a practice-driven instrument designed to fill a state requirement for an improvement strategy for the most underperforming schools in Boston, the survey has been restructured to tighten its connection to the relevant research base and to reflect most recent knowledge on accountability and school improvement.

Core Elements of Internal Coherence

The three domains we assess in the ICAP are Leadership Practice, Efficacy Beliefs and Organizational Processes. These three factors predict the likelihood that a school will behave as an organization around the core function of instructional practice.  Each of these factors bears a strong relationship to school performance in the research literature.  Taken together, however, they enable us to construct a developmental theory that places schools on a continuum on each dimension, and allow practitioners to focus their instructional improvement work on their specific organizational needs.  In addition, we suggest that these factors embody a general theory of how schools improve—a provisional causal order—that is helpful to practitioners in understanding which factors to address in which order. 

Leadership practice focused on instructional improvement is an essential initial condition that precedes both the creation of organizational structures and processes and the development of individual and collective efficacy in an organization.  Likewise, it is difficult to develop collective efficacy without organizational processes that create a safe and stable learning environment for teachers and students.  Relationships among these factors are probably also recursive, in the sense that an organization that has high efficacy is probably more likely to take an active role in altering its organizational structures and processes in response to evidence of student learning, and will be more agile in developing and using leadership roles within the organization.

We discuss each of the core elements of the Internal Coherence framework, in detail, below.

School Leadership

There is general agreement in the field that strong leadership is a prerequisite of school improvement and performance (Hallinger & Murphy, 1985; Fullan, 2007). Over the past two decades, visions of school leadership have fallen into two categories; instructional and transformational. Instructional leadership traditionally holds the principal as the primary source of educational expertise, responsible for maintaining high expectations for teachers, coordinating curriculum, supervising instruction and monitoring student progress (Marks & Printy, 2003; Barth, 1986). Transformational leadership aims to transform school cultures, providing intellectual direction that fosters innovation and empowers and supports teachers (Marks & Printy, 2003; Firestone & Louis, 1999; Leithwood & Jantzi, 1999). Most recent research indicates that school leadership able to integrate aspects of the two will have the most potent effect on teachers’ instructional practices (Goddard et al., 2010). 

Leadership practices associated with high Internal Coherence combine the focus on culture associated with transformational leadership, particularly as it relates to the creation of a learning environment (Edmondson, 2007), with the conception of “shared  instructional leadership” (Marks and Printy, 2003). Shared instructional leadership is characterized by the active, ongoing collaboration of principals and teachers on issues of teaching and learning. Here, principals involve teachers in sustained dialogue and decision making around instruction and student learning, while remaining central agents for change (Marks & Printy, 2003; Darling-Hammond, 1988; Rowan, 1990).

Leadership practices focused on culture include the creation of an environment conducive to ongoing adult learning and supportive of the risk-taking inherent for teachers in making fundamental changes to practice. There are two scales in the leadership section of the survey designed to tap culture-building practices: modeling learning as a core value, through asking questions, listening attentively, and acknowledging gaps in personal knowledge, and; establishing psychological safety, or an environmentin which teachers are supported in experimenting with untested approaches, able to publicly seek help with new practices and speak candidly about challenges they face in their work.      

Further leadership practices assessed by the ICAP emerge in relation to organizational processes at the level of the school and team. Specifically, we are concerned with the extent to which principals provide the expectation and the opportunity for teachers to collaborate effectively around issues pertaining to instruction. Existing research suggests the importance of teachers’ collaboration to adult learning, instructional practice, student learning, and individual and collective efficacy beliefs (Rosenholtz, 1989; Ross et al., 2004; Goddard et al., 2007; Louis et al., 2009). Specific leadership practices in this domain include setting the expectation that teachers focus on issues of instruction in their time together, protection of the structures and support for the processes which are prerequisites for productive collaboration, requiring that decision-making be grounded in student data, and the practice of holding teachers accountable for all of the above.

Finally, Internal Coherence requires that school leaders have a baseline competence in instructional practice. Items on the IC Survey solicit teachers’ perceptions of their principals’ understanding of how students learn, and of their principal’s ability to not only articulate a vision for teaching and learning, but to take steps to effect its realization. These questions surface the extent to which principals are a presence in teachers’ classrooms, have a handle on the instructional practice and professional development needs across their organization, and utilize student achievement data in the assessment of teachers’ practice. 

Organizational Processes

Challenges like whole-school improvement require organizational responses, and therefore depend not only on the knowledge and skill of the people in the organizations but on the integrative structures and processes of those organizations for their success (Honig & Hatch 2004; Childress et al., 2005; Elmore et al., 2007). The literature on organizational learning stresses the importance of group-level structures and processes as places where knowledge is created, stored, and used, and of integrative structures and processes at the top of the organization as places where competing views and interests are negotiated and resolved (Edmondson, 2002; Crossen and Berdrow, 2003). Despite the fact that many school systems devote extensive resources to creating the structures of collaboration – shared teacher time, vertical and horizontal teaming, leadership teams, etc. – practitioners frequently lack the skills and processes to capitalize on their time within these structures for powerful ends.

Pilot phases of this work have revealed teachers’ differential perceptions of organizational processes at the team and whole school levels, leading us to parse out these items on the survey. At the level of the school, we are interested in the degree to which teachers feel improvement processes in the organization are realistic, measurable, and in alignment. This section also addresses teachers’ involvement in collaborative processes for school improvement, a condition associated in the research with higher student achievement in both math and reading (Goddard & Goddard, 2007). Questions regarding organizational processes at the team level attempt to get inside the formal structures to the substance of teachers’ collaborative work, including the content of the talk and team member’s facility with the norms and processes of effective collaboration. 

Finally, we include questions at the team and whole school levels about the development of a shared understanding of effective instructional practice. Schools high in Internal Coherence are schools in which educators work systematically together to build the knowledge and the skill for ongoing instructional improvement. Regardless of the specific process schools use to move student and adult learning out of the privacy of the classroom and into the public spaces of the organization (City et al., 2009), this culture-building exercise is a critical component of building coherence and improving collective practice.

Individual and Collective Efficacy Beliefs

Individual teacher efficacy, or a teacher’s expectation that he or she will be able to bring about student learning, is a well established predictor of teaching behaviors that foster academic achievement (Gibson & Dembo, 1984; Hoy et al., 2003). These behaviors include a willingness to undertake classroom experimentation and innovation, particularly with regard to techniques that are difficult to implement and involve risks, such as sharing control with students. High-efficacy teachers try harder, use management strategies that stimulate student autonomy, attend more closely to low-ability student needs and modify students’ ability perceptions. Not surprisingly, these behaviors result in higher student achievement in both core academic subjects and in affective goals like motivation and self-esteem (Ross et al., 1998; Ross et al., 2004).

Collective efficacy is a related construct treating schools, rather than individual teachers, as the unit of analysis. Collective efficacy is a measure of group-referent efficacy beliefs: “I believe the faculty as a whole has the ability to successfully educate students,” and emerges as a group-level attribute from the interactive dynamics of a faculty (Bandura, 1997). A faculty’s perception of its collective ability to improve student outcomes sets expectations for action that influence how teachers pursue organizational goals and the effort they put forth to attain success (Bandura, 1997; Goddard 2001; Goddard et al., 2004). Where aggregate measures of individual efficacy beliefs do not vary greatly across schools, aggregate measures of teachers’ beliefs about the collective capability of their faculties vary greatly across schools, and are strongly linked to student achievement (Goddard et al., 2004).

The power of collective efficacy perceptions to influence the organizational life of a school lies in the socially transmitted expectations for action, or normative press (Sampson et al., 1999). In schools with high levels of perceived collective efficacy, teachers learn that extra effort and educational success are the norm. These expectations for action create a normative press that encourages all teachers to do what it takes to excel, and discourages them from giving up when faced with difficult obstacles (Goddard et al., 2004). Collective efficacy repeatedly emerges in research as a powerful predictor of student achievement, able to offset the effect of student demographic variables and explain high proportions of between-school variance in student achievement across a variety of grades and subjects (Goddard et al., 2000; Goddard, 2001; Goddard, et al., 2003; Goddard et al., 2004; Ross et al., 2004).

The Internal Coherence framework focuses on both individual and collective efficacy, because, as a matter of organizational development, individuals “learn” the effects of their practice on students both by reflecting on their independent work in classrooms and by working collaboratively on common instructional goals. We presume there to be an ongoing, reciprocal growth process between individual and collective efficacy beliefs as the leadership and organizational domains of Internal Coherence improve. As faculty collaboration becomes more effective and more tightly linked to instruction and student learning, opportunities for individual teachers to augment their instructional repertoires and confidence in their abilities increase (Rosenholtz, 1989; Ross et al., 2004). Should improved classroom practice lead to improvements in student achievement, schools will experience a collective mastery experience, one of the most powerful sources of efficacy-shaping information (Bandura, 1997).

Clinical Practice

Assessment Protocol

One of the problems with educational practice, as compared with other fields, is that education has a relatively weak basis in clinical practice.  That is, in fields like medicine, social work, and counseling, practitioners have access to a collection of assessment tools and practices that support their work with clients, and they are trained to use these clinical tools as part of their practice.  Despite the fact that there is a relatively solid and useful research base related to school improvement, the field of education has been slow to develop useable clinical tools to support practitioners in their work.  The ICAP is an attempt to develop one such tool.

The ICAP grows out of more than a decade of research on school improvement and accountability, and, using well-tested and -documented survey items and observational protocols from the research, creates a way for system-level and school-level leaders to create a common body of data for the support of school improvement. Data generated by the teacher survey is corroborated and contextualized by data derived from classroom observation, as well as teacher and principal interviews, and compiled into a profile of a specific school’s capacity to engage in deliberate improvements in instructional practice and student learning across classrooms. In addition to providing a platform for individual school improvement, tools like the ICAP produce interesting evidence on the process of school improvement, and on the variety of responses that schools have to attempts to support their improvement, which can be used to inform the larger knowledge base in the field.  Our aim is to produce a protocol that can be used primarily to support practitioners in their work, but also to help improve our longer-term understanding of the processes of school improvement and how to support them.

The first generation of the protocol was field-tested in the school year 2007-2008 with 21 low-performing schools in Boston, focused on the middle grades.  As part of the field test, Elmore and Forman worked for a full school year not just on the development of the protocol but also in close collaboration with one chronically underperforming school to document and support their response to the process.  The second generation of the protocol is currently being administered in 13 Boston schools that have been identified by system-level administrators for intensive support over the 2010-2011 and 2011-1012 academic years.  This work will include support for Boston’s Chief Academic Officer and team of Academic Superintendents in how to integrate the ICAP data into their supervisory relationships with principals, in service of developing and supporting instructional improvement efforts in their schools.

The internal coherence assessment is intended to be a clinical assessment, rather than a purely research-oriented instrument.  That is, it is designed to generate information about the school that can be used by people in the school and their system-level supervisors to inform decisions about what the school might do to improve the quality, consistency, and effectiveness of instructional practice and student performance.  It is designed to provide one source of information, among many, for school improvement, and it should not be seen as displacing other important sources, such as school improvement plans, student assessment data, analysis of student work, or curriculum-based assessments.

The components of the ICAP are described below.

Teacher Survey

The teacher survey is administered to all teachers in the school and requires approximately 30 minutes to complete.  The survey generates evidence on teachers’ perceptions in the three main domains of Internal Coherence:  (1) school leadership, broadly defined to include administrators’ role in organizing, managing, and supporting teachers in their instructional roles; (2) individual and collective efficacy, broadly defined as teachers’ perceptions of the degree to which they influence factors that determine student achievement, individually and collectively, in the school; and (3) organizational structures and processes, broadly defined as the structures, routines, and practices that the school uses to make instructional decisions.

The data from the teacher survey is used to produce a school profile of teachers’ responses on the three dimensions that can be used by teachers, school leaders, supervisors, and support staff to identify key areas on which individual schools can focus their school improvement work.  Because the teacher survey constitutes the major source of evidence on the school as an organizational environment, it is critical that all teachers respond to it. We have placed the minimum response rate at 65% for the production of a profile. The product of this part of the assessment will be a report of teacher survey results, grouped by the major categories of the assessment. (See Appendix B for survey scales).

Interviews / Focus Groups

In addition to the teacher survey, the internal coherence assessment requires a set of interviews, and optional focus groups, that serve three purposes:  (1) to corroborate, with the voice of teachers, some of the broader themes and findings of the teacher survey; (2) to provide direct evidence from school leaders, who are not included in the teacher survey, on the major dimensions of the internal coherence framework; and (3) to provide specific contextual information about the school that is useful in understanding its school improvement work. Currently, the selection of teachers for both interview and observation is left to the discretion of the principal. Because of the small sample size (typically 4-6 teachers are interviewed and 4-6 observed, with the possibility of overlap) results of this component of the diagnostic serve purely for the learning of the research team and district leadership: interview data is not reported back to administrators in schools.

Classroom Observations

The purpose of the classroom observation component of the assessment process is to obtain a thumbnail sketch of the level of cognitive demand of instruction and the degree of variability from classroom to classroom in a given school (City, Elmore, Fiarman, and Teitel 2009).  It is not designed to provide an exhaustive profile of instructional practice, nor is it designed to be used in an evaluative fashion to judge how well the school is doing, but to provide additional information on the organizational context of the school.

School Profiles

The data from the survey, interviews, and observations are developed into a school profile for each of the schools in the sample. All profiles are provided to district leadership, and individual principals receive copies of the profile for their school. Profiles contain summary-level data on each of the dimensions of the organizational coherence framework, as well as selected item-level data from the survey.  In the pilot phase, these school-site summaries have also included comparisons to mean scores by dimension within the sample, and summary comparisons where selected items were anchored on nationally-normed surveys such as the School and Staffing survey.   

Intervention

The second major element of the Internal Coherence framework is the connection between the ICAP profile and the practice of participating teachers, principals and district leaders. Once the ICAP has been completed, school profiles are used as a point of departure for a series of professional development events and supports for school and district leaders. The Internal Coherence professional development model is designed to engage practitioners in an analysis of the conditions of their own organizations, and the processes by which they can build sustained learning environments within them. In other words, the goal of the IC professional development is to use ICAP data to enhance the ability of leaders at the school and system levels to create the conditions necessary for educators to engage in the continuous improvement of their classroom practice over time.

The problem of linking the system, the school, and the classroom around school improvement is not so much a problem of the lack of knowledge as a problem of putting the knowledge we have to work around critical gaps in theory and practice. Hence, the professional development intervention anchored in the ICAP data focuses not on developing new models of school leadership or accountability, but in filling in gaps in the existing research and in connecting existing research more tightly to the practice of school improvement.  The IC professional development responds to three main gaps in the connection between research and practice in the leadership of school improvement:

• Weak theories of action linking leadership practice to improvements in the instructional core of schools.  There is a rich and useful literature around the characteristics of effective educational leaders, and the role of leaders in improving low-performing schools.  Where the literature is weakest, however, is in explicit linkages between the practices that leaders use to improve schools and the instructional core, defined as the teacher and the student in the presence of content (City, Elmore, Fiarman and Teitel, 2009).  As the number of schools in need of improvement increases, it becomes more critical to focus on the actual practices that support improvement of classroom instruction, not just the general characteristics of successful leaders.
• Weak linkages between existing assessments of organizational capacity and the practice of educational leadership.  There have been several lines of research over the past fifteen years on the internal characteristics of schools that seem to predict success with student learning, beginning with high-level correlational studies, continuing with more focused qualitative studies, and, lately, with more powerful statistical models that focus on clearer operationalization of individual and organizational factors that predict student performance.  While this literature informs the work of school improvement and the preparation and professional development of leaders at a global level, it has not yet been connected to concrete assessment practices and professional development strategies that can be used by school leaders to build and implement school improvement strategies.
• Weak models of professional development linking multiple assessments, including assessments of organizational capacity, to the practice of educators in their own organizational settings.  One consequence of the shift to performance-based accountability is that most schools now have an abundance of data on student performance with which to make decisions.  Most state and local accountability systems focus on providing student performance data; few, if any, focus on providing schools with information about how their organizations actually work, and how they might work more powerfully to support instructional improvement.  Most school leaders know, when they look at evidence of student performance, that they are supposed to “do something” to bring the resources of the organization to bear on instructional practice; they are usually less clear about what to do.

Research on professional development for teachers has, for decades, emphasized that the most effective training for teachers – measured in terms of its impact on teacher practice and student learning – is close to the classroom, involves direct interaction with teachers individually and in groups around concrete instructional problems, and results in increased knowledge of content and pedagogy that can be directly applied in the classroom.  The research also suggests that the most effective professional development practices are the least-frequently employed by local schools and districts (Darling-Hammond, et al. 2008; Desimone, Porter, et al. 2002; National Research Council 2000).

The Internal Coherence professional development model is designed around three basic principles:  (1) Developing a safe environment for practitioners to examine evidence from the ICAP profile and to connect that evidence to their overall improvement plans, the internal structures and processes of their schools, and their relationships with system-level supervisors and teachers;  (2) Using data from multiple sources, including the ICAP and measures of student performance, help leadership teams to develop their faculties’ ability to make binding commitments at the classroom, group, and school level to engage in specific actions leading to improvements in performance; and (3) Developing the self-monitoring and self-correcting practices necessary to assess whether the agreed-upon actions are leading to the desired results in instructional practice and student performance.

The goals of the IC professional development are to build school and district leaders’ knowledge of and ability to act on the characteristics of effective school organization.  We will build the knowledge base of school and system leaders around the ways in which schools as organizations affect teachers’ learning, teachers’ practice, and their students’ achievement over time. Specifically, we will use the data from the assessment as a platform for a professional development for school leadership teams and system-level leaders around building shared goals for student learning, collaboration and collective responsibility for instructional practice and reflective professional inquiry. We anticipate that over time, one effect of the IC professional development will be increased levels of collective efficacy in school improvement.

Our initial work with schools suggests that the effect of the Internal Coherence process occurs in three stages:  (1) Recognition, or using the assessment data to engage in structured inquiry about the conditions of teaching and learning in the school, and the leadership practices and organizational processes that surround them; (2) Planning, or using the Internal Coherence data, coupled with other data sources, to frame a set of specific actions that will lead to improved instruction and student performance; and (3) Commitment, or a formal process of agreeing on specific actions and the processes by which teachers and administrators will hold each other accountable for carrying them out.

This intervention is intended to foster a strong professional community in participating schools, characterized by a faculty able to collectively run organizational processes around improving instruction and student learning in an ongoing fashion. This effect should be visible in both survey data in the domains of leadership practice, individual and collective efficacy and organizational processes as well as in qualitative interview data and the observation of team meetings.  Our work, and the work of others, suggests that these processes have to be repeated several times before they begin to reshape the culture and processes of schools.  The expected effects of the Internal Coherence assessment and professional development model should be seen within two or three cycles.  As part of the process, schools are required to track their own improvement and our qualitative research will also track their improvement over the course of their participation in the project.

Because the organizational learning literature suggests that organizations improve when they build capacity for learning and leadership broadly in the organization (Edmondson 2002; Crossen and Berdrow 2003), the IC professional development model adopts a functional definition of leadership. This means that we define leadership roles in terms of their functional relationship to the task at hand, in this case school improvement.  The implication for the project design of this definition is that we will engage people across leadership roles within the system and the organization as a whole, rather than focusing on people traditional leadership roles (Elmore 2004; Spillane 2006; Fullan 2007). Leadership teams, rather than individual leaders, participate in PD sessions with IC facilitators. In between sessions, leadership team members are charged with developing strategies for bringing the learning back into their organization. Their “ticket of entry” for the subsequent session includes some record of the work they have done with faculty (presentation slides, video, etc.) as well as a formal reflection on the process.

A final component of the conceptual model is what we have come to call in our practice “transfer of agency” (City, Elmore, Fiarman, and Teitel 2009; Honig and Hatch 2004). The purpose of the project is to build capacity at the classroom, school, and system levels, and to document how that process occurs, rather than to institutionalize a continuing consulting or research relationship between the IC research team and individual schools.  The ICAP and professional development process is designed to create the conditions in which a system and its component schools will learn to manage and assess their own processes in pursuit of a strategy for ongoing instructional improvement. Currently, the assessment and professional development process is intended to run for two years. The third year is envisioned as a “turn key” year in which we continue the assessment and professional development work, but we work with a team of school and system level leaders to develop a process for turning the work over to the organization at the end of the project.  Our goal is to leave the organization with enhanced capacity to monitor its own work, and to demonstrate how that might work in other settings through documentation of the process.