WASHINGTON — A new report from the National Research Council offers a framework for evaluating the effects of a 2007 reform law on the District of Columbia’s public schools. The evaluation program must include systematic yearly public reporting of key data as well as in-depth studies of high-priority issues, such as teacher recruitment and retention, the report says. As part of the evaluation program, the Mayor’s Office should produce an annual report to the city on the status of the public schools, including an analysis of trends and all the underlying data.
The evaluation program should be structured to provide ongoing, independent monitoring and feedback on a number of key aspects of D.C.’s public education system, which are described by the committee that wrote the report. “Although the immediate goal for the District is to evaluate the changes enacted in 2007, we also see an opportunity to build a continuing program of analyses that will be useful regardless of future political or personnel changes,” said Christopher Edley, committee co-chair and dean of the School of Law at the University of California, Berkeley.
In 2007 the Public Education Reform Amendment Act (PERAA) shifted control of D.C. Public Schools (DCPS) from an elected school board to the mayor and created the position of chancellor, among other changes. The act also mandated an independent five-year evaluation to determine whether enough progress has been achieved to warrant continuation of the new structure. The D.C. City Council asked the Research Council to provide guidance on how to structure the evaluation and to offer initial feedback about implementation of PERAA, to the extent possible.
Report Offers First Impressions, With Cautions
The report offers some first impressions of school reform under PERAA and notes that the city and DCPS have made a good-faith effort to implement the required changes, but the committee cautions that it is premature to draw general conclusions about the reforms’ effectiveness at this time.
Data suggest that a modest improvement in student test scores has continued since PERAA was enacted. However, the report stresses that scores on their own cannot identify the reasons for that improvement. Further examination is necessary to determine whether they can be linked to the reforms or are the result of other factors, such as demographic changes in D.C.’s highly mobile student population. Also, because the rising scores are averages, further analysis and monitoring are needed to determine how various subgroups of students — such as particular grade levels or students in particular areas — are faring; some could be making gains while others are not.
The report emphasizes that student test scores should not be the only factor used to judge improvement in the school system. A comprehensive evaluation should address the full range of district responsibilities: quality of teachers, principals, and other personnel; quality of classroom teaching and learning; capacity to serve vulnerable children and youth; promotion of family and community engagement; and quality and equity of operations, management, and facilities.
Although the committee found that the administrative and governance changes required by PERAA have generally been carried out, the report notes that the Office of the Ombudsman was defunded three years after its creation. Since the office was intended to be the primary channel for parents to communicate with school officials and seek redress for complaints, its absence is significant, said the committee.
Evaluation Program Should Be Long-Term, Independent
The report offers a framework for evaluating key elements of reform under PERAA that affect student outcomes — including management structures and roles and conditions for learning — as well as the student outcomes themselves.
A set of specific, measurable indicators should be established as soon as possible, building on suitable indicators already collected in the District, the report says. These would track how well the city’s public schools are doing, how well the key strategies the city has chosen to improve education are working, the conditions for student learning, and the capacity of the system to produce valued outcomes.
In addition, in-depth studies should be designed to provide deeper analysis of specific questions about high-priority issues, including both the targets of PERAA and long-standing challenges in District schools, such as special education. These studies can answer questions about the causes of trends seen in indicator data, explore potential reasons for disappointing outcomes, provide information to help alter existing strategies, or help explain why some strategies appear to be working better in some schools than in others. They are key, the report notes, because they are the means by which evaluators can answer policymakers’ questions about the effects of particular policies and practices.
To develop and sustain an infrastructure for ongoing research and evaluation of its public schools, D.C. will need to engage local universities, philanthropic organizations, and other institutions, the report says. The program must be independent of school and city leaders and responsive to the needs of all stakeholders, and its research should meet the highest standards for technical quality. There is no one model for the city to follow, but examples of research and evaluation programs that have been created in other cities — such as Chicago, New York City, and Baltimore — could be useful as D.C. develops its own program, the report notes.
“An independent evaluation program that can be an ongoing source of objective information and analysis will be an invaluable resource for the city,” said committee co-chair Robert Hauser, executive director of the National Research Council’s Division of Behavioral and Social Sciences and Education. “There are no quick answers. Education reform itself is a long-term process, and the evaluation of its outcomes also has to be seen in the long term.”
The committee will discuss the report at a public briefing to be held at 2 p.m. on Sunday, March 6, at the National Academies’ Keck Center, 500 Fifth St., N.W., Washington, D.C. A live audio webcast of the briefing will be available at http://national-academies.org.
The study was sponsored by the government of the District of Columbia, the National Science Foundation, National Research Council, Spencer Foundation, Diane and Norman Bernstein Foundation, World Bank Community Outreach Program, Philip Graham Fund, City Bridge Foundation, and the Kimsey Foundation. The National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council make up the National Academies. They are private, nonprofit institutions that provide science, technology, and health policy advice under a congressional charter. The Research Council is the principal operating agency of the National Academy of Sciences and the National Academy of Engineering. For more information, visit http://national-academies.org. A committee roster follows.
Contacts:
Sara Frueh, Media Relations Officer
Luwam Yeibio, Media Relations Assistant
Office of News and Public Information
202-334-2138; e-mail [email protected]
Pre-publication copies of A Plan for Evaluating the District of Columbia’s Public Schools: From Impressions to Evidence are available from the National Academies Press; tel. 202-334-3313 or 1-800-624-6242 or on the Internet at http://www.nap.edu. Reporters may obtain a copy from the Office of News and Public Information (contacts listed above).
NATIONAL RESEARCH COUNCIL
Division of Behavioral and Social Sciences and Education
Center for Education
Committee on Independent Evaluation of D.C. Public Schools
Christopher Edley Jr. (co-chair)
Professor of Law and Dean
School of Law
University of California
Berkeley
Robert M. Hauser 1 (co-chair)
Executive Director
Divison of Behavioral and Social Sciences and Education
National Research Council, and
Vilas Research Professor of Sociology
Center for Demography of Health and Aging
University of Wisconsin
Madison
Beatrice F. Birman
Managing Research Scientist
Education, Human Development, and Workforce Program
American Institutes for Research
Washington, D.C.
Carl A. Cohn
Research Practitioner
School of Educational Studies
Claremont Graduate University
Claremont, Calif.
Leslie T. Fenwick
Dean
School of Education
Howard University
Washington, D.C.
Michael J. Feuer
Dean
Graduate School of Education and Human Development
George Washington University
Washington, D.C.
Jon Fullerton
Executive Director
Project for Policy Innovation in Education
Harvard University
Cambridge, Mass.
Fernando A. Guerra 2
Director of Health
San Antonio Metropolitan Health District
San Antonio
Jonathan Gueverra
Chief Executive Officer
Community College of D.C.
Washington, D.C.
Jonathan Guryan
Associate Professor of Human Development and Social Policy, and
Faculty Fellow
Institute for Policy Research
Northwestern University
Evanston, Ill.
Lorraine M. McDonnell
Professor and Chair
Department of Political Science
University of California
Santa Barbara
C. Kent McGuire
President and CEO
Southern Education Foundation
Atlanta
Maxine F. Singer 1, 2
President Emeritus
Carnegie Institution of Washington
Washington, D.C.
William F. Tate IV
Chair and Professor of Education
Department of Arts and Sciences
Washington University
St. Louis
RESEARCH COUNCIL STAFF
Patricia Morison
Study Director
1 Member, National Academy of Sciences
2 Member, Institute of Medicine