A growing body of evidence indicates that many urban charter schools boost the standardized test scores of disadvantaged students markedly. Attendance at oversubscribed charter schools in Boston for example—those with more applicants than seats—increases the test scores of low-income students by a third of a standard deviation a year, enough to eliminate the black-white test score gap in a few years of attendance.
The achievement gains generated by Boston charters are in line with those generated by urban charters elsewhere in Massachusetts, as we have shown in studies of a Knowledge Is Power Program (KIPP) school in Lynn, Massachusetts and in an analysis of charter lottery results from around the state. Similar effects have been found in New York City and in a nationwide study of oversubscribed urban charter schools.
A defining feature of most of Massachusetts’ urban charter schools is No Excuses pedagogy, an approach to urban education described in a book of the same name. No Excuses schools emphasize discipline and student conduct, traditional reading and math skills, extended instruction time, and selective teacher hiring. Massachusetts’ No Excuses charters also make heavy use of Teach for America corps members and alumni, and they provide extensive and ongoing feedback to teachers.
Although the positive effects of urban charter school attendance on test scores is increasingly well documented, the interpretation of these results is disputed. High test scores need not reflect additional educational value. They might instead reflect an emphasis on state-mandated standardized testing. In other words, teachers at charter schools might simply “teach to the test,” at the expense of focusing on the development of skills with a longer-term payoff. Charter schools would appear to have a particularly strong incentive to teach to the test because schools whose students do poorly on state tests can be closed.
We assess whether attendance at charter high schools produces meaningful long-term gains for historically disadvantaged urban youth, focusing on outcomes linked to college attendance.
We assess whether attendance at charter high schools produces meaningful long-term gains for historically disadvantaged urban youth, focusing on outcomes linked to college attendance. Because of the strong link between college and earnings, schools that boost college attendance are very likely to generate lasting gains for their students.
Data
Our study is based upon applicants to admissions lotteries at six Boston charter high schools. These six schools account for the bulk of charter high school enrollment in Boston today.
A total of 3,685 applicants sought a seat at these charter schools from 2002 through 2009. Applicants in our sample tend to have higher baseline test scores than the traditional Boston Public School population, are more likely to be black, and are less likely to have limited English proficiency.
Controlling for differences in traditional and charter school students
Our aim is to measure the effect on student achievement caused by charter school attendance. Yet we face a critical challenge: there are additional differences beyond the “yes-or-no” of charter school attendance between students attending charters and students attending traditional district schools. Motivation, ability, and family background are some examples of potential differences between charter and non-charter students that may drive not only the decision to attend a charter school but also student achievement.
How is it possible to isolate the extent to which charter student achievement is caused by charter attendance and not these other factors? The task is particularly challenging due to data limitations on the factors characterizing the differences in the two groups of students
To overcome this problem, our study exploits a “natural experiment” that arises because of Massachusetts state laws. Specifically, the state mandates an admissions lottery when there are more charter applicants than seats. The lottery allows us to compare students offered a seat and students not offered a seat, removing from our analysis confounding factors such as motivation, ability, and family background.
There are some nuances; not all students who are offered a seat in the lottery enroll in a charter school, and some who are not offered a seat find a way into the school through the waitlist or re-application in subsequent years. Thus, there is a difference between the students receiving offers and students who actually attend charter schools. To account for this, we use the econometric technique of instrumental variables to convert an estimate of the effect of receiving an offer into an estimate of the effect of attending a charter school. This amounts to dividing the difference in average outcomes between offered and non-offered by the difference in average enrollment in these two groups.
Most of the applicants to charter schools who do not receive an offer go on to matriculate at traditional public schools. Our estimates, then, can be interpreted as the effect on achievement caused by charter attendance for students who would take a charter seat when offered one in a lottery, but would otherwise enroll in a traditional public school.
A major advantage of this approach is that it is possible to test its validity. If the approach is valid, then there should be no difference in characteristics between applicants offered a seat by lottery and applicants not offered a seat. We find exactly that; there is no evidence of differences in race, gender, special education, limited English proficiency, or subsidized lunch status. This fact suggests that we have a valid comparison.
A potential threat to our approach involves follow up differences between offered and non-offered students. Here, too, our study is on firm ground due to consistent collection of achievement data by the state of Massachusetts. We observe standardized test scores for roughly 80% of our sample two years after charter seats are allocated. Three-quarters of our sample is still in a Massachusetts public school in grade 12. Importantly, there is no difference in observation rates by offer status.
Empirical Findings
The High-Stakes Test
The Massachusetts state government evaluates schools according to student scores on a high-stakes exam: the Massachusetts Comprehensive Assessment System (MCAS). Consistent with our findings in earlier studies of charter schools in Massachusetts, charter school attendance has a large effect on the likelihood that applicants score in the upper-two score categories for the MCAS.
This is shown in Figure 1A. The figure compares the estimated distribution of English Language Arts (ELA) scores for applicants who take a charter seat when offered one in a lottery (labeled treated compliers in the diagram) to applicants who are not offered a seat and enroll in a traditional public school (untreated compliers). Charter attendance shifts the ELA test score distribution to higher values and boosts the likelihood of meeting competency standards by 15 points. Figure 1B shows charter attendance similarly pushes math test scores to higher values.
Beginning with the high school class of 2005, the state has used the MCAS to determine who qualifies for public university tuition waivers, an award known as the Adams Scholarship. Charter attendance increases the chance of qualifying for an Adams Scholarship by 24 percentage points for a mostly poor, minority population.
A. English Language Arts
B. Maths
FIG. 1.—Charter (“treated”) and non-charter (“untreated”) distributions for MCAS scaled scores in English language arts and math. The sample is restricted to lottery applicants projected to graduate between 2006 and 2013. Dotted vertical lines indicate MCAS performance category thresholds (220 for Needs Improvement, 240 for Proficient, and 260 for Advanced). See original paper for details.
AP and SAT Exams
If charter school teachers “teach to the test” at the expense of spending time developing skills with a longer-term payoff, then the gains on the MCAS should not carry over to gains on exams that are correlated with long-term success, but are not part of the state’s accountability system. But we find that the gains on high stakes exams for charter school students (relative to students in traditional schools) also translate to gains on Advanced Placement (AP) exams, which count towards college credit, and the SAT, a globally recognized college admission test.
Almost 60% of charter students take at least one AP test, compared with 28% of students in traditional public schools.
Charter attendance doubles the likelihood that a student sits for an AP exam, with especially large gains in the share of students taking science exams. Almost 60% of charter students take at least one AP test, compared with 28% of students in traditional public schools. Attending a charter school quadruples the likelihood of taking an AP Calculus exam. Charter attendance increases the fraction of students scoring high enough on AP Calculus to qualify for college credit from 2% for non-charter attendees to 13% for charter attendees.
Attending a charter school quadruples the likelihood of taking an AP Calculus exam. Charter attendance increases the fraction of students scoring high enough on AP Calculus to qualify for college credit from 2% for non-charter attendees to 13% for charter attendees.
As shown in Figure 2, charter attendance also boosts SAT scores sharply, especially in math. The figure compares the distribution of SAT scores for treated compliers – students who attended a charter school as a result of winning a lottery – to untreated compliers – students attended a traditional public school after not winning a seat in a charter school lottery. As the figure shows, the distribution shifts to the right with charter school attendance. Specifically, charter attendance boosts average math scores by 52 points. The estimated SAT gains are about as large as the estimated gains on the state’s high-stakes high school exit exam, despite the fact that SAT scores are unrelated to state-mandated accountability standards. The score gain in verbal and writing is about 26 points in each subject.
A. Composite SAT score distribution
B. SAT Reading score distribution
C. SAT Writing score distribution
D. SAT Math score distribution
FIG. 2.—Charter (“treated”) and non-charter (“untreated”) distributions for SAT scores. The sample is restricted to lottery applicants who are projected to graduate between 2007 and 2013. See original paper for details.
High School Graduation and College Enrollment
Though charter attendance initially reduces the likelihood a student graduates on time by 14.5 percentage points, this negative estimate falls to zero when the outcome is graduation within five years of ninth-grade entry. It appears that many charter students take an additional twelfth-grade year to graduate, perhaps due to more rigorous graduation requirements at charter schools.
Although overall college enrollment effects are not statistically significant, charter attendance induces a clear shift from 2-year to 4-year colleges, decreasing 2-year college-going by 11 points and increasing 4-year college-going by 13 points. These gains are most pronounced at 4-year public institutions in Massachusetts.
We estimate the effects of charter attendance on college selectivity as measured by Barron’s rankings. College selectivity appears unchanged by charter enrollment.
Additional Results
Our analysis also links gains on accountability assessments to gains in later outcomes, finding that effects on the two sets of outcomes are highly positively correlated. In other words, whether or not state assessments are of intrinsic interest, gains on state tests predict gains elsewhere.
Next, we investigate the characteristics of students who benefit most from charter attendance. Overall, attendance at Boston’s charter high schools boosts key outcomes for most subgroups, with large effects on at-risk groups including boys, special education students, and those who enter high school with low achievement.
Overall, attendance at Boston’s charter high schools boosts key outcomes for most subgroups, with large effects on at-risk groups including boys, special education students, and those who enter high school with low achievement.
A criticism sometimes directed at charter schools is that the schools boost student achievement by retaining high-performing students and asking low-performing students to leave. If this “selective retention” were substantial, it would lead to a concentration of high-performers at charters who may have positive effects on their peers. The question, then, is whether this mechanism explains why charter attendance has positive effects on students.
We find that the gap in average peer performance between charter and non-charter schools actually shrinks over the course of high school. Non-charter high schools experience high dropout rates among the lowest-performing students, which means a higher average over time. These results suggest that the mechanism for positive charter effects is not the peer effect from selective retention.
Looking Ahead
Our results suggest that the gains from attendance at Boston’s high-performing charter high schools extend well beyond high-stakes tests. The math skills carry over to exams that represent college readiness, and more students attend 4-year colleges. Since most of the students in our study first enter a charter school when they are 14 or 15 years old, these results also weigh against the view that high school is too late for cost-effective human capital interventions.
The estimates reported here show gains for recent cohorts of charter applicants. As these cohorts continue to progress through college and enter the labor market, we plan to use our lottery-based research design to determine whether the effects reported here extend to college completion, employment, and earnings.
This article summarizes “Stand and Deliver: Effects of Boston’s Charter High Schools on College Preparation, Entry, and Choice” by Joshua Angrist, Sarah Cohodes, Susan Dynarski, Parag Pathak, and Christopher Walters.
