We demonstrated in part one that Success Academy has very high achievement relative to the economic needs of the network’s students. However, Success Academy skeptics believe their high scores might be explained by other factors: student opt-in, student attrition, and smaller populations of English learners and students with disabilities. All these factors involve questions of selected student populations. We consider each one below, as well as a comparison to schools that clearly have a selected population: citywide gifted programs.
Opt-In Bias
Charter schools are different from zoned schools because parents must choose to send their child and indicate their interest by submitting an application several months before the start of the school year.
Research suggests students who apply to NYC charter schools, on average, scored higher in the previous year than the students who didn’t submit an application. Therefore, a portion of the over-achievement that charters experience reflects the fact that applicants are a select group, a motivated group with the awareness and wherewithal to apply in advance. An estimate of this “opt-in” effect suggests it may be as high as 9 scale score points on the NY exams. (The true opt-in effect is likely lower than 9. Please see the footnote for more details.)[1]
As shown in the graph below, once the opt-in effect is take into account, it is unclear whether some charters which initially appeared to be positively impacting their students are actually getting results beyond the opt-in effect. Other charters including Success Academy, Ichan, and Uncommon Schools show overachievement significantly above the estimated opt-in effect.
Attrition Bias
A second issue that has been raised as potentially biasing Success Academy’s achievement is the suggestion that the school is “pushing out” students who it suspects will not score well on exams. This loss of students is referred to as attrition.
In December of 2012, Beth Fertig and her colleagues at WNYC’s Schoolbook published an analysis of attrition across all NYC charters along with comparisons to the five NYC Bureaus as well as the CSD’s with the greatest number of charters.
The analysis revealed a few important things. First, attrition is significant citywide and particularly high in communities with larger populations of low-income students. Second, it showed that indeed some charters experience attrition rates above the citywide average and above the CSD’s where they operate.
As shown in the chart below, Success Academy schools experienced annual attrition of 10 percent, approximately 3 percent less than the citywide average and about 4 percent less than would be expected based on the boroughs where they operate.
Backfill, another component of attrition that is discussed in a thorough piece by Chalkbeat’s Sarah Darville, relates to whether charter schools choose to fill vacated seats or whether they let cohorts shrink as students leave the school. The Success Academy schools have a policy of backfilling grades K thorough 3 but not the older grades.
If leaving students on average are lower performers, the choice not to backfill would raise the achievement in later grades. However, limiting analysis to the third grade, where Success Academy does backfill, results in overachievement of 33 points versus 39 for all grades. This indicates the choice not to backfill does not explain much, if any, of Success Academy’s overachievement.
ELL & Special Needs Bias
A third area of potential bias flows from the percentage of students served who are English Language Learners and qualified for special needs.
Success Academy schools serve fewer ELL students than zoned schools in the districts where they operate. Reviewing the percentage of ELL students at Success Academy’s tested schools indicates they serve about half as many ELL students (6 percent) as zoned schools nearby (12 percent).
While this is an important starting point, there is a confounding factor at play that must be considered. The basic idea of ELL interventions is that the students eventually learn English and then move out of the program. At Success Academy, 27 percent of students passed the reading & writing ELL exam compared to 19 percent at the district schools. Similarly, 65 percent of Success Academy students passed the spoken ELL exam compared to 50 percent at their zoned counterparts. This suggests that part of the reason participation rates are lower at Success Academy schools is because students attending the schools are more likely to master English and move out of the program.[2]
It is also worth noting that the performance gap between Success Academy ELL students and non-ELL students is relatively small, suggesting that if the schools were to serve a larger number of ELL students, scores would likely only fall marginally. Ninety-one percent of ELL students passed the math exam while 41 percent passed the English exam. Both rates are substantially higher than the city.[3]
Special needs is another area which often receives attention in the media; however, administrative data indicate that Success Academy schools serve a similar share of special needs students relative to their zoned counterparts. Both serve approximately 12%. Further, the data show the Success Academy schools are more effective at mainstreaming the students, with twice as many students moving to less restrictive settings.[4]
Comparison to Citywide Gifted and Talented
A final context within which to consider bias and the Success Academy achievement is in comparison to other schools with clear selection bias. Recall in part one of this analysis, we highlighted the performance of citywide gifted programs.
In order to gain admission to a citywide gifted program, students must take a standardized test. InsideSchools reported that for the 2014 year, three times as many students scored in the 99th percentile as there were seats available at the five most selective citywide programs.
As a result of the limited number of seats and the large number of test-takers, students admitted to a citywide Gifted & Talented program are a select bunch. In fact, they start out wholly in the top 1%.
For the issues of opt-in and ELL status discussed above, the Success Academy students may to some extent also be a select bunch. However, it is quite challenging to believe they could ever be a more curated group than students required to score in the 99th percentile on an admissions exam. Yet, surprisingly, the overachievement seen at Success Academy is double the overachievement seen at Gifted and Talented schools.
Consideration of potential sources of bias indicates that a portion of Success Academy’s overachievement may be explained by factors related to the student population served. However, a majority of Success Academy’s overachievement remains unexplained by these factors; therefore, it seems the schools’ curriculum and operations are responsible for much of their success. In part three of this analysis, we explore the Success Academy approach and look for elements that may contribute to the network’s performance.
by John Keltz and Jarod Apperson
John Keltz (@keltz_) is a researcher for the Atlanta Public Schools. He graduated from Case Western Reserve with a bachelors degree in Economics and Math. He earned a Masters of Economics from The University of Wisconsin - Madison.
Jarod Apperson (@gradingatlanta) is a Graduate Research Assistant at Georgia State University. He graduated from New York University with a degree in Finance and Accounting.
[1] This is likely an upper bound. Stanford economist Caroline Hoxby’s 2009 NBER working paper “Charter Schools in New York City: Who Enrolls and How They Affect Their Students’ Achievement” showed that students who applied to charter schools from tested grades scored 0.22 standard deviations (median) in math and 0.23 standard deviations (median) in reading above their peers who did not apply. The standard deviation of scores on New York exams is approximately 35 points. This suggests an opt-in effect may be somewhere around 7 to 9 scale score points. Since the time that Hoxby’s data was collected, the rates of application have risen significantly in some of the CSD’s with large numbers of charter schools. With applicants representing a greater share of total students today than they did in 2006, the effect using 2006 data is likely greater than the effect today. Recent lottery data would need to be investigated further in order to more accurately estimate the current effect.
[2] Data collected from the New York City Charter School Center. The composite data reported above was developed weighting by the number of tested students from each school.
[3] Subgroup data was reported in a Success Academy press release.
[4] Data collected from the New York City Charter School Center. The composite data reported above was developed weighting by the number of tested students from each school.
nisbidotco liked this
gradingatlanta posted this