By Joshua Benton and Holly K. Hacker
Staff Writers
Page 21A
The News’ analysis found more than 50,000 students whose answer sheets on the TAKS in 2005 and 2006 appear to have been involved in cheating. But there are several reasons to believe that underestimates the problem – perhaps severely.
The News’ data did not include about 20 percent of the state’s answer sheets.
That’s because the Texas Education Agency withheld those students’ scores because of federal privacy laws. If those missing kids were flagged at the same rate as the rest of the state, about 12,000 additional tainted answer sheets would have been detected.
The News’ analysis used conservative assumptions at each step of the way.
The News and researcher George Wesolowsky set a high threshold for how similar student answer sheets had to be to be flagged for cheating. The intention was to minimize the chances of a false positive – students being flagged improperly.
The threshold for most of the analysis was set so that the probability that a school with no cheaters will have students flagged is approximately less than one in 10,000. That means there has to be extensive copying before a pair of students is flagged. The side effect is that some students – including those who copied only a few answers – will go undetected.
The methodology used by The News is substantially more conservative than the one used by the test-security firm Caveon in its analysis of Texas scores.
As a for-profit company, Caveon keeps much about its methods secret. But by analyzing the technical parts of its report on Texas, it’s possible to see how many students the company flagged for possible cheating on the 2005 TAKS. In its search for improper collaboration among students, Caveon consistently found more cheating than The News’ analysis did.
Caveon provided comparable data for only five tests. In 11th-grade math in 2005, Caveon flagged 6.1 percent of all answer sheets statewide. The News flagged 1.7 percent. In sixth-grade reading, Caveon flagged 1.9 percent of answer sheets. The News flagged 0.4 percent.
The methods used by The News don’t identify students or teachers who copy mostly correct answers.
The News’ method looks for students who share large numbers of unusual incorrect answers. But if students – or teachers – copy only right answers, resulting in perfect or near-perfect scores, it’s unlikely they’ll be detected.
For example, at Jesse Jackson Academy last year, 11th-graders did very poorly on the science test. Nearly every student bombed the test with almost identical wrong answers, seemingly copied from a single – and very bad – source. More than 90 percent of the students were flagged for cheating; only 5 percent passed.
But on social studies tests, Jackson’s students were superstars. At 10th grade, students had the highest average score of any school in the state – beating out even the state’s best schools. Thirty-two students did not miss a single question. (To put that in context, on the same test at Dallas’ School for the Talented and Gifted, which is about the same size, only three students had perfect scores.) And in 11th grade – the same group of students who bombed the science test – Jackson had the 16th-highest score in the state, out of nearly 1,500 high schools.
Those are remarkable results for a school that has earned the state’s lowest rating in six of the last seven years and caters primarily to recovered dropouts. But because the students got so few questions wrong on the social studies tests, none of those answer sheets was flagged.
There is no way to know how many students might be missed by The News’ analysis because they cheat effectively.
The News’ analysis uses only one detection method.
Caveon, for instance, used four different methods, including ones that look for unexplained sudden gains in performance and high levels of erasures on answer sheets. The News’ analysis looks only for unusually high similarity among answer sheets. That’s the cheating-detection method with the most support from researchers, but it leaves behind cheaters who might be detected through other means.