By Joshua Benton
Staff Writer
Page 1A
The Texas Education Agency is leaning toward severing ties with the company it hired to look for cheating on the TAKS test, in part because the results have generated negative publicity for the state.
The agency also has some concerns about some methods used by the company, Caveon, officials said.
“I don’t have a lot of confidence in them anymore,” state Education Commissioner Shirley Neeley said. “Right now, I’m sure not inclined to ask Caveon for anything anymore.”
TEA hired the company after a series of stories in The Dallas Morning News that found evidence of educator-led cheating on the 2003 and 2004 Texas Assessment of Knowledge and Skills. Its job was to use statistical analysis to identify schools where cheating might have occurred on the 2005 administration of the test.
Caveon flagged 699 schools, 171 in North Texas, as suspicious in one way or another – for example, because some students’ test scores shot up too quickly or because a group of students had identical or nearly identical answer sheets. That’s nearly one-tenth of all Texas schools.
In response, TEA created a task force on test security, added to its test-security staff and has announced plans to investigate all 699 schools to varying degrees.
But Dr. Neeley and other state officials have repeatedly said they had not planned to investigate any of the schools and that they have done so primarily in response to media coverage of Caveon’s findings.
“It’s how it’s been misconstrued that’s the problem,” said Robert Scott, deputy commissioner of TEA. “The statistical analysis may be fine. But the implications have been ‘everybody’s cheating.’ ”
Even though investigations are coming, state officials have said that Caveon’s methods are not reliable enough to evaluate the test scores of individual students and were intended to uncover “anomalies,” not cheating.
“Is it worth the trauma to put districts through an investigation if a flag ultimately doesn’t turn up anything?” agency spokeswoman Debbie Graves Ratcliffe said.
The agency has declined to give investigators Caveon’s detailed findings, such as how many students in a given school are suspected of improper acts and what was suspicious about their answer sheets.
The agency has said it believes Caveon’s claims can be fully and fairly investigated without knowing which students are under suspicion. Dr. Neeley said Caveon’s findings were never intended to be broken down to the student level.Experts who study cheating, however, say such data would be key to determining what happened on test day.
Firm backs work
Caveon, in its original contract for the work, said it would provide valid results for individuals. And both in its contract and in its report to the district, the company expressed confidence in the accuracy of its findings.
According to Caveon’s contract, its duties were to provide “summary and detailed results” that include “cheating and piracy activities by individual examinees,” “the incidence of test fraud/theft by classroom and school,” and “anomalous test results in schools that are most likely due to cheating by test administrators or outside sources.”
And in its report, Caveon terms its methods “a very conservative statistical approach” that ensures that schools flagged “will be so anomalous that reasonable explanations of these inconsistencies by referring to normal circumstances become improbable.”
Don Sorensen, Caveon’s vice president of marketing, said Friday that the company would have no comment on its work with Texas. A representative of Pearson, the state’s testing contractor, also declined to comment.
Methodology criticized
Criticisms of Caveon’s methods have centered on how it detects which schools have made suspiciously large gains in performance.
Caveon uses statewide data to determine how big a jump in TAKS performance a student typically has from year to year. Then it filters out students whose gains were more than a certain amount above that state average. Schools that have too many of those students get flagged as suspicious.
But because it uses the same standard for all schools, Caveon’s method puts additional scrutiny on high-achieving and high-wealth schools. Students at those schools tend to have higher gains from year to year than schools with lower performance.
The result is that Caveon flagged a large percentage of the high schools in well-off suburbs – schools where students generally achieve high TAKS passing rates without having to resort to cheating. Some superintendents have said they don’t trust Caveon’s gain-score methodology.
Caveon also flagged some schools for having unusual numbers of erasures on their answer sheets, where disproportionate numbers of wrong answers were changed to correct ones. But Ms. Ratcliffe also said that’s not necessarily a sign of improper activity.
“There’s nothing illegal about erasing answers,” she said.
However, the validity of another flag that Caveon used to detect possible cheating – schools where very similar answer sheets suggest students copied answers off one another – is well established in academic research.
What’s next?
Cutting ties with Caveon would leave the agency with several options. It could stop looking for cheaters through statistical analysis altogether. It could try to find another company to provide the services. Or, Ms. Ratcliffe said, the agency could try to perform the statistical analysis in-house using TEA staff.
In any event, the delay will probably push back the timetable for examining scores from the spring 2006 TAKS tests.
The current investigations into the 2005 TAKS will be looking at tests given 18 months ago, long after many memories of improper activity have faded. Agency officials had said they hoped to reduce that lag time with the 2006 tests.
But Ms. Ratcliffe said that a decision on who, if anyone, will analyze those tests probably won’t be made until at least October, when the test-security task force meets again and considers the state’s options.
Breaking ties with Caveon would seem to conflict with the agency’s original stated plan, which was to get multiple years of the company’s analysis before considering whether it was worth taking any action based on Caveon’s findings.
Meanwhile, the agency has begun investigations into “close to 20 schools” based on Caveon’s report, Ms. Ratcliffe said. Those investigations began before the commissioner’s task force on test security had its first meeting last month, she said.
Ms. Ratcliffe would not say what those investigations entailed or how the schools were selected – for example, if they were the schools set to receive state money from a special incentive program for high test scores.
“It’s a mix, schools we had some concern about,” she said. “It’s part of the first wave. We have a lot more work to do.”
The agency has not yet announced which of the Caveon schools will receive on-site visits as part of its investigation.