By Joshua Benton
When he saw that six Richardson schools were on the state’s list of potential TAKS cheaters, Superintendent Jim Nelson wanted to investigate. But to do so, he needed to know how Caveon – the company that built the list – did its work.
He e-mailed state Education Commissioner Shirley Neeley, whose agency paid Caveon to do the analysis: “Commissioner, how do I get detailed information as to how Caveon reached their conclusions? All we got were the conclusions.”
He added, according to documents obtained by The Dallas Morning News: “Anger and frustration aimed at the agency is palpable. I want to help, but we must have access to their analysis.” Without those details, the Texas Education Agency is doing “nothing more than a hit and run,” he said.
Mr. Nelson and other Texas educators have tried to get the information they think they need to clear their schools’ names. But the TEA hasn’t been able to give it to them. That’s because agency officials never got the data themselves.
As a result, few, if any, thorough cheating investigations have begun – nearly two months after Caveon determined that 609 schools had suspicious test scores.
As Frisco Superintendent Rick Reedy wrote in a statement on the district’s Web site: “We did take the report seriously, and we did try to investigate the findings … without much luck.”
TEA officials say they never wanted the findings from the $500,000 Caveon analysis to lead to large-scale investigations. The agency expected the analysis to be the first part of a multiyear study that might improve test security down the line.
But faced with anxious superintendents and political leaders, the agency has been moved to action. Now it’s scrambling to figure out how far, exactly, it’s willing to pursue suspected cheaters.
Dr. Neeley counsels patience. “We just need more information,” she said.
Perhaps Dr. Neeley summed up the agency’s frustration over reaction to the Caveon list best in one of her e-mail replies to Mr. Nelson: “I am livid; this whole thing has totally gotten out of hand.”
Stories brought probe
The TEA brought in Utah-based Caveon last year after a series of Dallas Morning News stories uncovered educator-led cheating in a number of Texas schools.
Caveon was supposed to analyze 2005 scores on the Texas Assessment of Knowledge and Skills to look for evidence of cheating by students or educators. That could include inexplicable jumps in student test scores, duplicate answer sheets or suspicious patterns in student answers.
But the problems identified for the 609 schools aren’t all the same. Some had TAKS results that seem beyond the pale – like the one where the answer sheets of 89 of a classroom’s 91 students were virtually identical. Caveon flagged other schools for having only a few students with suspiciously similar answer patterns.
The problem is that schools don’t know which group they fall in – and for now, there’s no way for them to find out.
When the TEA hired Caveon, it didn’t ask the company to provide the detailed data necessary for an investigation. Such data could say, for example, which students’ answer sheets at a given school were suspect and why they were flagged. The data could also indicate which students had duplicate answer sheets suggesting copying and which score gains were suspicious.
That’s the kind of information that superintendents say they want.
“We cannot get enough information or data from Caveon to conduct any kind of investigation – if in fact one is even warranted,” Mr. Nelson wrote via e-mail last week from Ireland, where he was vacationing. “My schools on the list are relatively high-performing, so I must admit to being very skeptical of their analysis.”
The lack of supporting data from the TEA is particularly noteworthy because Dr. Neeley, in a letter to superintendents, told districts to “conduct any investigations you deem necessary.”
In their e-mail conversation, which took place on June 13, Dr. Neeley told Mr. Nelson that Caveon’s methods of detecting cheaters are proprietary and, thus, couldn’t be shared with districts.
Mr. Nelson expressed disbelief: “That really causes a problem. It’s like Alice in Wonderland. ‘Our analysis shows you may have cheated, but we won’t show you how the analysis is done.’ Crazy.”
A Caveon official told The News that the company was never asked to provide the detailed data.
“What we furnished to Texas was exactly what was asked for,” said Don Sorensen, Caveon’s vice president of marketing. The additional information could be provided, he said, but probably would cost the agency more. “If they want further detail, that would require a new contract.”
To this point, the TEA hasn’t formally requested the information, Mr. Sorensen said.
“If we can, TEA will get that data,” Dr. Neeley said. She indicated there may have been confusion over how much of Caveon’s data is proprietary.
To Mr. Nelson, she wrote last month: “Caveon owns the software – it is proprietary. In other words, we do NOT have access to the programs that generate the results.”
But Mr. Sorensen said that while Caveon’s specific algorithms are proprietary, the classroom- and student-level detail they produce would be available.
The difficulty in obtaining the information extends beyond the TEA to the company that administers the TAKS test for the state. When one San Antonio-area principal called Caveon last month to seek more information about why his school was on the list, his request was forwarded to Walter Sherwood, who heads the Texas assessment contract at the testing company Pearson. (Caveon is a subcontractor to Pearson.)
“No information was provided to this person, and I’ve instructed Caveon not to respond to this message or any others that they receive,” Mr. Sherwood wrote in an e-mail to TEA officials.
Dr. Neeley, a former superintendent in suburban Houston, said she understands the frustrations of school districts having trouble investigating without the data. “I agree with them 100 percent,” she said Friday. She said districts and the public should be patient while the TEA formulates a plan of action.
“Everybody should just take a chill pill,” she said.
A stronger pursuit
One district, Conroe ISD outside Houston, has even hired an attorney to pursue testing records that the TEA has not provided. Three Conroe high schools landed on the Caveon list, all for unusual test gains. One is The Woodlands High School, one of the state’s wealthiest.
After the district filed an open records request seeking some of the data, TEA attorneys appealed to the state attorney general’s office, arguing that the agency shouldn’t have to provide the information.
Conroe Superintendent Don Stockton didn’t respond to several calls from The News. But in an interview with a Conroe newspaper, he said the TEA has told him the allegations aren’t worth investigating.
“Our scores are not alarming because TEA has told us there’s nothing to be alarmed about,” he told The Courier of Montgomery County.
That’s the approach the TEA has taken with many of the suburban schools that Caveon said had unusually large gains. Outsize gains were one of the four “flags” that the Caveon analysis assigned to the 609 schools. It is also the flag most commonly assigned to well-off suburban high schools, including in Richardson and Frisco.
Dr. Neeley said those scores – which Caveon considers statistically unlikely to have occurred naturally – are instead indications of hard work by educators.
“I’m not trying to say it should be a badge of valor to be on that list, but every superintendent should be able to explain why those student gains were so good,” she said in the interview Friday. “As much effort as we concentrate on closing the achievement gap, I would be disappointed if we didn’t have significant gains.”
Dr. Neeley said the wealthy districts on the list – including many considering self-investigations – are unlikely to cheat.
“You look at Highland Park, Richardson, Eanes,” she said, naming some of the state’s wealthiest districts in the Dallas and Austin areas. “Do they have to cheat to have good scores? I gave a talk in Eanes not long ago and said, ‘Do you people think Westlake High School had to cheat to get good scores?'”
One of Dr. Neeley’s top deputies, Susan Barnes, said schools that think they need to investigate possible cheating could instead try to figure out how their teachers and administrators did their jobs so effectively.
“They can ask, ‘What did we do in our district that helped our students learn so well?'” said Dr. Barnes, associate commissioner for standards and programs. “They could certainly say, ‘Here are some of the reasons our students did such a good job on their assessment.'”
But the TEA is explaining away score gains without any idea how unusual or widespread those gains may be. State officials don’t know if a school’s Caveon flag was caused by 10 students with unusually big score gains – or 100.
Dr. Neeley has maintained that cheating on state tests is rare and that some in the media, including The News, and the public have overreacted.
As she put it in her e-mail exchange with Mr. Nelson, himself a former state education commissioner: “Am so sorry about all the anguish this has caused, because the bottom line is we are being punished for working hard and doing a good job to close the achievement gap.”
Bonuses change context
Agency officials received draft versions of Caveon’s findings last fall but didn’t pursue any investigations – even though the study made it clear that some of the high school seniors who graduated this May cheated on state tests required for graduation.
The initial plan was to gather several years’ worth of data before judging whether some schools might house improper behavior.
When the Caveon list was released in May, TEA officials said it would let districts decide whether to investigate themselves. But that changed when it became clear that 14 schools on the list were due state bonuses because of their high test scores.
Now the agency plans to investigate all 14 of those schools, plus an uncertain number of others. “We need to figure out where to draw the line,” Dr. Neeley said. Officials said those decisions should be made within a month.
Dr. Neeley said TEA teams would visit the schools selected for investigation to check for signs of cheating. In addition, an outside appeals panel evaluate the work of the TEA teams.
On-site visits of the sort Dr. Neeley is promising are relatively rare for cheating allegations. The last major such visits were to Wilmer-Hutchins ISD in late 2004 and early 2005. Those visits were prompted by News stories outlining apparent cheating in several district schools.
In the end, the TEA team found that two-thirds of the district’s elementary teachers were helping students improperly.
The agency dissolved the district last month.
“We take this very seriously,” Dr. Neeley said.