By Joshua Benton and Holly K. Hacker
Last of three parts
In 1975, a social scientist named Donald Campbell came up with the idea that would eventually be called Campbell’s Law. He wrote like an academic, but you could boil the concept down to this:
The higher the stakes, the more likely people are to cheat.
It makes intuitive sense. The Dallas Morning News’ analysis of TAKS scores found that cheating is almost twice as common on the 11th-grade test – which is required for graduation – as on the 10th-grade test.
But experts say that Texas has missed the lesson of Campbell’s Law. Over the past two decades, the state’s tests have become the dominant force in Texas education. But they say the state’s test-security system – the rules and tools officials use to prevent cheating – hasn’t kept up with the increasing importance the TAKS test now has in students’ and educators’ lives.
As a result, they say, the TAKS is given in a far more permissive environment than other high-stakes tests like the SAT, bar exams or graduate-school admissions tests. And much of the cheating on the TAKS could be prevented with a series of reforms based on what those other tests do.
“It seems to me like terribly bad leadership,” said Robert Frary, a professor emeritus at Virginia Tech who has studied cheating for more than 30 years. “The people in charge have to make it their priority.”
Testing in Texas is leagues away from where it was in the late 1970s, when statewide tests first came into prominence.
Back then, results weren’t filtered into school ratings or published in the newspaper. Tests were only given in a few select grades. Realtors didn’t pitch houses based on how the neighborhood elementary school had fared.
In the early 1970s, a young man named Jim Impara helped develop Florida’s first statewide testing program. He started attending national conferences on the subject, and there was one subject that was almost never discussed.
“I can’t remember security ever being a topic,” he said. “There wasn’t really any incentive to cheat.”
But in the decades since then, Texas, Florida and the entire nation have begun to take tests much more seriously. Texas’ alphabet soup of tests – first TABS, then TEAMS and TAAS, and now TAKS – evolved into the dominant force in public education. Some teachers’ salaries are now tied to their students’ performance on test day. School ratings are a mark of public pride or shame. And decisions like whether a teen graduates or a third-grader gets promoted now hinge on test scores.
As Campbell’s Law would predict, with higher stakes came a greater temptation to cheat, at all levels of the system. (As Dr. Campbell himself originally put it: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”)
There are a variety of potential penalties on the books for cheating in Texas, all the way up to an entire school being shut down. But punishments are exceedingly rare.
Teachers bear burden
The security of the TAKS depends, in large part, on the many thousands of teachers and other school staffers who handle the exams. For the vast majority, of course, honesty isn’t an issue. But recent years have shown there are exceptions.
“You can’t have the same kind of control you have when you have everyone come to a neutral location and have outside proctors,” said H.D. Hoover, a testing legend who was the primary overseer of the Iowa Test of Basic Skills.
Texas sets no standards for the physical arrangement of students on test day. It has no rules for how far apart students should sit from one another. It also has no rules on seating arrangements, so that in many schools, best friends are allowed to sit within easy view of each other’s answer sheets. (Some school districts voluntarily set tighter rules.)
During tests with higher levels of security, such as the SAT, there are usually firm standards on the distances between students, and students are generally not allowed to choose where they sit. Would-be cheaters are sentenced, at least, to eyestrain.
Texas also does not require schools to keep charts recording where students sit. If two students have identical and highly unlikely answer sheets – but there is no way to know whether they sat near each other – it can be difficult for an investigation to proceed. Texas doesn’t even require schools to record what classroom a student was in when he or she took the TAKS.
“Knowing where they sit can take the evidence from a statistic and make it stronger,” said Dr. Frary, a retired professor of educational measurement.
Some local students said their teachers let them sit pretty much wherever they wanted during TAKS testing. Others said their teachers were more aggressive about assigning seating. But even when teachers take preventive steps, students find ways around them.
Priscilla Ramirez, a student at DISD’s Adamson High School, said that on one test day in her class, students were assigned numbers as they walked in. Each number corresponded to a desk, which was intended to randomize seating. But, she said, some students just traded their numbers to sit next to their friends.
Cheating “is sort of like an epidemic,” she said. “It’s not going to stop unless you really, really, really try.”
Perhaps the single most effective cheating-prevention move the state could make, experts say, would be to produce multiple forms of the TAKS. Currently, on each TAKS test, every student gets all the same questions in the same order. That makes cheating easy, since one child’s Question 12 is also his neighbor’s Question 12.
Scrambling the order of questions in each test booklet makes copying much harder. Many high-stakes tests produce four different versions of exams and distribute them in such a way that no test-taker is seated next to anyone with the same version of their exam.
Diane Birdwell, a teacher at Bryan Adams High in Dallas, said she started producing multiple versions of her classroom tests when she realized kids in her first-period class were sharing answers with their friends in her third-period class.
“It’s a great deterrent,” said Ms. Birdwell, a board member of the teachers’ group NEA Dallas. “You tell the kids, look, there’s no point in looking at Johnny’s No. 14 because his isn’t going to look like your No. 14.”
When engineering professor Charles Bernardin first started teaching at the University of Texas at Dallas, he wasn’t particularly proactive about preventing cheating on his exams. But he quickly noticed that some of students were taking advantage of his lax security.
“If you leave a hundred dollar bill on the floor, people are going to pick it up,” said Dr. Bernardin, who is also the university’s director of academic assessment. “There’s going to be a temptation.”
But then he started producing multiple versions of his exams and mandating where students sit. Now, he said, “I never have any cheating problems.”
Switching to multiple forms does come with a few drawbacks.
For example, printing different versions of test booklets can cost more money and make the test harder to administer. But Texas already prints multiple versions of its TAKS test booklets. Each TAKS contains a number of “field test” questions that don’t count toward a student’s score, but are instead evaluated for inclusion on future editions of the TAKS. Those questions differ from student to student. Only the questions that are identical in every test booklet count toward a student’s score.
For Texas, state officials said, the primary concern would be the added expense and complication. More versions of a test would slow down the grading process and increase the chance, however small, of a grading error.
“Certainly we have the ability to make things more and more complex and generate more forms of the test,” said Criss Cloudt, a TEA associate commissioner who oversees the state’s testing program. “But there’s a price you pay in complication.”
Scrambled versions of the same test would also have to be statistically equated so officials can ensure they are all of equal difficulty. That can take precious time, considering that in Texas, a legislative mandate requires TEA to score the entire TAKS within 10 days of test day.
Still, Dr. Impara said that despite the potential difficulties, he recommends multiple forms for statewide assessment programs. So does the company he helped found, years after he left Florida – Caveon, the test-security firm TEA hired last year to examine the veracity of its TAKS scores.
“It has drawbacks, but the one benefit is better security,” he said.
Dr. Cloudt – who has overseen testing only since February – and other TEA officials defended the state’s test-security system. Traditionally, the state has seen its role as providing training to district testing coordinators through manuals, seminars and other avenues. Those district coordinators are then left to define many of the specific rules for test day. That leaves some districts – like, as of this year, Dallas – with relatively strict rules about things like adult supervision and who can proctor tests. Other districts can have less stringent standards.
“We feel we do a lot to keep testing secure,” Dr. Cloudt said. “We’re certainly open to looking at other ideas.”
A role model for Texas?
When it comes to security on state tests, Mississippi is one of the gold standards. It requires at least two adults in the testing room at all times – and in some cases, even more. Violators of state test-security rules can, in the most extreme cases, be fined or face jail time.
State auditors make at least one unannounced visit to every school district each year on testing days to inspect the security procedures in place. If students are seated too close together to meet state guidelines, for instance, schools are told to spread them out. Seating students at cafeteria tables so they face each other – which is allowed in Texas – is prohibited in Mississippi.
And each year, before scores on the graduation test are released, the state runs statistical checks to look for suspicious scores. One method, similar to The News’, looks for pairs of students with too-similar answer sheets. School districts are told to investigate the most unusual scores; in about half of those cases, according to state testing director Cindy Simmons, the suspicious scores are invalidated. If state officials disagree with a district’s findings, the state Board of Education can overrule the decision.
“There are big consequences attached to these tests, and we want to make sure they’re fair for all students and that no one has an unfair advantage,” Ms. Simmons said. School districts have been cooperative, she said. “It’s not, ‘We’re trying to catch you.’ It’s, ‘We want you to know.'”
Mississippi officials have discussed switching to multiple forms of the test to increase security even more, she said. But so far, the state has decided not to, citing the cost. “We are not a state that is very wealthy,” Ms. Simmons said. “We feel good about our security system as it is.”
Little to gain
What keeps state exams from having a system as secure as other big-time tests? For cheating researcher George Wesolowsky, the key question is one of will.
With so much to gain from high test scores, it’s not always in a school’s or district’s best interest to find cheating, he said. Reports of cheating bring schools embarrassing negative publicity. Students and parents become upset.
“Instructors have no encouragement or incentives to actively look for cheating, and prosecuting cheaters is a time-consuming and unpleasant activity,” said Dr. Wesolowsky of McMaster University, who worked with The News on its analysis of TAKS scores.
Among researchers who have devised methods of detecting cheating, the biggest roadblock often is resistance from school officials who either deny that cheating exists or think it’s not a problem worth fighting.
Dr. Wesolowsky said he’s given his cheating detection program to several private and governmental testing organizations. But those groups are also aware of the negative consequences of discovering cheating.
“Usually when they get the program up and running, an iron curtain goes up,” he said. Concerns about falsely flagging individual students can lead educators to ignore the even more important evidence a statistical analysis can provide: information on which schools have serious test-security problems.
Dr. Frary has faced similar obstacles. “It’s so difficult to institutionalize it,” he said. “It’s a very hard, uphill battle to convince people to go after the problem.”
Even with the wide availability of security measures, some researchers are pessimistic that cheating can ever be completely eliminated as long as someone has something to gain from a high score.
“The basic ground rule is, whatever security measures go in, somebody is going to find out how to evade them,” said David Berliner, a professor of education at Arizona State University and a critic of high-stakes testing. “Banks are still broken into.”
In any event, Dr. Hoover – father of the popular Iowa test given in many Texas schools – has a different recommendation for how to stop cheating: Reduce the high stakes attached to tests.
“That’s what leads to the problems with security: the pressure to raise scores,” he said. “It’s led to cases where teachers, rightly or wrongly, feel that they are being judged on the basis of these instruments. And some of them cave in and do things they shouldn’t.”