By Holly K. Hacker and Joshua Benton
Staff Writers
Page 21A
The Dallas Morning News wanted to check whether cheating occurs on the Texas Assessment of Knowledge and Skills. So it turned to an expert in the field: George Wesolowsky, a professor of management science at McMaster University in Canada.
Nearly a decade ago, Dr. Wesolowsky developed a software program that uses statistical methods to detect cheating on multiple-choice exams. (An academic paper about his method is available on dallasnews.com.) He used the program to analyze 2005 and 2006 TAKS answer sheets for evidence that students may have copied answers from each other. The News analyzed those results to see how much cheating occurs across various schools, grades and subjects.
Here’s how Dr. Wesolowsky’s program works. Let’s say two students take a multiple-choice test with 50 questions, and answer 48 identically. The program calculates the chances that could happen if they were answering independently, with no cheating. It examines how common those shared answer choices are among other students. Sharing only popular right answers won’t trigger red flags – but a long string of uncommon identical wrong answers could. If the odds are extremely unlikely, the students’ answers are flagged as suspect.
If two students are flagged, it doesn’t mean both are cheaters. In many cases, one could be the innocent victim of the other’s wandering eyes.
Dr. Wesolowsky’s method considers several factors, including the difficulty of each question and how the entire class performed. Other researchers in the field said that Dr. Wesolowsky’s is the best or among the best methods for cheating detection yet devised.
Tests examined
Using open records laws, The News requested answer data from the 2005 and 2006 TAKS tests for all public schools in the state. The data covered grades three through 11 and included student responses on each test. (The Texas Education Agency withheld information for about 20 percent of students because of federal privacy laws.)
Student names and other identifying information were not included.
The analysis examined reading and math answers in each grade, plus social studies and science for grades eight, 10 and 11.
Unusual cases
Dr. Wesolowsky’s method assumes that most students in a school are not cheating, so exceptions to the rule stand out. But in a small number of cases, many students taking a test had very similar wrong answers. That could indicate widespread cheating. Many of those schools were among the group the test-security firm Caveon had considered most suspicious in its analysis.
Dr. Wesolowsky analyzed those unusual cases again, this time lumping them with dozens of other schools. That allowed cheating at each school to be properly detected, because the larger pool included enough noncheaters from other schools for the model to work.
To review the results of the analysis, The News turned to two more experts: David Harpp, a professor at McGill University, and Robert Frary, a professor emeritus at Virginia Tech.
The two men examined the results independently and both supported Dr. Wesolowsky’s findings. Dr. Harpp, using a different method, also performed an independent analysis of several schools, which supported Dr. Wesolowsky’s findings.
It’s important to note that the purpose of The News’ study was not to make cases against specific individuals, but to estimate the extent of cheating in Texas schools.