Education research is under the microscope; Quality of data crucial as schools base more decisions on studies

By Joshua Benton
Staff Writer

Page 1A

Education research has been a punching bag for decades – largely for good reason.

Streams of studies serve up contradictory facts. Anecdotes and hearsay are often as prominent as hard data. And some scholars, motivated by ideology, seem to reach their conclusions before the research even starts.

“When you look back, there have been entire movements in education launched on the basis of a few anecdotes, a lot of rhetoric, and not much evidence,” said Gerald Sroufe, director of government relations for the American Educational Research Association.

But now the field is facing unprecedented pressure that could change the nature of research into what works in schools.

A new federal law requires schools to base dozens of policy decisions on research or risk losing federal money, and a bill pending in Congress would, for the first time, attempt to set quality standards for government-funded education research.

“There’s some very good work out there, but there’s a lot of very bad work, too,” said Stanford professor Richard Shavelson. “You just have to figure out how to separate the two.”

The federal education bill passed in January uses some variant of the phrase “scientifically based research” 110 times.

It requires states and districts to use research to determine their approaches to everything from teacher training to the hiring of security guards. The Bush administration has spoken often about its desire to make education an “evidence-based field.”

That, in turn, has fostered the desire to establish quality-control measures for research.

“I’ve seen some colossal missteps in education – things like New Math and ‘schools without walls’ – that were seemingly never tested before being tried,” said Rep. Michael Castle, R-Del., chairman of the House education reform subcommittee and author of the pending bill.

Teachers looking for ways to improve their skills find that different researchers give contradictory evidence for what works and what doesn’t. Academics line up on both sides of every major issue – vouchers, testing, social promotion – and are often accused of cooking the data to get the results they want.

Peer reviews

Reformers such as Mr. Castle would like the field of education research to look more like the world of medical research.

Until the 1950s, doctors operated largely on anecdotal evidence about what worked and what didn’t. As independent professionals, many felt that they – not some distant academic researcher – were best able to decide the effectiveness of treatment.

“The prevailing attitude was that each doctor was his own experimenter,” said Robert Baruch, an education researcher at the University of Pennsylvania. That’s not far from how some teachers feel today, he said.

Over time, physicians were convinced that controlled clinical trials could lead to better medicine. Now, medical research is strictly regulated by government – the Food and Drug Administration – and by an academic community that conducts rigorous peer reviews of scholarly work in prestigious publications such as The Journal of the American Medical Association.

Mr. Castle’s bill, which passed the House on April 30, would require peer reviews of all federally funded studies and would model research regulations on those of other federal agencies, such as the National Institutes of Health and the National Science Foundation.

The bill is now sitting in the Senate committee chaired by Sen. Edward Kennedy, D-Mass. A Kennedy spokesman said that Senate work on the bill will begin in September, but that the committee will likely create its own bill rather than work from Mr. Castle’s.

Some academics agreed that the overall quality of education research needs to be raised but said they are wary of government standards.

“The intent is appropriate, but how it gets translated into law is questionable,” Dr. Shavelson said. “I don’t think it’s the role of the federal government to define science.”

Raising bar

For example, the most successful style of medical experiment has been the randomized trial, in which a group is randomly divided into two parts: One gets a certain treatment and the other doesn’t. But local control of schools makes random assignment difficult for education studies, he said.

“When the researchers come in and say, ‘We want you to abandon the way you teach and try this new way for three years so we can see if it works,’ are parents going to go along with that? Or will local control come into play and say ‘We’re in charge, we’re not going to [do] that’?”

Maintaining strict research conditions in a school is notoriously difficult: students move, principals get fired, policies and demographics change. In one recent long-term experiment led by Boston University researcher Christine Rossell, half of her subjects had withdrawn from the study within three years.

“The so-called hard sciences – chemistry, physics – are misnamed,” she said. “Those should be called the easy sciences. You can keep ions and molecules in a beaker for decades. They don’t have any human rights. They don’t move from one school district to another in the middle of a research study and mess up your data. Researching real human beings who don’t always follow directions – that’s a hard science.”

“People think you can just look at test scores and figure out what works,” she said. “Doing a strict scientific study is extremely difficult and gets very, very expensive.”

According to a National Research Council study released this year, federal funding for educational research has dropped from more than $400 million in 1973 to about $130 million today. Only about one-tenth of 1 percent of education funding in America goes to research.

“If we were a drug company or GM or Ford, spending that little money on research, we wouldn’t stay in business,” Dr. Baruch said.

Even without government intervention, some private sources are trying to improve research quality. Dr. Baruch leads the Campbell Collaboration, a 2-year-old organization whose goal is to evaluate existing education research and “screen the good stuff from the poor stuff.” He said that sometimes fewer than 10 percent of the studies they evaluate meet the highest standards the collaborative looks for.

“One of our biggest problems is that it’s all called ‘research,’ no matter how good it is,” Dr. Sroufe said. “Once you see the headline ‘Researchers show such-and-such,’ it’s difficult for the public to get much deeper than that.”

Trying to turn tide of dropouts; Perry, Sanchez tout plans that capitalize on tested methods

By Joshua Benton
Staff Writer

Page 41A

When the Education Commission of the States moved offices not long ago, Kathy Christie came across an old storage box and a lot of bad memories.

“The entire box will filled with nothing but information on state dropout prevention efforts from the 1980s,” said Ms. Christie, vice president of the Colorado-based research organization.

“I looked through them all and saw all the time and energy that went into them. And none of them had accomplished much. We ended up throwing them all out.”

For decades, states have been trying to tackle the dropout problem and keep kids in school all the way to graduation. Real, sustainable success stories have been few.

Mindful of that, the two major candidates for governor – GOP incumbent Rick Perry and Democrat Tony Sanchez – have come up with their own proposals for how the state can reverse its dropout rates.

Both plans chiefly rely on methods previously tried, with varying success.

Mr. Perry’s proposals focus on identifying and expanding successful, existing programs and putting more counselors into schools with high dropout rates. He also wants to expand after-school programs and summer-school programs for at-risk students.

“My dropout prevention plan will emphasize proven strategies to help schools,” he said.

He estimates his dropout proposals will cost $20 million, which will come from new funding attached to the recently passed federal education bill.

Mr. Sanchez, a Laredo banker and oilman, has put forward a less detailed plan centered on better diagnostic testing in early grades to catch academic problems before they develop into crises.

“Teachers will identify areas of weakness and target the remediation a child needs to progress to the next grade,” he said.

Mr. Sanchez’s major dropout proposal connects to his plan to make Texas’ testing system less focused on accountability and more on diagnostics.

He recommends allowing some students to take the state’s new test, the Texas Assessment of Knowledge and Skills, when they are ready, not when the state testing calendar dictates. Students who need help can be matched with a mentor to work through academic difficulties.

Teachers and administrators already use the state’s tests for diagnostic purposes to find what areas a student needs to improve in to pass. But Mr. Sanchez’s plan would add further “ongoing informal diagnostic checkpoints” at all grades to better track students’ skills.

Sanchez spokesman Mark Sanders said his plans will cost $43 million, funded primarily through a new restitution levy on convicted criminals. It would need legislative approval.

“People who’ve been convicted, you often find they’re people who’ve dropped out,” Mr. Sanders said. “So they’ll help pay to solve the problem.”

He also said Mr. Sanchez hoped to find private sector funds to match a portion of the state’s dropout prevention spending.

Good points to both

Ms. Christie, of the Colorado education organization, lauded Mr. Perry’s emphasis on identifying best practices – efforts that have worked – and duplicating them elsewhere.

“You don’t want to be giving kids more of the same that hasn’t worked for them,” she said.

She also supported the governor’s proposal to hire more counselors. She pointed to a recent survey of Colorado schoolchildren who, when asked how schools could best keep more kids in school, gave “more counselors” as their top response.

Mary Reimer, a consultant for the National Dropout Prevention Center at Clemson University, said Mr. Perry’s plan was too limited.

“They’re pretty much the standard things that everyone’s tried,” she said. “They’re good programs, but it’s kind of like trying to stick your finger in the dam after the water’s run out. It’s not hopeless, but most folks are finding out that prevention needs to start much earlier.”

She said she prefers Mr. Sanchez’s emphasis on early identification of problems.

“Most kindergarten and first-grade teachers can tell you who the children are who will struggle all the way through school,” she said. “If you can help them early, you can prevent them from dropping out later.”

Perry spokesman Ray Sullivan rejected the criticism and said changes under former Gov. George W. Bush and Mr. Perry have emphasized early grades, including major elementary reading and math initiatives and better diagnostic testing. He said Mr. Perry also supports more early diagnostics, primarily through online tests.

“In the coming years, those younger children will be entering junior high and high school, and they’ll be better prepared than ever before,” Mr. Sullivan said.

Turning to the Democrat, Mr. Sullivan called Mr. Sanchez’s plan vague and said the flexible testing system he wants would damage the integrity of the state’s accountability system.

Mr. Sanders, the Sanchez spokesman, disputed that, saying the added student data gained from flexible testing would make schools “more accountable, not less.”

Ninth-grade emphasis

Both Mr. Sanchez and Mr. Perry recommend strengthening the state’s current efforts aimed at ninth-graders, the year often cited as the most dropout-prone.

Texas freshmen are four times as likely to repeat the year than students in other grades.

Both want to expand the Ninth Grade Success Initiative, which gives grants to school districts to help freshmen’s academic performance.

Mr. Perry already has implemented one item on his platform: a restructuring of the Texas Education Agency to create a single dropout prevention division.

Laredo Superintendent Paul Cruz was hired this month to lead the new division.

Previously, piecemeal responsibility for keeping kids in school fell to many different parts of the agency.

But some researchers say that state-level dropout prevention plans, no matter how well-intentioned, are usually less effective than their proponents claim.

Mark Dynarski, a senior education researcher with Mathematica Policy Research in New Jersey, has scrutinized dropout efforts for the last 12 years and found that the majority have negligible impacts.

He said the most effective efforts are small-scale and target individual students, not broader populations.

“You can always find people who say, ‘We’re really excited about this program,’ or give anecdotal evidence,” he said. “But when you go back and look at the hard data, the impacts are usually very small, if they’re there at all.”

State hires educator to keep kids in school

By Joshua Benton
Staff Writer

Page 41A

Not long ago, a woman in her late 20s came up to Paul Cruz, the Laredo school superintendent. She wanted to say thanks.

Thanks for the teachers who had taken an interest in her, years after she dropped out of high school. The teachers who had persuaded her to re-enroll, take her tests and finally to get her diploma.

“It was because she got personal attention from a teacher after all these years that she was able to do it,” Dr. Cruz said. “Now she’s going to community college.”

Dr. Cruz – at 36, not much older than the woman – will soon have a chance to help many more than a single student. He has been appointed to a new position in the Texas Education Agency: deputy commissioner for dropout prevention and initiatives.

The position is an attempt to tackle one of the traditional complaints about how states try to keep kids in school.

Since so many parts of the educational process affect the dropout rate – testing, social promotion, adult education, counseling services and others – it can be difficult to coordinate a unified state policy.

“We want to have the agency organized in such a way to make dropout prevention as effective as possible,” state Education Commissioner Felipe Alanis said in April.

Advocates argue about the magnitude of the problem, but all sides agree that thousands of Texans drop out of school every year. Last year, Texas enrolled 360,000 high school freshmen, but only 220,000 seniors.

Tasks ahead

In his new position, Dr. Cruz will oversee the state’s assessment system, GED testing and the state’s math and reading initiatives.

“The important thing is coordinating the work being done in different parts of the agency,” he said. “One student moving through the system may be getting help from many different areas, and they all should be talking to each other.”

Dr. Cruz is considered one of the state’s rising education stars. He was only 32 when he became Laredo’s superintendent in 1998, after a brief stint at TEA and stops in four other Texas school districts.

“He was quite young, but he was extremely articulate,” said Dennis Cantu, president of the Laredo school board. “He looked you in the eye. He had plans and goals.”

He lauded Dr. Cruz for a strong focus on student performance and depoliticizing the school system. Previous superintendents had been criticized for running the district as a cog in the local political machine, Dr. Cantu said.

“He wasn’t interested in that – he just wanted to focus on education,” Dr. Cantu said.

When Dr. Cruz arrived in Laredo, the district had four schools rated “recognized” or “exemplary” in the state’s accountability system. Last year, it had 12.

Laredo’s dropout rate also improved during his stay. In 1998, Laredo reported an annual dropout rate of 3.3 percent. By 2001, that had dropped to 1 percent.

Dr. Cantu said Dr. Cruz found money to increase the number of attendance officers in Laredo, which allows schools to keep better track of students who don’t show up every day. He also made it clear to district officials that lowering the dropout rate was a high priority.

“He made sure principals knew they would be held responsible for addressing the dropout problem,” he said. “It’s something Dr. Cruz and his staff were never reluctant to talk about in public. If there were problems, they brought them out and faced the public.”

Guiding principles

Dr. Cruz said he’d use the same guiding principles to lead state dropout prevention as he did in Laredo. Primary among them: Setting coherent curriculum standards at all grade levels. If schools can do a better job of standardizing what all third-, fourth- or fifth-graders know in Texas, they will prevent children from falling too far behind when they reach high school, he said.

“All principals should, whether they’re elementary school, middle school or high school principals, know and understand the requirements of the graduation plans,” he said. “If a child is 7 years old and in the second grade, we need to be thinking where he needs to be now so he can take a rigorous math program when he reaches high school.”

Dr. Cruz said his top priority is using the new federal education bill to improve dropout prevention. The bill gives states more flexibility in how they use federal funds, and Dr. Cruz said that could open up new avenues for programs to keep children in school.

He also said he wants to find ways to improve the way the state counts its dropouts, long a matter of debate among education activists. The official state annual dropout rate is 1.3 percent, which many observers believe doesn’t accurately reflect the size of the problem.

“I think the methodology has improved, but there’s more we can do,” he said.

Jackson says he can transfer skills; Longtime politician’s supporters say he’s a good fit for UNT post

By Joshua Benton
Staff Writer

Page 23A

This time, Lee Jackson got the job.

The Dallas County judge had been up for top education posts three times before: twice for superintendent of Dallas’ school system and once for Texas education commissioner. Each time, the job went to someone with more experience in education.

But the soon-to-be chancellor of the University of North Texas System said he always knew the leadership and management skills he demonstrated in government could be transferred to education.

“I’ve proven I can lead effectively in the public arena, and now I’ll bring those experiences to bear to benefit the UNT community,” he said Wednesday after regents named him as the lone finalist to succeed outgoing Chancellor Alfred Hurley.

Mr. Jackson has managed to be one of Dallas’ most powerful political figures for more than two decades without compiling a lengthy list of enemies. His friends credit his ability to build coalitions and draw disparate forces together.

“I think it’s a brilliant choice,” said Tom Luce, Dallas attorney and chairman of the National Center for Educational Accountability. “I can’t conceive of anybody who could do a better job.”

Born in Austin and raised in Dallas, Mr. Jackson decided on public service at an early age. A year after getting his bachelor’s degree from Duke University, he earned a master’s in public administration at Southern Methodist University. Soon after, he started work in the Dallas city manager’s office.

In 1976, he was elected to the Texas House of Representatives, where he served five terms, including time on the public education committee. He was elected Dallas County judge in 1986.

Mr. Jackson said the job has forced him to learn about areas of expertise unfamiliar to him, like transportation policy – a skill he said will be useful in learning about the university. He has been praised for stabilizing the county’s juvenile department and improving relationships between the commissioners court and other county officials.

“Lee has the ability to work with people,” said Commissioner Jim Jackson, a UNT alumnus. “He’s smart, he’s analytical, he can listen. He can help bring people together.”

Lee Jackson said he hasn’t particularly been trying to get a position in education – the three previous job opportunities were presented to him, he said, and he didn’t pursue them. When he announced he wouldn’t seek re-election last fall, he said he was “open to a variety of things. I wanted something challenging that would let me be as enthusiastic as I’ve been in this job.”

After UNT’s search consultant contacted him in April, it became apparent it would be a good match, he said.

In 1999, when Mr. Jackson was first up for the Dallas schools job, Dallas County GOP Chairman Bob Driegert declared it a 95 percent certainty that Mr. Jackson would get the job. But the school board ended up hiring San Francisco Superintendent Bill Rojas.

A year later, when the position opened again, the board voted 6-3 not to hire Mr. Jackson.

Mr. Jackson said he doesn’t want to talk about the DISD experience – “that’s an ancient issue for me,” he said – but one of the trustees who voted against him said it shouldn’t be interpreted as a criticism of his abilities.

“Everyone was very impressed with Judge Jackson,” said Ken Zornes, the current board president and chairman of the search committee in 2000. “He’s an outstanding person, completely qualified.”

That’s indicative of an apparent trend with Mr. Jackson: Even those who oppose him at one time or another say they respect his abilities and leadership skills.

“Judge is going to do an excellent job wherever he lands,” said County Commissioner John Wiley Price, the only Democrat on the Commissioners Court. “He’s got the interpersonal skills, the people skills, the leadership skills.”

Mr. Jackson’s experience working with universities is limited. In the mid-1970s, he spent two years as SMU’s assistant director for alumni giving. He also has taught history and government at Dallas community colleges.

As a nonacademic without a doctorate, he’s aware that some people may question choosing him to lead a university system. He predicted that he’ll prove any naysayers wrong.

“That’s the kind of issue that usually goes away in six months if the person in the job is effective,” he said. “And I expect to be effective.”

Irony marks teachers’ meeting; Largest union in nation begins annual gathering in state that shuns label

By Joshua Benton
Staff Writer

Page 17A

Ten thousand members of America’s largest teachers union are in Dallas this week. But if they start looking for the union label, they could be searching a while.

That’s because Texas’ teachers unions are among the nation’s weakest. A combination of state laws and large nonunion competitors has pushed them further into the background here than almost anywhere else.

“People who are coming from union states are going to have real trouble understanding how different Texas is,” said Ignacio Salinas Jr., president of the Texas State Teachers Association. The 70,000-member TSTA is the state affiliate of the National Education Association, the 2.7 million-member union holding its annual meeting at the Dallas Convention Center.

The biggest difference is the absence of collective bargaining, the traditional contract negotiation that goes on between a teachers union and schools. Bargaining over issues such as salary, benefits and working conditions is a given for teachers in most states.

Texas is one of six states with no collective bargaining for teachers and one of two states that explicitly ban the practice by law. That makes some wonder why the National Education Association is coming to town.

“Why are we bringing 10,000 delegates and millions of dollars of convention business to a state that denies the basic right of collective bargaining to its teachers?” said Louis Malfaro, president of Education Austin, that city’s teachers union.

Nationally, the NEA and its smaller, friendly rival, the American Federation of Teachers, dominate the world of teacher associations. The vast majority of local and state teacher organizations are affiliates of one or the other. Some, such as Education Austin, are affiliated with both.

Texas is different

But in Texas, four state organizations compete for teachers’ attention. The largest is the Association of Texas Professional Educators, with almost 100,000 members, and it is vocally anti-union. The 44,000-member Texas Classroom Teachers Association is also opposed to collective bargaining. TSTA and the 40,000-member Texas Federation of Teachers represent the unionized side.

Having four groups means that the Legislature sometimes hears opposing arguments from opposing teachers groups – a far cry from some states, where a single state union dominates the conversation.

Representatives from all four groups say that when they unite on an issue, they can be an effective force. They cite the new teacher health insurance plan passed in the last Legislature and the $3,000 pay raises given in the previous one.

“When we do link arms and go to the Legislature, it can be very powerful,” said Jeri Stone, executive director of the nonunion TCTA.

But unity is often hard to come by. Each side regularly refers to the other as “the enemy,” and the bad blood seems to run deep. “It’s a very real divide between the two sides,” said Larry Comer, spokesman for the nonunion ATPE.

Officials cite various reasons that nonunion groups have had such success in Texas, including Texan ideals of individuality and conservatism. There’s also a financial incentive: annual dues for unions are often three or four times higher than for nonunion groups.

Texas unions hope that the state will repeal its ban on collective bargaining, but they say the political climate in the Legislature isn’t likely to make that possible soon, if ever. In the meantime, some teacher unions have settled on a half-measure called exclusive consultation.

If a district agrees to exclusive consultation, its teachers hold an election to select a local union as its sole representative in negotiations with the district. That group negotiates with the school board.

District can go own way

There’s one big difference from collective bargaining, though: The district is not obligated to listen to their demands. There’s no system of mediation or arbitration if the sides can’t reach an agreement. And teachers can’t strike if they don’t like what the district is doing.

“Collective bargaining makes a huge difference, but we’ve been able to make some major advancements with consultation,” said Aimee Bolender, president of Alliance/AFT, the local union that has consulted with the Dallas school district since 1997. “There’s one clear and focused voice speaking for everyone.”

Most big-city districts across the state use exclusive consultation, something nonunion groups oppose.

“Consultation is just a pale shadow of collective bargaining, and it’s not something we’re supportive of,” Ms. Stone said.

The two state unions, TFT and TSTA, have been in on-again, off-again merger talks for several years. A merged union could be more forceful in pushing for collective bargaining and other issues, Mr. Malfaro said, and it could be stronger financially. But Mr. Comer said he wouldn’t expect a merger to change the sometimes-icy relationship among the state’s teacher groups.

“That would just mean we’d have one fewer enemy,” he said. “We’d only be looking over one shoulder.”