TEA gives more information on STAAR essay machine scoring after spike in zeros, concerns from school leaders – Houston Public Media

AP Photo/Ty Wright, File

After a spike in the number of students scoring a zero on STAAR writing questions and widespread concern from public school leaders about the shift to automated scoring this year, the Texas Education Agency is offering more information about the changes to the state’s standardized test.

The agency said district testing coordinators will have “the opportunity to receive additional information about (student writing) that received a score of zero” if they contact the agency before Wednesday, March 13. In the message to testing coordinators, the TEA credited “educator inquiries about this first use of the hybrid scoring model.”

Automated scoring is the latest change in a complete overhaul of the state’s standardized test. Student composition scores plummeted after the STAAR 2.0 test was introduced last school year, even when writing was still graded by humans.

Over the past few weeks, school leaders have sounded the alarm about the shift to machine scoring for student writing this school year. They argued the automated scoring system has led to even lower scores, and they’ve criticized what they see as a lack of advance notice from the TEA.

“No one knew this was happening,” said H.D. Chambers with the Texas School Alliance, which represents about 50 large public school districts. “The education field wasn’t told. There was no ‘Heads up, hey, this is happening,’ and so it was just kind of discovered.”

TEA officials have pushed back on that framing.

“For some reason, it’s being portrayed as a quiet implementation,” said assessment division director Jose Rios. “We did communicate during the fall (of 2023) to several stakeholders. We personally communicated this to all district testing coordinators in the fall during our big training that we call the kickoff of the year.”

How does the new scoring model work?

The TEA changed a lot about the STAAR last school year when the test shifted entirely online. The STAAR 2.0 revamp marked the end of the separate writing exam in 4th and 7th grade; instead, writing is now included at every grade level.

The writing prompts are also different. Before last year, students generally wrote self-reflective responses to open-ended prompts, like “Write about the best teacher you know.”

Now, students are asked to read passages, and then write a response to more narrow questions. The new grading rubrics are more stringent, making it more likely for a student response to score a zero. For example, a technically correct answer could earn a zero if it “lacks an introduction and conclusion” or “an organizational structure is not evident.”

Even before the introduction of automated scoring this year, those changes contributed to a dramatic spike in the number of students scoring a zero for writing questions last year. Before the STAAR 2.0 revamp, 5% or less of students typically scored a zero on constructed writing responses. When students took the test in the spring of last year, more than 40% of third and fourth-graders across the state scored a zero, along with more than 20% of students in middle school and high school.

This school year brought another change — automated scoring for most student writing, first reported by the Dallas Morning News.

The early results from the December 2023 testing window have led to tension between school leaders and the TEA.

“We noticed, comparing those scores to … when it was scored the old way, that the automated scoring engine was grading them much more difficult,” Chambers said.

As Chris Rozunick with the TEA pointed out, the December testing window included many high schoolers who were “retesting” because they previously failed the exam. She also emphasized the sweeping changes to the writing prompts and grading rubrics.

“That more than anything else is driving up the zeros, particularly in December where you see a slightly larger retest population — kids who have tried the test and not passed,” Rozunick said.

Back in spring 2023 — before the introduction of automated scoring — 25% of high schoolers scored a zero on writing compositions. Nineteen percent of first-time testers scored a 0, and 50% of retesters scored a 0.

Automated scoring was rolled out for the first time earlier this school year. For high schoolers who took the STAAR in December, almost 70% scored a 0 on the English I composition and almost 80% on English II. For first-time testers, more than 40% scored a zero, compared to more than 70% for retesters.

The TEA insists that machine scoring is in line with human grading.

“We’re confident in our results,” Rozunick said. “We continue to look at things, you know, as they’re raised, and we haven’t seen any red flags to date.”

When did the TEA actually announce automated scoring?

School leaders have argued the TEA did not provide enough advance notice about the shift to machine scoring.

“The concern with the specific issue of a machine scoring writing isn’t the true act of the machine scoring it, it’s more of the way it’s rolled out,” Chambers said. “The reason that’s such a concern is because the stakes are so high with the accountability system.”

STAAR results play a major role in the TEA’s A through F accountability system, which assigns campuses and districts a letter grade based on student test scores and post-graduation readiness. The TEA revamped the accountability system last year. More than 100 school districts across the state sued the TEA over the changes, temporarily blocking the release of more stringent ratings.

In the eyes of Chambers, the machine scoring and A through F revamp represent the same thing — the TEA “moving the goalposts in the middle of the game.”

A TEA spokesperson pointed to a state legislative hearing in August 2022 as the first time automated scoring was made public. Education Commissioner Mike Morath presented updates on standardized assessments to the House Public Education Committee. During his more than an hour-long presentation, he mentioned automation once.

“We are exploring cost-savings everywhere we can,” Morath told lawmakers. “So, automation is a big way to save costs. If, for some reason, people don’t like automation, then we have to un-automate, which will cost you more money just to stay the course.”

According to the last slide of his presentation, “full human scoring” would cost up to $20 million more per year.

The agency spokesperson also pointed to training for testing coordinators in September 2023, presentations to “various stakeholder groups” in November, and the release of the “scoring process documents” in December.

According to the March 1 message from the TEA, district testing coordinators who contact the agency before March 13 will have the opportunity to view student responses that earned a 0 in December, but they won’t be able to see the actual test questions or explanations of the grading rubrics “due to test security requirements.”