Sorry, Educrats, the SOL Numbers Are Truly Dismal

Written for Bacon’s Rebellion by James A. Bacon

The Standards of Learning pass rates for Virginia public school students were grim this year, showing declines of  20 to 25 percentage points in history, math and science from the 2018-19 school year. English reading and writing pass rates weren’t as poor, but that’s in large measure because the State Board of Education had reduced the cut score — the number of correct answers required to get a passing grade.

Don’t make too much of the numbers, warned the Virginia Department of Education. In past years 99% of Virginia students took the SOL exams. This spring, due to lingering fears about COVID-19, participation declined to the 75%-to-80% range. Therefore, concluded the VDOE in its press release today, “making comparisons with prior years would be inappropriate.”

That is a fair point. If the “smart kids” were more likely to drop out of testing for some reason, that could have biased the results downward. Conversely, if the poor performers were more inclined to skip the exams, that would imply the opposite.

So, let’s take a closer look. Is the VDOE’s cautionary statement justified?

Two variables tracked by VDOE are known to correlate fairly tightly with SOL pass rates — socioeconomic status and disability. If kids classified as economically “disadvantaged” or “disabled” were significantly more or less likely to take the exams than their non-disadvantaged and non-disabled peers, the VDOE warning not to compare apples to oranges would be validated.

The following chart shows the number of test takers for five subjects in the 2018-19 and 2020-21 school years, comparing disadvantaged students with non-disadvantaged students.

The percentage decline in test taking between the two groups was pretty darn close across the board. Any variability seems to be statistical noise. Indeed, I am surprised at how little the difference was — I was expecting that poor students would show up for the tests in disproportionately small numbers, but that was not the case. Socioeconomic status does not appear to have been an influence on whether a student took the test.

The next chart compares the number of test takers who were classified as “disabled” and not-disabled.

We see the same pattern — no meaningful difference in the likelihood of disabled or not-disabled students to participate in the exams. Hence, we can say also that disability status was not a factor in whether a student took the test.

It is possible to drill deeper into the numbers than I have done, and I would love to see anyone else’s findings. But if disabled and/or disadvantaged kids were no more or less likely to participate in the SOLs, it looks like a comparison of the 2018-19 and 2020-21 school years is all apples to apples. The VDOE warning that results are not comparable is just a rhetorical ploy to avoid accountability for spectacular, mind-numbing failure.