Monday, May 16, 2011

Telling Time: The Long and the Short of It

One aspect of education reform that is incredibly difficult to deal with in the highly politicized world that education reform occupies is time.  Time complicates everything both for those aggressively pushing new tactics and for those who believe the new tactics being introduced are a waste of resources.  Time, or more precisely a lack of it, creates false narratives, meaningless debates and premature decisions.  One thing that doesn’t exist when it comes to aggressive reform is patience whether it is the reformers pushing for change or the traditionalists pushing to evict the reformers.  Something missing from the larger debate is a firm understanding of time.

Life is subject to randomness.  Some days you are lucky and some days you are not.  Some days go better than expected and some days go worse.  Some years test scores rise and some years test scores fall.  This will happen no matter what.  The world is a random place full of the inexplicable.  Large sample sizes are required to reach confident conclusions.  Consistent trends take years to see.  Variance is the enemy of reliability and consistency.  And a consistently good education is what our students need to succeed over the course of 12-plus years in school.

It is easy to jump to conclusions after the short term.  But those conclusions are short term and nothing else.  What we are all in search of are long term solutions and those require long term conclusions.  Every year we see the release of test scores as a finish line, a referendum on newborn policies.  Every year someone like Jay Matthews will chime in with a voice of reason and say that a single year’s test scores don’t really mean much.  What matters is a three year trend, a five year trend, or, god forbid, a ten year trend that actually encompasses the bulk of a child’s education.

In a world full of competing interests, a need for instant gratification and education as political issue it is exceedingly difficult to give enough room for success to blossom.  Everyone is so focused on one off anecdotes that signal educational Armageddon or one number that represents the Absolute Truth of what is happening or aberrant results whether good or bad that they fail to look at the whole of an incredibly complex, impossible to fully understand picture.  No matter what anecdote a person finds there will be a matching opposite.  No matter what number a person identifies there will be an argument about what it really means.  No matter how extreme a success or failure is, it is likely to be less extreme next year.  Only after a number of years can we trust the numbers.  Only after a number of years of site visits, retrospective anecdotes, and injustices addressed do we start to see a narrative that may add some value.  Only after a number of years do outliers regress to the mean or signal once and for all failure and defeat.

School districts, and therefore school reform, operate on a school year schedule.  In year one a new intervention or reform or change is an idea.  In year two it will be implemented poorly because it is impossible to do something new perfectly.  In year three it will be implemented better and start showing usable results.  In year four maybe a legitimate evaluation can take place.  In year five you can toss the program if it is no good or try to scale it up to other locations if it is great. 

Every year, each program, each change, each idea, has one chance.  If that chance is missed the year is lost.  A focus on short term results compresses the timeline of responsible implementation, undermining any new program from the start and leading to evaluation of results without a firm understanding of the extent to which those results are noise, or random.  And the focus on early difficulties prevents proper learning by the school or the district.  Instead of learning from past mistakes, instead of learning how to deliver consistently, instead of evaluating long term data from many perspectives, a school or district or policymaker must defend their past decisions without truly knowing what works and what doesn’t.  Similar to the way teachers tend to be evaluated it is difficult to get past the voodoo and into the substantive and meaningful.  And sometimes someone gets stuck defending the wrong thing, or lets go of a promising program too soon, or a disaster too late.

This is why research and development is important.  This is why having the patience to try new ideas and see if they actually work is both important and difficult.  It requires time and political capital and credibility that nobody has.  Of course, this is how it is supposed to be, highly urgent.  We are failing kids every day.  Preaching patience is morally questionable.  But if we take a long term view, if we are serious about making the future a better place, we have to see that focusing exclusively on next year or even the next three years won’t help us build something better to outlast us all.  That doesn’t mean ignore the present.  No way.  Never.  But as we pull our hair out about short term results, recent battles, upcoming elections, all of us have to keep our eye on the future prize, the one where slow, steady solving might actually show us a place where skin color, zip code and parent’s occupation don’t define life outcomes.

And there is a small way everyone who cares about the education conversation can help get the long view instead of limiting themselves to just the short view.  And it isn’t to stop talking about yearly test scores.  That’s too unrealistic.  It is to learn basic statistical concepts.  I don’t care if you know stat or not.  I don’t.  I’m not statistician by any stretch of anyone’s imagination.  But understanding concepts like variance, regression to the mean, confidence intervals, statistically significant will go a long way toward understanding a clearer picture of what is and isn’t happening.  It will give a fuller picture of the difficulty of trying to identify successes and failures.  

Understanding common statistical fallacies like the Simpson’s Paradox or the Gambler’s Fallacy can make a huge difference in understanding what the numbers mean.  It can give anecdotes and headlines a broader context, a textual story to illustrate.  It can take some of the nitty gritty argument about numbers out of the arena of he-said, she-said and into the arena of explaining a nuanced approach to data analysis.  A better educated conversation could help us give the space to the experts to provide a better education to those who can benefit more from a discussion of solutions rather than a discussion of politics and personalities.

No comments:

Post a Comment