Correct Data for Data Driven Decisions

Being a "data-driven decision maker" is as cliche today as being a "servant leader." Both descriptions need to be prevalent in job applications, but in reality, if you are not both of these, then you should not be in leadership. Servant leadership is another blog post, where I defend that servant leadership is only sometimes good, but I quickly digress.

If you are not in the business of optimizing decision-making while providing knowledge and resources to others, it is hard to defend calling yourself a leader. You may have connived some people into giving you a title, but that does not mean you are leading. Being "data-driven" means having enough high-quality data analyzed through a critical method to tell a true story, regardless of the results.

Common downfalls of data-driven decisions are using too small or homogenous a sample size, ignoring relevant data, changing too many variables, or manipulating data to show what we hope to see rather than reality. If the reality does not match the hypothesis or desired outcome, ignoring or skewing the data to fit what's wanted only leads to problems.

Too little data

I have seen too many district and school-based leaders overemphasize one exceptionally good or bad year of test results, ignoring the trend. One grade level of students not meeting standards after a decade of exceeding standards should raise some concern with targeted interventions for that class of students. It should not dictate a need for wholesale changes to programming to ensure it never happens again. Conversely, one incredibly strong performance after years of mediocrity should be celebrated. But it should not lead to subsiding the push toward excellence. One important question all research studies must address is the "n-size," which is the number of participants and/or trials completed to make a decision.  Causation and correlation are very different ideas! Check out these "spurious correlations" from tylervigen.com as examples:

AND 

Too many changes

Schools are notorious for changing initiatives one after another. The whiplash that comes with this leads many people to avoid implementing any change and the status quo prevails. Ideas keep being added, but little is intentionally stopped and removed. After a few cycles of this, teachers' attitudes of "this too shall pass" sink in. Some may play along for appearances, and a small few may actually try, but most will close their doors and do what they have always done. Why would anyone invest time and energy into one thing when something else will replace it in six months or a year?

Too many changes also mean trying to change too many variables in the equation at once. Perhaps a culture survey reveals that a school demographic lacks a sense of belonging. In a great attempt to remedy this gap, a club is created, phone calls home are made to engage families, student schedules are altered, and culturally relevant music is played throughout the school. All of those are great ideas! But when the next survey data is analyzed which of those mattered? Was one of those positively impactful but one was seen as tone-deaf and patronizing, and they cancelled each other out? Good decisions about what to continue or stop cannot be made accurately when more than one variable has changed. 

Ignoring or manipulating data

Ignoring bad data or making excuses for it only exacerbates the issue at hand. Take for instance data showing a lack of growth in reading scores year over year. Blaming a changing demographic, COVID-19, or low attendance are easy scapegoats to make staff and leadership feel better. It also allows for everyone to avoid change. There may be some causation between any of those and students failing to grow, but without digging deeper and taking a hard look at the instructional practices and resources will never turn the results around. 

The next easy area to blame in this scenario would be the reading resource. Again, it may not be the right one for students. But readers of this blog know (or will soon) that I firmly believe in people over resources. Changing from one phonics program to another without improving the instruction behind phonics will just mean a lot of money was spent for no change.

Conclusion

Data is abundant, necessary, and often misused. As a leader analyzing data, make sure that the data set is large and diverse enough to be significantly important. Look for trends more than outliers, and react with precision not scattershot. Do not fear or ignore results that tell a different story than the narrative being sought. It may be painful to not get the desired results, but those gaps are where the growth can happen. Make that growth happen one change at a time, and measure that impact when possible. Otherwise, wasted resources are used on well-intentioned but low-impact actions.