Some hopefully useful questions to guide data-driven decisions

I’ve mentioned in multiple posts on this blog that the primary “point” of data is to help inform decisions. There are probably occasions where it’s useful to just report out on broad compliance-y things, or where we want to explore data with no clear decision in mind in the hopes that some revelation will jump out and whack us over the head. But mostly, if we’re devoting any considerable amount of time to analyzing data, we ought to have some potential decision in mind.

Data-driven decision making obviously isn’t anything new. People have probably been doing some version of this since we started connected cooking our meat and getting sick less. And if you Google “data driven decision making,” you’ll get about a gorillion hits, with results from Harvard Business School and Tableau and IBM and Forbes and countless self-proclaimed “thought leaders” and everyone in between.

And although I’m much too self-aware and have far too much dignity to ever (unironically) call myself a thought leader…yes, I am also about to write about data-driven decision making. And I’m even going to have the audacity to propose a framework/set of principles to help you, dear reader, support data-driven decision making in your own school or school district, only for the low price of…I kid, I kid.

For real though, a big part of my job is working with my colleagues to help them refine their maybe-not-fully-baked requests for data analyses. It’s incredibly rare that someone comes to me with, like, a detailed proposal delineating exactly what analyses they want conducted and which students ought to be sampled and where the data lives and who owns the data and what sociopolitical factors are spurring them to come knock on the wall of my cube and apologize for intruding and ask for this set of analyses at this particular moment. Usually people slink around the corner with a bindle full of ideas and their own subject matter expertise and ask if we can maybe set up a meeting to sort through everything together.

Which is great – this is actually one of the aspects of my job I enjoy the most. I get to work with smart people who want to do something meaningful for students but who also need some help figuring out the data piece of it all.

And so, a couple of years ago, my teammates and I put together an internal framework for managing these conversations. Since this whole “data-driven decision making” enterprise has been around basically forever, there are oodles of resources available. My favorite one was (and still is) Caitlin Hudon’s data-driven decision making framework. And so, when we were creating our own internal guidance, we liberally adapted and outright stole from Caitlin’s framework.

Thanks Caitlin!

We ended up creating a set of guiding questions that we ask our other colleagues when they come to us with requests. These guiding questions fall into four categories, and we usually broach them during meetings in roughly this order:

Problem/Concern

These questions are all just intended to help people kick ideas around. Maybe narrow something that’s too broad down to the most important cognitive kernels. Maybe articulate some underlying concerns or questions that are actually at the heart of the matter.

Some of these guiding questions include:

• Tell me about your concerns about X?
• Why are you interested in looking into data about X?
• What specifically are you trying to learn about X from the data?
• Can you state your problem/concern about X in a single sentence?

Define the Decision

After rolling around the ideas a bit, we’re hopefully ready to define an actual decision we want to make. The point of this part of the conversation is to move from a vague feeling of "it would be interesting to learn more about X" to a concrete "I want to make a decision about X."

Some of the guiding questions in this category are:

• What is the decision you’d like to make? And how does this relate back to the stated problem?
• Who is impacted most by this decision?
• What are your possible choices?
• What does a “good” outcome look like? (This is trying to get at the underlying thing(s) we want to maximize or minimize)
• Are there any outcomes that are unacceptable?

Define Additional Context and Constraints

Once we have the broad strokes of the decision defined, we next discuss some additional context. This gets into things like deadlines, timings of relevant contextual factors, potential collaborators or other interested parties, relevant policies, etc.

Some of the guiding questions in this category are:

• Do you have a decision deadline? What happens if you don’t meet it?
• Is your decision reversible?
• What are the potential costs of one decision vs another? (Note that this could be part of the later data analysis if these aren’t known upfront)
• Beyond the people/groups you represent, is there anyone else this decision would have a large impact on?
• What policies, if any, exist that could influence this decision?
• Are there any other critical practical constraints?

Determine Data Needed

The last (!) thing we discuss is data. Well, that’s the idea at least, although sometimes data comes up earlier. This isn't a script. But I really do believe that it’s not worthwhile to start talking about what data is available, how we ought to analyze it, and all of that until we’ve determined what decision we want to make and walked through all (or at least most) of the contextual factors influencing the decision.

Anyway. These questions get into the data, but you might notice they’re mostly not nuts-and-bolts questions. Even when we’re discussing data, what I need most from the person making this request is still value- and subject-matter-centric stuff. I’m not asking about the details of, say, an individual column in a spreadsheet – I can figure this out later, or else I can send an email.

Here are some of the questions that fall into this category:

• Is there data that would unequivocally lead to any specific decision?
• If you’re leaning in one direction, what data would change your mind?
• What kind of deliverable would be most helpful (e.g. a number, a spreadsheet, a dashboard, a report, a visualization, etc)?
• If the “ideal” data isn’t available, what would it cost (in terms of money, effort, time, and social capital) to collect it? Is the data worth this cost?
• What specific metrics do you think are most useful to consider? (Ideally we want to limit this to a few well-defined metrics)

Since I’m a real person and not an automaton, I’m not literally going through all of these questions verbatim during our conversations, but I do try to make sure I’m more or less getting all of this information. I have a note-taking template, too, that has neat little sections that correspond to those above, and each section has the guiding questions printed in it because I very often get sidetracked and need a visual cue to get me back on track. It may also take a few conversations or follow-up emails to get everything, especially for larger projects that have the potential to affect multiple people.

To summarize – this framework for thinking about data-driven decision making has helped me, and it’s helped me support my colleagues. Maybe it’ll help you too. I’m not promising it will change your life or your career trajectory. I’m not claiming any of this is revolutionary or visionary or any other kind of -ary. I’m certainly not claiming it’s “disruptive” or a “force-multiplier” or any other thought-leader bullshit. 

If you’re enjoying reading these weekly posts, please consider subscribing to the newsletter by entering your email in the box below. It’s free, and you’ll get new posts to your email every Friday morning.