Why Data Teams Fail
You're moving too fast or too slow. Here's how to reframe how you work and find your Goldilocks way of building.
Most data teams sit at one end of the extreme. They either move way too slow, unable to prioritize what the business needs in the moment, or they move too fast, not thoughtfully designing a scalable data platform.
The teams that move slowly are too busy worrying about the “what-ifs” before they even have a chance to see if what they are building provides value to the business.
The teams that move too fast are stuck in a never-ending fury of ad-hoc requests, unable to think about anything but the current moment. They don’t have the capacity to take a step back and identify the pitfalls in what they are building.
I’ve worked as an analytics engineer on both types of teams, and neither is sustainable if you want the business to be truly data-driven and care about the things you build.
The problem with too fast
Every data team I’ve joined has been overwhelmed by data requests. They were teams of just a data analyst and a data manager, taking orders like they were fry cooks. They weren’t focused on building systems that would help them scale, but just getting stakeholders what they needed, fast.
As an analytics engineer, I’ve been hired to help these teams slow down. I’ve built data models at the right grain to reduce complex code in BI tools, promote modularity, and allow them to dig deeper into patterns in the data.
If you think you are building a scalable foundation without the help of an analytics engineer, you probably aren’t. Your infrastructure is most likely crumbling underneath you because you haven’t had the time to think 5 steps ahead.
If you are…
building datasets in your BI tools
writing complex transformations within your BI tools
using R for transformation instead of SQL
focusing on multiple projects rather than defining one key initiative
not taking the time to document and write style guides
…you are not building for the long-term. This will lead to loss of trust from the business (due to slow or inaccurate data), more complex systems that will eventually need to be untangled, and a platform that’s impossible to onboard future data team members onto.
The problem with too slow
Now, some people may disagree with me here, especially if you come from a data engineering or higher education background. I don’t think you need to build to protect against every worse-case scenario.
I’ve talked with people who both agree and disagree with this statement. Some think building with protection is what defines the role of an engineer. You need to put up defenses in your code for things that haven’t happened but could happen some day. Others agree with me and think it’s better to first provide value and then iterate and strengthen what you build over time as it proves itself useful.
A lot of this depends on the maturity of data within an organization. If the business has barely any core data models to work with, focus on building those out first and iterating on them. A well-done MVP is better than an overly-planned multiple-month-long project.
If the business has a lot of core data models built out, you can afford to be more methodical in how you build out your data models.
But, again, data people frequently end up building things that don’t ever get used. Don’t spend a crazy amount of time perfecting something that hasn’t yet proven its value. Get it out into the world, let the business use it, and then work on making it bulletproof once its proven helpful.
Data myth: build it and they will use it.
Reaching the next milestone in data
You need to find your sweet spot with how the data team builds at your company. It can’t be too slow, but it also can’t be too fast… It needs to be just right.
Ask yourself the following questions:
Is the way I’m working sustainable long-term? Am I burning myself out?
Does the business trust our data?
How long does it take us to provide value to the business when they tell us they need something? Is the solution scalable?
Do I have the expertise needed on my team to build something that can last?
Do I have systems in place to produce high-quality, trusted data?
If you find you don’t have the team needed to set the foundational practices needed to build something that can really scale and make true business impact, reach out to me. I’m happy to offer a free consultation call.
Have a great weekend!
Madison
For more on crafting a modern data stack from scratch using analytics engineering best practices, check out my ebook, The ABCs of Analytics Engineering.
The main reason that data teams fail is that they don't take my advice. I have not had a failed project in 34 years by any customer who has taken my advice. And still men won't take my advice. It is very simple. Any public for profit company who wants to sustainably increase their profit over a long period of time can simply use my free software and free methodology and take the advice I have given publicly. That alone will more than double their chances of success.