Stop Letting Data Errors Compound
The most important decision is the one you make after a failure
We always boast about our successes and rarely talk about our failures. However, I’m a huge believer that our biggest failures lead to the most valuable lessons in life.
This is why I try to frequently talk about the failures I’ve made as an analytics engineer. Nobody is perfect! I’ve broken data pipelines, written development data to production, and pushed untested changes to the main branch.
Mistakes happen. Failures happen.
When we turn mistakes into lessons, we become better and stronger as individuals, and as data people. It’s when we don’t learn from them, and continue to make them, when things start to take a downturn.
Sahil Bloom recently wrote about the compounding mistake.
This is the idea that the more frequently you make a mistake, the more likely it is to compound into a negative outcome.
One workout skipped becomes ten workouts skipped. The next thing you know you are sedentary and 20lbs overweight.
One data model undocumented becomes ten data models undocumented. The next thing you know none of your models are documented and legacy knowledge is sprinkled among different people throughout the company.
In my last two analytics engineering roles, when I joined the data team, the data modeling was unorganized and more tangled than a bowl of spaghetti.
There were data models with names like “transform 1” and “transform 2”. It was impossible to tell the orders of the models, what they did, and which models referenced one another.
It was so bad, that “transform 1” and “transform 2” weren’t even subsequent models. Someone had clearly added patches between them to attempt to fix any gaps, making it more difficult to read in the process.
The dbt project lacked guidelines on naming conventions, data governance, or best practices. It was a free-for-all. The project was half used by engineering and half used by analytics. Whoever wanted to add a “model” to it, could. The models weren’t documented, properly named, tested, or had clear owners.
In both cases, the companies needed to bring an experienced analytics engineer onto their team to fix the mess they had gotten themselves into. However, if they realized the project was spiraling to a place where it could no longer be properly maintained, they could have avoided future headaches.
In many cases, we can’t avoid the first mistake- you don’t know what you don’t know. Whether you accidentally upgrade a tool without reading the documentation or need to push a change quickly to make a stakeholder happy, little slips happen.
However, Sahil Bloom says,
“After a failure or mistake, the next decision has elevated importance—this is where you can stop the negative compounding and start a move in the right direction”.
As analytics engineers, we need to have awareness of our failures and mistakes, so that we can fix them. Awareness prevents you from traveling down the wrong path.
When something isn’t working, don’t continue doing it. Find a better solution and iterate on it.
In my upcoming course, Transform Your Data Stack with dbt, I will teach you how to build your dbt project following best practices. You won’t have to worry about figuring it out as you go or racking up a ton of tech debt.
You’ll learn from all of the mistakes I’ve made in the past, and the mistakes of others that I’ve had to fix, so you don’t make the same ones.
Some of the hands-on projects include:
Creating your own dbt style guide (we will learn why this is crucial in building a scalable project!)
Documenting using doc blocks to centralize metric definitions
Adding full-coverage tests to your data models
Writing macros to reduce redundant code in your project and achieve modularity
The course will consist of 4 live sessions, where you will have direct access to me and a community of data professionals who are all there to learn and grow as analytics engineers. It will be supplemented with reading material to help you complete the projects and apply the knowledge to your daily work.
I’m a firm believer that learning from others is an excellent way to reduce compounding mistakes. When you harness the knowledge of those who have done what you are trying to do, you catapult your learning and growth.
This is why I’ve invested money in courses from those at the top of their industry. I’ve bought courses from Justin Welsh for solopreneurship,
for data engineering, and Jessica Stone for life purpose/spirituality (where are my woo-woo people at?).Don’t hesitate to reach out with any questions on the course! And, if you haven’t talked to your company or manager about a learning development budget, this is a great chance to do that.
Have a great week!
Madison Mae
With all due respect, but yet another advertisement of the course without the real value of the article (: