4 Analytic Engineering Fundamentals That Haven't Changed
Because these are still things that AI can't replace and become more important now than ever
Analytics engineering is changing rapidly.
We can now do so much more in such little time. Data models can be built in minutes, pipelines can be debugged in seconds, and the overall pace of work is accelerating.
With all the doom and gloom out there, it can seem like AI is coming for our jobs. While AI is really good at the technical skills that once took decades to perfect, AI is still lacking in the thing that makes analytics engineering so special- the ability to encode and extract messy business processes from data.
Understanding the business and using that insight to create scalable systems has always been the unique value proposition of analytics engineers.
That hasn’t just gone away.
It is more important now than ever to harness our skillset as analytics engineers. If you’ve been doing this for awhile now like I have, you certainly have a leg up, but that doesn’t mean you shouldn’t stop learning.
It also means that any aspiring analytics engineers shouldn’t feel discouraged. Now is the time to start learning these skills! Especially if you’re a data analyst.
Data modeling
Stakeholders are now turning towards AI agents to calculate their metrics and give them interesting insights. This means that the datasets that power these AI agents need to be clean, clear, and concise. Otherwise, agents will have a hard time knowing which model to turn to for specific metrics.
Sadly, I don’t know one data team (maybe Anthropic’s lol) that has a clean data model foundation. Many have similar fields with no clear distinction in multiple models, unclear definitions, or barely any data modeling at all.
Anthropic recently came out and said that canonical datasets with enforced standards is one of the key factors in allowing them to achieve 95% accuracy with self-service analytics. Because data modeling requires a strong understanding of business processes, it’s not something you can delicate to AI.
Accurate data modeling involves many business conversations, a ruthlessness when it comes to dimensional data modeling design principles, and time. A lot of time. Teams that have been working on this for years are still in the trenches!
Most companies will really need to step up their data modeling efforts before they can even think about self-service analytics. This means more analytics engineers will be needed to create this solid foundation!
Data governance
With everyone wanting to touch data with AI tooling, data governance knowledge becomes a prized possession. Those that know how to deal with data safely are going to be the ones that win. It’s only a matter of time before the ones who gave these agents access to their data without thinking twice start to see the consequences.
Once again, the data team will be critical in deciding how everyone across the company can access data with these tools. Analytics engineers who own the data warehouse will be at the forefront of creating safe data governance policies.
I wish data governance wasn’t so complicated, but it is.
Rather than depending on AI to tell you what/who to give read and write access, you need to harness documentation and have a strong understanding of permission hierarchies.
Even without a strong understanding, you can be the one to play “bad cop” and question all the unsafe data and AI practices happening, being the one to figure it out the right way.
Business communication
You can almost guarantee that AI will not ask stakeholders the questions that need to be asked. It doesn’t know the in’s and out’s of the business like someone who sits in company-wide calls, reads product briefs, and observes what’s happening in Slack channels.
Because AI is replacing the more technical, logical pieces of our work, understanding the business is more important now than ever. We need to know what questions to ask to get to the root of their problems. The more we understand the in’s and out’s, the more we can bake them into our data processes.
I often think this is the hardest skill to learn because it requires practice. The more you talk with stakeholders, the more you sharpen it. I had to go through many projects that felt like the back-and-forth’s never ended before I finally started to realize what things I needed to ask in the very beginning.
However, when you finally start to get the hang of it, it’s a skill that makes you unstoppable.
Query optimization
AI can help you get to the right answer, but it doesn’t always do so efficiently. It’s important to have a strong understanding of query optimization so you can spot when this is the case.
Optimized queries mean faster performance and lower costs- two things even more important when using AI tools with your data warehouse.
Basic concepts related to query optimization include:
indexing
data pruning
partitioning
materializations
These are all things that an agent isn’t thinking about when it spits out some code.
For paid subscribers, I’ll be diving deep into these fundamentals over the next few weeks. If you’re looking to up-level your foundational analytics engineering skills, be sure to follow along.
Have a great week!
Madison