Analytics Engineers Are Taking Over
How I accidentally built a career in analytics engineering and why its popularity only continues to grow
I was first introduced to analytics engineering by a little data transformation tool called dbt. At the time dbt was a purely open-source tool and quite new to the market.
A data engineer at the time, I used it as an engine to apply SQL to large CSV files. This isn’t how dbt is meant to be used, but it worked for our use case.
Little did I know that this introduction would lead to an entire career path in analytics engineering, a career I didn’t even know existed at the time.
I actually stumbled upon an analytics engineer role while browsing LinkedIn, trying to find a job that combined both my data engineering experience with what I was most interested in at the time- data analysis.
I applied for the job, went through multiple rounds of interviews, landed the role, started writing about what I was learning, and the rest was history.
Analytics engineering has changed a lot over the years. It’s evolved from a role that centered around dbt to so much more. Now, it encompasses data warehousing, data modeling, building pipelines, communicating with stakeholders, defining metrics, and powering dashboards.
Analytics engineers are no longer tied soley to dbt.
They are truly the ones that connect data to business value and revenue outcomes.
What is an analytics engineer?
An analytics engineer transforms raw data into insights in a way that is scalable. There is nothing an analytics engineer does that can’t be directly tied to a business outcome.
Every piece of data that is ingested, transformed, and designed should drive some type of business initiative. Analytics engineers help move data to make this possible.
Of course, there is some ad-hoc analysis and infrastructure work here and there, but the primary responsibility is to provide repeatable value with data.
This sounds a bit vague, but that’s honestly intentional. Analytics engineers can really do anything. The thing that differentiates them from data engineers is the business understanding. The thing that differentiates them from data analysts is the scalability and repeatability.
They aren’t writing a query once, but building something that can grow in value over time.
When I first started as an analytics engineer, I focused on tools and coding languages. That helped me get familiar with different foundational principles like access control in data warehouses and incremental models, but it missed the point of what analytics engineering really is about.
Compounding value.
Day in the life of an analytics engineer
Ok, so you understand that analytics engineers help turn data into insights and drive value for the business. Well, what does this look like in practice?
The truth is that every day looks different.
However, most days are spent communicating back and forth with the business, trying to understand what they need, and then building the data models that support that need.
For example, let’s say I’m working on a brand budgeting dashboard for the sales team. The first step of this project is meeting with the team, understanding their current processes, and getting to the root issue.
At first, it may seem like they want to mimic the process that they currently have. But, after probing a bit more, you learn that they want to add a few more complexities that they aren’t currently able to do manually. So now, you aren’t just figuring out how to automate a manual process, but also how to layer on additional complexities.
This often involves many iterations of building a data model, presenting it to them, and discussing what should be changed, validated, added, etc.
Analytics engineering is collaborative, but it’s also about building something scalable.
When you’re building your data models, most likely using SQL and dbt, you need to think about how it fits in with your other data models. You also need to identify any gaps in the data you have available. It may involve making changes to current models, or creating new ones. With every change or new model, you also need to balance complexity and scalability.
As analytics engineers, we don’t build something to only serve one purpose. We build something that can be reused for many different business problems.
How we differ from data engineers and data analysts
Data engineers focus on moving and processing large amounts of data. They don’t look at this data and how it relates to the business. It’s strictly about infrastructure.
Data analysts are consumers of the datasets that analytics engineers build. They don’t model the data to scale, but rather use it for one-off queries and dashboards. Data analysts are very connected with the business, but lack scalability.
You can think of analytics engineers sitting somewhere in the middle. Similar to data engineers, they build processes that are reliable and can last. Similar to data analysts, they work closely with the business to drive outcomes using data.
Analytics engineers own data modeling, which I like to think of as the perfect intersection between data engineering and data analysis.
Why data modeling is more important now than ever
We are continuing to see more and more analytics engineers because of the necessity of data modeling. Data modeling was always important, but now with AI, many companies are finally starting to prioritize it.
Data modeling makes sense of raw data and layers in important business context that AI models need to leverage data to answer business questions. Without data modeling there is no semantic layer, no agentic analytics, and no source of truth.
The future of analytics engineering
I believe we will begin to see more and more data teams comprised soley of analytics engineers. Companies now have the option to replace data engineers and data analysts with tools like Airbyte, Fivetran, and AI-assisted BI.
We are already seeing a move to “jack-of-all-trades” roles, like analytics engineers. With the right tools, analytics engineers can cover things like schema changes and dashboarding. In fact, these things become easier when the people modeling your data are so connected to them.
Analytics engineers know exactly what will break if a field is removed or named in a database.
Analytics engineers know exactly how to model the data so it can easily be consumed by a dashboard.
By owning the entire end-to-end data process, communication gaps no longer exist and pipelines can be built to scale.
I’m currently on an all-analytics engineers data team and the process is much smoother than when I worked directly with data engineers and data analysts. We know exactly what is going on with the data at all times.
What do you think? What does the future of analytics engineering look like?
Have a great week!
Madison