The Key to AI-Ready Data: Metadata
What is context engineering, and why is it so important to the success of analytics engineers and AI implementation?
To succeed as an analytics engineer, you need strong technical skills and even better business understanding and communication skills. This is because of all of the context that comes with data.
Data without a deep understanding of the who, what, why, and how won’t help you get the right answers to help drive revenue, scale your business, and make the best decisions. The greatest strength of analytics engineers is AI’s greatest weakness; without context, it is useless.
Analytics engineers bridge the gap not only between business stakeholders and engineers, but they are also in a position to do this with AI.
When given the right context, AI has the power to greatly support business needs through strong data practices, 10x-ing your productivity and speed as a data team. This practice has become so crucial that it has its own name: context engineering.
What is context engineering?
Context engineering refers to the art of delivering the most relevant metadata to AI at query time. Just like anything else, it takes a strong understanding of how the technology behaves to ensure you are giving it the information it needs to make the most informed decisions.
For analytics engineers, context engineering helps AI make sense of the information that already exists rather than being a black box for data users.
Adding context engineering to the analytics engineering workflow involves giving it the metadata it needs to make informed decisions without explicitly being given a restricted number of resources. Instead, you feed AI your data models with important details such as ownership, data quality status, upstream/downstream dependencies, and usage within external platforms like the BI layer. This replaces the step of pre-approving ready-to-use data models and allows AI and your data infrastructure to work seamlessly together.
For example, context engineering in practice gives AI access to the metadata that I would otherwise manually search for in external tools. When deciding on which model to use in a dashboard, I look in dbt Cloud’s catalog. When deciding which model should be prioritized due to high amounts of downstream usage, I check the BI tool. If I have two equally compliant, well-documented models, each powering multiple reports, but one driving 500 daily queries while the other hasn’t been accessed in months, I’m always going to prioritize that one.
A metadata intelligence tool like Euno provides the data I need to do context engineering the right way. It ensures AI recommends the model that’s actively delivering value, not just the one that looks good on paper, while eliminating all of the analytics engineers’ manual overhead.
How metadata helps with context engineering
You can think of metadata as data quality, ownership, lineage, usage, total run time, and more. With context engineering, it’s essentially all of the details about a data model that indicate whether it’s healthy and ready to be used by the business. If something goes wrong, metadata is the first thing analytics engineers check.
Successfully leveraging this metadata helps to unlock the confidence to use AI within your data stack. You no longer have to worry about AI using data owned by a certain business unit or locked in a certain environment. Harnessing your metadata eliminates the need to second guess whether a dataset is trusted and reliable because all of the information needed is available.
Metadata for governance
Metadata platforms like Euno can use ownership status, number of dependencies, whether or not there is documentation, and more to tag your data models and other assets as AI ready or not.
For example, you may only want to allow access to AI if the data has an owner, uniqueness tests in place, and only depends on verified gold standard sources upstream. You can configure these rules and definitions within the platform based on your organizational standards and let AI do the rest.
In turn, this allows analytics engineers to move faster and build trust with stakeholders. There is no more thinking that needs to be done every time a stakeholder wants to run an AI model on top of your data because Euno has the proper context engineering covered.
Metadata as quality control
Metadata also unlocks the ability to predict potential errors and breaking changes before merging code to production. Using Euno, you can make quicker and more seamless code changes by taking advantage of usage mapping and pre-processed lineage.
Instead of needing to assess which models are dependencies, you can assess how often those models are used and how much impact the change will truly have. This allows you to focus on the code change itself rather than the implications to the business.
A new analytics engineering workflow
When you add metadata and context engineering into your AI-enabled analytics engineering workflow, your workflow looks something like this:
Accept request from your stakeholder for a new data model
Set up rules for what defines a high-quality dataset in Euno using run time length, existence of tests and documentation, and recent freshness
Model your data using details from stakeholder
Make a change to upstream model to support new data model that you’re building
Ask AI in natural language to check the upstream and downstream dependencies to prep for production
Ask Euno to identify owners and usage levels of these models so you can properly communicate your changes
Ship your data model with altered dependencies with confidence
Do a happy dance because your model will be labeled as healthy thanks to the rules you set and this information is now available to other team members and AI agents
AI and metadata will soon be baked into everything we do as analytics engineers, from stakeholder management to data modeling to orchestration debugging.
How metadata, context engineering, and analytics engineering work together
Analytics engineers aren’t going anywhere. We are still the ones powering AI agents with strong business-centric data models and clean raw data. Now, we get to also play the role of context engineers by off-sourcing the more manual work to metadata tools like Euno.
Euno allows us to automatically add context like tags and precomputed lineage to our AI agents, ensuring confidence in the data we give AI access to and allowing us to focus on more technical data modeling problems.
Heading to Coalesce? I unfortunately won’t be there but the Euno team will be! Swing by their booth for a demo and a chat on all things analytics engineering and AI. They also happen to have my favorite swag (wink).
Have a great week!
Madison
Thanks for including my work!