My REAL Thoughts on Real-Time Data

Monthly Q&A #1: technical interviews, mentorship, CI checks, and dbt docs

This is the first post in my monthly Q&A series, where I answer any analytics engineering questions on your mind. I want this to be a safe space to express the problems you are struggling with and how I can help.

Whether you are a business stakeholder trying to understand the data development process, a data analyst looking to transition to analytics engineering, or a long-time analytics engineer looking for a different perspective, your questions are welcome here!

Let’s dive in!

How do I learn real-time visualization?

Any time anyone asks about real-time anything, I always start by asking why. Stakeholders tend to ask for real-time data, and many data engineers always assume real-time is the best way to deliver it.

However, this couldn’t be further from the truth. Real-time data is rarely necessary. Whether anything should be real-time or not depends on the data itself and how it’s being used. If a stakeholder checks a dashboard once a day, or even once a week, that visualization does not need to be real-time.

When you deliver real-time data for something only looked at daily, you increase costs, decrease performance, and increase the likelihood that something goes wrong. With your data updating more frequently comes more possible issues.

Real-time data is only really needed for situations like financial trading, fraud detection, and customer-facing experiences. It’s rarely needed for business stakeholders outside of these use cases.

However, if you wanted to visualize data in real-time, the first step would be to work with data engineers to ensure you are ingesting and transforming data in real-time. Everything further up the pipeline needs to be handled in real-time before you can think about visualizing the data in real-time.

Would you ever be open to being a mentor for analytics engineers?

I consider my weekly newsletter my form of mentorship. I put hours of work into the content that I share each week. I love it when people ask questions, and I am always happy to answer them. So if you are struggling with something, please reach out! I also want to continue doing these monthly Q&A posts for paying subscribers, so I can dive deeper into your questions and help you directly.

I have 2 analytics engineer interviews coming up. It will be a live coding challenge in SQL and Python with questions testing some statistics knowledge and business acumen. They would like to see how I take a business problem from a PM, solve it technically, and communicate insights. Any pointers?