Fundamentals outlast waves

Over the last twenty years the analytics industry has gone through wave after wave of innovation and hype, producing along the way a flood of new needs, concepts, companies, tools, paradigms and professions.

We saw the move to the cloud, distributed systems and big data frameworks, followed by the rise of data science, analytics engineering and the modern data stack. And now, with the return of generative AI, endless discussions about RAG, vector databases, LLMs, semantic layers and knowledge graphs.

In the middle of all this, a restless market where hundreds of companies with similar pitches compete for room in every layer of the analytics stack.

I parachuted in

I’m a child of this generation myself. Like so many, I parachuted into analytics without much background, learning just enough to solve problems as they came. That’s normal at the start: you learn what’s being asked of you and get good at the tools and platforms.

With so many moving targets and so much diffuse information, it’s easy to lose sight of the fundamentals and walk in circles, learning how to materialize tables, draw charts or orchestrate DAGs in each new framework.

Turn the perspective around

At some point you have to stop, take a step back and turn the perspective around: learn the concepts underneath today’s tooling, and understand the motivations behind the innovations.

Using a tool well takes more than solving problems with it. It matters to understand its limits and traits, the context it was created in, the technologies it relies on. Fundamental ideas and concepts tend to last longer, travel better between implementations, and let us design solutions without depending on specific technologies.

Orthodoxy and speed

For anyone doing data modeling, reading Joe Reis is close to mandatory. Not because he owns some truth, but because, the way things are, a lot of basic knowledge seems to have been lost in a dark alley behind so many colorful ads for new tools.

In The Tension of Orthodoxy and Speed he touches on a problem I see all the time in data delivery: the rush to deliver “immediate value” leads to models that don’t sustain “continuous value”, and the result is a pile of backlog to validate and rework models nobody planned up front. Balancing orthodoxy with flexibility matters too, because today not everything fits a relational or dimensional model, and you need other tools where those models hit their limits.

Maturity before adoption

“The lesson here is timeless: adoption should be driven by alignment with organizational maturity.” A tale as old as time, and always current.

We’re living through a convoluted period in which hype, essential innovation and speculative innovation mix intensely, across many layers of technology and operations, in organizations of every size. Learning to navigate these abstract spaces of digital information is more and more necessary. And nobody has shown up with a proper map yet.

Don’t confuse superficial hype with what is fundamental and transformative. And don’t confuse convenience with utility.