Data engineering is building traps
It's not about tools, and not just about data. It's about producing abstractions of meaning, and every abstraction has a cost of understanding.
Data engineering is not about tools. It’s not just about data. It’s not even about pure “value”, financial or otherwise.
It’s about producing abstractions of meaning.
And it necessarily involves modulating action, so that action produces more precise and delicate effects. Data engineering is the process of designing and building abstractions that organize those patterns of effect: so that we can actually perceive something.
The cost of understanding
The beauty of orchestrating information with words is that abstractions have an energy cost, and not only in the physical sense. They also have a cost of understanding.
That’s why the cost of modeling a data system should be inversely proportional to its return and to its maintenance cost. And it should be justifiable to keep a technical foundation strongly oriented to consolidating the information produced by everything we can capture of the organization’s effects on the world.
Vital signs
Clearly the main effects are financial: they’re like the vital signs of a living being. Every organization depends on inputs, and the cost of those inputs can, inevitably, be converted into financial cost. That doesn’t capture many social aspects of an idea’s strength, but it does capture a real energy cost of the existence of any social group set on transforming the space outside itself.
A person’s time is always a measure of available energy. And the ways of leveraging it are at the heart of everything we do with data.
A culture, not a department
Data engineering is one aspect, one quality, of a spectrum of techniques and skills needed to develop a data culture: a social organism oriented to producing systems of knowledge about its own actions and their results, in every way that’s possible and feasible for the organization, at any point in time.
Arapucas
An arapuca is a trap of sticks stacked into a pyramid, common in the Brazilian countryside. The bird walks in, touches the trigger, and the structure comes down. It’s an elegant mechanism: simple materials, one precise action, and the one who triggers it is the one who gets caught. And unlike a snare, an arapuca takes the bird alive.
A good data model works like that. It isn’t taxidermy, pinning meaning down, dissecting it and freezing it into a schema. It’s a simple structure that living meaning walks into, where it can be observed, shared and let go again.
Data engineering is building traps.