The data Tower of Babel

At offsite retreats, morning stand-ups, sales kickoffs, 1:1 catch-ups and small-team weeklies, the message is the same: we need to get on the same page, sing from the same hymn book — level-set, sync up, get aligned. You would swear the goal of a business is not to make money, but to reach common ground.  

It’s a well-worn joke about corporate culture. But behind it, there’s an important truth. Large projects really do depend on clear communication. If corporate types are constantly banging on about the need for mutual understanding, that’s because it’s so important — and so hard to achieve.  

The data Tower of Babel 

Sci-fi humorist Douglas Adams imagined a creature called the Babel Fish: a tiny universal translator that curled around its host’s inner ear, rendering even the most obscure languages understandable. 

Sadly, the reality in large organizations is less Babel Fish than Tower of Babel. Businesses are made up of many teams that nominally pull toward the same lofty goal but are hindered by their lack of a common language. The languages in question are not spoken tongues, but different ways of communicating. More and more, they are communicating about data. 

One often hears data referred to as “the new oil” — a valuable resource capable of improving corporate decisions and making the entire organization more nimble and productive. Much modern business revolves around obtaining, analyzing and acting on data. Businesses create entire teams and hire elite strategic, technical and operational talent just to better leverage it. Each of these teams has its own culture, its own priorities, its own level of verbal and quantitative sophistication and, consequently, its own preferred way of communicating.  

That’s where the trouble starts. Miscommunication leads to friction. Friction leads to delay. Delay leads to expense.A Harris Poll research report found miscommunication costs U.S. businesses a staggering $1.2 trillion per year.  

To each his/her own 

To understand why, we need to take a closer look at the different groups who work with data in the enterprise. They tend to cluster into four categories. 

Executives communicate data needs and ideas in natural language text in emails, conversations and presentations. Their thinking is generally anchored to business concepts like market share and growth, departmental efficiency and sales forecasts. Their use of data often revolves around metrics, like the difference between Net Dollar Retention and Account-based Churn, or margin vs. gross margin. They think about data as a resource. They don’t generally worry about where or how it’s stored or how it gets refined, only what they need to do with it. 

Analysts are more fluent with data. Like executives, they mostly focus on data consumption and value. But they worry about details executives don’t: data that’s dirty or out of date. They look for new data sources to enrich existing analytics. They experience data not as a concept but as something that physically exists. Something that needs to be cleaned and rationalized for proper use. They communicate their data needs in formal or informal requirements documents. They define what data they need, where it comes from, what’s required in terms of cleanup and how they would like to use it. They care about documentation, structure and history. If a given supply chain efficiency metric is suddenly calculated differently or sourced from another system, analysts want a paper trail showing the history as well as the impact of that change. 

Power users are essentially tech-savvy analysts. They may not know how to optimize SQL, but they know they need to use the CUSTOMER_ID field to integrate order data with forecast data. They think about how to access data from the database, but are generally insulated from “mechanical” data problems like management, storage and the potential for runaway cloud costs. They often use visual design tools to describe how different datasets should be processed and integrated. 

Data engineers think about the “physics” of data: Where it’s stored, how it’s secured, who gets to see it, how clean (or not) it is, whether a given dataset includes Personally Identifiable Information (PII) and the costs associated with its management. They do the “heavy lifting” of providing access to potentially hundreds of data sources. Because they’re the ones who usually handle the physical movement of data, their “native language” is code — often SQL, sometimes Spark, Python, or other languages. Code gives them precision to control everything from how data is reformatted to where it is processed to minimize cloud costs.  

‘Building the tower’ in the real world 

Now imagine all these stakeholders trying to work together to understand, say, inventory carrying costs across a supply chain. 

The analyst writes a requirements document, explaining the new metric, how it’s calculated and what data is needed for the calculation. A power user on the analyst team then designs a visual workflow that logically maps the concepts expressed by the analyst (“end of month inventory levels from every warehouse”) to describe the extraction of the data from warehouse systems, and the integration and transformation steps needed to rationalize it. Since that data could come in many formats or have empty records and errors, these steps may be complex. The power user may rely on a variety of tools to deliver the clean, trusted data the analyst requires. The visual design might be executed in Google Slides, a PowerPoint deck, a diagram from a workflow design tool, or even a data preparation tool that provides a visual connection between data concepts and physical data access. 

A data engineer then takes the visual workflow and translates it into fast, scalable, cost-efficient code that retrieves and delivers the data in the visual spec. In most cases, the data engineer is entirely divorced from the business goal of minimizing inventory costs. Their concern is entirely about maximizing the use of the data platform. 

It should be clear at this point that each of these stakeholders is effectively speaking a different language about data. And, as in the Tower of Babel, every time information moves from one mode to another, it adds friction and loses fidelity.  

Errors propagate upstream.The engineer delivers data that doesn’t match what the power user had in mind. The power user delivers results to the analyst that omit key aspects of the metric in the requirements doc. The analyst, unable to use the data the others retrieved, restates the requirement with more specificity, and the cycle begins again. 

Can’t AI solve this problem? 

Anyone who has used Google Translate has experienced the exhilarating “superpower” of being able to “speak” dozens of languages. (It’s no coincidence that an early Web translation service adopted the Babel Fish moniker.) Recent research shows that more than half of the organizations applying generative AI to data management are achieving 31-50% productivity increases. Given that, can’t we just use AI to translate between the “data languages” of these different groups?  

Unfortunately, it’s not that simple. 

AI is good at translation. The problem is that “speaking” data is not just a translation problem. Yes, in some environments, using one of the many variations of Text2SQL, you can type “top 10 customers by Q1 spend” and get a viable response from auto-generated SQL. But enterprise data requests also need to account for factors like lineage (“where did this data come from and when was it refreshed?”). They need documentation, testing, versioning and governance. And they need to support multiple iterations as analysts, business users and data engineers work together to get the right data. AI can generate the right answers, but it can’t get everyone on the same page.  

Mitigating disconnects 

So, if AI isn’t the answer, what is? You need to apply different approaches to bridge the gap between groups. This doesn’t mean teaching each group the other’s language, though there is an element of that. It’s more about designing communications with an awareness of the needs of whoever is receiving the message — crystallizing best practices and making communications consistent and effective.  

For instance, when possible, analysts should develop visual prototypes using live data and actual data structures, rather than sketching approximations in PowerPoint. Pseudo-code can obscure edge cases that quickly become apparent when working with the actual database. More importantly, engineers work in code, and code requires specifics. The goal isn’t to make power users abandon their visual medium. It’s to include enough context that the visual metaphor can be cleanly translated to code. The less engineering needs to infer, the more accurate the results are likely to be. Most modern tools will let you do this with only a small representative “slice” of the data. 

Another helpful practice is to assign cross-functional teams to high-value data products — say, a report showing total monthly revenue for all active customers. Instead of playing a game of telephone relaying specs between groups, with a different “translation” of the task at each step, assign the task to a team that includes members of each group. This doesn’t eliminate the problem of multiple languages, but you do gain the efficiency from having a dedicated team working on the same problem together, over time. 

It’s a big undertaking, but organizations can get huge leverage by taking the time to develop a shared semantic layer across data assets. The semantic layer provides a map to what data lives where. It also provides meaning. With a shared semantic layer, an analyst looking to analyze customer profitability is far more likely to reuse the right source data and calculations for profitability as defined in the semantic layer, instead of hunting for customer data and hand-rolling a new profitability calculation. 

And yes, there is a place for improving data literacy across groups — training each other in each other’s language. This is not a small undertaking, but it can be transformative, especially for line-of-business users. Colgate Palmolive upskilled thousands of employees on data and analytics, specifically focusing on data fundamentals. When embarking on a project like this, it’s important not to try to do too much. You can accomplish a lot with a two-hour “SQL for analysts” clinic. Familiarity, not fluency, is the goal.  

Forging data clarity: Impact beyond the babble

We may never eliminate the Tower of Babel problem entirely. After all, that is the point of the story. Analysts think in metrics, power users in visuals and engineers in code. But with thoughtfully constructed practices and well-designed systems, we can greatly accelerate data delivery and drive real business impact. 

Lance Walter has over two decades of experience in enterprise data management and analytics. He has held a wide variety of roles in product management, product marketing and marketing leadership at organizations including Oracle, Business Objects and Neo4j. As CMO at Prophecy.io, Lance gets to work with large organizations to understand how they’re modernizing their infrastructure to get data to users more quickly. Outside of work, Lance is a Dad and musician and you might find him learning Taylor Swift covers for his three daughters.

This article is published as part of the Foundry Expert Contributor Network.
Want to join?