Data Platform Evolution: From Paper Ledgers to the Lakehouse

Data has always been essential to business. Collecting it was never the real problem; storing it, organising it, and making sense of it is where the struggle begins. This article tells the story of that struggle: how each generation of data technology solved the limitations of the one before it, and how, along the way, those solutions built the discipline we now call data engineering.

The Physical Era

Before computers, data lived on paper. Sales records, inventory logs, and financial ledgers were maintained by hand, stored in filing rooms, and retrieved by clerks. The system worked at small scale, but as organisations grew, so did the chaos. Answering even a simple question meant hours of manual cross-referencing, and the risk of loss was constant: a single fire or flood could erase years of records. Data existed, but actually using it was close to impossible.

Databases and the Birth of SQL

The turning point came in 1970, when IBM mathematician Edgar F. Codd published a paper titled 'A Relational Model of Data for Large Shared Data Banks'. Instead of forcing programmers to think about how data was physically stored, Codd proposed organising it into tables of rows and columns, linked by shared values, and querying it in plain, logical terms. This was the birth of the relational database.

To communicate with these databases, two IBM researchers built a prototype language called SEQUEL (Structured English Query Language), later shortened to SQL after they discovered the name SEQUEL was already trademarked. Its central idea remains elegant today: you describe what you want, and the database engine works out how to fetch it efficiently. More than fifty years later, SQL is still the shared language of data, used daily by every data engineer, analyst, and scientist, and it remains one of the most in-demand technical skills in the job market.

Commercial adoption moved fast. Oracle shipped the first commercial SQL database in 1979, and companies like Informix and Sybase built entire businesses on Codd's ideas before IBM brought its own product to market in 1981. Records could now be stored, updated, and retrieved in milliseconds, and the paper filing room was finished.

But these systems were built for operations, not analysis. Running a heavy report on the same live database that processes customer orders would slow both to a crawl. These two workloads now have names: OLTP (online transaction processing) covers the many small, fast reads and writes that keep a business running, while OLAP (online analytical processing) covers the big, heavy queries that scan millions of rows to answer questions. Much of the history that follows is, at heart, the story of separating the two.

The Data Warehouse

The solution to the operations-versus-analysis problem was separation: give the organisation a dedicated platform for analysis, away from the live systems customers depend on. This became the data warehouse.

In 1990, Bill Inmon, often called the father of data warehousing, formalised the term. He defined a warehouse as a 'subject-oriented, integrated, time-variant, non-volatile collection of data' built to support management decisions. Inmon's approach was top-down: build one central, tightly modelled warehouse first, then let departments branch off it. Slower to deliver, but highly consistent as the organisation grows.

A few years later, Ralph Kimball popularised the alternative: a bottom-up approach built from smaller, business-focused 'data marts', delivered department by department and then linked together. Kimball's designs brought value much faster, but consistency across the business became harder to maintain as the data grew. To this day, the Inmon-versus-Kimball question remains a genuine design decision that every organisation building a warehouse must face.

To get data into the warehouse, it was extracted from source systems, transformed into a clean, consistent structure, and then loaded, a process known as ETL (Extract, Transform, Load). For the first time, different departments could pull from the same data and trust they were looking at the same numbers.

This architecture also forced a new discipline into existence. Data governance emerged to answer the questions that centralised data made unavoidable: Who can see this data? Who owns it? Which copy is the correct one?

The warehouse's great limitation was rigidity. It could only hold structured data: values that fit neatly into rows and columns. Photos, videos, audio, free text, and raw log files had no place in it, and the internet was about to produce those in unimaginable quantities.

The Big Data Era

Through the 2000s, the internet economy generated data of a volume and variety no warehouse could affordably handle: clickstreams, application logs, images, social posts, sensor readings. The breakthrough came from the companies drowning first. Google published landmark papers in 2003 and 2004 describing how it stored and processed data across thousands of cheap, ordinary machines, and in 2006 the open-source Hadoop project brought those ideas to everyone. Suddenly, storing everything was economically possible. The question became where to put it all.

The Data Lake

The answer was the data lake: one vast repository holding data of every shape in its raw, native format. The term was coined in 2010 by James Dixon, who likened a data mart to bottled water, processed and packaged for easy consumption, while the data lake was 'a large body of water in a more natural state' that many different users could dive into and examine as they pleased. As the internet era's explosion of new data types arrived, the lake became the first architecture truly capable of holding it all.

That flexibility created its own failure mode: a lack of discipline. Without enforced structure and governance, many data lakes degenerated into 'data swamps', vast collections of files that nobody could find, understand, or trust. Metadata was inconsistent or missing, performance was poor, and the governance frameworks so carefully built for warehouses had to be rebuilt from scratch in this new environment.

Alongside the lake, new database technologies emerged for data that never suited rigid tables. NoSQL databases such as MongoDB were designed for flexibility and scale with semi-structured data, while graph databases such as Neo4j served highly connected data like social networks and recommendation engines.

The Cloud Warehouse

While lakes were absorbing the world's raw data, the warehouse was being reinvented in the cloud. Amazon Redshift, Google BigQuery, and Snowflake arrived in the early 2010s and changed the economics of analytics. The crucial innovation was separating storage from compute: data sits cheaply in cloud storage, and querying power is rented only when needed, scaling up for a heavy month-end run and back down afterwards. A capable warehouse stopped being a multi-year enterprise project and became something a mid-sized company could stand up in weeks.

Even so, organisations now commonly ran two systems in parallel: a lake for raw and unstructured data, and a warehouse for governed analytics. Keeping the two in sync meant duplicated data, duplicated cost, and duplicated engineering effort.

The Data Lakehouse

The lakehouse emerged as the answer: a single platform combining the warehouse's structure, reliability, and fast querying with the lake's cheap, flexible, large-scale storage. The term had been floating around since about 2017, but it was Databricks that put it on the map in 2020, and the major cloud platforms quickly followed.

The practical appeal is consolidation. One copy of the data, one set of costs, one platform to secure and govern, and no more engineering effort spent keeping a lake and a warehouse reconciled. Data engineers, analysts, and data scientists can finally work from the same data rather than shuttling copies between systems, and modern engineering practices, such as automated testing, pipeline monitoring, and data quality checks, have become standard on top of it.

How It All Fits Together Today

As neat as this storyline is, these architectures did not simply replace one another. In most real organisations they coexist, each doing what it does best. A typical modern platform looks like this: source systems feed raw data into a lake landing zone; engineering pipelines ingest, clean, and transform it into governed warehouse tables; the warehouse serves BI teams and analysts with a finalised, trusted version of the data; and data scientists work closer to the raw data, often through a lakehouse, for exploration and model training.

Increasingly, data also arrives as a continuous stream rather than in daily batches, a shift significant enough that we cover it separately in Batch vs Streaming (coming soon!). The result is not a single product but a network of technologies, each connected, maintained, and monitored by strong engineering on a base of clear governance.

Conclusion

From paper ledgers to the lakehouse, every generation of data technology solved the limitation of the one before, and each introduced a new discipline along the way: SQL, ETL, data governance, schema design, DataOps. The technologies will keep changing; the pattern will not. The organisations best placed to capitalise on data today are not the ones chasing the newest tool, but the ones that have invested in strong, well-engineered data foundations.

This article is part of our data engineering knowledge base. New to the field? Start with What is Data Engineering? Ready to go deeper? Continue with ELT vs ETL (coming soon!).