Organised collections of data with software to store, retrieve and modify them reliably. Almost every application of any size is a user interface over a database, and the design decisions that made them dependable are unusually clear.
Storing data in ordinary files is possible and fails in predictable ways: two programs writing at once corrupt each other, a crash midway leaves the data inconsistent, finding a record requires reading everything, and every program must agree on the file format.
A database management system exists to solve those problems.
It enforces structure, so data has a defined shape that programs can rely on.
It provides indexes, so records can be found without scanning everything.
It handles concurrency, so many users can read and write simultaneously without interfering.
It guarantees recovery, so a crash leaves the data in a consistent state.
And it separates the logical view of the data from how it is physically stored, so applications need not change when storage does.
Before 1970, databases were navigational: data was linked by pointers and a program retrieved records by following those links, which meant the code depended on the physical arrangement. Changing the storage broke the programs.
Edgar Codd proposed the relational model in 1970. Data is held in tables of rows and columns, relationships are expressed by matching values rather than by pointers, and queries state what is wanted rather than how to fetch it.
The consequence is independence. A query asks for the rows meeting a condition, and the system decides how to find them. Storage, indexes and even table organisation can change without altering the query.
SQL became the standard language for expressing these queries and remains so more than fifty years later, which is an unusual survival in software.
Normalisation is the associated design discipline: organising tables so each fact is stored once, which prevents the contradictions that arise when the same information is recorded in several places and only some copies are updated.
A transaction groups operations so they take effect as a unit.
The standard guarantees are summarised as ACID. Atomicity means all operations in a transaction occur or none do. Consistency means the database moves between valid states. Isolation means concurrent transactions do not see each other's partial work. Durability means a committed transaction survives a crash.
The classic illustration is a transfer between accounts, which subtracts from one and adds to another. If the system fails between the two, money has vanished. Atomicity prevents that outcome existing.
These guarantees are the reason financial and administrative systems are built on databases rather than files, and providing them under concurrency and hardware failure is most of what a database system does.
An index is a separate structure allowing rows to be located without examining every row, most commonly a B-tree, which keeps data sorted and allows lookup in a number of steps proportional to the logarithm of the table size.
The trade is that each index speeds reads and slows writes, since every insertion must update it, so indexing is a deliberate choice rather than a default.
The query planner decides how to execute a query, choosing between available indexes and orderings using statistics about the data. Two queries returning identical results can differ in execution time by orders of magnitude depending on that choice, which is why database performance work concentrates on the planner's decisions.
From the late 2000s, systems collectively labelled NoSQL relaxed parts of the relational model for particular workloads.
Key-value stores map keys to values with minimal structure and extreme speed. Document stores hold nested records without a fixed schema. Column stores organise by column rather than row, which suits analytics scanning few columns across many rows. Graph databases represent relationships directly, which suits queries traversing connections.
The motivation was scale across many machines. Distributing data raises a genuine constraint, the CAP theorem, which states that a distributed system cannot simultaneously guarantee consistency and availability when the network partitions. Systems must choose which to sacrifice during a partition.
Many NoSQL systems chose availability, accepting eventual consistency in which replicas converge over time and may briefly disagree.
The pattern since has been convergence. Relational systems added horizontal scaling and support for document data, while several NoSQL systems added transactions and query languages resembling SQL, and the sharpness of the distinction has faded.
Databases are where an organisation's actual state resides, and the guarantees they provide are what allow software to be trusted with money, records and identity.
The relational model is also among the clearest cases of theory producing practice. It began as a mathematical formulation with no implementation, was resisted as impractical, and became the foundation of an industry once its independence from physical storage proved to be worth the cost.
