Introduction
In this post, we'll dive into how replication logs are implemented. This is a shorter version of what's in the book Designing Data-Intensive Applications by Martin Kleppmann.
What is a Replication Log?
A replication log is a list of all changes made to a database. It helps copy data from the main database to other databases. This log makes sure all other databases have the latest data, showing the same changes as the main one.
Statement-Based Replication
In statement-based replication, every write operation (such as INSERT, UPDATE, or DELETE commands) executed by the leader is logged and then sent to the followers. Followers execute these SQL statements to mirror the leader's state.
Challenges
Nondeterministic Functions: Functions like NOW() or RAND() can produce different results on each replica, leading to inconsistencies.
Execution Order: The order of actions is very important. Any change can cause different states in the databases.
Side Effects: Features like triggers or stored procedures could cause different results on each replica unless they're completely predictable.
Handling Nondeterminism
One approach to address these issues is for the leader to pre-calculate the results of nondeterministic functions, ensuring followers execute the same values.
Write-Ahead Log (WAL) Shipping
WAL shipping takes a lower-level approach by replicating the exact sequence of bytes written to the database's log. This method mirrors the database's internal state directly onto followers.
What Does "Replicating the Exact Sequence of Bytes" Mean?
When a change happens in the database, it's first written down in a special file called a log. This log includes the specifics of the change (such as a new entry in a table), but it notes the change in bytes. Bytes are the basic building blocks of data storage in computers. They detail precisely how the change affects the database's saved data on the disk.
In WAL shipping, this byte-by-byte record from the leader's log is sent directly to the followers. This means followers apply the same changes to their storage, ensuring they mirror the leader exactly.
What Is the "Database's Internal State"?
The database's internal state refers to all the information that defines the current condition of the database. This includes all the data in tables, indexes and other structures that the database uses to organize and retrieve information efficiently.
When we talk about replicating the database's internal state, we're talking about making sure that the followers have an exact copy of all this information, so they match the leader precisely.
Challenges with WAL Shipping
Tight Coupling
This method depends a lot on the details of how the database organizes and stores data. Since it's copying the exact way data is stored (those sequences of bytes), both the leader and followers must use the same storage engine, the part of the database that manages how data is stored and taken from the disk. This close connection between the replication method and the storage engine is what we call "tight coupling."
This coupling can create difficulties when you want to upgrade your database software. If the new version changes how data is stored or organized, the WAL logs may not be compatible between the leader and upgraded followers.
Operational Complexity
Upgrading without downtime is challenging with WAL shipping. All nodes (leader and followers) must understand the log format to stay synchronized, so different database software versions on nodes must support the same log format. This adds complexity to the upgrade process, as compatibility across all nodes must be ensured.
Logical (Row-Based) Log Replication
Logical log replication decouples the replication log from the storage engine's internal format. It records changes at the row level, detailing inserted, deleted, or updated rows, and then sends these records to followers.
Advantages
Flexibility: This approach lets different database versions or storage engines work between the leader and followers, making upgrades and migrations easier.
Extensibility: The logical log format is simpler for outside systems to understand, which helps with things like capturing data changes for data storage or indexing.
Trigger-Based Replication
Trigger-based replication provides the highest level of customization by allowing application-level logic to determine what gets replicated. Database triggers log changes to a separate table, which an external process can read and replicate to other systems.
It adds a lot of complexity but can come in handy because of its flexibility.
Challenges
Performance Overhead: This method can introduce significant performance penalties and complexity, as it involves additional processing for each change.
Risk of Bugs: Custom replication logic is more prone to errors and inconsistencies compared to built-in database replication mechanisms.
