API reference / @evolu/common / Evolu/Protocol
Evolu/Protocol
Evolu Protocol
Evolu Protocol is a local-first, end-to-end encrypted binary synchronization protocol optimized for minimal size and maximum speed. It enables data sync between a client and a relay. In the future, direct peer-to-peer (P2P) sync between clients will be possible without a relay.
Relays don't need to sync with each other—clients using those relays will sync them eventually. If a relay is offline (e.g., for maintenance), it will sync automatically later via client sync logic. For relay backup using SQLite, see https://sqlite.org/rsync.html (uses a similar algorithm to Evolu RBSR).
Evolu Protocol is designed for SQLite but can be extended to any database. It implements Range-Based Set Reconciliation. To learn how RBSR works, check Negentropy. Evolu Protocol is similar to Negentropy but uses different encoding and also provides data transfer, ownership, real-time broadcasting, request-response semantics, and error handling.
Message structure
| Field | Notes |
|---|---|
| Header | |
| - protocolVersion | |
| - OwnerId | Owner |
| - messageType | MessageType |
| Request (messageType=0) | |
| - hasWriteKey | 0 = no, 1 = yes |
| - OwnerWriteKey | If hasWriteKey = 1 |
| - subscriptionFlag | SubscriptionFlags |
| Response (messageType=1) | |
| - ProtocolErrorCode | |
| Broadcast (messageType=2) | |
| - (no additional fields) | |
| Messages | |
| - NonNegativeInt | A number of messages. |
| - EncryptedCrdtMessage | |
| Ranges | |
| - NonNegativeInt | Number of ranges. |
| - Range |
WriteKey validation
The initiator sends a hasWriteKey flag and optionally a WriteKey. The WriteKey is required when sending messages as a secure token proving the initiator can write changes. It's ok to not send a WriteKey if the initiator is only syncing (read-only) and not sending messages. The non-initiator validates the WriteKey immediately after parsing the initiator header, before processing any messages or ranges.
Synchronization
- Messages: Sends EncryptedCrdtMessages in either direction.
- Ranges: Determines messages to sync. Usage varies by transport—e.g., sent only on WebSocket connection open or with every fetch request.
Synchronization involves an initiator and a non-initiator. The initiator
is typically a client, and the non-initiator is typically a relay. Each
side processes the received message and responds with a new ProtocolMessage
if further sync is needed or possible, continuing until both sides are
synchronized.
The non-initiator always responds to provide sync completion feedback, even with empty messages containing only the header and no error. This allows the initiator to detect when synchronization is complete.
Both Messages and Ranges are optional, allowing each side to send, sync, or only subscribe data as needed.
When the initiator sends data, the OwnerWriteKey is required as a
secure token proving the initiator can write changes. The non-initiator
responds without a OwnerWriteKey, since the initiator’s request
already signals it wants data. If the non-initiator detects an issue, it
sends an error code via the Error field in the header back to the
initiator. In relay-to-relay or P2P sync, both sides may require the
OwnerWriteKey depending on who is the initiator.
Protocol errors
The protocol uses error codes in the header to signal issues:
- ProtocolWriteKeyError: The provided WriteKey is invalid or missing.
- ProtocolWriteError: A serious relay-side write failure occurred.
- ProtocolSyncError: A serious relay-side synchronization failure occurred.
- ProtocolQuotaExceededError: Storage or billing quota exceeded.
- ProtocolUnsupportedVersionError: Protocol version mismatch.
- ProtocolInvalidDataError: The message is malformed or corrupted.
All protocol errors except ProtocolInvalidDataError include the OwnerId
to allow clients to associate errors with the correct owner.
Message size limit
The protocol enforces a strict maximum size for all messages, defined by ProtocolMessageMaxSize. This ensures every ProtocolMessage is less than or equal to this limit, enabling stateless transports, simplified relay implementation, and predictable memory usage. When all messages don't fit within the limit, the protocol automatically continues synchronization in subsequent rounds using range-based reconciliation.
Database mutations are limited to 640KB, which is smaller than the protocol message limit to ensure efficient sync with defaultProtocolMessageRangesMaxSize.
Why Binary?
The protocol avoids JSON because:
- Encrypted data doesn’t compress well, unlike plain JSON.
- Message size must be controlled during creation.
- Sequential byte reading is faster than parsing and avoids conversions.
It uses structure-aware encoding, significantly outperforming generic binary serialization formats with the following optimizations:
- NonNegativeInt: Up to 33% smaller than MessagePack.
- DateIso: Up to 75% smaller.
- Timestamp Encoding: Delta encoding for milliseconds and run-length encoding (RLE) for counters and NodeIds.
- Small Integers (0 to 19): Reduces size by 1 byte per integer.
To avoid reinventing serialization where it’s unnecessary—like for JSON and certain numbers—the Evolu Protocol relies on MessagePack.
Versioning
Evolu Protocol uses explicit versioning to ensure compatibility between
clients and relays (or peers). Each protocol message begins with a version
number and an ownerId in its header.
How version negotiation works:
-
The initiator (usually a client) sends a
ProtocolMessagethat includes its protocol version and theownerId. -
The non-initiator (usually a relay or peer) checks the version.
- If the versions match, synchronization proceeds as normal.
- If the versions do not match, the non-initiator responds with a message
containing its own protocol version and the same
ownerId.
-
The initiator can then detect the version mismatch for that specific owner and handle it appropriately (e.g., prompt for an update or halt sync).
Version negotiation is per-owner, allowing Evolu Protocol to evolve safely over time and provide clear feedback about version mismatches.
Credible exit
The protocol specification is intentionally non-configurable to ensure universal compatibility. This design allows applications (users) to switch between any compliant relay without negotiation or compatibility checks beyond version matching. Relays are generic infrastructure that any application can use interchangeably making exit from any single provider technically feasible and economically viable.
Interfaces
| Interface | Description |
|---|---|
| ApplyProtocolMessageAsClientOptions | - |
| ApplyProtocolMessageAsRelayOptions | - |
| ApplyProtocolMessageAsRelayResult | Result type for applyProtocolMessageAsRelay. |
| ProtocolErrorBase | Base interface for all protocol errors. |
| ProtocolInvalidDataError | Error for invalid or corrupted protocol message data. |
| ProtocolMessageBuffer | Mutable builder for constructing ProtocolMessage respecting size limits. |
| ProtocolQuotaExceededError | Error when storage or billing quota is exceeded. Clients should prompt the user to upgrade their plan or expand capacity. |
| ProtocolSyncError | Error indicating a serious relay-side synchronization failure. Clients should log this error and show a generic sync error to the user. |
| ProtocolTimestampMismatchError | Error when embedded timestamp doesn't match expected timestamp in EncryptedDbChange. Indicates potential tampering or corruption of CRDT messages. |
| ProtocolUnsupportedVersionError | Represents a version mismatch in the Evolu Protocol. Occurs when the initiator and non-initiator are using incompatible protocol versions. |
| ProtocolWriteError | Error indicating a serious relay-side write failure. Clients should log this error and show a generic sync error to the user. |
| ProtocolWriteKeyError | Error when a OwnerWriteKey is invalid, missing, or fails validation. |
| TimestampsBuffer | - |
| TimestampsRangeWithTimestampsBuffer | - |
Type Aliases
| Type Alias | Description |
|---|---|
| ApplyProtocolMessageAsClientResult | Result type for applyProtocolMessageAsClient that distinguishes between responses to client requests and broadcast messages. |
| MessageType | - |
| ProtocolError | - |
| ProtocolMessage | Evolu Protocol Message. |
| ProtocolMessageMaxSize | - |
| ProtocolMessageRangesMaxSize | - |
| SubscriptionFlag | - |
Variables
| Variable | Description |
|---|---|
| decodeLength | - |
| defaultProtocolMessageMaxSize | Default ProtocolMessageMaxSize (1MB). |
| defaultProtocolMessageRangesMaxSize | Default ProtocolMessageRangesMaxSize (30KB). |
| MessageType | - |
| ProtocolErrorCode | - |
| ProtocolMessageMaxSize | Protocol message maximum size. |
| ProtocolMessageRangesMaxSize | Protocol message ranges maximum size. |
| ProtocolValueType | - |
| protocolVersion | Evolu Protocol version. |
| SubscriptionFlags | - |
Functions
| Function | Description |
|---|---|
| applyProtocolMessageAsClient | - |
| applyProtocolMessageAsRelay | - |
| createProtocolMessageBuffer | - |
| createProtocolMessageForSync | Creates a ProtocolMessage for sync. |
| createProtocolMessageForUnsubscribe | - |
| createProtocolMessageFromCrdtMessages | Creates a ProtocolMessage from CRDT messages. |
| createTimestampsBuffer | - |
| decodeNodeId | - |
| decodeNonNegativeInt | Decodes a non-negative integer from a variable-length integer format. |
| decodeNumber | - |
| decodeProtocolMessageToJson | Decodes a ProtocolMessage into a readable JSON object for debugging. |
| decodeSqliteValue | - |
| decodeString | - |
| decryptAndDecodeDbChange | Decrypts and decodes an EncryptedCrdtMessage using the provided owner's encryption key. Verifies that the embedded timestamp matches the expected timestamp to ensure message integrity. |
| encodeAndEncryptDbChange | Encodes and encrypts a DbChange using the provided owner's encryption key. Returns an encrypted binary representation as EncryptedDbChange. |
| encodeLength | - |
| encodeNodeId | - |
| encodeNonNegativeInt | Encodes a non-negative integer into a variable-length integer format. It's more efficient than encoding via encodeNumber. |
| encodeNumber | Evolu uses MessagePack to handle all number variants except for NonNegativeInt. For NonNegativeInt, Evolu provides more efficient encoding. |
| encodeSqliteValue | - |
| encodeString | - |