Simperium offers a streaming API that is accessible over a WebSocket or SockJS. This document describes the messages a client can send and receive as well as how to implement syncing for a client.
key and v (version)7u37290aaddfkkcc.c or command message.This assumes the client is starting with an empty index.
The client connects via WebSocket to wss://api.simperium.com/sock/1/APP_ID/websocket where APP_ID is replaced with the Simperium application id. Commands are sent over the WebSocket and can be prefixed with an integer which allows commands to be namespaced to a specific channel of communication to allow multiple buckets to be synced over the same socket connection.
After connecting a client can send various commands to retrieve a bucket's objects and changes to facilitate syncing a local representation of a bucket with the one that exists on the server.
The available commands are:
When a client is ready to connet to a user's bucket it sends the init command. The init command contains a JSON payload with the following key value pairs:
Optionally, the init request can contain a command to be executed upon initialization:
An example init command with a channel prefix 0:
0:init:{"api":1,"client_id":"android-1.0","token":"abc123","app_id":"abusers-headset","name":"notes"}
The Simperium server will respond with an auth command which will also contain either the authorized user's email address, or if authorization failed, a JSON error object.
Example failed auth:
0:auth:{"msg":"Token invalid", "code":401}
Possible error codes:
Example successful auth:
0:auth:ender@example.com
After authorizing the client can now issue commands for the initialized bucket.
The index command -- i -- allows the client to receive all of the keys and corresponding version numbers for the objects stored in the bucket. The index command takes four parameters sperated by colons ::
1 then the index will also return the data for each entity; otherwise omit entirelyWhen connecting for the first time with an empty index, the client should issue an i command with a sane limit. The following example requests the bucket's index with a page size of 500 items.
0:i::::500
The server will respond with the index results in a JSON payload prefixed with i:. Example response:
0:i:{"current": "5119dafb37a401031d47c0f7", "index": [{"id": "one", "v": 2}, ... ], "mark": "5119450b37a401031d3bfdb9"}
The JSON payload contains these keys:
cv/ccid. Clients should store this for future requests to indicate to the server which version of the index they have.If we also want to fill out the bucket data to mirror the server and do so on this initial load we can merge the requests into one by passing the data parameter.
0:i:1:::500
The server will now respond with a similar payload as before, but this time each item in the index arreay will also contain an additional key:
d : the value of the entity's data
0:i:{"current": "5119dafb37a401031d47c0f7", "index": [{"id": "one", "v": 2, "d": {"your": "data"}}, ... ], "mark": "5119450b37a401031d3bfdb9"}
Clients can request an entire entity at any version stored on the server. The e command takes one parameter which is an entity's key and version seperated by a dot .:
0:e:keyname.1
The server will respond with the same name and version followed by a new line \n and the response for the entity. If the entity represented by that key and version does not exist, the response is a single question mark ?:
0:e:keyname.1
?
Otherwise the response will be a JSON payload. The entity's data is stored in the payload's data key:
0:e:keyname.1
{"data": {"1": 2, "0": 1, "2": "Three"}}
To keep an existing index up to date, a client can request all changes since a specific change version or cv. The cv command takes one parameter: the change version to begin looking for bucket changes.
For example, the client may have stored an index locally that was received with change version abc123. The next time the client connects it will want to request all changes that may have happenned to the bucket while the client was disconnected. The client will send:
0:cv:abc123
The server will look for all changes on the bucket since cv of abc123. If the cv does not exist for that bucket, the server will respond with:
0:cv:?
Otherwise it will respond with a change c command that contains a JSON payload describing all of the changes that need to be applied to an index at change version abc123 to match the current change version:
0:c:[{"clientid": "sjs-2012121301-9af05b4e9a95132f614c", "id": "newobject", "o": "M", "v": {"new": {"o": "+", "v": "object"}}, "ev": 1, "cv": "511aa58737a401031d57db90", "ccids": ["3a5cbd2f0a71fca4933fff5a54d22b60"]}]
If the change version is up to date the server will respond with an empty c message:
0:c:[]
To communicate changes to the bucket index clients and servers should send and respond to change messages: c. A change message contains a JSON payload which is an array of hashes that describe each change version to be applied to the index in order to bring it to a current state.
A change has these keys:
change.ev minus change.sv will usually be 1 but is not always the case.
Possible operations o:
For modify operations the v key will contain an object diff compatible with jsondiff.
To keep a connection alive the client should send a heartbeat message while the connection is idle. This message should not be prefixed by a channel id since the heartbeat will maintain the connection for all channels.
The message takes one parameter, an integer that is incremented by the server and then sent back. Client sends:
h:0
Server responds with:
h:1
The client's next heartbeat message should increment the integer it received from the server. A heartbeat should be sent after 20 seconds of idle time and should expect an immediate response.
A client needs to perform a specific set of operations to successfully keep its index synced with the remote version. We're assuming messages are sent over a channel with the prefix 0 and that the client is starting with an empty index.
To authorize access to a bucket the client will first need to obtain a user's access token using the auth api. The client can then send an init command over the connection:
0:init:{"name":"mybucket" ... }
The client should then wait for an auth response:
0:auth:user@example.com
After a successful response the client should perform its first sync.
Upon first connection to a bucket (e.g. the client has no data in its local index) a client will need to request the current index from the server and sync each of the objects. The client can request the index using the i "index" command providing a limit for the page size.
Sending this message will request the bucket's latest index 100 items at a time:
0:i::::100
The server will respond with an i message containing the JSON payload that represents a page of entity keys and versions:
0:i:{"current": "5119dafb37a401031d47c0f7", "index": [{"id": "one", "v": 2}, ... ], "mark": "5119450b37a401031d3bfdb9"}
If there are more objects than fit in this page the server will send a cursor under the key mark that the client can use to request the next page. This command will request the next page of indexes from the server
0:i::5119450b37a401031d3bfdb9::100
The client will know when it has received the entire index when it receives an i message without a mark.
After receiving a set of index data a client can begin requesting entity data from the server by requesting each id.v in the index key from the server. The client will want to store which cv they are syncing (the value under current in an i message).
For each object in the index array of a i message, the client can request the entity's data using the e "entity". For example, this message asks the server to send version 2 of the entity stored at the key qwerty:
0:e:qwerty.2
If the server has this entity and version it will respond with:
0:e:qwerty.2
{"data":{"message":"hello world"}}
The entity's data is stored in the data key of the JSON payload. The client should store both the data and the version locally so it can request changes for the entity in the future.
After storing all of the objects from an index request the client will have a synced copy of the bucket and can now start sending and apply changes.
After sending an init message, if a client already has local index data stored for a bucket it should send a cv Change Version message instead of an i message. Downloading an entire index of data would be wasteful.
The client should have stored the current change version for the index so it can ask the server for all changes since that version in order to catch up.
0:cv:5119dafb37a401031d47c0f7
If the server knows about this change version for the connected bucket it will respond with the changes necessary to transform the local index to match the remote one:
0:c:[{"clientid": "sjs-2012121301-9af05b4e9a95132f614c", "id": "newobject", "o": "M", "v": {"new": {"o": "+", "v": "object"}}, "ev": 1, "cv": "511aa58737a401031d57db90", "ccids": ["3a5cbd2f0a71fca4933fff5a54d22b60"]}]
If the server doesn't have the requested change version it will send this cv message:
0:cv:?
At which point the client will need to reload the index. To save space the server starts aggregating older change versions so a single change version is not permanently stored forever.
A client will receive remote changes from the server either by explicitly asking for them (using the cv) or simply by being connected when a server receives a c change command. A remote change message will contain a JSON payload that is an array of changes and information about those changes (corresponding ccids and an index cv for the changes). An example incoming c message representing a single change:
0:c:[{"clientid": "sjs-2012121301-9af05b4e9a95132f614c", "id": "newobject", "o": "M", "v": {"new": {"o": "+", "v": "object"}}, "ev": 1, "cv": "511aa58737a401031d57db90", "ccids": ["3a5cbd2f0a71fca4933fff5a54d22b60"]}]
To apply these changes a client will want to loop through each change and perform the operation described in the change object:
change.id as the key
change.sv
change.o is - remove the entity from the local store only if the local entity has no pending changes.change.o is M apply change.v using jsondiffchange.evchange.cv for the indexApplying the change: If the client cannot apply the change, it needs to re-load the object
Change for missing entity: If a change is received referencing an entity that doesn't exist locally (and the change isn't creating the entity), client needs to re-load the object
In some cases the client needs to reload an object, when it does so it should do the following:
e "entity" commandWhen the client is ready to update an object's representation in the remote datastore it needs to generate the diff and write a c command to the server.
After sending the change, the client should flag the change as sent but not yet "acknowledged" until it receives a c response from the server containing the ccid that was sent. If the change was successful the client can consider the local representation as up to date.
In some cases the server will respond with a error response if it could not apply the change. An example error response:
c:[{"clientid":"offending-client","id":"object-key","error":"401","ccids":["abcdef123456"]}]
The server may ask the client at any time for some data
Server may send a command requesting the index for a particular bucket, this command will be sent on a particular channel that is already open
Example request from server on channel 0:
0:index
Response from client should follow:
0:index:{ current: <cv>, index: [ {id: <eid>, v: <version>}, ... ], pending: [ { id: <eid>, sv: <version>, ccid: <ccid> }, ... ], extra: { ? } }
Server may ask the client to start sending log messages from the client, this request will be sent without a channel prefix, responses should also be sent without a prefix.
We currently define 3 states of logging:
**0** : Client should stop sending remote logging messages
**1** : Client should send logging messages
**2** : Client should send "verbose" logging messages
The response from the client should be a json object containing at least a "log" field. The value from this field will be logged.
Example request from server (requesting normal logging):
log:1
Example response:
log:{"log":"<some log message>"}
Potential error responses:
Handling Change Errors:
400 : If it was an invalid id, changing the id could make the call succeed. If it was a schema violation, then correction will depend on the schema. If client cannot tell, then do not re-send since unless something changes, 400 will be sent every time.
401 : Grab a new authentication token (or possible you just don't have access to that document).
404 : Client is referencing an object that does not exist on server. By default client should send the change with full object (this will re-create the object)
405 : Bad version, client referencing a wrong or missing sv. This is a potential data loss scenario: server may not have enough history to resolve conflicts. Client has two options:
e command) then overwrite the local changessv409 : Duplicate change, client should stop sending this change and retrieve changes since its last syncd cv to process the changes list from the server. In normal case the remote changes list will contain the change that caused the duplicate error. If it doesn't, client will need to re-load this object. In either case, client will know that the server version of the object already incorporates the delta that this change contains.
412 : Empty change, nothing was changed on the server, client can ignore (and stop sending change).
413 : Nothing to do except reduce the size of the object
440 : Server could not apply diff, resend the change with additional parameter d that contains the whole JSON data object. Current known scenarios where this could happen: