4.1 Simplicity

The paged [JSON] data exchange standard is incredibly simple to implement, but conceptually requires some explanation.

4.2 Core concept

Consider an ordered list as shown in Fig. 2 Illustration of the deterministic list, with the following invariants:

• A continuous list of records that MUST be sorted deterministically and chronological (in the order they were updated). Either (i) ordered first by modified timestamp, and second by ID or (ii) ordered by an incrementing counter where records are assigned a new unique value on each update.

• Every record MUST only be represented once in this list at a given moment, with its position in the list depending on when it was last updated. Records can freely move position in the list as they are updated.

• This deterministic ordering based on timestamp allows for pages of arbitrary size to be sent without concern for race conditions; if a record is updated during the transfer of a page it MUST appear on a subsiquent page (i.e. simply reappear further down the list).

• Pages are defined using a "next page URL", which MUST contain enough information to identify a position in the list (e.g. by a "timestamp" and "ID" combination). It MUST NOT reference a specific record, as that record can change position in the list.

• If a "next page URL" is not used to access the list, the first page MUST be returned.

• If the consumer reaches the end of the list they consider themselves up-to-date at that moment, and can frequently revisit the end of the list in order to retrieve further updates.

  

4.3 Example Implementations

Two example implementations are described below, both adhere to the specified invariants. One of these two examples SHOULD be used as a basis for any implementation, unless the invariants are well understood and the implementation can be thoroughly tested.

   var query;
   if (queryParams.from) {
       query = Session.query().filter((Session.modified == queryParams.afterTimestamp && Session.id > queryParams.afterId) || (Session.modified > queryParams.afterTimestamp));
   } else {
       query = Session.query()
   }
   return query.sort([Session.modified, Session.id])

4.4 Related entities

This specification can be implemented for each relevant entity type within System 1 (e.g. club, courses, sessions).

Data from related entities can either be:

1. Embedded inside the parent entity. This requires the parent "updated" field to reflect the maximum value of all children - achievable via database triggers or similar.

2. Provided by separate endpoints implementing this specification, the combination of which can be reassembled by System 2.

4.5 Response grammar

It should be noted that although the <data> element is open to conformance with [JSON-LD], the paging has been deemed too trivial to require the application of any specific serialization.

   <response> => {
         items: [<item>,<item>,<item>,...],
         next: "/getSessions?afterTimestamp=<date>&afterId=<id>"
      }

   <item> => {
         state: 'updated' | 'deleted', (4)
         kind: "session", (2)
         id: "{21EC2020-3AEA-4069-A2DD-08002B303123}", (6)
         modified: Date(a),
         data: <data> (4)
      }

4.6 Example

A full example REST response from polling:

   /getSessions?afterTimestamp=1453931101&afterId={c15814e5-8931-470c-8a16-ef45afedaece}
   -> { items: [
           {
               state: "updated",
               kind: "session",
               id: "{c15814e5-8931-470c-8a16-ef45afedaece}",
               modified: 1453931101,
               data: {
                  lat: 51.5072,
                  lng: -0.1275,
                  name: 'Acrobatics with Dave',
                  clubId: "{fc1f0f87-0538-4b05-96a0-cee88b9c3377}"
               }
           },
           {
               state: "deleted",
               kind: "session",
               id: "{d97f73fb-4718-48ee-a6a9-9c7d717ebd85}",
               modified: 1453931925
           }
         ],
         next: '/getSessions?afterTimestamp=1453931925&afterId={d97f73fb-4718-48ee-a6a9-9c7d717ebd85}'
   }

5.1 Polling

A basic REST endpoint which accepts the from and after parameters is required to allow a full cache refresh / data download on demand (e.g. /getSessions?afterTimestamp=Date(b)&afterId={d97f73fb}).

For cases where only a polling endpoint is available, the client will poll the endpoint using heuristic backoff.

Implementation of at least one other type of endpoint is recommended in order to enable real-time updates.

5.2 Webhooks

Webhooks use the same mechanism as polling, except that pages are pushed from server to client, rather than requested explicitly by the client from the server.

The client registers an endpoint with the server, and the server repeatedly sends subsequent pages to the client. Using the same paging features as with polling allows the server to batch items to increase throughput.

When sending a particular page to the client the server is expected to wait for a successful acknowledgement of the page before sending the next page. If sending of a page fails that page should be continuously retried with an exponential backoff. The server should only proceed to the following page after successfully sending the previous one.

Note that during a full cache refresh the client will page the server for all data, and may simultaneously be receiving webhook requests from the client. Using the timestamp of each record to ensure records are only updated with newer data, the client is able to perform both the full cache refresh and receive webhook requests simultaneously. Alternatively the client can choose to drop webhook requests until its full cache refresh is complete, which should trigger the exponential backoff behaviour from the server, ensuring a good crossover in items between the end of the cache refresh and the webhook updates resuming.

   /putSessions
   -> { items: [{
          state: 'updated',
          kind: "session",
          id: "{c15814e5-8931-470c-8a16-ef45afedaece}",
          modified: Date(a),
          data: {
              lat: 51.5072,
              lng: -0.1275,
              name: 'Acrobatics with Dave',
              clubId: "{fc1f0f87-0538-4b05-96a0-cee88b9c3377}"
          }
       },{
          state: 'deleted',
          kind: "session",
          id: "{d97f73fb-4718-48ee-a6a9-9c7d717ebd85}",
          modified: Date(b)
       }]
   }

5.3 Server-Sent Events ([eventsource])

Server-Sent Events ([eventsource]) (over HTTPS) provides a simple and efficient channel through which a high volume of updates can pass. Although an additional library may be required depending on the servers' platform, it is a very light implementation.

Server-Sent Events require System 1 to implement a system to keep track of events and trigger the sending of data through this channel. This non-trivial implementation, together with the complexities of library support in various languages and frameworks compared with a straightforward [JSON] API implementation used in Polling and Webhooks is the reason that the other transport mechanisms are provided.

The responsibility is on the client to reestablish a connection to the server and inform it of the last retrieved record in order to continue the stream, which is closer to polling than to webhooks.

Response grammar / example:

   /stream?afterTimestamp=Date(b)&afterId={d97f73fb-4718-48ee-a6a9-9c7d717ebd85}
   ->
   event: itemupdate
   data: <item>

Only the <item> from the <response> is required here, and is passed as "data" (the Server-Sent Event specification of "data", which is different from the "data" part of the item). The explicit paging used with polling and webhooks is made redundant as this is a continuous stream over one connection.

The event type is always set to "itemupdate", as the state of each item is set with the "state" field consistent with polling.

   /stream?afterTimestamp=Date(a)&afterId={a97f73fb-4718-48ee-a6a9-9c7d717ebd85}
   ->
   event: itemupdate
   data: {
      state: 'updated',
      kind: "session",
      id: "{c15814e5-8931-470c-8a16-ef45afedaece}",
      modified: Date(a),
      data: {
          lat: 51.5072,
          lng: -0.1275,
          name: 'Acrobatics with Dave',
          clubId: "{fc1f0f87-0538-4b05-96a0-cee88b9c3377}"
      }

   event: itemupdate
   data: {
      state: 'deleted',
      kind: "session",
      id: "{d97f73fb-4718-48ee-a6a9-9c7d717ebd85}",
      modified: Date(b)
   }

5.4 AMQP

[OASIS AMQP] (RabbitMQ et. al.) provides a two way channel for events which includes buffering and multiple connections to increase throughput.

As with Server-sent events, only the <item> from the <response> is required to be sent in the message, as AMQP makes the explicit paging redundant.

For very high volumes, this allows the server to send multiple pages in parallel, as it can calculate the "afterTimestamp" and "afterId" parameter and control the send. The client can also process these in parallel (particularly useful in the case of shared or No-SQL data stores, with scaling queue processors), using the timestamp of each record to ensure records are only updated with newer data. However this requires additional infrastructure and the use of certificates (more complex to configure than HTTPS).