How to stream data from a web server to browser application.

These applications here use Python on web server and straight JavaScript

Before getting into SSE I thought it worth looking at the merits of three comms models in the web context

Polling
SSE
WebSocket

Visit this website for an interesting write up on the differences. Clearly for what we are doing in the Case Study, SSE is the most appropriate solution for the comms layer between the browser and the webserver. For the comms layer solution between the backend and webserver we are still undecided.

Server Side Events (SSE)

The idea behind SSEs may be familiar: a web app "subscribes" to a stream of updates generated by a server and, whenever a new event occurs, a notification is sent to the client. Created to overcome limitations of AJAX - polling (if data not available server returns an empty response - HTTP overhead), long polling (if the server does not have data available, the server holds the request open until new data is made available, "hanging GET" - implementation includes "hacks" and appending script tags to an 'infinite' iframe)

Server-Sent Events have been designed from the ground up to be efficient. When communicating using SSEs, a server can push data to your app whenever it wants, without the need to make an initial request. SSEs open a single unidirectional channel between server and client (unidirectional “push”), unlike long polling, SSEs are handled directly by the browser and the user simply has to listen for messages.

SSEs are better than WebSockets in scenarios when the flow of data is only in one direction - from the server to the client. SSEs are sent over traditional HTTP - that means they do not require a special protocol or server implementation to get working. Additional features like automatic reconnection, event IDs, ability to send arbitrary events.

SSE is a native HTML5 feature that allows the server to keep the HTTP connection open and push data changes to the client.

JavaScript API

To subscribe to an event stream, create an EventSource object and pass it the URL of your stream.

if (!!window.EventSource) { 
	var source = new EventSource('stream.php'); 
} else { 
// Result to xhr polling :( 
}

Note: If the URL passed to the EventSource constructor is an absolute URL, its origin (scheme, domain, port) must match that of the calling page.

var source = new EventSource("http://localhost:8000/hello");

Next, set up a handler for the message event. You can optionally listen for open and error:

source.addEventListener('message', function(e) {
  console.log(e.data);
}, false);

source.addEventListener('open', function(e) {
  // Connection was opened.
}, false);

source.addEventListener('error', function(e) {
  if (e.readyState == EventSource.CLOSED) {
    // Connection was closed.
  }
}, false);

When updates are pushed from the server, the onmessage handler fires and new data is be available in its e.data property. The magical part is that whenever the connection is closed, the browser will automatically reconnect to the source after ~3 seconds. Your server implementation can even have control over this reconnection timeout.

Browser Support

First browser version supporting SSE:

Chrome 6.0, Mozzilla 6.0, Safari 5.0, Opera 11.5

! NOT SUPPORTED IN Internet Explorer !

Event Stream Format

Sending an event stream from the source is a matter of constructing a plaintext response, served with a text/event-stream Content-Type, that follows the SSEformat. In its basic form, the response should contain a "data:" line, followed by your message, followed by two "\n" characters to end the stream:

If your message is longer, you can break it up by using multiple "data:" lines. Two or more consecutive lines beginning with "data:" will be treated as a single piece of data, meaning only one message event will be fired. Each line should end in a single "\n" (except for the last, which should end with two). The result passed to your message handler is a single string concatenated by newline characters. For example:

This will produce "first line\nsecond line" in e.data. One could then use e.data.split('\n').join('') to reconstruct the message sans "\n" characters.

Example using JSON data:

with the following client-side code:

Additional information about Event

You can send a unique id with an stream event by including a line starting with "id:"

Setting an ID lets the browser keep track of the last event fired so that if, the connection to the server is dropped, a special HTTP header (Last-Event-ID) is set with the new request. This lets the browser determine which event is appropriate to fire. The message event contains a e.lastEventId property.

Changing the reconnection time-out can be done by including a line beginning with "retry:", followed by the number of milliseconds to wait before trying to reconnect.

In case of different event types being generated by one event source, including a line "event:" followed by a unique name will associate this event with that name. On the client, an event listener can be setup to listen to that particular event. Note that the fields do not have to be in any order as long as there is a newline (\n) for each field and two (\n\n) at the end of them. Example of server output with three different event types:

The corresponding event listener can look like this:

Close connection

To cancel a stream from the client, simply call:

To cancel a stream from the server, respond with a non "text/event-stream"Content-Type or return an HTTP status other than 200 OK (e.g. 404 Not Found). Both methods will prevent the browser from re-establishing the connection.

Security

From he WHATWG's section on Cross-document messaging security:

Authors should check the origin attribute to ensure that messages are only accepted from domains that they expect to receive messages from. Otherwise, bugs in the author's message handling code could be exploited by hostile sites.

So, as an extra level of precaution, be sure to verify e.origin in your message handler matches your app's origin:

Another good idea is to check the integrity of the data you receive.

Python Generators and SSE

Python generators are a great fit to create a server stream because they can keeping looping and yielding data and handing it to the client very seamlessly. Here is an example code from a flask web application. The stream contains timestamp data generated every second

Python code to consume a stream of data

Useful websites

https://medium.com/code-zen/python-generator-and-html-server-sent-events-3cdf14140e56

https://www.html5rocks.com/en/tutorials/eventsource/basics/

https://pypi.org/project/sseclient-py/

https://www.w3schools.com/html/html5_serversentevents.asp

MAN - Software Development

HTML5 SSE (streaming data)

Server Side Events (SSE)

JavaScript API

Python Generators and SSE

Python code to consume a stream of data

Useful websites