Can we use a single line JSON stream for vector/webgl layers

[mapcentia]

The GC2 SQL API creates a valid GeoJSON doc server side and sends it to the browser. This approach puts a limit on how many data can be send - depending on the memory limit in GC2. And it doesn't scale well.

I've created a simple "stream" API, which sends one-line JSON, using a cursor in the database. This way there is no limit on data size.

SQL API: https://swarm.gc2.io/api/v2/sql/horsens?q=select%20*%20from%20test.togstation%20limit%2010

SQL Stream API: https://swarm.gc2.io/api/v2/sql/stream/horsens?q=select%20*%20from%20test.togstation%20limit%2010

Can we use the SQL Stream API instead of SQL API to populate a vector or webgl layer? Especially webgl is interesting because it can handle much more data.

One way of doing it, is to compile a valid GeoJSON doc from the one-lines and from that do what we do now. But a more elegant and less memory consuming way is to build the layer while data streams in. This way the user also have a visual indicator seeing the layer being build.

[sashuk]

@mapcentia According to https://github.com/mapcentia/geocloud2/blob/b696654778431961b4159475385bcbc490cfbeba/app/models/Stream.php, the streaming is organized by iteratively calling the ... flush(); ob_flush(); ..., so there are more than one output iterations for one single server request.

Seems like there are already packages that can help with streaming data (GeoJSON in particular) - https://github.com/node-geojson/leaflet-geojson-stream based on https://github.com/substack/hyperquest, so I think it is surely possible to implement!

Another interesting side of this question is to use the web sockets for streaming https://github.com/perrygeo/websocket-geojson-leaflet, but it can be probably implemented later.

[sashuk]

@mapcentia Could you please specify if this should be implemented?

[mapcentia]

@sashuk Yes, please start on this for WebGL, which is most relavant for big datasets. If you can get this to work, we can proced with styling, mouse-over and click for WebGL layers

[mapcentia]

@sashuk The endpoint is changed: https://swarm.gc2.io/api/v2/sql/horsens/stream?q=select%20*%20from%20test.togstation%20limit%2010
There're some debug info after each line - I'll get rid of that

[sashuk]

@mapcentia Great, thanks!