WIP: output extractor: infer data encoding from output metadata

[akhmerov]

As discussed with @minrk, with more media formats being added to the mimebundle, the logic of output extraction becomes harder to extend. For example if an application adds a new base64 encoded mimetype, ExtractOutputPreprocessor would need to be explicitly patched to handle this.

This PR defines an alternative mechanism for extensions communicating how to handle their outputs, specifically by defining output.metadata[mime_type]['encoding'] entry, which instructs the frontend how the output must be interpreted. Currently there are only 3 supported encodings:

• json for json objects
• utf8 for unicode strings
• base64 for binary data

This would for example allow to extend IPython.display.Audio and IPython.display.Video objects to publish the source files directly. One concern, however, is that they would still need to provide _repr_html_ and therefore would need to double the amount of data in the notebook.

[akhmerov]

This is WIP because of the following:

• It assumes a new part of the jupyter spec, which needs to be approved first
• Tests are missing

[SylvainCorlay]

Could this also enable the binary buffers in the messaging protocol?

[akhmerov]

I'm not sure: the protocol requires that everything should be string-serializable. Otherwise we wouldn't be able to store the json representation of the notebook.

[akhmerov]

...OTOH we could imagine other ways of specifying how the output data is provided, for instance by allowing to specify an URI.

[MSeal]

Removing 5.5 milestone as this PR isn't complete and it's unclear if it should be addressed as approached.

Overall I think having a stronger approach to encoding choices would be good. But it is difficult to do as there's a lot of undocumented behavior / mimetypes that rely on the accidental application of particular encoding patterns here :/. Perhaps a question to set a standard for encoding mimetypes should be made somewhere higher up in the specs that other library should enforce in future schema versions?

[akhmerov]

Indeed, this needs a discussing in http://github.com/jupyter/nbformat first.