[Proposal]: Thumbnails: Add supplementary metadata to getAvailableThumbnailTracks

[peaBerberian]

Based on #1496

Problem

We're currently trying to provide a complete[1] and easy to-use API for DASH thumbnail tracks in the RxPlayer.

Today the proposal is to have an API called renderThumbnail, to which an application would just provide an HTML element and a timestamp, and the RxPlayer would do all that's necessary to fetch the corresponding thumbnail and display it in the corresponding element.

The API is like so:

rxPlayer.renderThumbnail({ element, time })
  .then(() => console.log("The thumbnail is now rendered in the element"));

This works and seems to me very simple to understand.

Yet, we've known of advanced use cases where an application might not just want to display a single thumbnail for a single position. For example, there's very known examples where an application displays a window of multiple thumbnails at once on the player's UI to facilitate navigation inside the content.

Screenshot: this is the interface of a very popular web media player (the official one from the platform for which you installed newpipe instead) where multiple thumbnails are shown at once, under the seek bar. Interestingly, the video is also replaced by a thumbnail in that mode here, I guess to provide a smoother experience when rapidly navigating in the content.

To do that under the solution proposed in #1496, an application could just call renderThumbnail with several element and time values.
Yet for this type of feature, what the interface would want is not really to indicate time values, it actually wants basically a list of distinct thumbnails around/before/after a given position.

By just being able to set a time value, an application is blind on which time value is going to lead to a different thumbnail (i.e. is the thumbnail for the time 11 different than the thumbnail for the time 12? Nobody - but the RxPlayer - knows).

So we have to find a solution for this

[1] By complete, I here mean that we want to be able to handle its complexities inside the RxPlayer to ensure advanced DASH configurations like multi-CDN, retry settings for requests etc., while still allowing all potential use cases for an application.

Solution

In this solution, I experiment with a second thumbnail API, getAvailableThumbnailTracks (it already exists in #1496, but its role there was only to list the various thumbnail qualities, if there are several size for example). As this solution build upon yet stays compatible to #1496, I chose to open this second PR on top of that previous one.

I profit from the fact that most standardized thumbnail implementations I know of (BIF, DASH) seem to follow the principle of having evenly-spaced (in terms of time) thumbnails (though I do see a possibility for that to change, e.g. to have thumbnails corresponding to "important" scenes instead, so our implementation has to be resilient).

So here, what this PR does is to add the following properties (all optional) to a track returned by the getAvailableThumbnailTracks API:

• start: The initial time the first thumbnail of that track will apply to

• end: The last time the last thumbnail of that track will apply to.

  ⚠️ For live contents (which I guess should be the main type of contents relying on this, BIF thumbnails are good enough for VoD contents), for now that end will announce the future expected end of the Period if already known (for example thanks to a Period@end or a SegmentTemplate@endNumber attribute) - and undefined if unknown.
  In this current configuration, an application has to also request another API, like getLivePosition to know the maximum position it can currently load thumbnails from

• thumbnailsPerSegment: Individual thumbnails may be technically part of "segments" containing multiple consecutive thumbnails each.
  thumbnailsPerSegment is the number of thumbnails each of those segments contain.

  For example you could have stored on the server a segment which is a grid of 2 x 3 (2 horizontal rows and * 3 vertical columns) thumbnails, which the RxPlayer will load at once then "cut" the right way when calling renderThumbnail. In that example, thumbnailsPerSegment would be set to 6 (2*3).

  Note that the last segment of a content may contain less thumbnails as announced here depending on the duration of the content.

• segmentDuration: The "duration" (in seconds) each segments of thumbnails applies to (with the exception of the last one, which just fills until end)

Then, an application should have all information needed to calculate a time which correspond to a different thumbnail.

Though this solution lead to a minor issue: by letting application make the time operation themselves with start, end, segmentDuration and so on, there's a risk of rounding errors leading to a time which does not correspond to the thumbnail wanted but the one before or after. To me, we could just indicate in our API documentation to application developers that they should be extra careful and may add an epsilon (or even choose a time in the "middle" of thumbnails each time) if they want that type of thumbnail list feature.

Thoughts?

[peaBerberian]

Getting the timestamp of the latest available thumbnail in a live content is a hard thing to get in any case in the current solution:

/**
 * For the current content, returns the position in seconds that will
 * correspond to the currently last reachable thumbnail, or `undefined` if
 * unknown.
 *
 * That position may then be passed to the `rxPlayer.renderThumbnail()` method.
 *
 * @returns {number|undefined}
 */
function getLastThumbnailTime() {
  const livePosition = rxPlayer.getLivePosition() ?? undefined;
  const periods = rxPlayer.getAvailablePeriods();

  let lastSeekablePeriod;
  if (livePosition === undefined) {
    lastSeekablePeriod = periods[periods.length - 1];
  } else {
    // Most applications will not let users seek further than live edge even when possible
    // (e.g. when ads are pre-anounced in the Manifest)
    // So here find the last Period that exists before the live edge
    for (let i = periods.length - 1; i >= 0; i--) {
      if (periods[i].start >= livePosition) {
        lastSeekablePeriod = periods[i];
        break;
      }
    }
  }

  if (lastSeekablePeriod === undefined) {
    return;
  }

  // Just select the first thumbnail track for the last Period.
  const metadata = rxPlayer.getAvailableThumbnailTracks({
    periodId: lastSeekablePeriod.id,
  })[0];
  if (
    metadata === undefined ||
    metadata.start === undefined ||
    metadata.segmentDuration === undefined
  ) {
    return;
  }

  const maximumPosition = rxPlayer.getMaximumPosition() ?? undefined;
  if (maximumPosition === undefined) {
    return;
  }

  /**
   * Seconds at the end of the content for which a thumbnail has not yet been
   * generated.
   */
  const secondsWithoutThumbnailYet =
    (maximumPosition - metadata.start) % metadata.segmentDuration;

  /**
   * Position that will lead to the last available thumbnail being requested.
   */
  const maxThumbnailTime =
    Math.min(maximumPosition - secondsWithoutThumbnailYet, metadata.end ?? Infinity) -
    // To securize against rounding and/or precizion errors, we take a timestamp
    // at the middle of the thumbnail
    (metadata.segmentDuration / (metadata.thumbnailsPerSegment ?? 1)) / 2;

  if (livePosition !== undefined && livePosition < maxThumbnailTime) {
    return livePosition;
  }
  return maxThumbnailTime;
}

Even I, spending most of my time on core player matters, had trouble writing that (e.g. we have to potentially consider the live position several time and all that).

[github-actions]

Automated performance checks have been performed on commit af0c737bbc0a700d3e255741e498959b54259f92 with the base branch feat/thumbnail-tracks.

Tests results

✅ Tests have passed.

Performance tests 1st run output

No significative change in performance for tests:

Name	Mean	Median
loading	20.02ms -> 19.75ms (0.267ms, z: 2.77353)	27.75ms -> 27.45ms
seeking	13.47ms -> 14.80ms (-1.334ms, z: 1.47673)	11.55ms -> 11.55ms
audio-track-reload	26.63ms -> 26.68ms (-0.055ms, z: 0.21854)	39.00ms -> 39.15ms
cold loading multithread	48.80ms -> 48.41ms (0.391ms, z: 9.90690)	71.55ms -> 70.65ms
seeking multithread	18.71ms -> 19.46ms (-0.749ms, z: 0.72311)	10.50ms -> 10.50ms
audio-track-reload multithread	26.46ms -> 26.98ms (-0.519ms, z: 2.22423)	38.85ms -> 38.70ms
hot loading multithread	15.81ms -> 15.71ms (0.098ms, z: 2.57527)	22.95ms -> 22.80ms

If you want to skip performance checks for latter commits, add the skip-performance-checks label to this Pull Request.