proposal: integrating streams from sludge server

[drohen]

Hi!
Cool project, would be an interesting opportunity to combine this with streams coming from my NIME project, rivalium.
There's 4 components:

• splutter, which records audio in a browser and encodes it into segments of audio and sends it to a server
• sludge, which is the server receiving the files and can serve them in linear playback, live playback or randomly
• syllid, which receives the files from sludge, decodes and puts them into web audio buffer for playing
• sortition, provides a service to create collections of sludge streams that are served randomly on request

If its possible, would be awesome if we could plug in a sludge stream URL or sortition URL and decode+playback the audio within aleatora. Even more awesome would be to have the ability to encode + upload to a sludge endpoint too, but this can be facilitated via a virtual input device. For now, could also test with sending browser output to a virtual device then using that as input to aleatora, so basically the inverse of the previous suggestion.

Not sure if there's a better way to discuss this, but I will take a look at yr project further when I get a chance and see what is possible. Here's some more info on my project if you need it.

Cheers!

[ijc8]

Hey, thanks for your interest!
I think this is definitely doable in both directions, without relying on virtual devices.

Looks like this would involve pulling in something like opuslib or PyOgg for encoding/decoding.
For downloading, I guess we'd want a function that takes a sludge stream ID and returns a stream which would:

• GET https://play.rivalium.com/api/<stream ID>/?start=random
• GET each Opus file in the list from previous request, decode, and yield the samples.
• Repeat.

For uploading, it looks like we'll want to attach a side-effect to the stream of interest which:

• POSTs to https://play.rivalium.com/api/stream and gets the admin URL from the response
• Repeatedly encodes and POSTs Opus files to the admin URL, with timestamped filenames.

Does all of that sound right? I'm imagining an interface along the lines of

stream_from_rivalium = rivalium.recv("stream_id_here")
stream_with_side_effect_of_uploading = rivalium.send(my_interesting_audio_stream)

Although, I suppose rivalium.send() here should also have a way to return the public URL so you can know where your stream is!

That raises another question: should rivalium.send() create a new Rivalium stream when it's called, or when the Aleatora stream it returns is played? I'm guessing the former, so playing stream_with_side_effect_of_uploading multiple times should keep adding audio to the same Rivalium stream.
In that case, the interface should be more like:

stream_with_side_effect, public_url = rivalium.send(my_interesting_audio_stream)

Maybe it should also return (and optionally take) the admin URL, so multiple Aleatora streams can upload to the same Rivalium stream:

stream_with_side_effect, public_url, admin_url = rivalium.send(my_interesting_audio_stream)
another_effectful_stream, *_ = rivalium.send(another_audio_stream, admin_url)

Let me know what you think! I'm sure I'm missing some important details here, like working with sortition collections and the non-random modes of sludge.

[drohen]

Yo! Unexpectedly well researched response. After reading I'm now wishing I spent more time on documentation, but I think you've grasped the idea pretty well nonetheless.

Looks like this would involve pulling in something like opuslib or PyOgg for encoding/decoding.

I'm wondering now if it makes sense to build this as a standalone client with the initial focus on getting it to work nicely with Aleatora (btw, awesome project name).

Some notes:

• I'm in the process of updating how the system records/encodes/decodes/plays to accommodate live/normal playback modes, so I will push this asap and deploy a staging version for testing.
• During this process I have been trying to figure out things I didn't previously understand about the encoder/decoder. What I've since learned is that the encoder adds a 80ms "pre-skip" and also during decoding has empty samples at the end of the buffer due to (I think) the page sizes, not super relevant but worthwhile knowing when aiming for uniformity with encoding/decoding.
• Right now I'm looking at deciding between encoding audio at 1.08s segments or 3.08. The 0.08 is to compensate for the pre-skip, while originally I aimed for 1 second segments, but I noticed that most of the buffering and minimum random playback time was around 3+ seconds, so having audio less than that might not make sense, but I'll think about it more.
• Solid tones, like sine waves, don't get reproduced accurately due to a minor cross fade between segments, so keep that in mind if you start playing around with Rivalium and notice weird stuff like that.

Your downloading suggestion looks pretty straight forward, and can vary for live/normal/random modes but we can just start with implementing random mode. What would a signal look like from the audio buffer in case there's no audio yet available? As 0s are usually indicative on an empty signal, it could just stream those and Aleatora can decide whether to quit the stream or hold onto it until audio is available.

The main thing with creating the buffer is to possibly borrow my implementation for the decoding and joining segments, as there needs to be a small cross fade (~10ms, borrowed from the pre-skip data) to avoid clicks (there's no other simple way afaik). I'm not sure how all this works with python, most of my experience is with using the pyo dsp library.

I'm guessing the former, so playing stream_with_side_effect_of_uploading multiple times should keep adding audio to the same Rivalium stream.

In regards to uploading, I err against uploading more than a single stream, the idea is for the files to represent a sequence of 1 channel audio. If there is to be multiple inputs, they should each have their own Rivalium stream, and if multiple inputs should be combined, then I'm not sure if that should happen within the sending system or within the Rivalium client. Maybe that's what yr trying to work out, but I got a bit confused by stream_with_side_effect_of_uploading, so I figured I'd mention that caveat to the Rivalium design.

---

In regards to collections, to add streams its just sending a public URL and storing the code, then removing them by sending the code. Calling a collection endpoint just performs a redirect, so it's only a matter of following the redirect. Sortition will also be updated to handle different streaming modes soon too.

For the different modes, the main thing to think about is fetching the length (in seconds) and being able to play from a given second. As live and normal playback modes will generally have an "end" this will probably be handled similarly to an empty stream.

---

As mentioned before, maybe this would be good as a package that can be imported into Aleatora rather than built directly into it. I'm not as familiar with python as I am with JS, so I'm wondering how much time you'd be willing to put into helping implementing this as I can do as much as possible, but there might be things here and there I'd probably need some advice for.

Thanks again for your quick response, and let me know if you need any more details.

[drohen]

OH.
I almost forgot the additional feature I'm building into the browser client that I would think is also of value here (and relevant to the design of Aleatora streams may contain any kind of data type, not just samples). Each segment of audio in Rivalium is intended to be a content-addressed via it's URL (as they are all just regular opus audio files), so further data can be timed with via association with this address.

Therefore, it's worth considering how one might handle a returned segment URL after upload, and also the emitted URL from the Rivalium output when the buffer is playing a particular segment. I would imagine that Aleatora would be good for associating things like controller signals, OSC, MIDI, or any metadata generated from the coding environment with a segment, and also taking segment URL, mapping it to a pre-loaded hash table and using the values to control a plugin, etc etc.

[ijc8]

Yo! Unexpectedly well researched response. After reading I'm now wishing I spent more time on documentation, but I think you've grasped the idea pretty well nonetheless.

Great. My research mostly consisted of fiddling around with Rivalium with the dev console open on the "Network" tab. 🙂

I'm wondering now if it makes sense to build this as a standalone client with the initial focus on getting it to work nicely with Aleatora (btw, awesome project name).

As mentioned before, maybe this would be good as a package that can be imported into Aleatora rather than built directly into it.

Yes, this depends on your interests and how much work is needed for the integration.
For instance, if you want to make a standalone Python library for Rivalium, then Aleatora can just wrap that.
Or, if the integration ends up taking a lot of code, it might be preferable to contain it in an aleatora-rivalium package (with aleatora and an opus encoder as dependencies).

But I suspect the integration won't be that much work. If it can be nicely self-contained in a single module, I wouldn't mind hosting it here, perhaps under an integration or thirdparty submodule. (So usage would be something like from aleatora.thirdparty import rivalium.) The opuslib/PyOgg dependency could be declared optional (like the existing "speech", "plugins", and "foxdot" options in setup.py).

What I've since learned is that the encoder adds a 80ms "pre-skip" and also during decoding has empty samples at the end of the buffer due to (I think) the page sizes, not super relevant but worthwhile knowing when aiming for uniformity with encoding/decoding.

Ah, I was wondering about this. (I ran into a similar issue some time ago with MP3s... which unfortunately don't include a field saying how many samples to skip!)
Hopefully this is handled transparently by the opus decoder, so that those samples are already 'pre-skipped' before the application code sees them?

What would a signal look like from the audio buffer in case there's no audio yet available? As 0s are usually indicative on an empty signal, it could just stream those and Aleatora can decide whether to quit the stream or hold onto it until audio is available.

One option is for the stream to simply block until audio is available. This is how, for example, the networking streams work - and then they can be made nonblocking with net.unblock(), which can fill in 0's (or any other chosen value) until the underlying stream has something ready. Another option would be for the stream to end immediately, but that may be less appropriate here.

there needs to be a small cross fade (~10ms, borrowed from the pre-skip data) to avoid clicks (there's no other simple way afaik). I'm not sure how all this works with python, most of my experience is with using the pyo dsp library.

No worries. DSP can be done directly in Aleatora, and there is a ramp() stream function that may work for this purpose.

Maybe that's what yr trying to work out, but I got a bit confused by stream_with_side_effect_of_uploading, so I figured I'd mention that caveat to the Rivalium design.

Right. Aleatora streams can be played multiple times, with potentially different outcomes, so one question was whether a Rivalium stream should stick with a particular Aleatora stream across playthroughs, or if each separate playthrough should get its own Rivalium stream. It sounds like the first option is the right approach.

As for multiple inputs getting separate streams - yep, that makes sense. I think that should be the default, but it also seems plausible that someone might want multiple Aleatora streams to go to the same Rivalium stream (perhaps the streams are closely related, or maybe the user is experimenting and revising a stream live); hence the idea of allowing the user to optionally say "use this existing Rivalium stream rather than making a new one."

For the different modes, the main thing to think about is fetching the length (in seconds) and being able to play from a given second. As live and normal playback modes will generally have an "end" this will probably be handled similarly to an empty stream.

Yep, Aleatora streams can also end, so that can map over directly.

Each segment of audio in Rivalium is intended to be a content-addressed via it's URL (as they are all just regular opus audio files), so further data can be timed with via association with this address.
Therefore, it's worth considering how one might handle a returned segment URL after upload, and also the emitted URL from the Rivalium output when the buffer is playing a particular segment. I would imagine that Aleatora would be good for associating things like controller signals, OSC, MIDI, or any metadata generated from the coding environment with a segment, and also taking segment URL, mapping it to a pre-loaded hash table and using the values to control a plugin, etc etc.

Interesting. Yes; actually, the implementation I was imagining for rivalium.recv() would first create a stream of the segments (a stream of streams of samples), and then flatten that via Stream.join() into a stream of samples.
We could take that one step further and first create a stream of those URLs, then map() that into a stream of segments, and then join() that into a stream of samples. Associating other data with those segments would just be a matter of zip()ing it with that first stream or tweaking the function passed to map().

I'm not as familiar with python as I am with JS, so I'm wondering how much time you'd be willing to put into helping implementing this as I can do as much as possible, but there might be things here and there I'd probably need some advice for.

I'm happy to help out or accept a PR. I might take a stab at this over the weekend, if you don't beat me to it. 🙂

[drohen]

But I suspect the integration won't be that much work. If it can be nicely self-contained in a single module, I wouldn't mind hosting it here, perhaps under an integration or thirdparty submodule.
I might take a stab at this over the weekend, if you don't beat me to it.

If yr interested, I have some free time on the weekend too, maybe we can schedule a window for a chat and run through some ideas or whatever. Happy to work on it as just a PR to aleatora and we can see where it goes from there. I'm in Germany, so I'm about 6 hours ahead of you, if you're available somewhere between 9am and 2pm your time, let me know.

Hopefully this is handled transparently by the opus decoder, so that those samples are already 'pre-skipped' before the application code seems them?

Depends on the decoder, opusdec cli tool seems to remove the pre-skip and excess page data, whereas the decoder I'm using the browser doesn't, so just inspect the size of the output buffer and its contents to be sure.

One option is for the stream to simply block until audio is available.

This is interesting, I will look into how this works, feel free to send through any useful information to help me learn more.

Aleatora streams can be played multiple times, with potentially different outcomes

This is basically the same principle as rivalium but with "live" (as in, immediate but not real time) "streams" (as in streams of files, not packets), its funny to have designed projects around similar goals but with completely different approach.

One question was whether a Rivalium stream should stick with a particular Aleatora stream across playthroughs, or if each separate playthrough should get its own Rivalium stream. It sounds like the first option is the right approach.

Yeah I would suggest reusing rvm streams, as the longer the stream the more randomness and diversity there is (unless you want to eliminate any previously recorded audio, in which case a new rvm stream is necessary).

but it also seems plausible that someone might want multiple Aleatora streams to go to the same Rivalium stream
"use this existing Rivalium stream rather than making a new one."

Do you mean as above suggestion about reuse or as in sending 2 aleatora streams into the same rvm stream? This is what I'm mostly concerned about as the input audio should basically be combined before being segmented and encoded, that's all.

[ijc8]

Depends on the decoder, opusdec cli tool seems to remove the pre-skip and excess page data, whereas the decoder I'm using the browser doesn't, so just inspect the size of the output buffer and its contents to be sure.

👍

Do you mean as above suggestion about reuse or as in sending 2 aleatora streams into the same rvm stream? This is what I'm mostly concerned about as the input audio should basically be combined before being segmented and encoded, that's all.

Yes, more like re-use; someone might want an Aleatora stream to send to a Rivalium stream, and then they might want to send more to that same Rivalium stream from another Aleatora stream later on.

maybe we can schedule a window for a chat and run through some ideas or whatever.

Sure, that sounds good (and more efficient than back-and-forth on GitHub 🙂).
I'll shoot you an email to coordinate. Is the address listed in the WAC paper good, or would you prefer the one listed here?

[drohen]

Either is fine.

[ijc8]

Good talking to you! I created a feature branch + draft PR for this, and dumped the output of our call there as a starting point.
Feel free to comment on the PR (#3), checkout the branch for local experimentation, fork and submit a PR for it, etc.