Stuffing always audible #89
Comments
|
You are right that you can hear small crackles when Shairport Sync does basic stuffing on a sine wave. I can't really hear it in normal use, TBH. Using soxr stuffing helps a lot, as you might imagine. To minimise it, I suggest that you try to minimise the amount of stuffing necessary. If you enable the generation of statistics, you can see some figures, including "net corrections" and "corrections" in parts per million. A correction is a stuffing action – an insertion or deletion of a single audio frame in basic stuffing mode. If the number of "corrections" is greater than the number of "net corrections", then the system is over-correcting and doing more stuffing than is necessary. To reduce overcorrection, increase the level of "drift" permitted – this essentially allows the synchronisation to drift for longer before attempting to correct it using a stuffing action. |
|
When you say the "cracks are also audible when I change the volume", I suspect that some other phenomenon might be responsible for that. Is it possible that it is a kind of electrical noise? I notice that DACs and audio cables can easily pick up electrical interference from the power supply or nearby WiFi transceivers, and volume control changes are accompanied by extra network and processor activity, possibly generating electrical noise at the instant of volume control change. You could possibly test this by making the drift value insanely large, to stop stuffing, and then see if the volume-control-associated crackles remain... |
|
Just FYI, here is a link to a thread that, among other stuff, discusses the crackle: abrasive#296 (comment) |
|
Thanks for linking the thread - very interesting, although I do not understand everything. I changed the drift to 65535 and the crackles occur exactly when changing the volume. |
|
I plotted the spectrum of the output for the sine wave with a spectrum analyzer. This is a time-based graph were you can see that the amplitude gets much higher when changing the volume: |
|
When playing the sine wave with UPnP and changing the volume I can also here the crackles, though not so clearly. |
|
Hmm. I'm afraid I can't interpret the graph you sent. I don't really know what can be done about the crackle due to volume control changing. Were you able to reduce the amount of crackle due to stuffing by increasing the drift value? Did it have any useful effect? |
|
The x axes shows the frequency, the y axes the time. The red line is 1000 Hz. The light-blue parts in the upper half of the graph shows an increased ampiltude when changing the volume. However, I am not sure how it connects to the audible crackles. When increasing the drift value the crackles start later. For the default value it starts immediately, if I set it to 800 frames it starts about 40 - 60 seconds and does not stop after that. |
|
That sounds right regarding the drift setting. Tell me, can you hear the crackles due to stuffing in normal programme material? Regarding the volume control crackles, I'm not able to come up with anything much, except that it might be a function of whatever DSP is being done – after all, a sudden volume change is a bit like applying a step function, which could result in lots of harmonics, etc. |
|
Yeah! After changing my ALSA configuration it works much better now! The sine wave plays without crackles now. I added a dmixer (software mixer) and changed period_size and buffer_size. Surpisingly, no corrections are done now by shairport-sync (version 2.2.4, running with default options):
No stuffing, but underrun errors occur which are audible as small crackles: Furthermore, changing the volume produces almost no audible crackles now. |
|
Okay, obviously no stuffing is done now because I set the buffer_size for the dmixer to 4096 wich is smaller then |
|
I picked DAC_BUFFER_QUEUE_MINIMUM_LENGTH to be 5000 to stop low-powered devices from completely choking if they were stuffing or stripping. The sync error on your statistics looks extremely large, and the number of late packets and resend requests are high too. Here are some stats from some of my setups: iMac on ethernet to base station to Raspberry PI on WiFi with a Topping TP30 amp and with 880 frame "drift": That's right – no corrections whatever, 1 missing packet. Here's another: iPhone 6 on 5GHz WiFi to base station to Raspberry Pi on 2.4 GHz WiFi with 880 frame "drift": Finally: The TP-Link is in a marginal enough location for WiFi, but isn't great even in a good location. Also, BTW, it doesn't have the horsepower to do soxr-based stuffing. So I think these are pretty typical. What kind of device etc. are you using? |
|
Many thanks for running the tests! I'm using an iPhone 4s to base station to an ARMv5 device (450 MHz) with an on-board DAC. Maybe the device is to low-powered for stuffing? I played with the buffer_size parameter. If I set it lower then 5000, as expected, no stuffing is done at all, so no crackles are audible (of course). On the downside the drift increases and I'm loosing synchronization resulting in a large sync error after some minutes. If I set the buffer_size a little bit higher then 5000, some stuffing is done but the drift still increases over time. Some crackles, but almost not audible. After two minutes the devices get out of sync and shairport does resync. If I set the buffer_size higher then 5600, enough stuffing is done to keep the devices synchronized, but the crackles are audible very clearly for a sine wave and also for some music tracks with quiet parts. Increasing the tolerance does not really avoids the necessity of stuffing. It just delays it and the stuffing is done somehow later. Increasing the tolerance would only make sense for me if there is the possibility that the drift would decrease at some point of time instead of always increasing. I wonder if the audible crackles can be avoided in my case and how to debug the system to find the bottleneck. Do you hear crackles when streaming a sine wave with 0dB from your iPhone to the Pi and stuffing is done? And I do not understand why inserting or deleting a frame into the buffer can cause a crackle. Or the volume change? If read in the thread you linked that @yenchee1970 had the same problem: abrasive#296 (comment). He also wrote that soxr helped, if set to low quality for the Pi: abrasive#296 (comment). I tested soxr stuffing, but it does not help for me. I guess my device is not powerful enough. What is the setting of soxr use by shairport-sync. Can the quality by adjusted? |
|
Can you share a few lines of the statistics output? If a resynchronisation is being triggered in two minutes, it implies a net correction of about 400 ppm, which is very large, and would cause a good deal of stuffing or stripping and hence, on a 1 kHz sine wave, an audible hiss or crackle. It would also imply that the device's clock is really inaccurate. The noise seems to be unavoidable (see the graphs on that thread) – hence the move to soxr-based resampling, but with normal material and under normal listening conditions the noise is inaudible; that's what they seem to conclude in that thread, and is certainly my own experience. Offhand, I'd guess that the device is powerful enough for basic stuffing or stripping. For soxr-based operation, a lot depends on whether it has floating point hardware and whether it's supported by the compiler. My guess is that it's very unlikely. The processor in the TP-Link 710N above is a 400MHz device and is incapable of soxr-based operation. I did not explore any of the 'simpler' modes of operation of the soxr library. |
|
The is a statistics about two and a half minute from iPhone 4s to base station to shairport device with tolerance set to 882: I will certainly check if the system clock can be improved. Many thanks for the hint! Maybe the tool adjtimex will help... @yenchee1970 did some work on adding an option to set the soxr quality: yenchee1970@6d97953. The default value used by soxr is 'Low quality'. Five standard qualities are defined in soxr: http://sourceforge.net/p/soxr/code/ci/master/tree/src/soxr.h#l281. Maybe we can add such an option to shairport-sync. Certainly, it will not help me on my low-powered device, but it maybe useful for high-powered device to improve resampling. I've also learned that the alsa parameter |
|
Thanks. So the clock is a long way off alright. I don't think I'll have a look at the quality settings, but not too soon :) Regarding period size and buffer size, I did experiment with these early on, and came to the conclusion that they shouldn't be messed with, especially as the likely optimal parameters will change from device to device. In practice, this seems to have been the right thing to do. |
|
You're right with Can you explain the calculation of the actual delay, please? I'm still looking for the cause why the delay increases over time (without stuffing and resync). The systems crytal clock should not drift so much. |
|
I did some further investigations. I transposed the terms of calculating the delay to and plotted the three termes So |
|
This is good work, thanks; it's something I haven't done. Let me think about it for a couple of days. |
|
I repeated the test with a MacBook on ethernet and the iPhone on wifi with the WHAALE app (instead of iTunes). Both are showing similar results. I'll think about it, too. Many thanks so far! |
|
You must tell me how you generated the data and what tools you used to plot it. Anyway, here are my thoughts:
|
|
OK, so I've just pushed an update in The drift does not quite correspond with the net corrections figure produced as part of the statistics – they should be the same, give or take. I don't know why this is – maybe it's a simple calculation error, but it needs investigation. |
|
Many thanks for the explanation! I will try to fully understand the technical background. I added the following line to the code after this line https://github.com/mikebrady/shairport-sync/blob/development/player.c#L986. debug(1, "%lld %lld %lld", td_in_frames, current_delay, rt - nt);I'm using In the conf file I changed the following settings (no stuffing, no resync): I'm starting shairport-sync in non-daemon mode and copy the debug output after some minutes to a file, eg Starting So I will make a plot of |
|
Very useful, thanks. |
|
I need to modify the stuff I pushed yesterday to make it output raw data suitable for plotting. |
|
Thanks! I did a plot of the clock offset, the For this I added the following code snippet into rtp.c https://github.com/mikebrady/shairport-sync/blob/development/rtp.c#L402: int64_t current_delay = config.output->delay();
uint64_t offset = ((distant_receive_time - departure_time) + (distant_transmit_time - arrival_time)) >> 1;
debug(1, "%llu %llu %lld", offset, local_to_remote_time_difference, current_delay);https://www.dropbox.com/s/gw4xur95jf7zaxs/offset-local_to_remote_time_difference-alsa_delay.pdf?dl=0 The first graph shows the The x-axes represents the time in seconds where one interval represents three seconds (the timining interval). So the first 3*60 seconds the offset is decreasing, the for around 60 seconds it's increasing fast, after this it is increasing constantly, but slower. What do you think about this? |
|
Two obtain the raw data I changed the stuff from yesterday to: uint64_t clock_drift;
if (first_local_to_remote_time_difference>=local_to_remote_time_difference) {
clock_drift = (((first_local_to_remote_time_difference - local_to_remote_time_difference) * 1000000000) / (get_absolute_time_in_fp() - first_local_to_remote_time_difference_time));
//debug(1,"Clock drift is -%lld ppb.",clock_drift);
}
else {
clock_drift = (((local_to_remote_time_difference - first_local_to_remote_time_difference) * 1000000000) / (get_absolute_time_in_fp() - first_local_to_remote_time_difference_time));
//debug(1,"Clock drift is %lld ppb.",clock_drift);
}
debug(1, "%llu", clock_drift);This is the corresponding plot: The same plot with truncated y-axes: y-axes shows ppb, y-axes shows time with an interval of 3 seconds (timing interval). |
|
Thanks for all these fantastic plots – they are very rich in terms of visualisations, and hopefully will hep in understanding what's happening. A couple of things. First, the first Third, the last plot is of clock drift averaged over the entire interval, so I think it probably doesn't tell us enough about how the drift is evolving. If the code you cited above was replaced by this: it would fix that slight error and give the drift in microseconds, unaveraged. |
|
Sorry – didn't mean to close the issue! |
|
Nothing wrong with either of those parts :) About the timestamps, I never thought about it too much, but I imagine that communication with each extra airplay device would be handled by separate threads, each going about its business by itself. I'd have been a bit surprised if they were the same, frankly. |
|
Full of silly questions, sorry, but here's another: does your CPU go into a low-power mode or anything of that nature? |
No, any low-power features are disabled in the Kernel config: |
|
Yeah!!! I have it! The divider of the SAIF clock is without software interaction not precise enough for synchronisation. A guy from the Freescale community have had the same problem. Fortunalty he shared his solution just today :-) I will post a graph after some tests later. @mikebrady Many thanks for all your help and work! It helped me really a lot to understand the implementation and the synchronization part. It is indeed a lot of good work you've done! I hope the work done by @mikebrady will help others to quickly identify audio problems because of bad clock synchronization! [Edit: Added missing link to the solution.] |
|
One quick question: What drift value did you use for the last test #89 (comment)? |
|
It was left as the default – 88 frames. Looking forward to seeing some graphs :) |
|
By the way, thanks are due to you, @joerg-krause, for providing the setting, motivation and a good deal of help in understanding the synchronisation system more completely. |
|
I applied the patches and rerun the test two times. It looks much better now compared to my last test. With one exception:
After around 145 timing packets a lot of corrections are done. Why? This was the same in both tests. |
|
That looks somewhat better alright. Can you plot the sum of Clock Drift + Source Drift + Corrections? That should tend to zero, but allowing for some slop due to the drift figure. I'm just wondering if the error begins to cross the threshold at around 142. |
|
I removed DAC delay in favour of Net (= Clock Drift + Source Drift + Corrections):
Just for clarification: all parameters are set to default (eg drift is 88 frames).
Which error? |
|
Ok, I enabled debug output for the Linux audio driver to see when the frequency is adjusted. In the current test I can see that around the frequency is adjusted (faster) and the number fo corrections rapidly is growing. After 52 seconds the frequency is adjusted to run slower (it is the same value as before the adjustment). In the current running test the number of corrections before the adjustment was done was 0! |
|
The "error" I meant was just the latency error calculation. But that would not account for the duration of the change in the rate of corrections – there seems to be some kind of hysteresis effect working there, but I don't know where it could be coming from. Trying to understand the patches for that clock problem on the Freescale i.MX28 CPU, is it possible that the dividing ratios are being changed from time to time? Also, that clock drift figure is still rough – it might be a further clue... |
|
Ah – our posts crossed. So my guess might be right? |
|
Yes, it is. Every 420.1 seconds the frequency is adjusted for 52 seconds to maintain audio synchronization within +/-20ms of sample rate. I will run a test now with setting the drift to 882 (= 20ms). |
|
That looks like it then. The first 420 seconds is around 144 samples. The rapid change lasts to around sample 161, thus 15 samples or 45 seconds. It's plausible. |
|
This is the graph with MCLK frequency adjustment. Drift is set to 882:
You can see clearly that every 420 seconds the frequency (the clock is slower then the desired clock) is adjusted for 52 seconds (clock runs faster then the desired clock). What do you think about this? [Edit]: The tests lasts about 3000 seconds. |
|
That graph looks exactly as I would expect. The clock drift still looks rough. What does it sound like, I wonder? I think it should sound pretty okay, though you might hear bursts of crackle every 420 seconds... |
|
Sounds much better now! Far less crackles. This is a graph a made when I measured the roundtrip time. Maybe that's why the clock drift looks so rough?
|
|
Yep. The filter should be discarding the outliers, but there is still plenty of variation below 10,000 (10 milliseconds?). Do you have any ideas about the round-trip time variations yourself? |
|
Yes, 10 ms.
Not sure about this. Maybe some wifi driver issues? Do you think it's worth to look at this? |
|
Off the top of my head, yes, I do – something is going on there. I guess I should do a few tests myself here to compare it with, but my vague memory is that the times were in the low single milliseconds, often much lower. |
|
Okay, I'll have a look at this tomorrow. Many thanks so far! |
|
Here's an example on a TP-Link 710N using its built-in WiFi here at home. Seems to be just over 2 ms on average, and pretty even.
|
|
Thanks for the test! I run several tests om my own, but did not succeed in getting an average ping time under 5ms. I will discuss this on the linux-wireless mailing list, as it is not shairport specific. Nevertheless, I will post any results here. |
|
Thanks. We might be able to close this issue soon, I'm thinking. |
|
I close this thread until any new breakthrough discoveries are made. Again, @mikebrady many thanks for all the work and investigations! |
|
Thanks – it was very interesting. |
When streaming from iTunes to my custom device I always hear small cracks when shairport-sync does basic stuffing. I changed several parameters but nothing helped. The cracks are also audible when I change the volume. You write in the README that
The cracks are very good to hear when playing a sinus tone.
How can I influence the number of frames inserted/deleted? Or do you have another idea what might be wrong here?
The text was updated successfully, but these errors were encountered: