[Twitter] How does the extractor.twitter.unique option work?

[Hrxn]

I just noticed this from reading configuration.rst, this must be somewhat recent and I've not used this in my own config yet.

The description simply states:

Ignore previously seen Tweets.

But how, does it rely on the archive file? Wouldn't this be some overlap in functionality then?

What is the point of this option?

[mikf]

extractor.twitter.unique tries to fix a problem outlined in #2712 (comment), where /with_replies timelines can return the same Tweet multiple times, which might potentially interfere with -A, for example.

It is implemented by keeping a list of previously seen Tweet IDs in memory and rejecting any Tweet that has an ID already in said list. (1d14928)

It is kind of redundant, but you can turn it off if you don't like it / don't want to waste extra memory.

[Hrxn]

Okay, so first of all, this only affects the /with_replies timeline, yes?

And if you get the same tweet multiple times, would that break anything, or is the issue simply that it would increase the total time of the extraction run? Or is there a nonzero probability that it would not terminate at all?

[mikf]

Okay, so first of all, this only affects the /with_replies timeline, yes?

The option is active by default for all Twitter extractors, but its effect should mostly be seen for /with_replies since for example /media doesn't really return any Tweets twice, even without this option.

And if you get the same tweet multiple times, would that break anything, or is the issue simply that it would increase the total time of the extraction run? Or is there a nonzero probability that it would not terminate at all?

Processing the same Tweet multiple times can't really break anything, but it could stop the extraction process prematurely if you you are using -A and get the same Tweet with 4 images 3 times in a row. That's 4 regular downloads followed by 8 unnecessary skipped downloads.

To a certain extend, it also affects the total time of the extraction run, but only by a negligible amount. Anything that is not an HTTP request can more or less be ignored when considering time needed.

[nisehime]

Probably this also should affect performance, since gallery-dl would be checking the existence of a file on a disk or in archive multiple times. At least checking it in RAM should be faster.