Installing ffmpeg

[MarqFJA87]

Installing and updating yt-dlp proved easy to do through the command line, but I'm confused as to how to properly install ffmpeg, since it appears crucial for downloading certain video files from Twitter; trying to do it through pip command like I did with yt-dlp only results in downloading a very outdated version.

I managed to obtain the latest release from here. Do I put the folder in Python's library of apps, where the folder of yt-dlp files is located?

[Hrxn]

Well, you can basically put the FFmpeg binaries in any directory which is present in your PATH, including one of the Python directories, yes.

[MarqFJA87]

Well, I did exactly that, and it while it seems to work fine with nominally GIF tweets (downloading them as mp4 as usual), it seems unable to handle links to YouTube videos. Eh, not much of an issue, but I think it's worth noting if it hasn't been reported before, in case it's actually easy to fix.

[Hrxn]

Okay, thanks.

Things tried so far:

• Can open and display the Tweet in an Incognito window
• Works with using yt-dlp directly

PS D:\> yt-dlp --ignore-config -F 'https://twitter.com/gggrande/status/1432472212313817094'
[twitter] 1432472212313817094: Downloading guest token
[twitter] 1432472212313817094: Downloading JSON metadata
[youtube] OI_osEFZuYI: Downloading webpage
[youtube] OI_osEFZuYI: Downloading android player API JSON
[youtube] OI_osEFZuYI: Downloading MPD manifest
[youtube] OI_osEFZuYI: Downloading MPD manifest
[info] Available formats for OI_osEFZuYI:
ID  EXT  RESOLUTION FPS |  FILESIZE   TBR PROTO | VCODEC        VBR ACODEC     ABR  ASR    MORE INFO
--- ---- ---------- --- - --------- ----- ----- - ----------- ----- --------- ---- ------- ---------------------
139 m4a  audio only     |  1.53MiB    48k https |                   mp4a.40.5  48k 22050Hz low, m4a_dash
140 m4a  audio only     |  4.05MiB   129k https |                   mp4a.40.2 129k 44100Hz medium, m4a_dash
251 webm audio only     |  4.49MiB   143k https |                   opus      143k 48000Hz medium, webm_dash
17  3gp  176x144    8   |  1.77MiB    56k https | mp4v.20.3     56k mp4a.40.2   0k 22050Hz 144p
160 mp4  256x144    30  |            108k dash  | avc1.4d400b  108k                        DASH video, mp4_dash
278 webm 256x144    30  |             95k dash  | vp9           95k                        DASH video, webm_dash
133 mp4  426x240    30  |            242k dash  | avc1.4d400c  242k                        DASH video, mp4_dash
242 webm 426x240    30  |            220k dash  | vp9          220k                        DASH video, webm_dash
134 mp4  640x360    30  |  3.64MiB   116k https | avc1.4d401e  116k                        360p, mp4_dash
18  mp4  640x360    30  |  7.43MiB   237k https | avc1.42001E  237k mp4a.40.2   0k 44100Hz 360p
243 webm 640x360    30  |            405k dash  | vp9          405k                        DASH video, webm_dash
135 mp4  854x480    30  |           1155k dash  | avc1.4d4014 1155k                        DASH video, mp4_dash
244 webm 854x480    30  |            752k dash  | vp9          752k                        DASH video, webm_dash
136 mp4  1280x720   30  |  8.50MiB   271k https | avc1.4d401f  271k                        720p, mp4_dash
22  mp4  1280x720   30  |            399k https | avc1.64001F  399k mp4a.40.2   0k 44100Hz 720p
247 webm 1280x720   30  |           1505k dash  | vp9         1505k                        DASH video, webm_dash
PS D:\>

Interestingly, it also does not work for me in gallery-dl:

PS D:\> gallery-dl --ignore-config --verbose 'https://twitter.com/gggrande/status/1432472212313817094'
[gallery-dl][debug] Version 1.19.0
[gallery-dl][debug] Python 3.10.0 - Windows-10-10.0.19042-SP0
[gallery-dl][debug] requests 2.26.0 - urllib3 1.26.7
[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/gggrande/status/1432472212313817094'
[twitter][debug] Using TwitterTweetExtractor for 'https://twitter.com/gggrande/status/1432472212313817094'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/2/timeline/conversation/1432472212313817094.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweet=true&count=100&ext=mediaStats%2ChighlightedLabel HTTP/1.1" 200 3124
[twitter][info] No results for https://twitter.com/gggrande/status/1432472212313817094
PS D:\>

I tried it with my standard config as well (contains authentication for Twitter), but this does not work either.
So, seems like the invocation of yt-dlp by gallery-dl is somehow off?

As far as I see, this should be the interesting part (with "raw-options": {'verbose': true}):

PS D:\> gallery-dl --verbose 'https://twitter.com/gggrande/status/1432472212313817094'
Debug  : gallery-dl -> Version 1.19.0
Debug  : gallery-dl -> Python 3.10.0 - Windows-10-10.0.19042-SP0
Debug  : gallery-dl -> requests 2.26.0 - urllib3 1.26.7
Debug  : gallery-dl -> Starting DownloadJob for 'https://twitter.com/gggrande/status/1432472212313817094'
Debug  : twitter -> Using TwitterTweetExtractor for 'https://twitter.com/gggrande/status/1432472212313817094'
Debug  : urllib3.connectionpool -> Starting new HTTPS connection (1): twitter.com:443
Debug  : urllib3.connectionpool -> https://twitter.com:443 "GET /i/api/2/timeline/conversation/1432472212313817094.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&send_error_codes=true&simple_quoted_tweet=true&count=100&ext=mediaStats%2ChighlightedLabel HTTP/1.1" 200 3129
Debug  : twitter -> Using download archive 'D:\Test\gldl-archive-twitter.db'
Debug  : twitter -> Active postprocessor modules: [ClassifyPP]
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] yt-dlp version 2021.10.10
[debug] Python version 3.10.0 (CPython 64bit) - Windows-10-10.0.19042-SP0
[debug] exe versions: ffmpeg n4.4-154-g79c114e1b2-20210929, ffprobe n4.4-154-g79c114e1b2-20210929
[debug] Optional libraries: Cryptodome, mutagen, sqlite, websockets
[debug] ANSI escape support: stdout = True, stderr = True
[debug] Proxy map: {}
Debug  : downloader.ytdl -> [debug] [twitter] Extracting URL: https://twitter.com/i/web/status/1432472212313817094
Debug  : downloader.ytdl -> [twitter] 1432472212313817094: Downloading guest token
Error  : downloader.ytdl -> ERROR: [twitter] 1432472212313817094: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
Error  : downloader.ytdl ->   File "C:\Users\Hrxn\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\extractor\common.py", line 698, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "C:\Users\Hrxn\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\YoutubeDL.py", line 3228, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "C:\Users\Hrxn\AppData\Local\Programs\Python\Python310\lib\urllib\request.py", line 525, in open
    response = meth(req, response)
  File "C:\Users\Hrxn\AppData\Local\Programs\Python\Python310\lib\urllib\request.py", line 634, in http_response
    response = self.parent.error(
  File "C:\Users\Hrxn\AppData\Local\Programs\Python\Python310\lib\urllib\request.py", line 563, in error
    return self._call_chain(*args)
  File "C:\Users\Hrxn\AppData\Local\Programs\Python\Python310\lib\urllib\request.py", line 496, in _call_chain
    result = func(*args)
  File "C:\Users\Hrxn\AppData\Local\Programs\Python\Python310\lib\urllib\request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

Error  : download -> Failed to download ytdl:https://twitter.com/i/web/status/1432472212313817094
PS D:\>

But first of all: The title of this discussion is clearly wrong. This is apparently not about FFmpeg, so can you please change the title to something more in line with the actual issue here?

[Hrxn]

@mikf

Just noticed something else, although not directly related. Forgive me for posting it in here anyway, but before I forget this...

[2021-10-20__16:53:34][error] �[0;31mERROR:�[0m [twitter] 1432472212313817094: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
[2021-10-20__16:53:34][error] Failed to download ytdl:https://twitter.com/i/web/status/1432472212313817094
[2021-10-20__17:03:41][error] �[0;31mERROR:�[0m [twitter] 1432472212313817094: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
[2021-10-20__17:03:41][error] Failed to download ytdl:https://twitter.com/i/web/status/1432472212313817094
[2021-10-20__17:07:07][error] �[0;31mERROR:�[0m [twitter] 1432472212313817094: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on  https://github.com/yt-dlp/yt-dlp . Make sure you are using the latest version; see  https://github.com/yt-dlp/yt-dlp  on how to update. Be sure to call yt-dlp with the --verbose flag and include its complete output.
[2021-10-20__17:07:07][error] 

There are some ANSI escape codes in my logfile... I use both custom log and logfile options (can post this part of the config here also, if you want it), but the logfile format options seems to be the correct ones ({asctime}{levelname} {message}), with my custom time formatting etc...
But, there are somehow some ANSI codes in the {message}, it seems.
I mean, I have these in my options for log, but not for logfile ?

[MarqFJA87]

It was originally about ffmpeg, but we managed to unwittingly discover an issue with a different component of the process. Unfortunately, I can't change the title on my end.

[mikf]

@Hrxn the ANSI codes are from youtube-dl/yt-dlp. Adding "no_color": true to downloader.ytdl.raw-options should disable this.

You need to enable the cards option to download the youtube video from https://twitter.com/gggrande/status/1432472212313817094 with gallery-dl.

I've also found a bug in the download progress indicator when attempting to download this video. It doesn't account for youtube-dl not reporting a total_bytes in its progress hook.

[downloader.ytdl][debug] Traceback
Traceback (most recent call last):
  File "/home/mike/dev/python/gallery_dl/downloader/ytdl.py", line 122, in _download_video
    ytdl.process_info(info_dict)
  File "/home/mike/.local/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 1970, in process_info
    partial_success = dl(fname, new_info)
  File "/home/mike/.local/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 1915, in dl
    return fd.download(name, info)
  File "/home/mike/.local/lib/python3.9/site-packages/youtube_dl/downloader/common.py", line 366, in download
    return self.real_download(filename, info_dict)
  File "/home/mike/.local/lib/python3.9/site-packages/youtube_dl/downloader/dash.py", line 48, in real_download
    success, frag_content = self._download_fragment(ctx, fragment_url, info_dict)
  File "/home/mike/.local/lib/python3.9/site-packages/youtube_dl/downloader/fragment.py", line 104, in _download_fragment
    success = ctx['dl'].download(fragment_filename, fragment_info_dict)
  File "/home/mike/.local/lib/python3.9/site-packages/youtube_dl/downloader/common.py", line 366, in download
    return self.real_download(filename, info_dict)
  File "/home/mike/.local/lib/python3.9/site-packages/youtube_dl/downloader/http.py", line 352, in real_download
    return download()
  File "/home/mike/.local/lib/python3.9/site-packages/youtube_dl/downloader/http.py", line 301, in download
    self._hook_progress({
  File "/home/mike/.local/lib/python3.9/site-packages/youtube_dl/downloader/common.py", line 374, in _hook_progress
    ph(status)
  File "/home/mike/.local/lib/python3.9/site-packages/youtube_dl/downloader/fragment.py", line 246, in frag_progress_hook
    self._hook_progress(state)
  File "/home/mike/.local/lib/python3.9/site-packages/youtube_dl/downloader/common.py", line 374, in _hook_progress
    ph(status)
  File "/home/mike/dev/python/gallery_dl/downloader/ytdl.py", line 140, in _progress_hook
    info["total_bytes"],
KeyError: 'total_bytes'
[download][error] Failed to download 1432472212313817094_1.webm

[Hrxn]

@mikf Thanks for the hint that these ANSI sequences came from yt-dlp!

Also, I had "cards": true in my config file for Twitter, so I think that this alone does not trigger the bug, maybe?