All requests are now detected by Youtube as BOT

[NannoSilver]

The problem started about 1 hour ago. All requests to download audio are detected as BOT, even at my local machine with a residential IP (never detected as bot before).

Here is the code to test the issue:

import sys
from pytubefix import YouTube


list_of_video_ids = ['PSPb2AAl4O0', 'sTzkmEmUPIc', 'O0PRQamYAZk', 'F6ayhSvu9Rk', 'n95Cqx5V4rQ', '5J0yEnJ4B1o']

for youtube_episode_id in list_of_video_ids:
    try:
        yt = YouTube(f'https://www.youtube.com/watch?v={youtube_episode_id}')
        stream = yt.streams.get_by_itag(140)
        stream.download(filename=f'{youtube_episode_id}.m4a', skip_existing=True, max_retries=0)
    except:
        error_type, e, error_traceback = sys.exc_info()
        print(f'Failed with Error: {e}')

Output:

Failed with Error: PSPb2AAl4O0 This request was detected as a bot. Use `use_po_token=True` or switch to WEB client to view. See more details at https://github.com/JuanBindez/pytubefix/pull/209
Failed with Error: sTzkmEmUPIc This request was detected as a bot. Use `use_po_token=True` or switch to WEB client to view. See more details at https://github.com/JuanBindez/pytubefix/pull/209
Failed with Error: O0PRQamYAZk This request was detected as a bot. Use `use_po_token=True` or switch to WEB client to view. See more details at https://github.com/JuanBindez/pytubefix/pull/209
Failed with Error: F6ayhSvu9Rk This request was detected as a bot. Use `use_po_token=True` or switch to WEB client to view. See more details at https://github.com/JuanBindez/pytubefix/pull/209
Failed with Error: n95Cqx5V4rQ This request was detected as a bot. Use `use_po_token=True` or switch to WEB client to view. See more details at https://github.com/JuanBindez/pytubefix/pull/209
Failed with Error: 5J0yEnJ4B1o This request was detected as a bot. Use `use_po_token=True` or switch to WEB client to view. See more details at https://github.com/JuanBindez/pytubefix/pull/209

Looks like youtube found a way to detect pytubefix or they changed something else that is triggering the BOT exception.

---

Tested with:
Pytubefix 8.12.0
Python 3.9
Windows 11 and Ubuntu 20.04

[jacknakoom]

I suddenly ran into this issue too, even though the IPs are residential and everything was fine just half an hour ago. The library definitely needs cookie support, like yt_dlp, or else it’s game over for this library

[Nelson0213]

Same here

[JuanBindez]

Yesterday I had started some tests related to headers, Cookies and IPs but I don't have time at the moment, if anyone discovers something let us know

[JuanBindez]

I suddenly ran into this issue too, even though the IPs are residential and everything was fine just half an hour ago. The library definitely needs cookie support, like yt_dlp, or else it’s game over for this library

Do you know how it's done in them?

[Yangseok]

Same here

[davidtwchn]

It took me two days to install the node on my Linux server, upgrade pytubefix to 8.12.0, use client=WEB, let node generate poToken, and download videos smoothly.

[davidtwchn]

It took me two days to install the node on my Linux server, upgrade pytubefix to 8.12.0, use client=WEB, let node generate poToken, and download videos smoothly.

The biggest challenge in installing node on Linux is upgrading GLIBC. The server was close to crash twice.

[NannoSilver]

I did a preliminary test, without poToken, with all possible clients and this is the result.

Code to test with Pytubefix 8.12.0:

import sys
from pytubefix import YouTube


youtube_episode_id = 'PSPb2AAl4O0'

list_of_clients = ['WEB', 'WEB_EMBED', 'WEB_MUSIC', 'WEB_CREATOR', 'WEB_SAFARI', 'ANDROID', 'ANDROID_MUSIC', 'ANDROID_CREATOR', 'ANDROID_VR', 'ANDROID_PRODUCER', 'ANDROID_TESTSUITE', 'IOS', 'IOS_MUSIC', 'IOS_CREATOR', 'MWEB', 'TV', 'TV_EMBED', 'MEDIA_CONNECT']

for client in list_of_clients:
    try:
        yt = YouTube(f'https://www.youtube.com/watch?v={youtube_episode_id}', client=client)
        stream = yt.streams.get_by_itag(140)
        stream.download(filename=f'{youtube_episode_id}___{client}.m4a')
        print(f'Success with client {client}\n\n\n\n')
    except:
        error_type, e, error_traceback = sys.exc_info()
        print(f'Failed client: {client} with Error: {e}\n\n\n\n')

I run the code multiple times and the result is erratic.

The clients ANDROID and WEB_EMBED always download the audio successfully.
In some runs one or another client could download too.

Here is the outcome of one of such runs:

Unable to run botGuard. Skipping poToken generation
Unable to run botGuard. Skipping poToken generation
The WEB client requires PoToken to obtain functional streams, See more details at https://github.com/JuanBindez/pytubefix/pull/209
The WEB client requires PoToken to obtain functional streams, See more details at https://github.com/JuanBindez/pytubefix/pull/209
Failed client: WEB with Error: HTTP Error 403: Forbidden




Unable to run botGuard. Skipping poToken generation
Unable to run botGuard. Skipping poToken generation
The WEB_EMBED client requires PoToken to obtain functional streams, See more details at https://github.com/JuanBindez/pytubefix/pull/209
The WEB_EMBED client requires PoToken to obtain functional streams, See more details at https://github.com/JuanBindez/pytubefix/pull/209
Success with client WEB_EMBED




WEB_MUSIC client returned: This video is not available
Switching to client: MWEB
Failed client: WEB_MUSIC with Error: HTTP Error 403: Forbidden




Failed client: WEB_CREATOR with Error: PSPb2AAl4O0 requires login to view, YouTube reason: Please sign in




Unable to run botGuard. Skipping poToken generation
Unable to run botGuard. Skipping poToken generation
The WEB_SAFARI client requires PoToken to obtain functional streams, See more details at https://github.com/JuanBindez/pytubefix/pull/209
The WEB_SAFARI client requires PoToken to obtain functional streams, See more details at https://github.com/JuanBindez/pytubefix/pull/209
Failed client: WEB_SAFARI with Error: HTTP Error 403: Forbidden




Unable to run botGuard. Skipping poToken generation
Unable to run botGuard. Skipping poToken generation
The ANDROID client requires PoToken to obtain functional streams, See more details at https://github.com/JuanBindez/pytubefix/pull/209
The ANDROID client requires PoToken to obtain functional streams, See more details at https://github.com/JuanBindez/pytubefix/pull/209
Success with client ANDROID




Failed client: ANDROID_MUSIC with Error: PSPb2AAl4O0 requires login to view, YouTube reason: Please sign in




Failed client: ANDROID_CREATOR with Error: PSPb2AAl4O0 requires login to view, YouTube reason: Please sign in




Failed client: ANDROID_VR with Error: PSPb2AAl4O0 This request was detected as a bot. Use `use_po_token=True` or switch to WEB client to view. See more details at https://github.com/JuanBindez/pytubefix/pull/209




Failed client: ANDROID_PRODUCER with Error: HTTP Error 403: Forbidden




ANDROID_TESTSUITE client returned: This video is not available
Switching to client: MWEB
Failed client: ANDROID_TESTSUITE with Error: HTTP Error 403: Forbidden




Failed client: IOS with Error: PSPb2AAl4O0 This request was detected as a bot. Use `use_po_token=True` or switch to WEB client to view. See more details at https://github.com/JuanBindez/pytubefix/pull/209




Failed client: IOS_MUSIC with Error: PSPb2AAl4O0 requires login to view, YouTube reason: Please sign in




Failed client: IOS_CREATOR with Error: PSPb2AAl4O0 requires login to view, YouTube reason: Please sign in




Failed client: MWEB with Error: HTTP Error 403: Forbidden




Success with client TV




Failed client: TV_EMBED with Error: PSPb2AAl4O0 is unavailable




Failed client: MEDIA_CONNECT with Error: HTTP Error 403: Forbidden





Process finished with exit code 0

[NannoSilver]

It took me two days to install the node on my Linux server, upgrade pytubefix to 8.12.0, use client=WEB, let node generate poToken, and download videos smoothly.

Are you using proxies?

[crapomat]

Using ' client=WEB ' lets me download videos.

[duong-dt]

Using client='WEB' on my machine (residential IP, Linux Mint 21.3) works fine as well, both with or without NodeJS.

[NannoSilver]

Using ' client=WEB ' lets me download videos.

Unable to run botGuard. Skipping poToken generation The WEB client requires PoToken to obtain functional streams, See more details at #209

I just tested it, and I’m getting this error

It is not an error, but a warning as the message itself explains.
It skips the poToken generation and move forward without it.

[nannerpusser]

Since this is active within the hour and closely related to a youtube search library question I can't seem to get an answer to, I'm gonna ask here and hope for the best:

Is it possible to search for playlists without using the official Data v3 API and dev key? I have written several yt CLI search tools and used yewtube for years now, but in the last several months every library I've tried (python-search-youtube, pafy, etc. etc.) is no longer able to get return results for playlist-only searches (like for music). I was able to make my own tool again but had to use the official API and it sucks huge balls.

[felipeucelli]

Hello @NannoSilver

Test this PR and tell us the result: #425

[davidtwchn]

It took me two days to install the node on my Linux server, upgrade pytubefix to 8.12.0, use client=WEB, let node generate poToken, and download videos smoothly.

Are you using proxies?

No, my linux server is in Los Angeles.I use this server to download youtube videos and then fetch these videos from this server.

[NannoSilver]

Hello @NannoSilver

Test this PR and tell us the result: #425

Hi @felipeucelli

I applied the modified files of #425 on Pytubefix 8.12.0.

It is running in my production server and so far it is working very well.
I am observing the logs regarding the rate of detection as BOT, but so far seems it is similar to how it was until weeks ago.

Thank you for fixing one more issue!

[NannoSilver]

Hello @NannoSilver

Test this PR and tell us the result: #425

Here is an additional feedback after a few days using the PR #425 on top of Pytubefix 8.12.0.

The requests detected as BOT reduced massively, but some of my proxies detected by Youtube as bot remain blocked.
I requested the proxy seller to replace the blocked proxies and the new ones are working fine with a very low rate of detection as bot. I am using "datacenter" proxies that are cheaper.

[GVCoder09]

Yesterday I had started some tests related to headers, Cookies and IPs but I don't have time at the moment, if anyone discovers something let us know

As a temporary solution, disable this check in the check_availability function (pytubefix/__main__.py class YouTube)

...
            elif status == 'LOGIN_REQUIRED':
                if reason == (
                        'Sign in to confirm your age'
                ):
                    raise exceptions.AgeRestrictedError(video_id=self.video_id)
                elif reason == (
                        'Sign in to confirm you’re not a bot'
                ):
                    return
                    raise exceptions.BotDetection(video_id=self.video_id)
                else:
                    raise exceptions.LoginRequired(
                        video_id=self.video_id, reason=reason)

Oddly enough it works. Despite ignoring the error, the download continues

[Sandwich2001]

Oddly enough it works. Despite ignoring the error, the download continues

Won't continuous requests end up in Youtube blocking your IP anyway?

[nannerpusser]

Oddly enough it works. Despite ignoring the error, the download continues

Won't continuous requests end up in Youtube blocking your IP anyway?

From experience, yes and no. Absolutely yes if your IP is a "known quantity" and they've had to take any kind of action on it or just had it flagged internally for whatever reason. Most people don't really understand that even using a VPN or proxy to smash the API isn't very effective in maintaining anonymity for a long list of fingerprinting reasons that Google has about you, not to mention most people are finger printed alongside various IPs their fingerprint has been seen with , etc. That said, if you do things above board and are using a real API key with a developer account, you get a shit load of leeway and a lot of abuse before an actual IP or account ban. Sure, you get rate-throttled and free accounts only get 10k or so request tokens (see also: does not mean 10k API calls) but even when you do stupid shit like I do with a free or paid account and fuck up a for loop in your method for hitting and parsing the API and use 10,000 tokens in 45 seconds, they don't give a fuck. You just come back tomorrow.

[GVCoder09]

Won't continuous requests end up in Youtube blocking your IP anyway?

Several days have passed, a dozen or two videos have been downloaded and nothing has been blocked.

[Ixiko]

Initially, I was able to avoid BotDetection by changing the IP address, e.g. by temporarily switching off the VPN or changing the VPN server. In the meantime, this no longer helps. At least not immediately. I wait at least 1-2 hours and then the download starts again. I had the impression that the more often I download, the longer I have to wait. But this may also be due to my impatience and could therefore be more a feeling than a measurable fact.