Filter Impossible Domains

[Fir3God]

Please describe your feature request:

If using subfinder, often times lists can end up with impossible domains such as "-azure.tumblr.com"
Currently if you pass a list containing impossible domains, httpx will print them into your file which is problematic.
Using -fc wouldn't be the best solution as you may end up missing out on genuine results.

For example,

cat 1.txt
-azure.tumblr.com
tumblr.com

The following command prints:

httpx -l 1.txt -sc

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/              v1.1.4

                projectdiscovery.io

Use with caution. You are responsible for your actions.
Developers assume no liability and are not responsible for any misuse or damage.
https://tumblr.com [301]
https://-azure.tumblr.com [200]

Recommendation:

httpx should automatically filter out impossible domains from any given input/list.

[ehsandeep]

@Fir3God looks like that's the standard behavior, let me know if have more examples.

curl -sI https://-azure.tumblr.com:443
HTTP/2 200 
server: openresty
date: Mon, 20 Dec 2021 15:32:37 GMT
content-type: text/html; charset=UTF-8
vary: Accept-Encoding
vary: Accept-Encoding
x-rid: 41d77b1f46ce55a5ce9c7f66b233187b
p3p: CP="Tumblr's privacy policy is available here: https://www.tumblr.com/policy/en/privacy"
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
strict-transport-security: max-age=15552001
x-tumblr-user: -azure
x-tumblr-pixel-0: https://px.srvcs.tumblr.com/impixu?T=1640014357&J=eyJ0eXBlIjoidXJsIiwidXJsIjoiaHR0cDovLy1henVyZS50dW1ibHIuY29tLyIsInJlcXR5cGUiOjAsInJvdXRlIjoiLyJ9&U=HNMIOBCFDM&K=e8a055cf1023ece125c705d24b7ec6ebaa43e0ee8ccf66fb39ddfab359f62294--https://px.srvcs.tumblr.com/impixu?T=1640014357&J=eyJ0eXBlIjoicG9zdCIsInVybCI6Imh0dHA6Ly8tYXp1cmUudHVtYmxyLmNvbS8iLCJyZXF0eXBlIjowLCJyb3V0ZSI6Ii8iLCJwb3N0cyI6W3sicG9zdGlkIjoiMTU1Njg0ODk0NTEiLCJibG9naWQiOjQ1MzQ1MzAsInNvdXJjZSI6MzN9LHsicG9zdGlkIjoiMTQyNjk1MzEwMDYi
x-tumblr-pixel-1: LCJibG9naWQiOjQ1MzQ1MzAsInNvdXJjZSI6MzN9LHsicG9zdGlkIjoiMTIwOTkwNzc1NDciLCJibG9naWQiOjQ1MzQ1MzAsInNvdXJjZSI6MzN9LHsicG9zdGlkIjoiMTIwOTg5NTI5NTkiLCJibG9naWQiOjQ1MzQ1MzAsInNvdXJjZSI6MzN9LHsicG9zdGlkIjoiMTIwMTczOTk4NTUiLCJibG9naWQiOjQ1MzQ1MzAsInNvdXJjZSI6MzN9LHsicG9zdGlkIjoiMTIwMTY4NDYzOTUiLCJibG9naWQiOjQ1MzQ1MzAsInNvdXJjZSI6MzN9LHsicG9zdGlkIjoiMTE0NjYyMTE4NjgiLCJibG9naWQiOjQ1MzQ1MzAsInNvdXJjZSI6MzN9LHsicG9zdGlkIjoiMTEzNTI1NDA4MzAiLCJibG9naWQiOjQ1MzQ1MzAsInNvdXJjZSI6MzN9LHsicG9zdGlkIjoiMT
x-tumblr-pixel-2: EyMjMzMDIyMzkiLCJibG9naWQiOjQ1MzQ1MzAsInNvdXJjZSI6MzN9LHsicG9zdGlkIjoiOTUzNDc0NTU5OSIsImJsb2dpZCI6NDUzNDUzMCwic291cmNlIjozM31dfQ==&U=EGFCPFHODK&K=8e7b638bfe7dc6529f168a1b3796490d601fa8225eb8c9d5f244f3c00579c8d9
x-tumblr-pixel: 3
link: <https://64.media.tumblr.com/avatar_a0426893fae8_128.png>; rel=icon
x-ua-compatible: IE=Edge,chrome=1

[Fir3God]

@ehsandeep

I get a different response to you.

curl -I https://-azure.tumblr.com:443
curl: (6) Could not resolve host: -azure.tumblr.com

curl --version
curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3
Release-Date: 2020-01-08

[sullo]

It might be your nameservers? This is Google's.

╰─$ nslookup                                                                                                                                                                                                         1 
> server 8.8.4.4
Default server: 8.8.4.4
Address: 8.8.4.4#53
> -azure.tumblr.com
Server:		8.8.4.4
Address:	8.8.4.4#53

Non-authoritative answer:
Name:	-azure.tumblr.com
Address: 74.114.154.22
Name:	-azure.tumblr.com
Address: 74.114.154.18

FYI the above command works for me with curl 7.64.1 so it's not related to your version.

[Fir3God]

This is my result.

root@server:~# nslookup
> server 8.8.4.4
Default server: 8.8.4.4
Address: 8.8.4.4#53
> -azure.tumblr.com
nslookup: '-azure.tumblr.com' is not a legal IDNA2008 name (string start/ends with forbidden hyphen), use +noidnin

[sullo]

That might be your nslookup client complaining before it ever gets to the DNS server?

It seems like DNS rules are now across several RFCs (and I can't read them all!), but my understanding was always that names are disallowed from starting or ending with a hyphen.

However, given that the Google nameservers are successfully resolving names with leading hyphens, I'd say we should let the resolution attempt happen and possibly fail.

I am also able to access it with curl from my Mac, but not in Chrome. I don't really have an explanation as to why Google is resolving, but apparently some clients will accept it.