Custom DNS resolver callback?

[weusego]

When you perform a lot of req requests for crawling you hit the problem that DNS timeouts take a long time and slow you down.

This results in net/opError or similar.

Our idea to get around that is with a custom DNS resolver callback that decides if req should go to network layer BEFORE it actually does that in net standard packages.

So far we have not found this idea implemented.

Of course we could filter this on a domain level before even starting the req requests, as a workaround.

But it would maybe be interesting to have req give you the flexibility to do that instead.

An example custom resolver could work as follows

1/ receive the FQDN
2/ look up FQDN in a custom hosts file
3/ return IP4 or IP6 OR return NOHOST error

essentially the basic DNS client interface,nbut without traversal and timeouts...

...the maintenance of that file would happen elsewhere

[imroc]

This is very interesting. In theory, it can be implemented in req, but it requires a certain amount of development work. I will add this to TODO.

If you need it urgently, you can use client.SetDail to customize the dail function, where you can customize dns resolve and connection establishment.