httpx scan progress keeps 100% for the long time

[madneal]

httpx version: v1.6.6

I used httpx to scan lots of targets. But the progress is 100% for a long time(more than 10 hours).

[madneal]

it had kept 100% over 100 hours

[GeorginaReeder]

Hi @madneal ! Thanks for this - 100 hours is definitely a long time and it does look like that's a lot of targets.

@dwisiswant0 - do we know if anything can be done about this?

[madneal]

It became 100% at the begining, about 10 hours. But it was still not finished after over 120 hours.

[dwisiswant0]

Can you provide more context and details? How do you run it; the command you use, and if you're using it as a library; what does the code look like?

[madneal]

func Scan(target) []string {
        result := make([]string, 0)
	options := runner.Options{
		Methods:         "GET",
		InputTargetHost: targets,
                 Showstats: true,
		OnResult: func(r runner.Result) {
			result = append(result, r.URL)
			if r.Err != nil {
				fmt.Printf("[Err] %s: %s\n", r.Input, r.Err)
				return
			}
			fmt.Printf("%s %s %d\n", r.Input, r.Host, r.StatusCode)
		},
	}

	if err := options.ValidateOptions(); err != nil {
		log.Fatal(err)
	}

	httpxRunner, err := runner.New(&options)
	if err != nil {
		log.Fatal(err)
	}
	defer httpxRunner.Close()

	httpxRunner.RunEnumeration()
        return result
}

the target is quite large, each item is IP:PORT. I have tried a small amount of target. If the amount is over 100, there will be only several seconds latency between 100% status and finished status.

[dwisiswant0]

You're feeding the target with nearly 2mio hosts, so you need to adjust the Threads option.

[madneal]

the thread I use is 2000. But the point is the progres is incorrect. What's meaning of the progress of 100%. It's still strange to cost over 120 hours for 2mio hosts.

[madneal]