Nuclei showing invalid matches upon 1st run

[marcelo321]

Nuclei version:

Using Nuclei Engine 2.7.2 (latest)

Current Behavior:

I am running a custom nuclei template over a small amount of hosts that I know nuclei is going to give a valid match (to test the template), so I run it over 30 hosts, and it properly alerts all the 30 hosts as "vulnerable". The problem is, then I run over all my targets (aproximately 800k hosts) and it only detects 2 hosts (the 800k hosts includes all the hosts given in the short scan + a lot more, it is a tech detect).

Expected Behavior:

So I don't know what is happening, it's the third time I run nuclei over the large list of hosts and it is not detecting all of them, it is detecting only a few.

Steps To Reproduce:

Not sure how to explain it or help reproduce it. the template I am running is:

id: github-detection

info:
  name: github-detect
  author: me
  severity: info
  reference:
    - https://none.com
  tags: tech

requests:
  - method: GET
    path:
      - "{{BaseURL}}/"

    matchers:
      - type: word
        words:
          - 'Server: GitHub.com'
        part: header

Something must be off since there can't be only 2 subdomains hosted in github (I know there are more from previous scans).

[ehsandeep]

@marcelo321 you can use the below options while running your scan and debug the logs / raw response once scan gets completed.

   -sresp, -store-resp           store all request/response passed through nuclei to output directory
   -tlog, -trace-log string  file to write sent requests trace log
   -elog, -error-log string  file to write sent requests error log

[marcelo321]

The thing is I am kinda afraid if this happens to other templates, that when mass scanning something fails. I can use -store-resp in this scan and then grepping for Server: GitHub.com manually to see if it is nuclei's fault or what is happening, but I am not sure if that's the solution. The big scan includes the hosts of the "test scan", so unless I am being blocked by the hosts (strange because it is only 1 request per host) then nuclei is failing somewhere.

[ehsandeep]

@marcelo321 that's definitely not a solution, but more about knowing the problem that needs to be solved, I'm moving this to a discussion as there is no sufficient information to look further, please add more details to a discussion once you get to know about the issue using the flag shared above.

[marcelo321]

Running it again one more time with the flag you recommended, yesterday I lowered the threads to 120 and got 6 results instead of 2. I really think something messes up with nuclei, might be the long list, the amount of threads or the big list of hosts.. I will update tomorrow, but it's strange that -store-resp flag created executable files instead of normal text files.

[marcelo321]

Ok so I run it again with the -store-resp flag, it only detected like 6 hosts again.

But it doesn't resolve my doubt, it should have detected at least +30 cases, (probably around 200). I am not sure what is failing but something definitely is, can anyone check this one? run this template on all of their hosts to check if it is working properly nuclei at large scale?

[marcelo321]

Ok last update, I run it with different VPSs, at -rl 100 and I got 100 matches with 1 vps, 6 matches with other vps and 80 matches with the last vps.

Nuclei is not working well with large amount of hosts, something is breaking and I am not sure why, someone should re-check this.

[ehsandeep]

@marcelo321 there is a known issue from the last release that has been reported by a few users, i.e displaying invalid matches upon 1st run and can not be reproduced after 2nd run, for the same reason we did not manage to reproduce but something we are exploring at this point.

since you manage to reproduce a few times, do you have any feedback to reproduce this scenario? are you running these on fresh vps to experience this behavior or something else that you can share could be useful to confirm and fix this issue.