remove fullurl from req-res

[dogancanbakir]

This PR removes fullURL storage when using -store-response. Possible solution to #1317.

before:

$ cat test.txt
https://pastebin.com/raw/TYsVwM0n

GET /raw/TYsVwM0n HTTP/1.1
Host: pastebin.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20100101 Firefox/9.0
Accept-Charset: utf-8
Accept-Encoding: gzip


HTTP/1.1 200 OK
Connection: close
Transfer-Encoding: chunked
Age: 45
Cache-Control: public, max-age=1801
Cf-Cache-Status: HIT
Cf-Ray: 7ff435236ac3286e-OTP
Content-Type: text/plain; charset=utf-8
Date: Thu, 31 Aug 2023 09:22:11 GMT
Last-Modified: Thu, 31 Aug 2023 09:21:26 GMT
Server: cloudflare
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Xss-Protection: 1;mode=block

38
line1
this is a line containing HTTP/1.1 FOO BAR
line3
0

$ cat main.go
package main

import (
	"bufio"
	"bytes"
	"fmt"
	"net/http"
	"os"
)

func main() {
	data, _ := os.ReadFile("test.txt")
	br := bufio.NewReader(bytes.NewReader(data))
	req, err := http.ReadRequest(br)
	fmt.Println(err, req)
	fmt.Println()
	res, err := http.ReadResponse(br, req)
	fmt.Println(err, res)
}

$ go run .
malformed HTTP request "https://pastebin.com/raw/TYsVwM0n" <nil>

malformed HTTP response "" <nil>

after:

$ cat test.txt
GET /raw/TYsVwM0n HTTP/1.1
Host: pastebin.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20100101 Firefox/9.0
Accept-Charset: utf-8
Accept-Encoding: gzip

HTTP/1.1 200 OK
Connection: close
Transfer-Encoding: chunked
Age: 45
Cache-Control: public, max-age=1801
Cf-Cache-Status: HIT
Cf-Ray: 7ff435236ac3286e-OTP
Content-Type: text/plain; charset=utf-8
Date: Thu, 31 Aug 2023 09:22:11 GMT
Last-Modified: Thu, 31 Aug 2023 09:21:26 GMT
Server: cloudflare
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Xss-Protection: 1;mode=block

38
line1
this is a line containing HTTP/1.1 FOO BAR
line3
0

$ cat main.go
package main

import (
	"bufio"
	"bytes"
	"fmt"
	"net/http"
	"os"
)

func main() {
	data, _ := os.ReadFile("test.txt")
	br := bufio.NewReader(bytes.NewReader(data))
	req, err := http.ReadRequest(br)
	fmt.Println(err, req)
	fmt.Println()
	res, err := http.ReadResponse(br, req)
	fmt.Println(err, res)
}

$ go run .
<nil> &{GET /raw/TYsVwM0n HTTP/1.1 1 1 map[Accept-Charset:[utf-8] Accept-Encoding:[gzip] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:9.0) Gecko/20100101 Firefox/9.0]] {} <nil> 0 [] false pastebin.com map[] map[] <nil> map[]  /raw/TYsVwM0n <nil> <nil> <nil> <nil>}

<nil> &{200 OK 200 HTTP/1.1 1 1 map[Age:[45] Cache-Control:[public, max-age=1801] Cf-Cache-Status:[HIT] Cf-Ray:[7ff435236ac3286e-OTP] Content-Type:[text/plain; charset=utf-8] Date:[Thu, 31 Aug 2023 09:22:11 GMT] Last-Modified:[Thu, 31 Aug 2023 09:21:26 GMT] Server:[cloudflare] Vary:[Accept-Encoding] X-Content-Type-Options:[nosniff] X-Frame-Options:[DENY] X-Xss-Protection:[1;mode=block]] 0x400006a240 -1 [chunked] true false map[] 0x400012a000 <nil>}

[tarunKoyalwar]

lgtm !

$ ./httpx -u projectdiscovery.io -screenshot && nuclei -u output/response/projectdiscovery.io -passive

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/

		projectdiscovery.io

[INF] Current httpx version v1.3.4 (latest)
https://projectdiscovery.io

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.9.13

		projectdiscovery.io

[INF] Current nuclei version: v2.9.13 (latest)
[INF] Current nuclei-templates version: v9.6.2 (latest)
[INF] New templates added in latest release: 61
[INF] Templates loaded for current scan: 881
[INF] Targets loaded for current scan: 1
[metatag-cms] [] [info] output/response/projectdiscovery.io/8dc78052bc714ad90b93ebabae752a988f4807e4.txt [Framer 7aa0232]
[tech-detect:cloudflare] [] [info] output/response/projectdiscovery.io/8dc78052bc714ad90b93ebabae752a988f4807e4.txt

[Mzack9999]

This was an explicit choice to be consistent with https://github.com/tomnomnom/meg indexing. Indeed it's would break every RFC compliant request/response reader. What do you think if we put the full url at the very bottom of the file like this:

[request]

[response]
\n
\n
[full url]

In this way it should be ignored by parser as it should end processing the body respecting the content-length header or transfer encoding start/end markers.

[dogancanbakir]

@Mzack9999 Make sense.

$ cat test.txt
GET / HTTP/1.1
Host: scanme.sh
User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 15_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/101.0.4951.58 Mobile/15E148 Safari/604.1
Accept-Charset: utf-8
Accept-Encoding: gzip

HTTP/1.1 200 OK
Connection: close
Content-Length: 2
Content-Type: text/plain; charset=utf-8
Date: Tue, 05 Sep 2023 12:58:52 GMT

ok


https://scanme.sh

$ cat main.go
package main

import (
	"bufio"
	"bytes"
	"fmt"
	"net/http"
	"os"
)

func main() {
	data, _ := os.ReadFile("test.txt")
	br := bufio.NewReader(bytes.NewReader(data))
	req, err := http.ReadRequest(br)
	fmt.Println(err, req)
	fmt.Println()
	res, err := http.ReadResponse(br, req)
	fmt.Println(err, res)
}

$ go run .
<nil> &{GET / HTTP/1.1 1 1 map[Accept-Charset:[utf-8] Accept-Encoding:[gzip] User-Agent:[Mozilla/5.0 (iPhone; CPU iPhone OS 15_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/101.0.4951.58 Mobile/15E148 Safari/604.1]] {} <nil> 0 [] false scanme.sh map[] map[] <nil> map[]  / <nil> <nil> <nil> <nil>}

<nil> &{200 OK 200 HTTP/1.1 1 1 map[Content-Length:[2] Content-Type:[text/plain; charset=utf-8] Date:[Tue, 05 Sep 2023 12:58:52 GMT]] 0x40000d0240 2 [] true false map[] 0x40000f2000 <nil>}