plans to improve embedded lib usage?

[mikerott]

Hi. My team is using Nuclei embedded as a library (documented here).

As I was integrating the sample code, I noticed several things;

1. the sample code assumes a "catalog" of templates hosted on the filesystem:

   home, _ := os.UserHomeDir()
   catalog := catalog.New(path.Join(home, "nuclei-templates"))
   

   In my case, the lib may be embedded in an AWS lambda. Since Catalog is not an interface, I have no way to override it to acquire templates from anywhere other than the filesystem. Perhaps Catalog can be refactored to be an interface/impl to support non-filesystem sources of templates?
2. Why does ExecutorOptions have a struct field TemplateID (and other Template*), implying one template, but also a Catalog field, implying many templates? Should this be refactored for clarity?

I guess I should ask a more general question about embedded usage, which I realize may lead to a much deeper requirement of refactoring for the embedded use case. Ultimately, I'd like sample code that:

1. does not assume a filesystem is present
2. uses a "Catalog" interface so that templates can be provided by a bring-your-own-Catalog implementation
3. shows how to load all protocols for all templates

This is what I am ultimately running at this point. Notice I'm not showing functions like parseSelfContainedRequests and convertRequestToProtocolsRequest but due to how I had to embed Nuclei, I had to copy these private (lower-case) functions out of the Nuclei codebase. I'll gladly take your critique!

func runScan(addresses []string, template templates.Template) (resultEvent []*output.ResultEvent, failureEvent []*output.InternalEvent, err error) {

	// this implementation code is copied (and modified) from
	// https://github.com/projectdiscovery/nuclei/blob/master/DESIGN.md#using-nuclei-from-go-code

	// beware of what is happening here.  By yaml unmarshaling the template ourselves, and other calls below,
	// we only support HTTP requests, which may be good enough for now.  To fully parse a template, we'd rather use
	// "github.com/projectdiscovery/nuclei/v2/pkg/templates".Parse, but that function takes a filesystem path, not a []byte
	// of yaml.  The other options are:
	// * copy most (all?) of "github.com/projectdiscovery/nuclei/v2/pkg/templates" here (which would probably spiral out of control with other private funcs also needing to be copied)
	// * edit the code in the vendor folder (risky!  next upgrade of nuclei would wipe it out and you wouldn't know)
	// * start a discussion to start the process of contributing nuclei code improvements for the embedded use case: https://github.com/projectdiscovery/nuclei/discussions and start improving it

	cache := hosterrorscache.New(30, hosterrorscache.DefaultMaxHostsCount)
	defer cache.Close()

	mockProgress := &testutils.MockProgressClient{}
	reportingClient, _ := reporting.New(&reporting.Options{}, "")
	defer reportingClient.Close()

	outputWriter := writer{}

	defaultOpts := types.DefaultOptions()
	defaultOpts.Timeout = 2
	defaultOpts.Retries = 0
	if protocolstate.Dialer == nil { // protect against multiple unnecessary Init calls
		err = protocolinit.Init(defaultOpts)
		if err != nil {
			return nil, nil, err
		}
	}

	interactOpts := interactsh.NewDefaultOptions(&outputWriter, reportingClient, mockProgress)
	interactClient, err := interactsh.New(interactOpts)
	if err != nil {
		return nil, nil, err
	}
	defer interactClient.Close()

	executerOpts := protocols.ExecuterOptions{
		Output:          &outputWriter,
		Options:         defaultOpts,
		Progress:        mockProgress,
		IssuesClient:    reportingClient,
		RateLimiter:     ratelimit.New(150),
		Interactsh:      interactClient,
		HostErrorsCache: cache,
		Colorizer:       aurora.NewAurora(true),
		ResumeCfg:       types.NewResumeCfg(),
		TemplateID:      template.ID,
		TemplateInfo:    template.Info,
	}

	requests := convertRequestToProtocolsRequest(template.RequestsHTTP)

	template.Executer = executer.NewExecuter(requests, &executerOpts)

	if template.Executer != nil {
		if err = template.Executer.Compile(); err != nil {
			return nil, nil, err
		}
		template.TotalRequests = template.Executer.Requests()
	}
	if template.Executer == nil && template.CompiledWorkflow == nil {
		return nil, nil, fmt.Errorf("The template executor is still nil")
	}

	parseSelfContainedRequests(&template)

	engine := core.New(defaultOpts)
	engine.SetExecuterOptions(executerOpts)

	workflowLoader, err := parsers.NewLoader(&executerOpts)
	if err != nil {
		return nil, nil, err
	}
	executerOpts.WorkflowLoader = workflowLoader

	for i, address := range addresses {
		if !strings.HasPrefix(address, "http") {
			addresses[i] = fmt.Sprintf("http://%s", address)
		}
	}

	input := &inputs.SimpleInputProvider{Inputs: addresses}
	_ = engine.Execute([]*templates.Template{&template}, input)
	engine.WorkPool().Wait() // Wait for the scan to finish

	return outputWriter.GetResults(), outputWriter.GetFailures(), nil
}

[Ice3man543]

Hi @mikerott, so yes your point is valid, to allow more diverse use-cases the best course of action would be to convert the Catalog to an interface.

This will fix all your questions about first two points. The third point can be easily added by changing the templates.Parse function to accept an io.Reader. Regarding why the Options structure contains a Catalog, it is for loading workflow templates later down the execution chain and other purposes. The TemplateID is populated with individual template IDs by creating a copy of the ExecuterOptions.

We can create an issue to convert Catalog to interface and add a io.Reader and proceed from there, doing changes as required.

[mikerott]

That would be quite helpful! I appreciate you taking constructive feedback. I anticipate my team may offer code contributions as well, and am familiar with how to fork and PR back into the repository. Looking forward to continuous improvement!

[mikerott]

I opened an issue: #2214