[BUG]Issue using multiple websitescrapper tools in an agent

[aliyasir]

Description

I am attempting to build a system that gathers information froma curated list of sites. I am using the default crewAI project as a base example.. When I give the agent a single webscrapertool for a particular site, it works fine. But when I provide it with two of these for different sites, the first works but the second fails to scrape anything, The agent proceeds with a few attempts and then ultimately returns no information at all, from the provided resources, even ignoring the first successful scrape.

I have tried different prompts and changes but the result is always the same. I am only providing it multiple websitescraper tools as a single tool does not accept multiple website addresses.

Steps to Reproduce

1. Setup an agent to extract information from websites
2. Setup the associated task accordingly
3. Provide it with two webscrapertools, each for a different site
4. Run the crew

Expected behavior

The agent should scrape multiple websites, and proceed to use the consolidated information as instructed in its prompt

Screenshots/Code snippets

CrewBase
class Knowledgebot():
"""Knowledgebot crew"""

# Learn more about YAML configuration files here:
# Agents: https://docs.crewai.com/concepts/agents#yaml-configuration-recommended
# Tasks: https://docs.crewai.com/concepts/tasks#yaml-configuration-recommended
agents_config = 'config/agents.yaml'
tasks_config = 'config/tasks.yaml'

techcrunch_scraper_tool = ScrapeWebsiteTool(website_url='https://techcrunch.com/')
venturebeat_scraper_tool = ScrapeWebsiteTool(website_url='https://venturebeat.com/')
wired_scraper_tool = ScrapeWebsiteTool(website_url='https://www.wired.com/')
mitreview_scraper_tool = ScrapeWebsiteTool(website_url='https://www.technologyreview.com/')

# If you would like to add tools to your agents, you can learn more about it here:
# https://docs.crewai.com/concepts/agents#agent-tools
@agent
def researcher(self) -> Agent:
	return Agent(
		config=self.agents_config['researcher'],
	#	tools=[self.techcrunch_scraper_tool, self.venturebeat_scraper_tool, self.wired_scraper_tool],
		tools=[self.mitreview_scraper_tool, self.techcrunch_scraper_tool],
		allow_delegation=False,
		verbose=True
	)

@agent
def reporting_analyst(self) -> Agent:
	return Agent(
		config=self.agents_config['reporting_analyst'],
	#	allow_delegation=True,
		verbose=True
	)

@agent
def editor(self) -> Agent:
	return Agent(
		config=self.agents_config['editor'],
	#	allow_delegation=True,
		verbose=True
	)

# To learn more about structured task outputs, 
# task dependencies, and task callbacks, check out the documentation:
# https://docs.crewai.com/concepts/tasks#overview-of-a-task
@task
def research_task(self) -> Task:
	return Task(
		config=self.tasks_config['research_task'],
	)

@task
def analysis_task(self) -> Task:
	return Task(
		config=self.tasks_config['analysis_task'],
		output_file='report.md'
	)


@task
def editing_task(self) -> Task:
	return Task(
		config=self.tasks_config['editing_task'],
		output_file='report.md'
	)

@crew
def crew(self) -> Crew:
	"""Creates the Knowledgebot crew"""
	# To learn how to add knowledge sources to your crew, check out the documentation:
	# https://docs.crewai.com/concepts/knowledge#what-is-knowledge

	return Crew(
		agents=self.agents, # Automatically created by the @agent decorator
		tasks=self.tasks, # Automatically created by the @task decorator
		process=Process.sequential,
		verbose=True,
		# process=Process.hierarchical, # In case you wanna use that instead https://docs.crewai.com/how-to/Hierarchical/
	)

Operating System

Windows 10

Python Version

3.12

crewAI Version

0.95.0

crewAI Tools Version

0.95.0

Virtual Environment

Venv

Evidence

The output looks something like this:

Agent: AI Senior Data Researcher

Using tool: Read website content

Tool Input:

"{}"

Tool Output:

I tried reusing the same input, I must stop using this action input. I'll try something else instead.

Agent: AI Senior Data Researcher

Final Answer:

No information could be retrieved regarding the latest developments in AI from the provided tool resources.

The Final Answer was simply generic text generated by LLM, until I asked to to return this if it fails with the data collection task.

Possible Solution

None

Additional context

None

[CalvinHMX]

Can you provide yourLLM specific information?

[aliyasir]

Hello,

Yes. I am using openAI gpt-4o-mini

[CalvinHMX]

你好

是的。 我正在使用 openAI gpt-4o-mini
My suggestion is whether you can try 4O or more advanced model

[CalvinHMX]

try use 4o