docs/integrations/document_loaders/firecrawl/

[giscus[bot]]

docs/integrations/document_loaders/firecrawl/

FireCrawl crawls and convert any website into LLM-ready data. It crawls all accessible subpages and give you clean markdown and metadata for each. No sitemap required.

https://python.langchain.com/docs/integrations/document_loaders/firecrawl/

[VK-T]

params = {
"maxDepth": 1,
"scrapeOptions": {
"formats": ["markdown"]
}}

loader = FireCrawlLoader(
api_key=API-KEY, url="https://www.prnewswire.com/news-releases/news-releases-list/", mode="crawl", params=params
pages = []
for doc in loader.lazy_load():
pages.append(doc)

Getting error as:

AttributeError Traceback (most recent call last)
Cell In[29], line 2
1 pages = []
----> 2 for doc in loader.lazy_load():
3 pages.append(doc)

File ~/anaconda3/envs/taxonomy/lib/python3.11/site-packages/langchain_community/document_loaders/firecrawl.py:64, in FireCrawlLoader.lazy_load(self)
59 raise ValueError(
60 f"Unrecognized mode '{self.mode}'. Expected one of 'crawl', 'scrape'."
61 )
62 for doc in firecrawl_docs:
63 yield Document(
---> 64 page_content=doc.get("markdown", ""),
65 metadata=doc.get("metadata", {}),
66 )

AttributeError: 'str' object has no attribute 'get'