Inaccurate web scraping data - how to improve accuracy?

[lhexin]

Hi

I've just tried using Flowise to interrogate a website (using the Cheerio Web Scraper). I am asking it to find summer offers on Starbucks.com as an example.

The scraper works, and it is pulling in data from Starbucks but it didn't get the context of the offer right, i.e. the offer detail is wrong.

I am assuming there might be a way to improve the accuracy by changing up how the recursive character text splitter works? But I have no idea how.

Any help is appreciated!

[HenryHengZJ]

Try to use HtmlToMarkDown Text Splitter, because using recursive text splitter you will still have a lot of html gibberish embeddings.

Hopefully this helps - https://docs.flowiseai.com/use-cases/web-scrape-qna

[collasanta]

hey Henry, there are a lot of websites that have nested sitemaps under domain/sitemap.xml, I tested the feature scraping by XML and it seems to handle only sitemaps that have all the URLs in the root of sitemap.xml
Do you have plans to add support for that? thanks

[HenryHengZJ]

we dont have a near term plan for that as currently users dont have a clear way of selecting which links to use and which one not to. crawling nested sitemap could take really long time and doesnt fit well to current UI/UX.

we will improve this in future when we introduce workflow pipeline

[collasanta]

yes, I was trying it with websites with hundreds of pages and I got no clue of what is happening in the process when i got long loading times in the chat ui (also the feature that logs in all page that is going to be scrapped in the console did not work for me)

Thankss

[tomique34]

What about using Crawl4AI web scrapper ? I have read one article where they described Crawl4AI as very fast scrapper with ability scrape complete website in moments and have it in md format. Can you implement it into flowise workflow sometimes ?

[Charlotte-br560]

Improving the accuracy of web scraping data involves refining your scraping technique and understanding the website structure you're targeting. Here are a few tips to enhance accuracy:

Inspect the Website Structure: Before you can scrap the HTML structure of the website to understand how the data is organized. This will help you identify the elements you need to target accurately.

Use Targeted Selectors: Utilize precise CSS selectors or XPath expressions to target specific elements containing the desired data. Avoid broad selectors that may capture irrelevant information.

Handle Dynamic Content: Some websites load content dynamically using JavaScript. Ensure your scraper can handle dynamic content by using tools like Puppeteer or Selenium or by analyzing network requests to replicate AJAX requests.

Implement Error Handling: Build error-handling mechanisms into your scraper to gracefully handle unexpected situations, such as missing data or changes in website layout.

Regularly Update Scraping Logic: Websites frequently undergo updates that may affect scraping accuracy. Review and update your scraping logic regularly to adapt to any changes.

Test and Iterate: Test your scraper with different scenarios and iterate on your scraping logic based on the results. This iterative process can help refine the accuracy of your scraping.

By applying these strategies and experimenting with different approaches, you can improve the accuracy of your web scraping data extraction.

[ChatGurus]

Sounds like chatgpt ;)
But yes, this is more complicated than just adding a module in the workflow. It might also be a good idea to scrape the website seperately and upload the resulting document (after inspecting and optimizing it)

[toi500]

Apify is the solution for pro scraping. There is even a built-in integration to upsert the data to Pinecone.

Also, you can schedule your runs, so, per example, you can scraper 1 or 2 times per week any dynamic web.

If I have time, I would like to make a short tutorial about it. It is a perfect companion for Flowise.

[Vortigern-source]

When you make one please post it here,

thanks