[requestException] cURL error 60: SSL certificate problem: unable to get local issuer certificate

[GabMic]

Hi, all.
I am trying to figure out what am i doing wrong here, maybe a misconfigured ini setting or something, but am always getting:

cURL error 60: SSL certificate problem: unable to get local issuer certificate.

I am using this in laravel (latest version for now) 8.26
php version is 7.4.2
guzzle version is 7.2.0
spatie/crawler is 6.0

made a console command to run it:

public function handle()
    {
       $controller = new CrawlerController();
       $controller->index('https://taamtov.net');

    }

My class looks like this:

class CrawlerController extends CrawlObserver
{
    private $pages = [];

    public function index($site = null) {

         Spider::create()
            ->setCrawlObserver(new CrawlerController())
            ->setMaximumDepth(3)
            ->startCrawling($site);
    }

    public function willCrawl(UriInterface $url) {
        echo "Now crawling: " . (string) $url . PHP_EOL;
    }

    public function crawled(UriInterface $url, ResponseInterface $response, ?UriInterface $foundOnUrl = null) {
        $u = $url->getPath();
        $code = $response->getStatusCode();
        print_r([$u, $code]);

        $this->pages[] = [
            'path'=>$u,
            'code'=> $code
        ];

        print_r($this->pages);
    }

    public function crawlFailed(UriInterface $url, RequestException $requestException, ?UriInterface $foundOnUrl = null)
    {
        print_r(["url" => $url->getPath(), "requestException" =>  $requestException->getMessage(), 'foundOnUrl' => $foundOnUrl]);
    }
}

openssl in php.ini is activated, if that is something the you need to know.
what am i doing wrong here?
Thanks.

[rajakhoury]

here one way to do it. Extend the Crawler class and override the GuzzleHttp Client options.

e.g

MyCrawler::create()
	->setCrawlObserver(new CrawlObserver())
	->startCrawling('https://domain.com');

Then in MyCrawler do this:

use GuzzleHttp\RequestOptions;
use Spatie\Crawler\Crawler as BaseCrawler;
 
class MyCrawler extends BaseCrawler
{
	protected static array $defaultClientOptions = [
		RequestOptions::COOKIES => true,
		RequestOptions::CONNECT_TIMEOUT => 10,
		RequestOptions::TIMEOUT => 10,
		RequestOptions::ALLOW_REDIRECTS => false,
		RequestOptions::HEADERS => [
			'User-Agent' => self::DEFAULT_USER_AGENT,
		],
		'verify' => false // This is what you need to add  
	];
}

[rajakhoury]

ignore this answer. you don't need to extend the crawler class just pass the client options as an argument to create.

[devHarandi]

it's worked.

thanks

Crawler::create([
RequestOptions::COOKIES => true,
RequestOptions::CONNECT_TIMEOUT => 10,
RequestOptions::TIMEOUT => 10,
RequestOptions::ALLOW_REDIRECTS => false,
RequestOptions::HEADERS => [
'User-Agent' => "*"
],
'verify' => false
])
->ignoreRobots()
->setCrawlObserver(new class extends \Spatie\Crawler\CrawlObservers\CrawlObserver{

            public function crawled(UriInterface $url, ResponseInterface $response, ?UriInterface $foundOnUrl = null): void
            {
                $domCrawler = new \Symfony\Component\DomCrawler\Crawler(
                    (string) $response->getBody()
                );

                $test = $domCrawler
                    ->filter('#main-content')
                    ->first()
                    ->text();

                echo $test;
            }

            public function crawlFailed(UriInterface $url, RequestException $requestException, ?UriInterface $foundOnUrl = null): void
            {
                echo $requestException->getMessage() . PHP_EOL;
            }
        })
        ->setDelayBetweenRequests(500)
        ->startCrawling($url);

[dogeow]

\Spatie\Crawler\Crawler::create([
            RequestOptions::ALLOW_REDIRECTS => true,
            RequestOptions::TIMEOUT => 30,
            RequestOptions::VERIFY => false, // <----------It is this
        ])
            ->setCrawlObserver(new CustomCrawlerObserver())
            ->executeJavaScript()
            ->setCurrentCrawlLimit(1)
            ->startCrawling($url);