Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Want to capture the "div" which is created after scrolling the page #383

Open
N-pharmarack opened this issue Feb 15, 2022 · 1 comment
Open

Comments

@N-pharmarack
Copy link

Subject of the issue

I want to capture the items from a site, but half of the items are loaded when I scroll the page(link of the page : https://www.91mobiles.com/mobile-price-list-in-india?page=1 ). A new div ( page_scroll_1) is created after I scroll the the page, and the items I want to capture are there in that div.

I have tried ".scrollTo" method from 'x-ray-phantom' but It didn't work

Your environment

  • version of node: 16.13.0
  • version of npm: 8.1.0

Steps to reproduce

const Xray      = require('x-ray');
const puppeteer = require('puppeteer');
var phantom     = require('x-ray-phantom');

var phantom_opts = {
    webSecurity: false,
    images: false,
    weak: false
};

var x = Xray().driver(phantom(phantom_opts, function (nightmare, done) {
    done
        .useragent("Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.111 Safari/537.36")
        .goto(nightmare.request.req.url)
        .scrollTo(0, 500)
        .wait(3000)
        .scrollTo(0, 1000)
        .wait(3000)
        .scrollTo(0, 1500)
        .wait(3000)
        .scrollTo(0, 2500)
        .wait(3000)
        .scrollTo(0, 3000)
        .wait(3000)
        .scrollTo(0, 3500)
        .wait(3000)
        .scrollTo(0, 4000)
        .wait(3000)
}));


x('https://www.91mobiles.com/mobile-price-list-in-india?page=1', '.finder_snipet_wrap',[{
  mobile_model: '.name',
  Price: '.price',
  Url: '.name@href'
}])
.write('results.json')

Expected behaviour

It should capture all the item with '.finder_snipet_wrap'

Actual behaviour

Capture's only half items, and skips all the other '.finder_snipet_wrap' that is being created after page scrolling

@jmcoder1
Copy link

jmcoder1 commented Sep 7, 2023

Did you figure this out? I have a very similar issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants