Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conditional redirect? #280

Open
FezVrasta opened this issue Dec 30, 2017 · 2 comments
Open

Conditional redirect? #280

FezVrasta opened this issue Dec 30, 2017 · 2 comments

Comments

@FezVrasta
Copy link

Subject of the issue

I'm scraping a page that requires the user to click on a "thanks" button before I can access the info I'm looking for.
If the page has already been "thanked", an "unthank" button will be shown instead. I don't want the scraper to click on the "unthank" obviously.

Right now I'm using a code of this kind:

const x = Xray({
  filters: {
	// this filter makes the "thank/unthank" URL call become always a "thank" call
    // doing so I avoid to unthank anything that was previously thanked
    grateful: url => url && url.replace('withdrawthank', 'thank'),
  },
}).driver(driverWithAuth);

x(
  websiteUrl,
  '.subject > div > span',
  [
    {
      title: 'a',
      url: 'a@href',
      magnets: x(
        'a@href',
        x('.thank_you_button:first-child a@href | grateful', [
          'a@href',
        ])
      ),
    },
  ]
)

The problem with this setup is that I will always have to navigate to an additional page even when the page is already "thanked".

Ideally I'd like to be able to tell x-ray to not navigate to anything but just stay in the same page if the provided URL is maybe undefined or false.

Doing so I could have my filter do url.includes('thank') ? url : false, which will make the scraper hit the thank call only if needed.

Is it possible? If not, could this be added?

@lathropd
Copy link
Collaborator

lathropd commented Apr 17, 2019

Can your share here (or DM me on Twitter) the url of the site in question?

@xochilpili
Copy link

Any solution for this?
I am in the same situation, if there is a result or if not, then i can decide what to do in the filter, ie:

//my filter.js
export.isThere = (value)=>{
    return typeof value === undefined || value === '' ? value : ' No value';
}

x('sample_url','body',[{
  title: '.title | isThere ',
  description: '.title+div | isThere'
}]);

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants