Possible change to API

[ghost]

Kickstarter recently added a feature to drill down on the search. At about the same time a script that I had working stopped. Here is the core of what has stopped working. I accept possibility of operator error. Holidays prevent much more attention at the moment.

require 'kickscraper'

Kickscraper.configure do |config|
config.email = 'xxx'
config.password = 'xxx'
end

c = Kickscraper.client
rl = c.recently_launched_projects
rl.each {|p|
puts p.name
puts p.category.id
puts p.category.name
puts
}

[benrugg]

Hey @NolaMark, thanks for posting this. It looks like you're exactly right. Kickstarter seems to have changed (or removed) the urls for finding recently launched projects and also projects that are ending soon.

@markolson, if you have any time to do some detective work like you first did when you scouted out their API, it would be great to know if there are new urls to replace the old ones.

(I tried a few things in the brief time I have right now, but couldn't find any new urls)

[ghost]

WORKAROUND: I was able to get the info I wanted thru the new advanced search.
http://www.kickstarter.com/discover/advanced?sort=launch_date
will return all projects sorted by launch date

The category_id= will allow searching within a category, page= allows next group of 20 to be fetched
http://www.kickstarter.com/discover/advanced?page=16&category_id=16&sort=launch_date

On a whim, I appended format=json and got JSON response.
http://www.kickstarter.com/discover/advanced?page=16&category_id=16&sort=launch_date&format=json

The last part seems to open possibilities for non-api api like calls

[benrugg]

@NolaMark, that's great. Would you possibly have time to change the recently_launched and ending_soon methods so they use that advanced search?

[ghost]

I'll do what I can, but it is more likely a shortage of skill than a shortage of time that will be the limiting factor.

Also, recently_launched_projects had a (starting_at_timestamp = nil) parameter list that I've not [yet] discovered an equivalence in the advanced search, and the more_projects_available mechanism might require a kludge.

I'll share my discoveries, issues, and working notes if I feel like I get to a point where I can't generate the required code.

Edit: In case anyone else is looking into this.
I found that the following also work, so perhaps a url closer to original API is possible
http://www.kickstarter.com/discover/recently-launched?format=json
http://www.kickstarter.com/discover/ending-soon?format=json

[jamlen]

Wowzers, I really like the idea of calling a standard url (without requiring authentication) and just stating format=json as it means for us we remove the risk of having our accounts blocked for battering the system!

Maybe it's worth investigating if we can replace all or most of the api calls with these?

[markolson]

But where's the fun in that? :)
I'm firing up my proxy right now to dig through the API some more looking for changes.

[markolson]

Yep, the top level response from v1 no longer lists newest_projects, and I can't find anything like that by going through the app. I definitely like the idea of using non-authed stuff if possible, and combined with Issue #15 we might be able to work out a nice refactor for a lot of stuff.

[ghost]

Nice find @NolaMark! Using format=json or setting the Accept header to application/json works quite well for the advanced search. I crossed my fingers and tried that trick for several other requests through the web site (updates, comments, backers, main project pages, etc) and unfortunately I got HTTP 406 Not Acceptable everywhere except the advanced search. The site and API definitely seem to be evolving though, so perhaps the advanced search was just a prelude to more goodness coming.

[ghost]

In process of attempting to write new launched-recently and ending-soon, spec/project_spec failed because of differences project object as returned by find_project(id) and the same projects as returned by recently-launched. Further investigation reveals others
EDIT: the reload! mechanism may negate some of this as a problem, but that then requires refetching project thru api. Spoiling the ability to search not logged in.

• creator.urls.web.user
  • http versus https
• creator.urls.api.user
  • different signature
• urls.web
  • difference in http vs https
  • difference in value, missing from adv search results
  • message_creator
  • project_short
  • updates
• missing from adv search results
  • urls.api
  • "friends"
  • "is_starred":
  • "is_backing":
  • "updated_at":
  • "video":
  • "comments_count":
  • "updates_count":
  • "rewards":

[benrugg]

Dang, that's a lot of differences. Are you thinking that it's not worth coding around the web url, then?

[ghost]

I should be able push a feature branch later this week. But the implementation of load_more_projects and friends is likely to have a funky smell to it, since the urls.api returns results ascending from a parameterized starting point (cursor), where the urls.web returns results descending from the latest (back to the beginning of time, from what I can tell). Even if it allows people to continue to run their existing code unchanged, it is likely to conflict with any logic that worked with the baked in order from the original api.

That said, in a perfect world, as a consumer of the service, I'd prefer a versioned interface to match the reality of KS changing their calls and results. Perhaps more painful in the short run, but the gymnastics to maintain the look and feel of the original api seem like they will be a maintenance nightmare in the long run. (Maybe it's just the brain rot cause by punched cards and mag tape speaking, not so quick on the uptake with all these new fangled methodologies.)

[benrugg]

Yeah, I would definitely agree with that. A versioned interface would be great. If you want to spend the time on that, I think that would be the best way to move kickscraper forward!

I know that's a lot of work... I hope working on that sounds fun to you!

[ghost]

Is 'popular' broke now too? Or did I somehow muck it up? Is this error message new?
process_api_call
{:url=>"/v1/projects/popular", :params=>{}}
process_api_call
{
"error_messages": [
"The resource you are looking for does not exist."
],
"http_code": 404,
"disclaimer": "This endpoint is solely for the use of applications owned by Kickstarter Inc., any other use is against Kickstarter's terms and services."
}

WORKAROUND: http://www.kickstarter.com/discover/popular?format=json

[ghost]

This must be a very recent change. Calling the base API URL (https://api.kickstarter.com/v1/) used to give a bunch of links under urls.api. When I call it today, I only see self. That will break any call that doesn't stem from a specific user. It's looking like we'll have to use the API for authorized queries that can be traced back to a user, and web calls like your advanced search for all unauthorized 'discovery' type queries. At that point it almost sounds like two entirely different projects.

I'm curious what ideas the maintainers have about this.

[benrugg]

Holy crap, yeah, they've just made that change within the last 24 hours. And the disclaimer is new as well. Man, I knew that they could change their API whenever they wanted to, but I didn't actually expect Kickstarter to suddenly turn off access!

I would think that all the websites and services using Kickstarter's data would be a value to them. There is starting to be a whole ecosystem around Kickstarter, and by restricting that access, they're really thwarting that community!

I hope we can figure out a workaround that is within Kickstarter's terms.

[benrugg]

To your point @ajkerrigan, I haven't looked at the different API results enough to know if it would be best to use both the public urls and the authorized user ones for different things. If it's possible to use only the public urls, that would definitely be easier - even if that means that the data returned is different from the current version that we've been using.

If we do end up using both the public and signed in urls, I would still suggest that we keep it all within the one kickscraper project. My perspective is mostly just pragmatic. I think it's easiest to have Kickstarter data-gathering in one project so it can "black box" from the viewpoint of any other code using it. Does that make sense?

[markolson]

That disclaimer wording is pretty clear, and I'd rather not risk people getting their account banned for trying to get around it in some weasely way or another. There was a signature parameter on the URLs that the iPhone app was sending out, so this happening doesn't surprise me much.

My opinion would be that we pull out the existing API work (git will always have it for later) and refocus on just publicly accessible (or accessible with a user/pass/cookie) pages. Short term, I'm likely going to yank the rubygem, since it's not really functional anymore.

[benrugg]

Yeah, I think that's the way to go. If we can get it all functioning through the public URL's, let's just switch completely to that (or to whatever works).

By the way, now that the "popular" search in the API is broken, my project (www.jumpkick.me) is suddenly defunct. And since I spent the last 5 months making it, I'm pretty disappointed in Kickstarter's sudden change!

I'd like to get the new code working as quickly as possible. @markolson, what can I do to help?

[benrugg]

Oh yeah, one more quick thought. I'd say to leave the rubygem up while we change the code. It still functions for some of the methods, like searching for specific projects and users. And for anyone who has code deployed through something like heroku, it might break their entire ability to launch code if the gem isn't there. Is that ok with you guys?

[markolson]

Given the disclaimer, I've already yanked the gems. It's not something I'm comfortable distributing anymore, even if it breaks projects. That said, I've made a new v1 branch of this for archival purposes, because I suspect the scope of changes needed will necessitate a near total rewrite.

Short term, I'm going to make a few Issues were we can work on gathering information about replacement data - either via existing public JSON/Ajax requests or screen scraping, so that we can get a handle on what's where and what tools/frameworks we can use to gather it.

[benrugg]

Ok, yeah, that all sounds good. I've been looking at all the different url's and their responses, so I'll start adding some data to whatever issues you create.

[ghost]

I going forward with the work I was doing with the following changes in client.rb

    def initialize
      @user = nil
    end

    def process_api_call(request_for, additional_path, query = "", cursor = nil)
      abort

    def process_api_url(request_for, api_url, coerce_response = true)
      abort

and changing
./lib/kickscraper/connection.rb:34: :url => "https://api.kickstarter.com",
to www.kickstarter.com

I believe that will avoid all uses of api.kickstarter.com

[benrugg]

Sounds great. I will start adding some data to the other issue in a moment.

[benrugg]

So with the latest changes pushed to the branch we've been working on (https://github.com/markolson/kickscraper/tree/partially-using-public-search), it seems like that branch is essentially stable and working for all features that worked before Kickstarter's API change.

I would suggest that we could merge that branch into master and republish the gem. What do you guys think about that?

[benrugg]

Hey guys - anyone have an opinion on merging the new branch into master? @markolson, it might be especially up to you, since you're the owner of the gem.

As far as I can tell, this version should be a drop in replacement for the last stable version, and I would love to have it be publicly available again.

[ghost]

I don't have a strong opinion.
I don't have a big enough code base from before the API change to really tell if
thing are running smoothly or if there is a gremlin in the works with the newer
code.

NolaMark

On January 26, 2014 at 12:14 PM Ben Rugg [email protected] wrote:

Hey guys - anyone have an opinion on merging the new branch into master?
@markolson https://github.com/markolson , it might be especially up to you,
since you're the owner of the gem.

As far as I can tell, this version should be a drop in replacement for the
last stable version, and I would love to have it be publicly available again.

—
Reply to this email directly or view it on GitHub
#14 (comment) .

[benrugg]

Man oh man, I just realized that Kickstarter started enforcing ssl on their public search about 5 days ago. I've updated the branch to use ssl and also to follow redirects (because they were responding with a 301 redirect that the faraday connection was just ignoring).

[benrugg]

I'm going to create a pull request for the updated branch. We can continue our conversation there about whether to push all this code to master or not...

[benrugg]

We've solved enough of this to close this issue, I think. I just pushed the branch to master, and we can republish the gem now. Yay.