Save to Netlify

[argyleink]

i am trying to make a "Save to Netlify" feature but am a bit lost about how to hack this together. i edit using visbug and the dom changes in the dev tools, but when i run document.documentElement.outerHTML or document.body.innerHTML i still see the old (original) DOM printed out. is there somewhere i can just "download" all the post edited HTML/CSS?

thanks
Shawn

Originally posted by @sw-yx in #141 (comment)

[argyleink]

going to move our convo here since it's related but more specific!

i'm still not able to reproduce the error! can you give me more specific steps or conditions about your sandbox?

[argyleink]

#141 (comment) love how little you're going to use to get this task done!

[swyxio]

hm. ok so i tried it on Wikipedia again and now the visbug edited text shows up in the outerHTML dump. not sure what went wrong in my initial try. i can work with this. now to see how to send it to the netlify api.. and maybe add a button to visbug to do this

[swyxio]

looks like we basically need to POST to https://api.netlify.com/api/v1/drop/token with no payload to get a token, and then POST to https://api.netlify.com/api/v1/drop with a payload of both file and token:

const anonymousUpload = files =>
  withApi((dispatch, getState, api) =>
    api
      .dropToken() // https://api.netlify.com/api/v1/drop/token
      .then(response => response.data.token)
      .then(token =>
        api
          .dropFiles(getDigest(files), token) // https://api.netlify.com/api/v1/drop
          .then(response => ({ token, deploy: response.data }))
      )
      .then(({ token, deploy }) =>
        uploadDeployFiles(api, deploy, files, token)
          .then(() => api.waitForSiteDeploy(deploy.subdomain))
          .then(response => ({ token, site: response.data }))
      )
      .then(({ site, token }) => {
        setToken(token);
        dispatch(siteActions.siteSuccess(site));
        dispatch(loadScreenshot(site.name));
        dispatch(push(`/drop/${site.name}`));
      })
  );

getDigest is a list of files hashed by the rusha library's digest() method

[swyxio]

no bandwidth to do this now just leaving notes for myself

[swyxio]

netlify's open api swagger can be seen here: https://github.com/netlify/open-api https://open-api.netlify.com/#/default

[argyleink]

this upload to netlify feature is begging to be a plugin 😉
https://github.com/GoogleChromeLabs/ProjectVisBug/wiki/Plugins

visbug will call the "netlify" plugin function if/when the user types "/netlify" into the search bar, invoking this feature with the DOM, ready for pushing to netlify! could even open a new tab for them with the url loaded, ready to review and share. 💥

any change I can rekindle this flame and help ya push the feature in??!!

[swyxio]

100% lets do it! i’ll try to make a standalone demo of how to upload files to netlify without using the netlify ui without authentication. once i have that, we can lean on your visbug knowledge to connect the dots.

[argyleink]

branch with a netlify plugin added https://github.com/GoogleChromeLabs/ProjectVisBug/tree/netlify-plugin

[swyxio]

i investigated this today, and it seems like we really jealously guard that /drop/token endpoint - no third party site can request it so we can't automate that process (unauthenticated site hosting) directly. still we might be able to do it through more creative means but this isn't turning out as easy as i thought.

[argyleink]

what's the flow for getting a token?
what if i made a visbug account and let everyone piggy back off it (all plugin pushes use my token)? lol, when would i hit a scale limit?

[swyxio]

ok so i'm avoiding using that visbug account option first - i checked with someone on the backend team, and it turns out the endpoint was just a little opaque - it required a valid Referer field, that was all, and i just wasn't able to mock it on the front-end (still not sure if its a browser issue or i was just doing it wrong). We're now able to get a token using curl -X POST -H "Referer: https://app.netlify.com" https://api.netlify.com/api/v1/drop/token which is great!

I can mock that Referer field in Node, which is what I am doing now, but I'm not sure how we will go about this when inside VisBug. I assume that we will be able to solve this eventually, so I'm just proceeding for now to make a stripped down demo of the rest of the flow.

Step 2: the next step after getting the token is to sha1 all the files (internal link), and map it with their relative filepath in a POJO we call the "digest" (internal link). its also possible to do this for a zip file, which might be easier for visbug purposes.

so once we have the "digest", and the token, we can POST to https://api.netlify.com/api/v1/drop with the "digest" to create a "deploy":

fetch("https://api.netlify.com/api/v1/drop", {
      method: "POST",
      body: JSON.stringify({
        files: digest,
        token
      })
    });

(internal reference)

Step 3: we get back a reference to a "deploy" and use the Netlify js client to upload (internal ref)

Once this is done, its downhill from there. we get back a reference to the site with all the details we need.

Gonna pause on this today and pick it back up on some later date

---

if we have access to a node execution environment, we can use the Netlify CLI which makes this whole thing a lot easier: https://docs.google.com/document/d/1ACEYkiLiKwW6QEcZA7E9TMs9ju7xpo1WIErgEDxs3SE/edit

[argyleink]

sounds like a cloud function + a secret could make this all much less painful?..
how often does one need to fetch the referrer?

[swyxio]

you don't fetch the referrer, you set the Referer header on your request to get the deploy token. i reckon this has to be done every time someone Saves To Netlify from Visbug, unles we setup some more advanced persistence where it continually updates a single site (also possible, and frankly a much better experience, but we'd have to let the user tell us which site they want to use - maybe we can skip this step by adopting some sort of convention)

a cloud function + secret might work - would VisBug be able to chuck its data over to the cloud function? i'm not sure what underlying data structure VisBug uses

[argyleink]

VisBug works on the DOM 😉

we could totally setup a cloud function that owns a secret, question is, do i want to own and maintain it. even though it's serverless, doesnt mean it'll be free from hack attempts and abuse..

[swyxio]

ok, no problem, we'll do it the first way then, should be stateless and cost nothing.

just saw your comments on the original issue: #141 (comment)

how exactly will you serialize the diff? and is visbug set up to replay those diffs? because that would make things much easier... but im guessing you dont have any of this functionality yet right

[argyleink]

if visbug chooses the serialize/deserialize route, there's a few ways to attack it. and yep, in another branch i'm hacking on a pub/sub visbug api (which includes a serialization of changes) so other tools can either receive the changes visbug made, or visbug can consume them from an external caller and apply them.

it's an interesting problem, lots of ways to solve it, not enough time to pick any lol.