Generate contributor SVG widgets #4

Beanow · 2019-07-15T20:19:16Z

This PR adds an early version of the SourceCred widgets.
This widgets tool is intended for various styles of "widgets" like markdown tables or images like this SVG, see sourcecred/sourcecred#1047.

Specifically it will generate an SVG with the all-time top 50 contributors over 4.5 cred (based on the new timeline calculation), which we can use for placing in READMEs.

The order of avatars is descending by cred, taking into account our weights defined at weights.toml

Example from sfosc/sfosc today:

This file would be made available at
https://sfosc.org/sourcecred/widgets/sfosc-sfosc-contributors.svg
And update along with the scores on a daily cronjob, so we can include this file from the README.md

Beanow · 2019-07-18T18:10:53Z

@vsoch @nothingismagick are you able to help review this?

vsoch · 2019-07-18T18:12:28Z

yep!

vsoch

After this first round of comments, I'd be happy to test this out.

vsoch · 2019-07-18T18:13:53Z

mkweights/yarn.lock

@@ -0,0 +1,8 @@
+# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.


Do we want to include the yarn lock file? At least for jekyll, I usually don't add the Gemfile.lock because it adds more issues than helps.

Most of the time for node projects yes you should. Because the package.json dependencies allow you to install newer semver feature or patch updates at random.

"dependencies": { "toml": "^3.0.0" }

(^3.0.0 matches toml 3.1.3 just fine)

Committing a lock file ensures you have consistent environments between developers / CI jobs.

I agree. Push the lockfile.

Alright, 2 vs. 1 here! I should actually be advocating to include it, reproducibility etc.. Maybe this wouldn't be an issue if we interact via a container anyway.

vsoch · 2019-07-18T18:14:46Z

scripts/local-generate-svg.sh

+
+export SOURCECRED_DIRECTORY=$(pwd)/sourcecred_data
+SOURCECRED_GITHUB_TOKEN=`cat secrets/token`
+REPOS=`cat repositories.txt`


We should probably check here that this secrets file exists.

And then of course exit if the variables aren't what we expect.

The local-* scripts are mostly debugging helpers. But you're right it would be useful for others if it's consistent and produces clear errors like this. Going to improve those a little.

vsoch · 2019-07-18T18:17:15Z

scripts/local-generate-svg.sh

+
+cd mkweights
+yarn
+cd ..


Hmm, so I guess my worry here is that there are quite a few dependencies. For example, not everyone has yarn. and node is a beast. Is this generated on a user computer, locally? I'm thinking either we can provide a Dockerfile (and container?) to issue commands with (and the one dependency is docker of course) or the generation can be done in CI that has a recipe to install everything. Thoughts?

Hmm, so I guess my worry here is that there are quite a few dependencies.

That is definitely true. I'm involved with a couple of discussions to make depending on sourcecred simpler. For example sourcecred/widgets#8

For example, not everyone has yarn. and node is a beast.

With regards to yarn. Sourcecred has a yarn lockfile committed but not the npm package-lock.json counterpart. Thus your dependencies would be yarn + npm + node if I had kept the npm lock variant. That's why I switch to yarn here.

Is this generated on a user computer, locally?

As local-* is intended for debugging it's useful to have node installed. And yarn is pretty stable so npm install -g yarn would be sufficient if you do have node and npm.

The official node docker container has yarn installed. In .drone.yml I use node:10 as that's the active LTS node version. Should you prefer avoiding node you might try using node:10 or node:12 (current stable and tested by sourcecred test suite) and including the repository root as a bind volume. Personally I've not tested the local-* with this yet. But it's essentially what our Drone build does.

I'm thinking either we can provide a Dockerfile (and container?) to issue commands with (and the one dependency is docker of course) or the generation can be done in CI that has a recipe to install everything. Thoughts?

That is what this PR will do. The .drone.yml file configures the job run on a daily interval. That calls scripts/rebuild-site.sh with node:10 as the main docker image to do so.

I am not worried about lots of dependencies per sé - but then I have all of these installed on every machine I work on.

If local is for debugging, and there is a configuration with Drone to run programatically otherwise, I'm good here. What we don't want is an entirely un-reproducible thing.

Yes the main script used for CI is scripts/rebuild-site.sh
Which is also usable locally.

vsoch · 2019-07-18T18:20:42Z

scripts/local-generate-svg.sh

+yarn install
+yarn backend
+for repo in $REPOS; do
+	SOURCECRED_GITHUB_TOKEN=$SOURCECRED_GITHUB_TOKEN node bin/sourcecred.js load $repo --weights ../.weights.json


Could you walk me through what is going on here?

node mkweights create some original file, (and weights.toml is the configuration of weights?

yarn install installs, what does backend do?

then we iterate through the repos, and we issue a load command. Are the repos already included in weight.json (and we are loading) or are we adding data to weights.json for the repo?

Probably some comments in these sections would fit the bill, so others / future you can come back and remember what it does :)

node mkweights create some original file, (and weights.toml is the configuration of weights?

Yes. Sourcecred has a json weights format that is more verbose than the toml file format I'm using. I created the toml format as a more intuitive one to discuss. And lines 7-11 are to convert the toml format to the json format.

yarn install installs, what does backend do?

Yarn has a cli syntax yarn <script_name> so this is the equivalent of npm run <script_name>. In this case the script backend is defined here. Which will use webpack and babel to transliterate the sourcecred source code (using flow and esm) to javascript code that can be executed by node 10 or 12.

then we iterate through the repos, and we issue a load command. Are the repos already included in weight.json (and we are loading) or are we adding data to weights.json for the repo?

The load command fetches github data and runs the sourcecred scoring algorithm. We pass the weights arguments as configuration values. These weights essentially quantify: "How important is a comment? How important is a commit? How important is a PR?" ... etc.

That is the context to what these commands entail.

In broad stroke the workflow is:

Convert our simplified weights.toml to the weights.json format used by SourceCred.

yarn install + yarn backend to set up SourceCred.

sourcecred/bin/sourcecred.js load ... to fetch github data and score it using our weights.

sourcecred/bin/sourcecred.js scores ... to export those scores in a format we can consume.

widgets/bin/contributor-wall-svg.js to consume those scores and generate the SVG.

One final question here - just want to check that there is no kind of PI in the GitHub data?

That is a good question. My assumption would be that because no special permissions are required to fetch user info it could only be public data. However many people people use their full name and an email and make it public. So that may be PI either way. I don't think they are stored, only the ID and username for graphs and avatars and username for widgets. Though I would need to check.

vsoch · 2019-07-18T18:21:05Z

scripts/local-generate-svg.sh

+yarn
+export SVG_MIN_CRED=4.5
+export SVG_MAX_USERS=50
+for repo in $REPOS; do


This one is fairly intuitive!

Based on #4 feedback.

Beanow · 2019-07-18T22:06:48Z

Added the extra comments and checks to the local-* debugging scripts.

nothingismagick

I like it - but still think that having more than just an avatar would be nice (here is an example from a webpack page):

I know that having the name is actually something that would belong to the sourcecred/widgets project - but wanted to note that here.

nothingismagick · 2019-07-19T06:40:20Z

.gitmodules

@@ -1,3 +1,6 @@
 [submodule "sourcecred"]
 	path = sourcecred
 	url = https://github.com/Beanow/sourcecred.git
+[submodule "widgets"]


With submodules we need to be explicit about how to clone / fork and install submodules in the documentation. Some people might not immediately know that they need to recurse:

$ git clone --recurse-submodules

DEVELOPING.md does make mention of it recommending git submodule init > git submodule update.

There is an open issue to do away with the need for submodules upstream, or rather simplify depending on sourcecred: sourcecred/widgets#8

nothingismagick · 2019-07-19T06:40:49Z

mkweights/yarn.lock

@@ -0,0 +1,8 @@
+# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.


I agree. Push the lockfile.

nothingismagick · 2019-07-19T06:42:10Z

scripts/local-generate-svg.sh

+
+cd mkweights
+yarn
+cd ..


I am not worried about lots of dependencies per sé - but then I have all of these installed on every machine I work on.

Beanow · 2019-07-19T09:26:53Z

@nothingismagick agreed, variants with more information would be useful and implemented in the widgets package.

Feedback welcome, as expanding on this would be great for the utility of the package. And achieving the example you gave should be fairly easy if I dig into the SVG spec a little bit.

For SFOSC perhaps we should take it up to an issue in the main repo to discuss what we would like to have?

vsoch · 2019-07-19T14:58:31Z

I personally like the square graphics (without names) but I'm open to have my mind changed! @nothingismagick what do you think having the names adds?

nothingismagick · 2019-07-19T15:32:28Z

It removes the cult of personality. And furthermore, there are people with no personally identifiable, so you end up with stuff like this:

vsoch · 2019-07-19T15:38:55Z

I kind of like the grid, even with the missing pictures here and there :) It looks like a zoom in of one of those paintings / pictures that is made entirely of other pictures. Whoa! That would be an awesome integration for sourcecred - to make a picture based on matching pixels in some template image.

Beanow · 2019-08-07T16:10:56Z

@vsoch your review is marked requesting changes. Was there a particular point I hadn't addressed yet?

vsoch

Yes, it looks good! I had a bunch of original comments, all of which you addressed, and I mostly just forgot to approve :) Approved!

Beanow · 2019-08-07T16:42:59Z

Thanks!

I gave this another test run before merging and noticed the output CNAME file contained 'sfosc.org' instead of sfosc.org and fixed this.

It's updated daily along with the sourcecred data. See sfosc/sourcecred#4

Beanow added 2 commits July 15, 2019 22:02

Add widgets submodule.

4508471

Refactored scripts to include SVG widgets and sfosc/sourcecred repo.

12bcd4e

vsoch suggested changes Jul 18, 2019

View reviewed changes

Include more checks and comments to local-* scripts.

a81100e

Based on #4 feedback.

nothingismagick approved these changes Jul 19, 2019

View reviewed changes

vsoch approved these changes Aug 7, 2019

View reviewed changes

Bugfix, quote default CNAME correctly

176e089

Beanow merged commit ffa8cad into master Aug 7, 2019

Beanow added a commit to Beanow/sfosc that referenced this pull request Aug 7, 2019

Include the sourcecred widget in our readme.

b44cfea

It's updated daily along with the sourcecred data. See sfosc/sourcecred#4

Beanow mentioned this pull request Aug 7, 2019

Include the sourcecred widget in our readme sfosc/sfosc#111

Merged

Beanow deleted the pr/contributors-svg branch August 22, 2019 10:48

Beanow added a commit to sfosc/sfosc that referenced this pull request Aug 27, 2019

Include the sourcecred widget in our readme. (#111)

a889ada

It's updated daily along with the sourcecred data. See sfosc/sourcecred#4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate contributor SVG widgets #4

Generate contributor SVG widgets #4

Beanow commented Jul 15, 2019 •

edited

Loading

Beanow commented Jul 18, 2019

vsoch commented Jul 18, 2019

vsoch left a comment

vsoch Jul 18, 2019

Beanow Jul 18, 2019

nothingismagick Jul 19, 2019

vsoch Jul 19, 2019

vsoch Jul 18, 2019

vsoch Jul 18, 2019

Beanow Jul 18, 2019

vsoch Jul 18, 2019

Beanow Jul 18, 2019

nothingismagick Jul 19, 2019

vsoch Jul 19, 2019

Beanow Jul 20, 2019

vsoch Jul 18, 2019

Beanow Jul 18, 2019

vsoch Jul 19, 2019

Beanow Jul 20, 2019

vsoch Jul 18, 2019

Beanow commented Jul 18, 2019

nothingismagick left a comment

nothingismagick Jul 19, 2019

Beanow Jul 19, 2019

nothingismagick Jul 19, 2019

nothingismagick Jul 19, 2019

Beanow commented Jul 19, 2019

vsoch commented Jul 19, 2019

nothingismagick commented Jul 19, 2019

vsoch commented Jul 19, 2019

Beanow commented Aug 7, 2019

vsoch left a comment

Beanow commented Aug 7, 2019

		@@ -0,0 +1,8 @@
		# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.

Generate contributor SVG widgets #4

Generate contributor SVG widgets #4

Conversation

Beanow commented Jul 15, 2019 • edited Loading

Beanow commented Jul 18, 2019

vsoch commented Jul 18, 2019

vsoch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Beanow commented Jul 18, 2019

nothingismagick left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Beanow commented Jul 19, 2019

vsoch commented Jul 19, 2019

nothingismagick commented Jul 19, 2019

vsoch commented Jul 19, 2019

Beanow commented Aug 7, 2019

vsoch left a comment

Choose a reason for hiding this comment

Beanow commented Aug 7, 2019

Beanow commented Jul 15, 2019 •

edited

Loading