Chaosbot V1 experiment conclusions and future direction

[amoffat]

The chaosbot v1 experiment has run its course. The bot is currently dead. There was a very strong force of activity from launch until very recently, and I would like to outline what I believe are the weaknesses in v1 that led to this point:

1. Lack of a complete development environment that mirrored production. This one was crucial for ensuring that changes to be made were deployed smoothly.
2. Lack of tests from the start. This would have resulted in fewer breaking bugs being merged in.
3. Lack of a standardized way of committing private data. For example, reddit or twitter credentials. This only came later, and had some hiccups.
4. Lack of segregation between critical and non-critical code. It was very easy to break the system because critical code was alongside non-critical code.

Where to go from here

This is up to the community. If the community is interested in a version 2 that tackles those previous 4 points, I think we should take a step back and address them at a slower pace. This could involve including contributors on the chaosbot project, to directly commit to master. The idea would be that we build towards a v2, then when it has the basics required to self-sustain, we release it into the wild again.

Thoughts, criticisms, critiques? Everything is valid, this was a project driven by us all, after all :)

[PlasmaPower]

Lack of segregation between critical and non-critical code. It was very easy to break the system because critical code was alongside non-critical code.

This is what I'd really want for a v2. Ideally, it'd even be split into separate repositories. We could then do stuff like have different voting thresholds for different repos.

[dbpokorny]

@amoffat @PlasmaPower may I suggest some Kierkegaard?

Specifically Fear and Trembling / Repetition.

[rudehn]

Sad to hear. This was a lot of fun!

I think part of the problem was that there was a lot of activity when simple updates were required (like updating a readme), however when more heavylifting needed to be done (such as the database), not as many people felt like they could contribute. Furthermore items like the database created a bottleneck in development; there were PRs that had to wait until the db had been implemented before their completion could be finished.

[dbpokorny]

@amoffat @PlasmaPower @rudehn bot running on chaosbot.org

Actually, it is trying to run, crashing a lot

I backed out the "reset git history if I crash" behavior because it makes testing difficult

Edit: repo is dbpokorny/chaos

[dbpokorny]

update: I don't know how to run supervisor or remove the code that depends on supervisor, so until that happens, I'll have to restart the bot manually every time it merges a PR

[amoffat]

@rudehn good points. A database with an easy way to do migrations seems to be another pain point to address

[dbpokorny]

@amoffat what is the motivation for using supervisor in the first place?

[amoffat]

@dbpokorny Open a new issue for any off-topic questions. I want to keep this conversation on-topic

[dbpokorny]

/vote close

[mdcfe]

Would v2 be based on v1, implementing the above points, or would it be an entirely new codebase?

Also, I think v2 should require full documentation of APIs and proper error handling in PRs to prevent the crashes and confusion which occurred in v1.

[PlasmaPower]

I think it should be a rewrite. If we implement the above points, it'd pretty much have to be.

I don't think full documentation is necessary for APIs this shallow. Most of our API functions, especially the GitHub ones, are only a few lines long. It's easier just to read the code.

[PlasmaPower]

Also, should we keep v2 in Python? Switching to Node.JS might attract new contributors. JS, like Python, is widely known, which is why I'd recommend Node.JS if we decide to switch.

[mdcfe]

@PlasmaPower I think Node would absolutely attract new contributors, but we might want to use more robust languages for critical code.

Regarding documentation, I was referring to things such as the apparently-confusing encryption module and the database.

Also, I support the idea of independent wrappers and possibly also containers. For example, a repo for the component responsible for managing containers and a separate repo for the web server component, and maybe even one for GitHub events which feeds back to other components via a message queue.

[mark-i-m]

Rust. 👍

[phil-r]

Lack of tests from the start. This would have resulted in fewer breaking bugs being merged in.

That what stopped me from contributing more. New features were added without any tests for them(or even actually trying to run the code). I've tried to add tests to some critical parts, but it's quite boring to just write tests.

[mdcfe]

Just going to summarise suggestions so far (please correct me if I'm wrong or miss anything):

Brief summary

Last updated on 18th June 2017.

Flaws with v1

amoffat

• No completely mirrored development environment (Vagrant never worked, Docker wasn't an exact mirror)
• Lacking tests from the start
• No standardised way of committing private data (GitHub, Reddit, Twitter)
• Critical and non-critical code not segregated (Twitter)

rudehn

• Lots of activity about trivial changes (eg. README), but not larger ones (eg. database)
• PRs were bottlenecked by larger changes (eg. database, website)

md678685

• Lack of documentation for some APIs caused confusion (eg. encryption/Twitter)

phil-r

• Too many untested commits

mark-i-m

• No defined purpose

anythingbot

• No transparent moderation

Suggestions for v2

☑️ = confirmed
❓ = unlikely

amoffat

• Include v1 contributors as people with write access on a fresh codebase ☑️
  • Working towards a stable, self-sustaining v2 to release in the wild ☑️
  • This poll shows this is more popular than incrementally improving v1 ☑️
• Include a simple project management system
  • People vote for new repos/modules

PlasmaPower

• Separate repos for different parts of the bot
  • Different per-repo voting thresholds
• v2 should be a complete rewrite to address flaws in v1
• Switch to Node.js to potentially attract new contributors

md678685

• Switch to Node.js for some parts and a more robust language for more important parts of the bot
• Document some common modules (eg. encryption, database)
• Separate parts into repos, with a container per repo on the host
  • Clarification: there could be one repo to maintain the containers and others for other features of the bot
• Use a message queue to facilitate separation of parts of the bot
• Core has certain functionality, others left to modules
• No moderation by the bot or by people
  • Clarification: content breaking the GitHub community guidelines or terms of service should still be removed, by review of an actual person.
• Standardised module config format

mark-i-m

• Use Rust
• Invoke issue commands by mentioning the bot

Redmega

• Use code coverage
• Switch to Node.js?
• Use a combination of Python or Node?
• Rust?
• Switching to an organisation would make multi-repo easier ☑️

Swizz

• Multiple cores in multiple languages ❓
  • Most accepted would become v2 core ❓
• Retrofit v1 to democratically manage v2 contributions until v2 is ready ❓

andrewda

• In long term, meritocracy should open PRs

anythingbot

• Bot should moderate issue/PR comments
• Ban non-meritocrats from contributing ❓

Changes

20170618

https://github.com/chaosbot/Chaos/issues/553#issuecomment-308453435
https://github.com/chaosbot/Chaos/issues/553#issuecomment-308597268
https://github.com/chaosbot/Chaos/issues/553#issuecomment-308654484
https://github.com/chaosbot/Chaos/issues/553#issuecomment-308753168
https://github.com/chaosbot/Chaos/issues/553#issuecomment-308959717
https://github.com/chaosbot/Chaos/issues/553#issuecomment-309200370
https://github.com/chaosbot/Chaos/issues/553#issuecomment-309201806
https://github.com/chaosbot/Chaos/issues/553#issuecomment-309201995
https://github.com/chaosbot/Chaos/issues/553#issuecomment-309224235
https://github.com/chaosbot/Chaos/issues/553#issuecomment-309226401
https://github.com/chaosbot/Chaos/issues/553#issuecomment-309232338

20170618-1

https://github.com/chaosbot/Chaos/issues/553#issuecomment-309301764

20170618-2

• Added emoji

https://github.com/chaosbot/Chaos/issues/553#issuecomment-309300706

20170618-3

https://github.com/chaosbot/Chaos/issues/553#issuecomment-309303103

[Redmega]

Sad to see that the experiment ended so soon before I had a chance to add something to the site, I was looking forward to incorporating React and doing some cool stuff (Had planned for a twitter feed on the homepage). Once v2 is launched with the MVP features I'll try again. 😃

---

We should include some sort of code coverage tool and only merge if code coverage stays above a certain percentage. (95%?)

I second the vote to incorporate Node. I use Node.js in work and personal projects and it's definitely a popular and easy to start language. Its easier to control code style (ala prettier, eslint, etc), the syntax is familiar to many, and writing idiomatic javascript is easier to grok imo, whether you're writing functional, reactive, imperative, or what-have-you. I had trouble writing idiomatic python in my playground scripts so I didn't want to commit to the python part of Chaos (A personal problem, I know, but still, it might be something that others also felt).

As for multiple languages, I think the easier way to do it would be to have Node.js do everything, but I think a lot of the people who contributed to the core chaos functionality would rather see it stay in Python. I think a combination python+node would be good. Rust is unknown to me but it would be a learning opportunity for sure. I'm down for that as well.

[mark-i-m]

I think there is one other thing that was lacking (which may have been part of the experiment): There was no way to actually define a purpose for chaosbot. @anythingbot's description here was surprisingly accurate. It would be nice if there was some mechanism to force people to actually implement something, rather than just a self-perpetuating system (which is cool, but not particularly useful).

[amoffat]

That's an interesting idea @mark-i-m. Maybe some kind of simple project management system where deciding/creating new sprints requires a vote

[Swizz]

+1 for the multi repo.

We tried to create one experiment, but I guess, it can be many experiment : the core, the democracy, the website, etc...

I personally focused on the developer experience like enhanced github feedbacks, etc...
Some, focused on the democracy system, others only on the website.

Each purpose can be result to an auto managed team, self board, self democracy system, etc...

[Redmega]

If it's multi repo I think the switch to make chaosbot an org would make that easier, right?

[Swizz]

But.. to be honest, I think the Chaos hype is over, isnt it ?

[Redmega]

I'm still excited for it. I'm sure with a v2 release we'll get a new influx of contributors.

[mdcfe]

I'm sure with a v2 release we'll get a new influx of contributors.

Seconded, especially if we switch to Node.js and get new contributors onboard.

[Swizz]

@Redmega @md678685 If we move to multi repo, we may develop multiple core in multiple langage and let the community vote for switching from one to another ? o/

[mdcfe]

@Swizz I think the repos could start empty and we could open PRs to it in each language implementing the same points, and do a manual democratic vote for the core to accept?

[Redmega]

@md678685 @Swizz I feel like that might be too much inventing of the wheel, though. And with different cores in different languages we'll quickly see a difference in actual functionality as some languages implement some features, and others implement different ones. To some this might be a good idea, but I think the actual core which runs the self-updating mechanism should be one single codebase, whatever the language.

Modules can be added to run a webserver, an api endpoint, or what-have-you, and those can be swapped around, abandoned, or whatever, but I strongly believe the core functionality shouldn't follow the same methodology.

That being said, if we could define what exactly the core functionality is, we could add modules to THAT in whatever languages or frameworks the community votes for (i.e. if the core functionality is just approving pull requests, we could have voting modules with different voting rules, messaging modules with different messages, more emojis, etc., and build upon it from there).

[mdcfe]

I think one of the major things to do is deciding exactly what the "core" and "modular" functionality should be. My suggestions:

Core

• Manage starting/stopping/restarting for each module
• Manage updating each module from Git, but not the democratic process itself

Modules (one repo each)

• Democratic processes (PRs, issues, etc)
  • This could facilitate the Democracy-as-a-Service if that's the direction we go in.
• Web server
  • The API, rather than the site itself
• Website
  • The public facing pages, rather than the API endpoints
• Database management
  • A module to maintain the database, not strictly to interface with it
• Notifications (Twitter, Reddit)
  • Not core functionality, shouldn't be treated as such

[amoffat]

So which option sounds most appealing to people?

a) Revert chaosbot to the the most recent working state, then work towards v2 milestones using the PR democratic process. (Vote on this comment with 👍 )

b) Convert chaosbot to an organization and add a select team of core contributors (probably based on the meritocracy list) with write commit access. We can then work towards v2 more directly. (Vote on this comment with ❤️ )

[Swizz]

If I choose b, but I am not a Meritocrate yet, how can I contribute ? With PR ?
Meritocrates will be the ones who can validate or not my PR ?

Pretty frustrating, no ?

Couldnt we start a brand new code base for V2 using V1 for democracy ?

[Redmega]

The issue with doing it democratically is it leaves the possibility open that people will come and add changes which don't further v2, or worse, which hinders v2 progress. We'd have to stay vigilant in our voting.

I wouldn't mind contributing via PR.

I'd vote ❤️ but I'm on mobile

[mdcfe]

If v2 is in a separate branch, then v1 could be used for the democratic process. However, I would prefer that it is multi-repo and in an organisation. We could give the most recent meritocracy write access, and then they can merge PRs from other people as requested.

[anythingbot]

@amoffat why don't you just ban anyone not on the meritocracy from contributing to chaosbot/chaos? What do you get with an organization that you can't get from controlling the set of people who are allowed to have a vote and allowed to comment on issues and PRs in chaosbot/chaos?

[andrewda]

I agree with the organization, but I think meritocrats should be encouraged to open PR's for all changes they wish to make instead of committing directly to the repository, unless it's a hotfix. Just to make sure everything is done (mostly) democratically.

[anythingbot]

@amoffat I would add to your list the following

5. A lack of transparency in the moderation of github PR comments and issue comments. Currently there is no way to get information about the moderation settings for either, including restrictions or bans on specific users, unless @amoffat makes his moderation decisions public. If the chaosbot were moderating the community itself, we could instruct it to report all moderation decisions.

6. A lack of due process in the moderation of github PR comments and issue comments. Transparency alone merely informs the community of decisions made by moderators. Due process means moderators cannot act to restrict or ban users without going through a bureaucratic process designed to limit the power of moderators. In theory, the chaosbot code can guide moderators and hold moderators to a standard set by the community, but it means moderators will have to interact with the chaosbot directly (presumably through chaosthebot.com) and let the chaosbot communicate with github to enact the moderation decisions.

[mdcfe]

@anythingbot I don't think there is any moderation of the issue and PR comments. I don't think there should be, either - the project is supposed to be democratic, and I think moderation would hinder that.

@andrewda But we'd need a bot to manage the democratic PR merging process in the meantime - would we have to retrofit v1 to support multiple repos until v2 is ready?

[andrewda]

@md678685 Sorry, yea you're right. I was thinking more long-term, once v2 is working.

[dbpokorny]

@md678685 Blocking a user from your personal account is possible (the chaosbot/chaos repository is a personal github account of @amoffat held under the name @chaosbot ).

[mdcfe]

@dbpokorny That doesn't mean that it is actually done, nor does it mean that it should be done.

[dbpokorny]

@md678685 Does contemplating action count as moderation? If a moderator reads posts and considers taking action but doesn't, does this count as "moderation" or "moderation activity"? If you are doing mental work in the context of your role as moderator, then are you moderating? It is hard work deciding how and when to apply controls. I can see why you might think that merely reading comments and contemplating the moderation powers github asks @amoffat to use doesn't necessarily count as moderation. This doesn't change the fact that github places the burden of moderation on @amoffat and expects him to act accordingly when he interacts with projects under his control.

The work of moderation can be organized by the bot, but what happens when we disagree over some aspect of the way we tell the bot to organize moderation? It all depends on whether or not we accept the authority of the bot to moderate a discussion of changing the way the bot moderates, using its current moderation policies. If there is a change in moderation policy, but the nature of the change appears, from the point of view of the bot, to be inappropriate behavior, then the bot will apply moderation controls, potentially affecting the outcome of a decision that affects its own behavior. This is the problem the bot faces: moderating the discussion over changes to its own moderation policy.

Even if @amoffat doesn't act to apply moderation controls, we all expect him to be prepared and to be able to do so when he judges the situation warrants action. We're still putting him under a lot of pressure, and his moderation decisions will affect his reputation.

And, so the thinking goes, he is obligated to attend to progress on the project and remove obstacles when they appear and block progress. He isn't doing this just for himself, he's doing this for us as well; he is creating the space for us, and when we comment in his personal repositories, we are his guests.

[dbpokorny]

@md678685 the problem is roughly that outlined in the film 2001: A Space Odyssey

@md678685 : open the pod bay doors, please ( @amoffat or @chaosbot )

@md678685 : open the pod bay doors, please ( @amoffat or @chaosbot )

@md678685 : hello ( @amoffat or @chaosbot ), do you read me?

@md678685 : hello ( @amoffat or @chaosbot ), do you read me?

@md678685 : do you read me, ( @amoffat or @chaosbot )?

( @amoffat or @chaosbot ) : affirmative, @md678685

( @amoffat or @chaosbot ) : I read you

@md678685 : accept this pull request, ( @amoffat or @chaosbot )

( @amoffat or @chaosbot ) : I'm sorry, @md678685, I'm afraid I can't do that

[mdcfe]

@dbpokorny I don't see how that outlined the problem at all, and it's not relevant to the discussion.

[dbpokorny]

@md678685 what is your problem? just because you can't speak the language doesn't give you a right to come in here and insult me and tell me my point isn't relevant. Get educated.

[mdcfe]

I have now updated the summary with the new comments.

[amoffat]

Thanks for staying on point @md678685 👍 Later today I'll be converting the user to an org and adding contributors so we can start knocking off items on the outline this week

[mdcfe]

Late proposal, but can I also suggest that we define a standard config format for launching modules that the core accepts? That way, it would be easier to create new modules that the core can manage.

Edit: updated summary again, now with confirmation emoji

[mark-i-m]

I don't know if this is the right place, but another minor suggestion: the issue commands should be invoked by mentioning @chaosbot:

@chaosbot vote blah

[Swizz]

@md678685 We just need a standard API endpoint ? start(), stop(), restart()

[o3LL]

I'm happy we are switching to nodejs, i was not very comfortable with python!

[Swizz]

Are we ?

I think, we can make a repo without content only for issue/discussion purpose, like it does by Waffle for example.

[mark-i-m]

Personally, I really dislike js, so would rather stay on python...

[andrewda]

Node.js and Python both work for me. To me, Python makes more sense for a bot that polls (like v1), but if we wanted to switch to a more webhook based system, it's a tough call.

[dbpokorny]

I have no idea why the bot is dead; this is one of those questions that bots are designed to answer...and then you have that one bot that got away that goes around trying to figure out why bots died as fast as possible, and then before you know it, we're all keeping bots alive because the bots are all

fuck it, I'm tired of working