Skip to content

Commit

Permalink
Rename project from Night's Watch to Coal Mine
Browse files Browse the repository at this point in the history
  • Loading branch information
Jonathan Kamens committed Mar 20, 2015
1 parent 847fc31 commit be3bead
Show file tree
Hide file tree
Showing 8 changed files with 268 additions and 268 deletions.
162 changes: 82 additions & 80 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
Night's Watch - Periodic task execution monitor
Coal Mine - Periodic task execution monitor
===============================================

Track tasks that are supposed to execute periodically using "watchers"
that the tasks trigger when they execute. Alert by email when a
watcher is late. Alert again when a late watcher resumes. Keep a
partial history of watcher trigger times.
Track tasks that are supposed to execute periodically using "canaries"
that the tasks trigger when they execute. Alert by email when a canary
is late. Alert again when a late canary resumes. Keep a partial
history of canary trigger times.

Uses MongoDB for storage. Pull requests to add additional storage
engines are welcome.

The server notifies immediately when the deadline for an unpaused
watcher passes. Similarly, the server notifies immediately when a
previously late watcher is triggered.
canary passes. Similarly, the server notifies immediately when a
previously late canary is triggered.

All timestamps are stored and displayed in UTC.

Expand All @@ -29,19 +29,19 @@ API Usage examples

### Example commands

$ nights-watch &
$ coal-mine &
[1] 7564
$ curl 'http://nights-watch-server/nights-watch/v1/watcher/create?name=My+First+Watcher&periodicity=3600'
$ curl 'http://coal-mine-server/coal-mine/v1/canary/create?name=My+First+Canary&periodicity=3600'
{
"status": "ok",
"watcher": {
"canary": {
"deadline": "2015-03-19T02:08:44.885182",
"id": "fbkvlsby",
"paused": false,
"description": "",
"periodicity": 3600,
"name": "My First Watcher",
"slug": "my-first-watcher",
"name": "My First Canary",
"slug": "my-first-canary",
"emails": [],
"history": [
[
Expand All @@ -52,23 +52,23 @@ API Usage examples
"late": false
}
}
$ curl 'http://nights-watch-server/fbkvlsby?comment=short+form+trigger+url'
$ curl 'http://coal-mine-server/fbkvlsby?comment=short+form+trigger+url'
{
"recovered": false,
"unpaused": false,
"status": "ok"
}
$ curl 'http://nights-watch-server/nights-watch/v1/watcher/trigger?slug=my-first-watcher&comment=long+form+trigger+url'
$ curl 'http://coal-mine-server/coal-mine/v1/canary/trigger?slug=my-first-canary&comment=long+form+trigger+url'
{
"recovered": false,
"unpaused": false,
"status": "ok"
}
$ curl 'http://nights-watch-server/nights-watch/v1/watcher/get?name=My+First+Watcher'
$ curl 'http://coal-mine-server/coal-mine/v1/canary/get?name=My+First+Canary'
{
"watcher": {
"canary": {
"paused": false,
"name": "My First Watcher",
"name": "My First Canary",
"history": [
[
"2015-03-19T01:11:56.408000",
Expand All @@ -86,7 +86,7 @@ API Usage examples
"emails": [],
"id": "fbkvlsby",
"late": false,
"slug": "my-first-watcher",
"slug": "my-first-canary",
"deadline": "2015-03-19T02:11:56.408000",
"periodicity": 3600,
"description": ""
Expand All @@ -98,36 +98,36 @@ All API endpoints are fully documented below.

### Watching a cron job

0 0 * * * my-backup-script.sh && (curl http://nights-watch-server/fbkvlsby &>/dev/null)
0 0 * * * my-backup-script.sh && (curl http://coal-mine-server/fbkvlsby &>/dev/null)

Or use use the CLI!
-------------------

Use `nwcli` to send commands to the server from the same host or from
any other host where Night's Watch is installed. You can either
configure it:
Use `cmcli` to send commands to the server from the same host or from
any other host where Coal Mine is installed. You can either configure
it:

nwcli configure --host nights-watch-server --port 8080 --auth_key [auth_key in nights-watch.ini]
cmcli configure --host coal-mine-server --port 8080 --auth_key [auth_key in coal-mine.ini]

Or specify `--host`, `--port`, and/or `--auth-key` on the command line
of each invocation.

Some example commands:

nwcli create --help
cmcli create --help
# $((60*60*25)) is 25 hours
nwcli create --name 'My Second Watcher' --periodicity $((60*60*25))
nwcli delete --slug 'my-second-watcher'
cmcli create --name 'My Second Canary' --periodicity $((60*60*25))
cmcli delete --slug 'my-second-canary'

Run `nwcli --help` for more information.
Run `cmcli --help` for more information.

Server configuration
--------------------

Create <tt>nights-watch.ini</tt> in the current directory, or
Create <tt>coal-mine.ini</tt> in the current directory, or
<tt>/etc</tt>, or <tt>/usr/local/etc</tt>, or modify the list of
directories hear the top of <tt>main()</tt> in
<tt>nights-watch.py</tt> if you want to put it somewhere else.
directories hear the top of <tt>main()</tt> in <tt>server.py</tt> if
you want to put it somewhere else.

Here's what can go in the config file:

Expand Down Expand Up @@ -171,14 +171,14 @@ in responses are standard JSON, i.e., "true" or "false".

Timestamps returned by API are always UTC.

### Create watcher
### Create canary

Endpoint: /nights-watch/v1/watcher/create
Endpoint: /coal-mine/v1/canary/create

Side effects:

Adds watcher to database. Creates history record at current time with
"Watcher created" as its comment. Sets deadline to current time plus
Adds canary to database. Creates history record at current time with
"Canary created" as its comment. Sets deadline to current time plus
periodicity, unless "paused" was specified.

Required parameters:
Expand All @@ -192,13 +192,13 @@ Optional parameters:
* description - empty if unspecified
* email - specify multiple times for multiple addresses; no
notifications if unspecified
* paused - allows watcher to be created already in paused state
* paused - allows canary to be created already in paused state

Response is the same as shown for get().

### Delete watcher
### Delete canary

Endpoint: /nights-watch/v1/watcher/delete
Endpoint: /coal-mine/v1/canary/delete

Required parameters:

Expand All @@ -209,16 +209,17 @@ Response:

{'status': 'ok'}

### Update watcher
### Update canary

Endpoint: /nights-watch/v1/watcher/update
Endpoint: /coal-mine/v1/canary/update

Side effects:

Updates the specified watcher attributes. Updates deadline to latest
history timestamp plus periodicity if periodicity is updated and watcher
is unpaused, and sets late state if new deadline is before now. Sends
notification if watcher goes from not late to late or vice versa.
Updates the specified canary attributes. Updates deadline to latest
history timestamp plus periodicity if periodicity is updated and
canary is unpaused, and sets late state if new deadline is before
now. Sends notification if canary goes from not late to late or vice
versa.

Required parameters:

Expand All @@ -235,9 +236,9 @@ Optional parameters:

Response is the same as shown for get().

### Get watcher
### Get canary

Endpoint: /nights-watch/v1/watcher/get
Endpoint: /coal-mine/v1/canary/get

Required parameters:

Expand All @@ -247,7 +248,7 @@ Required parameters:
Response:

{'status': 'ok',
'watcher': {'name': name,
'canary': {'name': name,
'description': description,
'id': identifier,
'slug': slug,
Expand All @@ -258,41 +259,41 @@ Response:
'deadline': 'YYYY-MM-DDTHH:MM:SSZ',
'history': [['YYYY-MM-DDTHH:MM:SSZ', comment], ...]}}

### List watchers
### List canaries

Endpoint: /nights-watch/v1/watcher/list
Endpoint: /coal-mine/v1/canary/list

Required parameters:

* auth_key

Optional parameters:

* verbose - include all query output for each watcher
* paused - boolean, whether to list paused / unpaused watchers only
* late - boolean, whether to list late / timely watchers only
* verbose - include all query output for each canary
* paused - boolean, whether to list paused / unpaused canaries only
* late - boolean, whether to list late / timely canaries only

Response:

{'status': 'ok',
'watchers': [{'name': name,
'canaries': [{'name': name,
'id': identifier},
...]}

If "verbose" is true, then the JSON for each watcher includes all the
If "verbose" is true, then the JSON for each canary includes all the
fields shown above, not just the name and identifier.

### Trigger watcher
### Trigger canary

Endpoint: /nights-watch/v1/watcher/trigger
Endpoint: /coal-mine/v1/canary/trigger

Also: /_identifier_, in which case the "id" parameter is implied

Side effects:

Sets late state to false. Sets deadline to now plus periodicity. Adds
history record. Prunes history records. Unpauses watcher. Generates
notification email if watcher was previously late.
history record. Prunes history records. Unpauses canary. Generates
notification email if canary was previously late.

Required parameters:

Expand All @@ -307,19 +308,19 @@ Response:

{'status': 'ok', 'recovered': boolean, 'unpaused': boolean}

* recovered - indicates whether the watcher was previously late before
* recovered - indicates whether the canary was previously late before
this trigger
* unpaused - indicates whether the watcher was previously paused before
* unpaused - indicates whether the canary was previously paused before
this trigger

### Pause watcher
### Pause canary

Endpoint: /nights-watch/v1/watcher/pause
Endpoint: /coal-mine/v1/canary/pause

Side effects:

Clears deadline. Sets late state to false if necessary. Pauses
watcher. Adds history record about pause. Prunes history records.
canary. Adds history record about pause. Prunes history records.

Required parameters:

Expand All @@ -332,13 +333,13 @@ Optional parameters:

Response is the same as shown for get().

### Unpause watcher
### Unpause canary

Endpoint /nights-watch/v1/watcher/unpause
Endpoint /coal-mine/v1/canary/unpause

Side effects:

Sets deadline to now plus periodicity. Unpauses watcher. Adds history
Sets deadline to now plus periodicity. Unpauses canary. Adds history
record about unpause. Prunes history records.

Required parameters:
Expand All @@ -355,21 +356,21 @@ Response is the same as shown for get().
Quis custodiet ipsos custodes?
------------------------------

Obviously, if you're relying on Night's Watch to let you know when
something is wrong, you need to make sure that Night's Watch itself
stays running. One way to do that is to have a cron job which
periodically triggers a watcher and generates output (which crond will
email to you) if the trigger fails. Something like:
Obviously, if you're relying on Coal Mine to let you know when
something is wrong, you need to make sure that Coal Mine itself stays
running. One way to do that is to have a cron job which periodically
triggers a canary and generates output (which crond will email to you)
if the trigger fails. Something like:

0 * * * * (curl http://nights-watch-server/atvywzoa | grep -q -s '"status": "ok"') || echo "Failed to trigger watcher."
0 * * * * (curl http://coal-mine-server/atvywzoa | grep -q -s '"status": "ok"') || echo "Failed to trigger canary."

I also recommend using a log-monitoring service such as Papertrail to
monitor and alert about errors in the Night's Watch log.
monitor and alert about errors in the Coal Mine log.

Alternatives
------------

Alternatives to Night's Watch include:
Alternatives to Coal Mine include:

* [Dead Man's Snitch](https://deadmanssnitch.com/)
* [Cronitor.io](https://cronitor.io/)
Expand All @@ -390,8 +391,8 @@ there, for several reasons:
Contributors
------------

Night's Watch was created by Jonathan Kamens, with design help from
the awesome folks at Quantopian.
Coal Mine was created by Jonathan Kamens, with design help from the
awesome folks at Quantopian.

Developer notes
-----------------
Expand All @@ -413,20 +414,21 @@ for the sake of using a third-party package.

### Data model

For each watcher, we store:
For each canary, we store:

* name
* description
* slug - the name, lower-cased, with spaces and underscores converted
to hyphens and other non-alphanumeric characters removed
* random identifier, guaranteed unique
* periodicity - maximum number of seconds that can elapse before a
watcher is late.
canary is late.
* notification email address(es)
* late state (boolean)
* paused state (boolean)
* deadline for next update
* history of triggers, pruned when (>1000 or (>100 and older than one week)
* history of triggers, pruned when (>1000 or (>100 and older than one
week)

Timestamps in database are UTC.

Expand All @@ -446,8 +448,8 @@ Web UI.

Links to Web UI in email notifications.

Repeat notifications if a watcher remains late for an extended period of
time? Not even sure I want this.
Repeat notifications if a canary remains late for an extended period
of time? Not even sure I want this.

Better authentication?

Expand Down
0 nights_watch/__init__.py → coal_mine/__init__.py
View file
File renamed without changes.
Loading

0 comments on commit be3bead

Please sign in to comment.