-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat: A Bluesky feed for norwegian languages
Implements an ATProto feed server that subscribes to the Bluesky firehose and keeps track of all posts written in languages used in Norway. This includes norwegian bokmål, nynorsk, and sami languages. The server is implemented as a golang app that exposes commands for doing various actions. This includes serve, which serves the feed server and subscribes to the firehose, the publish command which publishes the feed to the Bluesky server, etc. Norsky is architected as several go routines that subscribe or publish to go channels. For example the firehose will be initialized with a go channel for passing event structs that tells the app that a post has been created or deleted. A db writer go routine subscribes to the channel and writes the post to or deletes it from the db. SQLite is used as a db because it is simple and does not require much setup. This should suffice for the norwegian language community of bsky as there are not that many posts. Goreleaser is used to create releases. Binaries are built for arm64, amd64, and linux and windows. It also builds docker images to make it simple to run Norsky in a docker container. Currently there are no tests because this is a fun hobby project and not a serious project with paying customers. Tests might be added later. Implement initial server handlers for atproto feed specs. describeFeedGenerator list the did of the feed generator and provide a list of feeds that can be subscribed to. getFeedSkeleton executes the feed logic itself getting the feed to execute from the query parameter. The server and algorithms are still work in progress, but getting closer. Use viper to provide environment variable and config file support. Cobra and Viper does not work 100% well together, but it is close enough that parameters can be provided and picked up in the commands relatively easily. Adds migration command to run db migrations as a one off task. Useful when testing application. Add Favicon and image to use as a feed avatar. Todo: Add publish functionality, complete the server part, and figure out the remaining atproto/bluesky feed spec stuff. Lots to learn. Signed-off-by: Snorre Magnus Davøen <[email protected]> Unfinished publish command and better feed endpoint The feed endpoint not returns the feed skeleton and handles cursors and all very well. Changed SQL to aggregate up languages to make limit and cursors work correctly. Cursor parsing now happens inside algorithms package so the fiber server logic can be completely ignorant of how cursors should work. This seems like more of a bluesky algorithm than server thing. Realized Bluesky has no, nb, and nn language codes meaning some users might not define if they write in bokmål or nynorsk. For now the feed will only return no posts for the algorithm that returns all norwegian languages. Todo: - Switch cli stuff to urfave cli to simplify command setup and env fallback - Finish setting up feed generator publisher based on the work by Furrysky feed generator server https://github.com/strideynet/bsky-furry-feed Signed-off-by: Snorre Magnus Davøen <[email protected]> Add commands for publish, unpublish, and tidying Signed-off-by: Snorre Magnus Davøen <[email protected]> Some adjustment to the cursor and sorting Signed-off-by: Snorre Magnus Davøen <[email protected]> Fix Docker release issues Add the org/x/crypto/x509roots/fallback module to get statically linked certs in the go binary. This is necessary as a Docker scratch image does not have any system certs to fall back on. Fix regression in the db migration scripts. Signed-off-by: Snorre Magnus Davøen <[email protected]> Add github action to run goreleaser Signed-off-by: Snorre Magnus Davøen <[email protected]> Fix go version in github action Signed-off-by: Snorre Magnus Davøen <[email protected]> Add publish workflow Signed-off-by: Snorre Magnus Davøen <[email protected]> fix publish action Signed-off-by: Snorre Magnus Davøen <[email protected]>
- Loading branch information
Showing
34 changed files
with
1,276 additions
and
391 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| name: Pull Request | ||
|
|
||
| on: | ||
| pull_request: | ||
|
|
||
| permissions: | ||
| contents: write | ||
|
|
||
| jobs: | ||
| goreleaser: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - | ||
| name: Checkout | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 0 | ||
| - | ||
| name: Set up Go | ||
| uses: actions/setup-go@v4 | ||
| with: | ||
| go-version: "1.21.1" | ||
| - | ||
| name: Login to GitHub Container Registry | ||
| uses: docker/login-action@v1 | ||
| with: | ||
| registry: ghcr.io | ||
| username: ${{ github.actor }} | ||
| password: ${{ secrets.GITHUB_TOKEN }} | ||
| - | ||
| name: Run GoReleaser | ||
| uses: goreleaser/goreleaser-action@v5 | ||
| with: | ||
| distribution: goreleaser | ||
| version: latest | ||
| args: release --clean --snapshot | ||
| env: | ||
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| name: Publish | ||
|
|
||
| on: | ||
| push: | ||
| branches: | ||
| - main | ||
| tags: | ||
| - v* | ||
|
|
||
| permissions: | ||
| contents: write | ||
| packages: write | ||
|
|
||
| jobs: | ||
| goreleaser: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - | ||
| name: Checkout | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 0 | ||
| - | ||
| name: Set up Go | ||
| uses: actions/setup-go@v4 | ||
| with: | ||
| go-version: "1.21.1" | ||
|
|
||
| - | ||
| name: Set up Docker Buildx | ||
| uses: docker/setup-buildx-action@v3 | ||
| - | ||
|
|
||
| name: Login to GitHub Container Registry | ||
| uses: docker/login-action@v1 | ||
| with: | ||
| registry: ghcr.io | ||
| username: ${{ github.actor }} | ||
| password: ${{ secrets.GITHUB_TOKEN }} | ||
| - | ||
| name: Run GoReleaser | ||
| uses: goreleaser/goreleaser-action@v5 | ||
| with: | ||
| distribution: goreleaser | ||
| version: latest | ||
| args: release --clean | ||
| env: | ||
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,4 @@ | ||
|
|
||
| dist/ | ||
| tmp/ | ||
| feed.db |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,6 @@ | ||
| FROM scratch | ||
|
|
||
| ENTRYPOINT ["/norsky"] | ||
| CMD ["serve"] | ||
| EXPOSE 3000 | ||
| COPY norsky / |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,21 +0,0 @@ | ||
| MIT License | ||
|
|
||
| Copyright (c) 2023 Snorre Magnus Davøen | ||
|
|
||
| Permission is hereby granted, free of charge, to any person obtaining a copy | ||
| of this software and associated documentation files (the "Software"), to deal | ||
| in the Software without restriction, including without limitation the rights | ||
| to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
| copies of the Software, and to permit persons to whom the Software is | ||
| furnished to do so, subject to the following conditions: | ||
|
|
||
| The above copyright notice and this permission notice shall be included in all | ||
| copies or substantial portions of the Software. | ||
|
|
||
| THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
| AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
| LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
| OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | ||
| SOFTWARE. | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -28,51 +28,72 @@ docker run -d --name norsky -p 8080:8080 -v /path/to/db:/db ghcr.io/snorreio/nor | |
| ``` | ||
|
|
||
|
|
||
| ## Development | ||
| ## Usage | ||
|
|
||
| The feed server is a standalone application that you can run on your machine. | ||
| It is also available as a Docker image. | ||
|
|
||
| ### Commands | ||
|
|
||
| ``` | ||
| serve Serve the norsky feed | ||
| migrate Run database migrations | ||
| tidy Tidy up the database | ||
| subscribe Log all norwegian posts to the command line | ||
| publish Publish feeds on Bluesky | ||
| unpublish Unpublish feeds from Bluesky | ||
| help, h Shows a list of commands or help for one command | ||
| ``` | ||
|
|
||
| The application has been developed using go 1.21.1. | ||
| You should at least have go 1.18+ installed to build the application. | ||
| Example: | ||
|
|
||
| ``` | ||
| # Command line | ||
| norsky serve --hostname yourdomain.tld --port 8080 --database /path/to/db/feed.db | ||
| # Or docker run | ||
| docker run -d \ | ||
| --env=NORSKY_HOSTNAME="yourdomain.tld" \ | ||
| --env NORSKY_DATABASE="/db/feed.db" \ | ||
| --name norsky \ | ||
| -p 3000:3000 \ | ||
| -v /path/to/db:/db \ | ||
| ghrc.io/snorreio/norsky:latest | ||
| ``` | ||
|
|
||
| ## Development | ||
|
|
||
| The application has been developed using go 1.21.1 which is the required version to build the application as the `go.mod` file has been initialized with this version. | ||
|
|
||
| To get started clone the repository and run go run on the main file to list the available commands. | ||
|
|
||
| ```bash | ||
| go run main.go --help | ||
| ``` | ||
|
|
||
| The application uses the [cobra](https://github.com/spf13/cobra) library to manage commands and flags. | ||
| [cobra-cli](https://github.com/spf13/cobra-cli) should be used to generate new commands. | ||
|
|
||
| ### Structure | ||
|
|
||
| The application follows the standard way to structure a Go project and contains the following packages: | ||
|
|
||
| - `main` - The main package that contains the entrypoint for the application. | ||
| - `cmd` - The command package that contains the different commands for the application using cobra. | ||
| - `server` - The server package that contains setup code for initializing a Fiber server. | ||
| - `database` - The database package that contains the database connection, models, migrations, and queries. | ||
| - `db` - The database package that contains the database connection, models, migrations, and queries. | ||
| - `firehose` - The firehose package that contains the firehose client and the firehose subscription. | ||
| - `feeds` - The feeds package that contains the feeds and functions that generate the feed responses. | ||
| - `models` - The models package that contains the models for the application. | ||
| - `dist` - Where goreleaser puts the release artifacts if you build the application using goreleaser locally. | ||
|
|
||
|
|
||
| ## Contributing | ||
|
|
||
| Pull requests are welcome. | ||
| For major changes, please open an issue first to discuss what you would like to change. | ||
| If you find a bug or have a feature request, please open an issue. | ||
| I am also open to suggestions for improvements to the codebase as I'm not a golang expert. | ||
|
|
||
| If you want to provide feedback in private, you can send me an email at [[email protected]](mailto:[email protected]). | ||
|
|
||
| ``` | ||
| . | ||
| ├── cmd | ||
| │ ├── gc.go | ||
| │ ├── root.go | ||
| │ ├── serve.go | ||
| │ └── subscribe.go | ||
| ├── database | ||
| │ └── database.go | ||
| ├── firehose | ||
| │ └── firehose.go | ||
| ├── server | ||
| │ └── server.go | ||
| ├── go.mod | ||
| ├── go.sum | ||
| ├── LICENSE | ||
| ├── main.go | ||
| ├── norsky | ||
| ├── README.md | ||
| ``` | ||
|
|
||
| ## Release | ||
|
|
||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| /* | ||
| Copyright © 2023 NAME HERE <EMAIL ADDRESS> | ||
| */ | ||
| package cmd | ||
|
|
||
| import ( | ||
| "fmt" | ||
| "norsky/db" | ||
|
|
||
| "github.com/urfave/cli/v2" | ||
| ) | ||
|
|
||
| func migrateCmd() *cli.Command { | ||
| return &cli.Command{ | ||
| Name: "migrate", | ||
| Usage: "Run database migrations", | ||
| Description: `Runs database migrations on the configured database. Will create the database if it does not exist.`, | ||
| Flags: []cli.Flag{ | ||
| &cli.StringFlag{ | ||
| Name: "database", | ||
| Aliases: []string{"d"}, | ||
| Value: "feed.db", | ||
| Usage: "SQLite database file location", | ||
| EnvVars: []string{"NORSKY_DATABASE"}, | ||
| }, | ||
| }, | ||
| Action: func(ctx *cli.Context) error { | ||
| database := ctx.String("database") | ||
| fmt.Println("Database configured: ", database) | ||
| err := db.Migrate(database) | ||
| return err | ||
| }, | ||
| } | ||
| } |
Oops, something went wrong.