Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade Azure Storage SDK to "track 2" #459

Open
wants to merge 100 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
100 commits
Select commit Hold shift + click to select a range
fc897b4
Group replica wal segments by index
benbjohnson Jun 14, 2021
77274ab
Refactor shadow WAL to use segments
benbjohnson Jul 22, 2021
6db0606
README
benbjohnson Sep 21, 2021
aa2c684
Update contribution policy
benbjohnson Oct 2, 2021
cb33d8c
Replica.Restore fallback to DB.path
colin-sitehost Oct 3, 2021
755f54f
Update CONTRIBUTING & remove pull request template
benbjohnson Oct 10, 2021
61c80cb
README
benbjohnson Dec 5, 2021
d09f4ef
Fix FindMinSnapshotByGeneration() loop ref bug
benbjohnson Dec 9, 2021
ba6e13b
Sort output of snapshots in descending timestamp order
mtlynch Nov 28, 2021
531e19e
Refactor checksum calculation; improve test coverage
benbjohnson Dec 12, 2021
3f0ec9f
Refactor Restore()
benbjohnson Jan 4, 2022
f308e0b
CLI test coverage
benbjohnson Jan 11, 2022
84d08f5
Add end-to-end replication/restore testing
benbjohnson Jan 14, 2022
b8536fa
dependabot.yml
benbjohnson Jan 22, 2022
c7aa363
Parallelize GitHub Actions
benbjohnson Jan 21, 2022
0a6474f
Restrict CI jobs
benbjohnson Jan 22, 2022
b8d0495
Update CI
benbjohnson Jan 22, 2022
17831c7
Bump github.com/pierrec/lz4/v4 from 4.1.3 to 4.1.12
dependabot[bot] Jan 22, 2022
0b533e5
Bump github.com/aws/aws-sdk-go from 1.27.0 to 1.42.39
dependabot[bot] Jan 22, 2022
6c5fb2c
Bump cloud.google.com/go/storage from 1.15.0 to 1.18.2
dependabot[bot] Jan 22, 2022
d045b7b
Bump google.golang.org/api from 0.45.0 to 0.65.0
dependabot[bot] Jan 22, 2022
79b50c6
Update sqlite 3.36
anacrolix Jan 22, 2022
90715ef
Upgrade azure-storage-blob-go to v0.14.0
benbjohnson Jan 23, 2022
500cfd8
Upgrade shellwords, golang.org/x
benbjohnson Jan 23, 2022
55c475e
Upgrade github.com/pkg/[email protected]
benbjohnson Jan 23, 2022
5d24f91
Upgrade github.com/prometheus/[email protected]
benbjohnson Jan 23, 2022
3911450
Create codeql-analysis.yml
benbjohnson Jan 23, 2022
8950de8
Update dependabot.yml
benbjohnson Jan 24, 2022
8d759bb
Bump github.com/aws/aws-sdk-go from 1.42.39 to 1.42.40
dependabot[bot] Jan 24, 2022
ffaba87
Separate out GitHub Actions
benbjohnson Jan 25, 2022
1741c82
Produce build for every pull request
benbjohnson Jan 26, 2022
dbdde21
Use sqlite3_file_control(SQLITE_FCNTL_PERSIST_WAL) to persist WAL
benbjohnson Jan 28, 2022
f8382cf
Dispatch test runner in CI
benbjohnson Jan 28, 2022
26f219d
Add test runner request action
benbjohnson Jan 30, 2022
906ed9b
Revert "Add test runner request action"
benbjohnson Jan 30, 2022
0dfa5f9
Re-enable SFTP integration tests
benbjohnson Jan 30, 2022
f6c8590
Fix CodeQL warnings
benbjohnson Jan 30, 2022
e84994a
Add golangci-lint to CI
benbjohnson Jan 31, 2022
5d811f2
Fix golangci-lint issues
benbjohnson Jan 31, 2022
a2cf2e2
Skip some CI jobs for dependabot
benbjohnson Jan 31, 2022
ee77592
Skip dependabot CI using branches
benbjohnson Jan 31, 2022
fb3a3d9
Bump github.com/aws/aws-sdk-go from 1.42.40 to 1.42.44
dependabot[bot] Jan 31, 2022
d5c1559
Bump google.golang.org/api from 0.65.0 to 0.66.0
dependabot[bot] Jan 31, 2022
5f38134
Bump cloud.google.com/go/storage from 1.18.2 to 1.19.0
dependabot[bot] Jan 31, 2022
89560c8
Bump github.com/prometheus/client_golang from 1.12.0 to 1.12.1
dependabot[bot] Jan 31, 2022
4349398
Remove shadow WAL iterator
benbjohnson Jan 31, 2022
8009bcf
Remove Windows support
benbjohnson Feb 5, 2022
762c7ae
Implement FileWatcher
benbjohnson Feb 6, 2022
76e53dc
Remove built-in validation option
benbjohnson Feb 6, 2022
30a8d07
Add WAL overrun validation
benbjohnson Feb 6, 2022
54f3b94
Upgrade dependencies
benbjohnson Feb 7, 2022
006e4b7
Update index & offset encoding
benbjohnson Feb 8, 2022
8589111
Implement streaming WAL segment iterator
benbjohnson Feb 11, 2022
1a630ae
Add docker multiarch build and push to release
ohthehugemanatee Feb 11, 2022
fc42576
Add Docker arm/v7 to CI
benbjohnson Feb 15, 2022
fde17d0
Upgrade dependencies
benbjohnson Feb 15, 2022
4027c87
Fix Docker arch mismatch
benbjohnson Feb 15, 2022
6f8cd5a
Configurable monitor-delay-interval
benbjohnson Feb 18, 2022
4898fc2
Remove Docker linux/arm64 for PR builds
benbjohnson Feb 18, 2022
a090706
Implement live read replication
benbjohnson Feb 19, 2022
06ea1b1
Improve iterator Next() descriptions
tniessen Feb 26, 2022
62e301a
Change dependabot from weekly to monthly
benbjohnson Feb 26, 2022
c435b6b
Pass first DB path to child process
benbjohnson Mar 5, 2022
59de3a0
Upgrade mattn/go-sqlite3 to v1.14.12
benbjohnson Mar 5, 2022
1402642
Disable dependabot
benbjohnson Mar 5, 2022
7fe79d3
Add -addr flag to replicate command
benbjohnson Mar 5, 2022
8ee5fcb
Read config file from present working directory, if present
benbjohnson Mar 5, 2022
07d2200
Rename 'gcs' to 'gs' for consistency
benbjohnson Mar 5, 2022
d5792c4
Prevent double-close for SFTP client
benbjohnson Mar 5, 2022
00bad43
Set permission on file replica client on init
benbjohnson Mar 6, 2022
8d10881
Use database page size in read replication
benbjohnson Apr 2, 2022
6aba416
Remove CI task for executing long running test runner on each build
benbjohnson Apr 2, 2022
4688853
Default upstream path if not specified
benbjohnson Apr 3, 2022
2c3e28c
Improve http error logging
benbjohnson Apr 3, 2022
4466202
Allow read replication recovery from last position
benbjohnson Apr 3, 2022
f53857e
Add minimum shadow WAL retention
benbjohnson Apr 5, 2022
5d394bb
Document -addr flag on replicate command
benbjohnson Apr 5, 2022
80f8de4
Fix release workflow
benbjohnson Apr 9, 2022
ca07137
Re-add point-in-time restore
benbjohnson Apr 15, 2022
301e117
Add Go code coverage to CI
mtlynch Apr 17, 2022
6763e92
Fix path to coverage file
mtlynch Apr 17, 2022
88737d7
Add a unit test for internal.MD5Hash
mtlynch Apr 17, 2022
7d8b8c6
Remove verbose flag from restore docs
benbjohnson May 3, 2022
e6f7c60
Add two environments for overriding endpoint and region
hnakamur May 10, 2022
46597ab
Fix wal internal error log
hnakamur May 13, 2022
98673c6
Add environment variables for scheme and forcePathStyle
hnakamur May 13, 2022
2c0dce2
Use fsnotify
mattn May 6, 2022
7d0167f
Unwatch directory
mattn May 6, 2022
e9dbf83
Re-add Fileinfo()
benbjohnson May 12, 2022
4522c7b
implement Fileinfo for Windows and non-Windows
mattn May 13, 2022
31aa5b3
Fix build tag
mattn May 13, 2022
2acdab0
Improve readability
ryanrussell May 30, 2022
80cd049
Revert to correct `wal_downloader.go`
ryanrussell Jun 3, 2022
a8ab14c
Update dependencies
benbjohnson Aug 8, 2022
868d564
Remove streaming replication implementation
benbjohnson Aug 8, 2022
f50f03d
Update readme to note that this tool is for disaster recovery, not st…
josegonzalez Oct 14, 2022
64e535e
Handle errors when deleting objects from S3
lstoll Oct 31, 2022
5309304
fix: aws credential chain by using aws.Config
ekristen Jan 11, 2023
8dbdf2b
fix: remove debug code
ekristen Jan 11, 2023
0501dfa
Upgrade Azure Storage SDK to "track 2"
ItalyPaleAle Feb 3, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
29 changes: 15 additions & 14 deletions .github/CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,18 @@
## Open-source, not open-contribution
## Contribution Policy

[Similar to SQLite](https://www.sqlite.org/copyright.html), Litestream is open
source but closed to contributions. This keeps the code base free of proprietary
or licensed code but it also helps me continue to maintain and build Litestream.
Initially, Litestream was closed to outside contributions. The goal was to
reduce burnout by limiting the maintenance overhead of reviewing and validating
third-party code. However, this policy is overly broad and has prevented small,
easily testable patches from being contributed.

As the author of [BoltDB](https://github.com/boltdb/bolt), I found that
accepting and maintaining third party patches contributed to my burn out and
I eventually archived the project. Writing databases & low-level replication
tools involves nuance and simple one line changes can have profound and
unexpected changes in correctness and performance. Small contributions
typically required hours of my time to properly test and validate them.
Litestream is now open to code contributions for bug fixes only. Features carry
a long-term maintenance burden so they will not be accepted at this time.
Please [submit an issue][new-issue] if you have a feature you'd like to
request.

If you find mistakes in the documentation, please submit a fix to the
[documentation repository][docs].

[new-issue]: https://github.com/benbjohnson/litestream/issues/new
[docs]: https://github.com/benbjohnson/litestream.io

I am grateful for community involvement, bug reports, & feature requests. I do
not wish to come off as anything but welcoming, however, I've
made the decision to keep this project closed to contributions for my own
mental health and long term viability of the project.
7 changes: 0 additions & 7 deletions .github/pull_request_template.md

This file was deleted.

30 changes: 30 additions & 0 deletions .github/workflows/build_and_test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: "Build and Unit Test"
on: pull_request

jobs:
build:
name: Build
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v2

- uses: actions/setup-go@v2
with:
go-version: '1.19'

- uses: actions/cache@v2
with:
path: ~/go/pkg/mod
key: ${{ inputs.os }}-go-${{ hashFiles('**/go.sum') }}
restore-keys: ${{ inputs.os }}-go-

- name: Build binary
run: go install ./cmd/litestream

- name: Run unit tests
run: make testdata && go test -v --coverprofile=.coverage.out ./... && go tool cover -html .coverage.out -o .coverage.html

- uses: actions/upload-artifact@v3
with:
name: code-coverage
path: .coverage.html
38 changes: 38 additions & 0 deletions .github/workflows/codeql-analysis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
name: "CodeQL"

on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
schedule:
- cron: '20 16 * * 4'

jobs:
analyze:
name: Analyze
runs-on: ubuntu-latest
permissions:
actions: read
contents: read
security-events: write

strategy:
fail-fast: false
matrix:
language: [ 'go' ]

steps:
- name: Checkout repository
uses: actions/checkout@v2

- name: Initialize CodeQL
uses: github/codeql-action/init@v1
with:
languages: ${{ matrix.language }}

- name: Autobuild
uses: github/codeql-action/autobuild@v1

- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v1
18 changes: 18 additions & 0 deletions .github/workflows/golangci-lint.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
name: golangci-lint
on:
pull_request:

permissions:
contents: read

jobs:
golangci:
name: lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2

- uses: golangci/golangci-lint-action@v2
with:
version: latest
args: --timeout=10m
138 changes: 138 additions & 0 deletions .github/workflows/integration_test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
name: Integration Tests
on:
pull_request:
branches-ignore:
- "dependabot/**"

jobs:
s3-integration-test:
name: Run S3 Integration Tests
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v2

- uses: actions/setup-go@v2
with:
go-version: '1.19'

- uses: actions/cache@v2
with:
path: ~/go/pkg/mod
key: ${{ inputs.os }}-go-${{ hashFiles('**/go.sum') }}
restore-keys: ${{ inputs.os }}-go-

- run: go install ./cmd/litestream

- run: go test -v -run=TestReplicaClient ./integration -replica-type s3
env:
LITESTREAM_S3_ACCESS_KEY_ID: ${{ secrets.LITESTREAM_S3_ACCESS_KEY_ID }}
LITESTREAM_S3_SECRET_ACCESS_KEY: ${{ secrets.LITESTREAM_S3_SECRET_ACCESS_KEY }}
LITESTREAM_S3_REGION: us-east-1
LITESTREAM_S3_BUCKET: integration.litestream.io

gcp-integration-test:
name: Run GCP Integration Tests
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v2

- uses: actions/setup-go@v2
with:
go-version: '1.19'

- uses: actions/cache@v2
with:
path: ~/go/pkg/mod
key: ${{ inputs.os }}-go-${{ hashFiles('**/go.sum') }}
restore-keys: ${{ inputs.os }}-go-

- name: Extract GCP credentials
run: 'echo "$GOOGLE_APPLICATION_CREDENTIALS" > /opt/gcp.json'
shell: bash
env:
GOOGLE_APPLICATION_CREDENTIALS: ${{secrets.GOOGLE_APPLICATION_CREDENTIALS}}

- run: go test -v -run=TestReplicaClient ./integration -replica-type gs
env:
GOOGLE_APPLICATION_CREDENTIALS: /opt/gcp.json
LITESTREAM_GS_BUCKET: integration.litestream.io

abs-integration-test:
name: Run Azure Blob Store Integration Tests
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v2

- uses: actions/setup-go@v2
with:
go-version: '1.19'

- uses: actions/cache@v2
with:
path: ~/go/pkg/mod
key: ${{ inputs.os }}-go-${{ hashFiles('**/go.sum') }}
restore-keys: ${{ inputs.os }}-go-

- run: go test -v -run=TestReplicaClient ./integration -replica-type abs
env:
LITESTREAM_ABS_ACCOUNT_NAME: ${{ secrets.LITESTREAM_ABS_ACCOUNT_NAME }}
LITESTREAM_ABS_ACCOUNT_KEY: ${{ secrets.LITESTREAM_ABS_ACCOUNT_KEY }}
LITESTREAM_ABS_BUCKET: integration

sftp-integration-test:
name: Run SFTP Integration Tests
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v2

- uses: actions/setup-go@v2
with:
go-version: '1.19'

- uses: actions/cache@v2
with:
path: ~/go/pkg/mod
key: ${{ inputs.os }}-go-${{ hashFiles('**/go.sum') }}
restore-keys: ${{ inputs.os }}-go-

- name: Extract SSH key
run: 'echo "$LITESTREAM_SFTP_KEY" > /opt/id_ed25519'
shell: bash
env:
LITESTREAM_SFTP_KEY: ${{secrets.LITESTREAM_SFTP_KEY}}

- name: Run sftp tests w/ key
run: go test -v -run=TestReplicaClient ./integration -replica-type sftp
env:
LITESTREAM_SFTP_HOST: litestream-test-sftp.fly.dev:2222
LITESTREAM_SFTP_USER: litestream
LITESTREAM_SFTP_PATH: /litestream
LITESTREAM_SFTP_KEY_PATH: /opt/id_ed25519

- name: Run sftp tests w/ password
run: go test -v -run=TestReplicaClient ./integration -replica-type sftp
env:
LITESTREAM_SFTP_HOST: litestream-test-sftp.fly.dev:2222
LITESTREAM_SFTP_USER: litestream
LITESTREAM_SFTP_PASSWORD: ${{ secrets.LITESTREAM_SFTP_PASSWORD }}
LITESTREAM_SFTP_PATH: /litestream

long-running-test:
name: Run Long-Running Test
runs-on: ubuntu-22.04
steps:
- uses: actions/checkout@v2

- uses: actions/setup-go@v2
with:
go-version: '1.19'

- uses: actions/cache@v2
with:
path: ~/go/pkg/mod
key: ${{ inputs.os }}-go-${{ hashFiles('**/go.sum') }}
restore-keys: ${{ inputs.os }}-go-

- run: go install ./cmd/litestream

- run: go test -v -run=TestCmd_Replicate_LongRunning ./integration -long-running-duration 1m
51 changes: 51 additions & 0 deletions .github/workflows/release.docker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
on:
release:
types:
- published
pull_request:
types:
- opened
- synchronize
- reopened
branches-ignore:
- "dependabot/**"

name: Release (Docker)
jobs:
docker:
runs-on: ubuntu-latest
env:
PLATFORMS: "${{ github.event_name == 'release' && 'linux/amd64,linux/arm64,linux/arm/v7' || 'linux/amd64' }}"
VERSION: "${{ github.event_name == 'release' && github.event.release.name || github.sha }}"

steps:
- uses: actions/checkout@v2
- uses: docker/setup-qemu-action@v1
- uses: docker/setup-buildx-action@v1

- uses: docker/login-action@v1
with:
username: benbjohnson
password: ${{ secrets.DOCKERHUB_TOKEN }}

- id: meta
uses: docker/metadata-action@v3
with:
images: litestream/litestream
tags: |
type=ref,event=branch
type=ref,event=pr
type=sha
type=sha,format=long
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}

- uses: docker/build-push-action@v2
with:
context: .
push: true
platforms: ${{ env.PLATFORMS }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
build-args: |
LITESTREAM_VERSION=${{ env.VERSION }}
Loading