Implement multi-sourced postgres (#511)

* Support `postgres` config key to be either a list or an object * Use `PathBuf` instead of `String` where dealing with files * Merge `SrvConfigBuilder` into `SrvConfig` * Parse keep_alive as u64 * More config tests
maplibre · Dec 22, 2022 · 2b240b3 · 2b240b3
1 parent 3f41219
commit 2b240b3
Show file tree

Hide file tree

Showing 28 changed files with 1,113 additions and 711 deletions.
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/Cargo.toml b/Cargo.toml
@@ -48,6 +48,7 @@ semver = "1"
 serde = { version = "1", features = ["derive"] }
 serde_json = "1"
 serde_yaml = "0.9"
+subst = { version = "0.2", features = ["yaml"] }
 thiserror = "1"
 tilejson = "0.3"
 

diff --git a/README.md b/README.md
@@ -75,15 +75,15 @@ brew install martin
 You can also use [official Docker image](https://hub.docker.com/r/maplibre/martin)
 
 ```shell
-docker run -p 3000:3000 -e DATABASE_URL=postgres://postgres@localhost/db maplibre/martin
+docker run -p 3000:3000 -e DATABASE_URL=postgresql://postgres@localhost/db maplibre/martin
 ```
 
 ## Usage
 
 Martin requires a database connection string. It can be passed as a command-line argument or as a `DATABASE_URL` environment variable.
 
 ```shell
-martin postgres://postgres@localhost/db
+martin postgresql://postgres@localhost/db
 ```
 
 Martin provides [TileJSON](https://github.com/mapbox/tilejson-spec) endpoint for each [geospatial-enabled](https://postgis.net/docs/postgis_usage.html#geometry_columns) table in your database.
@@ -450,12 +450,12 @@ Options:
 
 You can also configure martin using environment variables
 
-| Environment variable          | Example                            | Description                                 |
-|-------------------------------|------------------------------------|---------------------------------------------|
-| `DATABASE_URL`                | `postgres://postgres@localhost/db` | Postgres database connection                |
-| `CA_ROOT_FILE`                | `./ca-certificate.crt`             | Loads trusted root certificates from a file |
-| `DEFAULT_SRID`                | `4326`                             | Fallback SRID                               |
-| `DANGER_ACCEPT_INVALID_CERTS` | `false`                            | Trust invalid certificates                  |
+| Environment variable          | Example                              | Description                                 |
+|-------------------------------|--------------------------------------|---------------------------------------------|
+| `DATABASE_URL`                | `postgresql://postgres@localhost/db` | Postgres database connection                |
+| `CA_ROOT_FILE`                | `./ca-certificate.crt`               | Loads trusted root certificates from a file |
+| `DEFAULT_SRID`                | `4326`                               | Fallback SRID                               |
+| `DANGER_ACCEPT_INVALID_CERTS` | `false`                              | Trust invalid certificates                  |
 
 ## Configuration File
 
@@ -468,93 +468,92 @@ martin --config config.yaml
 You can find an example of a configuration file [here](https://github.com/maplibre/martin/blob/main/tests/config.yaml).
 
 ```yaml
-# Database connection string
-connection_string: 'postgres://postgres@localhost:5432/db'
-
-# Trust invalid certificates. This introduces significant vulnerabilities, and should only be used as a last resort.
-danger_accept_invalid_certs: false
-
-# If a spatial table has SRID 0, then this SRID will be used as a fallback
-default_srid: 4326
-
 # Connection keep alive timeout [default: 75]
 keep_alive: 75
 
 # The socket address to bind [default: 0.0.0.0:3000]
 listen_addresses: '0.0.0.0:3000'
 
-# Maximum connections pool size [default: 20]
-pool_size: 20
-
 # Number of web server workers
 worker_processes: 8
 
-# Associative arrays of table sources
-tables:
-  table_source_id:
-    # Table schema (required)
-    schema: public
-
-    # Table name (required)
-    table: table_source
-
-    # Geometry SRID (required)
-    srid: 4326
-
-    # Geometry column name (required)
-    geometry_column: geom
-
-    # Feature id column name
-    id_column: ~
-
-    # An integer specifying the minimum zoom level
-    minzoom: 0
-
-    # An integer specifying the maximum zoom level. MUST be >= minzoom
-    maxzoom: 30
-
-    # The maximum extent of available map tiles. Bounds MUST define an area
-    # covered by all zoom levels. The bounds are represented in WGS:84
-    # latitude and longitude values, in the order left, bottom, right, top.
-    # Values may be integers or floating point numbers.
-    bounds: [-180.0, -90.0, 180.0, 90.0]
-
-    # Tile extent in tile coordinate space
-    extent: 4096
-
-    # Buffer distance in tile coordinate space to optionally clip geometries
-    buffer: 64
-
-    # Boolean to control if geometries should be clipped or encoded as is
-    clip_geom: true
-
-    # Geometry type
-    geometry_type: GEOMETRY
-
-    # List of columns, that should be encoded as tile properties (required)
-    properties:
-      gid: int4
-
-# Associative arrays of function sources
-functions:
-  function_source_id:
-    # Schema name (required)
-    schema: public
-
-    # Function name (required)
-    function: function_zxy_query
-
-    # An integer specifying the minimum zoom level
-    minzoom: 0
-
-    # An integer specifying the maximum zoom level. MUST be >= minzoom
-    maxzoom: 30
-
-    # The maximum extent of available map tiles. Bounds MUST define an area
-    # covered by all zoom levels. The bounds are represented in WGS:84
-    # latitude and longitude values, in the order left, bottom, right, top.
-    # Values may be integers or floating point numbers.
-    bounds: [-180.0, -90.0, 180.0, 90.0]
+# Database configuration. This can also be a list of PG configs.
+postgres:
+  # Database connection string
+  connection_string: 'postgresql://postgres@localhost:5432/db'
+
+  #  If a spatial table has SRID 0, then this SRID will be used as a fallback
+  default_srid: 4326
+
+  # Maximum connections pool size [default: 20]
+  pool_size: 20
+
+  # Associative arrays of table sources
+  tables:
+    table_source_id:
+      # Table schema (required)
+      schema: public
+
+      # Table name (required)
+      table: table_source
+
+      # Geometry SRID (required)
+      srid: 4326
+
+      # Geometry column name (required)
+      geometry_column: geom
+
+      # Feature id column name
+      id_column: ~
+
+      # An integer specifying the minimum zoom level
+      minzoom: 0
+
+      # An integer specifying the maximum zoom level. MUST be >= minzoom
+      maxzoom: 30
+
+      # The maximum extent of available map tiles. Bounds MUST define an area
+      # covered by all zoom levels. The bounds are represented in WGS:84
+      # latitude and longitude values, in the order left, bottom, right, top.
+      # Values may be integers or floating point numbers.
+      bounds: [-180.0, -90.0, 180.0, 90.0]
+
+      # Tile extent in tile coordinate space
+      extent: 4096
+
+      # Buffer distance in tile coordinate space to optionally clip geometries
+      buffer: 64
+
+      # Boolean to control if geometries should be clipped or encoded as is
+      clip_geom: true
+
+      # Geometry type
+      geometry_type: GEOMETRY
+
+      # List of columns, that should be encoded as tile properties (required)
+      properties:
+        gid: int4
+
+  # Associative arrays of function sources
+  functions:
+    function_source_id:
+      # Schema name (required)
+      schema: public
+
+      # Function name (required)
+      function: function_zxy_query
+
+      # An integer specifying the minimum zoom level
+      minzoom: 0
+
+      # An integer specifying the maximum zoom level. MUST be >= minzoom
+      maxzoom: 30
+
+      # The maximum extent of available map tiles. Bounds MUST define an area
+      # covered by all zoom levels. The bounds are represented in WGS:84
+      # latitude and longitude values, in the order left, bottom, right, top.
+      # Values may be integers or floating point numbers.
+      bounds: [-180.0, -90.0, 180.0, 90.0]
 ```
 
 ## Using with Docker
@@ -564,7 +563,7 @@ You can use official Docker image [`maplibre/martin`](https://hub.docker.com/r/m
 ```shell
 docker run \
   -p 3000:3000 \
-  -e DATABASE_URL=postgres://postgres@localhost/db \
+  -e DATABASE_URL=postgresql://postgres@localhost/db \
   maplibre/martin
 ```
 
@@ -576,7 +575,7 @@ For Linux, add the `--net=host` flag to access the `localhost` PostgreSQL servic
 docker run \
   --net=host \
   -p 3000:3000 \
-  -e DATABASE_URL=postgres://postgres@localhost/db \
+  -e DATABASE_URL=postgresql://postgres@localhost/db \
   maplibre/martin
 ```
 
@@ -585,7 +584,7 @@ For macOS, use `host.docker.internal` as hostname to access the `localhost` Post
 ```shell
 docker run \
   -p 3000:3000 \
-  -e DATABASE_URL=postgres://[email protected]/db \
+  -e DATABASE_URL=postgresql://[email protected]/db \
   maplibre/martin
 ```
 
@@ -594,7 +593,7 @@ For Windows, use `docker.for.win.localhost` as hostname to access the `localhost
 ```shell
 docker run \
   -p 3000:3000 \
-  -e DATABASE_URL=postgres://[email protected]/db \
+  -e DATABASE_URL=postgresql://[email protected]/db \
   maplibre/martin
 ```
 
@@ -612,7 +611,7 @@ services:
     ports:
       - "3000:3000"
     environment:
-      - DATABASE_URL=postgres://postgres:password@db/db
+      - DATABASE_URL=postgresql://postgresql:password@db/db
     depends_on:
       - db
 
@@ -664,7 +663,7 @@ services:
     image: maplibre/martin:v0.6.2
     restart: unless-stopped
     environment:
-      - DATABASE_URL=postgres://postgres:password@db/db
+      - DATABASE_URL=postgresql://postgresql:password@db/db
     depends_on:
       - db
 
@@ -750,7 +749,7 @@ The binary will be available at `./target/release/martin`.
 
 ```shell
 cd ./target/release/
-./martin postgres://postgres@localhost/db
+./martin postgresql://postgres@localhost/db
 ```
 
 ## Debugging
@@ -761,14 +760,14 @@ This will enable debug logging for all modules:
 
 ```shell
 export RUST_LOG=debug
-martin postgres://postgres@localhost/db
+martin postgresql://postgres@localhost/db
 ```
 
 While this will only enable verbose logging for the `actix_web` module and enable debug logging for the `martin` and `tokio_postgres` modules:
 
 ```shell
 export RUST_LOG=actix_web=info,martin=debug,tokio_postgres=debug
-martin postgres://postgres@localhost/db
+martin postgresql://postgres@localhost/db
 ```
 
 ## Development
@@ -810,21 +809,21 @@ Available recipes:
 just debug-page
 
 # Run Martin server
-DATABASE_URL=postgres://postgres@localhost/db cargo run
+DATABASE_URL=postgresql://postgres@localhost/db cargo run
 ```
 
 Open `tests/debug.html` for debugging. By default, martin will be available at [localhost:3000](http://localhost:3000/)
 
 Make your changes, and check if all the tests are running
 
 ```shell
-DATABASE_URL=postgres://postgres@localhost/db cargo test
+DATABASE_URL=postgresql://postgres@localhost/db cargo test
 ```
 
 You can also run benchmarks with
 
 ```shell
-DATABASE_URL=postgres://postgres@localhost/db cargo bench
+DATABASE_URL=postgresql://postgres@localhost/db cargo bench
 ```
 
 An HTML report displaying the results of the benchmark will be generated under `target/criterion/report/index.html`
@@ -839,7 +838,7 @@ You can use martin with [Managed PostgreSQL from DigitalOcean](https://www.digit
 First, you need to download the CA certificate and get your cluster connection string from the [dashboard](https://cloud.digitalocean.com/databases). After that, you can use the connection string and the CA certificate to connect to the database
 
 ```shell
-martin --ca-root-file ./ca-certificate.crt postgres://user:password@host:port/db?sslmode=require
+martin --ca-root-file ./ca-certificate.crt postgresql://user:password@host:port/db?sslmode=require
 ```
 
 ### Using with Heroku PostgreSQL

diff --git a/justfile b/justfile
@@ -97,10 +97,14 @@ docker-run *ARGS:
 git *ARGS: start-db
     git {{ARGS}}
 
+# Run cargo fmt and cargo clippy
+lint:
+    cargo fmt --all -- --check
+    cargo clippy --all-targets --all-features -- -D warnings -W clippy::pedantic
+
 # These steps automatically run before git push via a git hook
 git-pre-push: stop start-db
     rustc --version
     cargo --version
-    cargo fmt --all -- --check
-    cargo clippy --all-targets --all-features -- -D warnings -W clippy::pedantic
+    just lint
     just test