3.0

CHANGES.rst

@@ -1,3 +1,31 @@
+v3.0.0 - XXXX-XX-XX
+===================
+
+- Convert all logging to structured (#199).
+- Look for ``config.yaml`` configuration file by default, configuration file
+  can be optionally specified with ``--config``.
+- Support passing configuration options via environment variables.
+- Support loading ``.env`` file with environment variables for configuration.
+- [docker] Run exporter from the ``/config`` directory, supporting having
+  ``.env`` file there.
+- [snap] Run exporter from the ``$SNAP_DATA`` directory, supporting having
+  ``.env`` file there.
+
+**NOTE**:
+  This release introduces a few breaking changes from the 2.x series,
+  specificially:
+
+  - The ``--log-level`` option now takes lowercase names for levels.
+  - The configuration file is no longer a required option, since
+    ``config.yaml`` in the current directory is looked up automatically. If a
+    different file is specified, it should be done as an optional parameter
+    with ``--config``.
+  - Metrics of type ``counter`` are now set by default to values returned by
+    queries. To preserve the old default behavior of incrementing it by the
+    returned value, ``increment`` should be set to ``true`` in the metric
+    configuration.
+
+
 v2.11.1 - 2024-11-19
 ====================
 

Dockerfile

@@ -62,8 +62,10 @@ COPY --from=build-image /opt /opt
 ENV PATH="/virtualenv/bin:$PATH"
 ENV VIRTUAL_ENV="/virtualenv"
 ENV LD_LIBRARY_PATH="/opt/oracle/instantclient"
+# IPv6 support is not enabled by default, only bind IPv4
+ENV QE_HOST="0.0.0.0"
 
 EXPOSE 9560/tcp
 VOLUME /config
-# IPv6 support is not enabled by default, only bind IPv4
-ENTRYPOINT ["query-exporter", "/config/config.yaml", "-H", "0.0.0.0"]
+WORKDIR /config
+ENTRYPOINT ["query-exporter"]

README.rst

@@ -15,10 +15,13 @@ Each query can be run on multiple databases, and update multiple metrics.
 
 The application is simply run as::
 
-  query-exporter config.yaml
+  query-exporter
 
-where the passed configuration file contains the definitions of the databases
-to connect and queries to perform to update metrics.
+which will look for a ``config.yaml`` configuration file in the current
+directory, containing the definitions of the databases to connect and queries
+to perform to update metrics.  The configuration file can be overridden by
+passing the ``--config`` option, or setting the ``QE_CONFIG`` environment
+variable.
 
 
 Configuration file format
@@ -123,7 +126,7 @@ Each database definitions can have the following keys:
 
   All entries are optional, except ``dialect``.
 
-  Note that in the string form, username, password and options need to be
+  **Note**: in the string form, username, password and options need to be
   URL-encoded, whereas this is done automatically for the key/value form.
 
   See `database-specific options`_ page for some extra details on database
@@ -215,12 +218,10 @@ Each metric definition can have the following keys:
   for ``counter`` metrics, whether to increment the value by the query result,
   or set the value to it.
 
-  By default, counters are incremented by the value returned by the query. If
-  this is set to ``false``, instead, the metric value will be set to the result
+  By default, counters are set to the value returned by the query. If this is
+  set to ``true``, instead, the metric value will be incremented by the result
   of the query.
 
-  **NOTE**: The default will be reversed in the 3.0 release, and ``increment``
-  will be set to ``false`` by default.
 
 ``queries`` section
 ~~~~~~~~~~~~~~~~~~~
@@ -504,8 +505,9 @@ included by passing ``--process-stats`` in the command line.
 Debugging / Logs
 ----------------
 
-You can enable extended logging using the ``-L`` commandline switch. Possible
-log levels are ``CRITICAL``, ``ERROR``, ``WARNING``, ``INFO``, ``DEBUG``.
+You can enable extended logging using the ``-L`` (or ``-log-level``) command
+line switch. Possible log levels are ``critical``, ``error``, ``warning``,
+``info``, ``debug``.
 
 
 Database engines
@@ -539,6 +541,8 @@ To configure the daemon:
 
 - create or edit ``/var/snap/query-exporter/current/config.yaml`` with the
   configuration
+- optionally, create a ``/var/snap/query-exporter/current/.env`` file with
+  environment variables definitions for additional config options
 - run ``sudo snap restart query-exporter``
 
 The snap has support for connecting the following databases:
@@ -563,12 +567,10 @@ where ``$CONFIG_DIR`` is the absolute path of a directory containing a
 ``config.yaml`` file, the configuration file to use. Alternatively, a volume
 name can be specified.
 
-
-A different ODBC driver version to use can be specified during image building,
-by passing ``--build-arg ODBC_bVERSION_NUMBER``, e.g.::
-
-  docker build . --build-arg ODBC_DRIVER_VERSION=17
-
+If, a ``.env`` file is present in the specified volume for ``/config``, its
+content is loaded and applied to the environment for the exporter. The location
+of the dotenv file can be customized by setting the ``QE_DOTENV`` environment
+variable.
 
 The image has support for connecting the following databases:
 
@@ -582,6 +584,14 @@ The image has support for connecting the following databases:
 
 A `Helm chart`_ to run the container in Kubernetes is also available.
 
+ODBC driver version
+~~~~~~~~~~~~~~~~~~~
+
+A different ODBC driver version to use can be specified during image building,
+by passing ``--build-arg ODBC_bVERSION_NUMBER``, e.g.::
+
+  docker build . --build-arg ODBC_DRIVER_VERSION=17
+
 
 .. _Prometheus: https://prometheus.io/
 .. _SQLAlchemy: https://www.sqlalchemy.org/

pyproject.toml

@@ -41,20 +41,21 @@ dynamic = [
 ]
 dependencies = [
   "aiohttp",
-  "argcomplete",
   "croniter",
   "jsonschema",
-  "prometheus-aioexporter>=2,<3",
+  "prometheus-aioexporter>=3",
   "prometheus-client",
   "python-dateutil",
   "pyyaml",
   "sqlalchemy>=2",
+  "structlog",
   "toolrack>=4",
 ]
 optional-dependencies.testing = [
   "pytest",
   "pytest-asyncio",
   "pytest-mock",
+  "pytest-structlog",
 ]
 urls.changelog = "https://github.com/albertodonato/query-exporter/blob/main/CHANGES.rst"
 urls.homepage = "https://github.com/albertodonato/query-exporter"

query_exporter/__init__.py

@@ -1,3 +1,3 @@
 """Export Prometheus metrics generated from SQL queries."""
 
-__version__ = "2.11.1"
+__version__ = "3.0.0"

query_exporter/config.py

@@ -1,26 +1,23 @@
 """Configuration management functions."""
 
 from collections import defaultdict
-from collections.abc import Iterable, Mapping
+from collections.abc import Mapping
 from dataclasses import dataclass
 from functools import reduce
 from importlib import resources
 import itertools
-from logging import Logger
 import os
 from pathlib import Path
 import re
-from typing import (
-    IO,
-    Any,
-)
+import typing as t
 from urllib.parse import (
     quote_plus,
     urlencode,
 )
 
 import jsonschema
 from prometheus_aioexporter import MetricConfig
+import structlog
 import yaml
 
 from .db import (
@@ -38,6 +35,7 @@
     name=DB_ERRORS_METRIC_NAME,
     description="Number of database errors",
     type="counter",
+    config={"increment": True},
 )
 
 # metric for counting performed queries
@@ -47,6 +45,7 @@
     description="Number of database queries",
     type="counter",
     labels=("query", "status"),
+    config={"increment": True},
 )
 # metric for tracking last query execution timestamp
 QUERY_TIMESTAMP_METRIC_NAME = "query_timestamp"
@@ -94,14 +93,20 @@ class Config:
 Environ = Mapping[str, str]
 
 # Content for the "parameters" config option
-ParametersConfig = list[dict[str, Any]] | dict[str, list[dict[str, Any]]]
+ParametersConfig = list[dict[str, t.Any]] | dict[str, list[dict[str, t.Any]]]
 
 
 def load_config(
-    config_fd: IO[str], logger: Logger, env: Environ = os.environ
+    config_path: Path,
+    logger: structlog.stdlib.BoundLogger | None = None,
+    env: Environ = os.environ,
 ) -> Config:
     """Load YAML config from file."""
-    data = defaultdict(dict, yaml.safe_load(config_fd))
+    if logger is None:
+        logger = structlog.get_logger()
+
+    with config_path.open() as fd:
+        data = defaultdict(dict, yaml.safe_load(fd))
     _validate_config(data)
     databases, database_labels = _get_databases(data["databases"], env)
     extra_labels = frozenset([DATABASE_LABEL]) | database_labels
@@ -115,7 +120,7 @@ def load_config(
 
 
 def _get_databases(
-    configs: dict[str, dict[str, Any]], env: Environ
+    configs: dict[str, dict[str, t.Any]], env: Environ
 ) -> tuple[dict[str, DataBaseConfig], frozenset[str]]:
     """Return a dict mapping names to database configs, and a set of database labels."""
     databases = {}
@@ -147,7 +152,7 @@ def _get_databases(
 
 
 def _get_metrics(
-    metrics: dict[str, dict[str, Any]], extra_labels: frozenset[str]
+    metrics: dict[str, dict[str, t.Any]], extra_labels: frozenset[str]
 ) -> dict[str, MetricConfig]:
     """Return a dict mapping metric names to their configuration."""
     configs = {}
@@ -179,7 +184,7 @@ def _get_metrics(
 
 
 def _validate_metric_config(
-    name: str, config: dict[str, Any], extra_labels: frozenset[str]
+    name: str, config: dict[str, t.Any], extra_labels: frozenset[str]
 ) -> None:
     """Validate a metric configuration stanza."""
     if name in GLOBAL_METRICS:
@@ -194,7 +199,7 @@ def _validate_metric_config(
 
 
 def _get_queries(
-    configs: dict[str, dict[str, Any]],
+    configs: dict[str, dict[str, t.Any]],
     database_names: frozenset[str],
     metrics: dict[str, MetricConfig],
     extra_labels: frozenset[str],
@@ -239,13 +244,13 @@ def _get_queries(
 
 
 def _get_query_metrics(
-    config: dict[str, Any],
+    config: dict[str, t.Any],
     metrics: dict[str, MetricConfig],
     extra_labels: frozenset[str],
 ) -> list[QueryMetric]:
     """Return QueryMetrics for a query."""
 
-    def _metric_labels(labels: Iterable[str]) -> list[str]:
+    def _metric_labels(labels: t.Iterable[str]) -> list[str]:
         return sorted(set(labels) - extra_labels)
 
     return [
@@ -256,7 +261,7 @@ def _metric_labels(labels: Iterable[str]) -> list[str]:
 
 def _validate_query_config(
     name: str,
-    config: dict[str, Any],
+    config: dict[str, t.Any],
     database_names: frozenset[str],
     metric_names: frozenset[str],
 ) -> None:
@@ -313,7 +318,7 @@ def _convert_interval(interval: int | str | None) -> int | None:
     return int(interval) * multiplier
 
 
-def _resolve_dsn(dsn: str | dict[str, Any], env: Environ) -> str:
+def _resolve_dsn(dsn: str | dict[str, t.Any], env: Environ) -> str:
     """Build and resolve the database DSN string from the right source."""
 
     def from_env(varname: str) -> str:
@@ -347,7 +352,7 @@ def from_file(filename: str) -> str:
     return dsn
 
 
-def _build_dsn(details: dict[str, Any]) -> str:
+def _build_dsn(details: dict[str, t.Any]) -> str:
     """Build a DSN string from details."""
     url = f"{details['dialect']}://"
     user = details.get("user")
@@ -375,7 +380,7 @@ def _build_dsn(details: dict[str, Any]) -> str:
     return url
 
 
-def _validate_config(config: dict[str, Any]) -> None:
+def _validate_config(config: dict[str, t.Any]) -> None:
     schema_file = resources.files("query_exporter") / "schemas" / "config.yaml"
     schema = yaml.safe_load(schema_file.read_bytes())
     try:
@@ -385,38 +390,44 @@ def _validate_config(config: dict[str, Any]) -> None:
         raise ConfigError(f"Invalid config at {path}: {e.message}")
 
 
-def _warn_if_unused(config: Config, logger: Logger) -> None:
+def _warn_if_unused(
+    config: Config, logger: structlog.stdlib.BoundLogger
+) -> None:
     """Warn if there are unused databases or metrics defined."""
     used_dbs: set[str] = set()
     used_metrics: set[str] = set()
     for query in config.queries.values():
         used_dbs.update(query.databases)
         used_metrics.update(metric.name for metric in query.metrics)
 
-    unused_dbs = sorted(set(config.databases) - used_dbs)
-    if unused_dbs:
+    if unused_dbs := sorted(set(config.databases) - used_dbs):
         logger.warning(
-            f"unused entries in \"databases\" section: {', '.join(unused_dbs)}"
+            "unused config entries",
+            section="databases",
+            entries=unused_dbs,
         )
-    unused_metrics = sorted(
+    if unused_metrics := sorted(
         set(config.metrics) - GLOBAL_METRICS - used_metrics
-    )
-    if unused_metrics:
+    ):
         logger.warning(
-            f"unused entries in \"metrics\" section: {', '.join(unused_metrics)}"
+            "unused config entries",
+            section="metrics",
+            entries=unused_metrics,
         )
 
 
-def _get_parameters_sets(parameters: ParametersConfig) -> list[dict[str, Any]]:
+def _get_parameters_sets(
+    parameters: ParametersConfig,
+) -> list[dict[str, t.Any]]:
     """Return an sequence of set of paramters with their values."""
     if isinstance(parameters, dict):
         return _get_parameters_matrix(parameters)
     return parameters
 
 
 def _get_parameters_matrix(
-    parameters: dict[str, list[dict[str, Any]]],
-) -> list[dict[str, Any]]:
+    parameters: dict[str, list[dict[str, t.Any]]],
+) -> list[dict[str, t.Any]]:
     """Return parameters combinations from a matrix."""
     # first, flatten dict like
     #