Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Define syntax for escaping declarative config env var substitution #4375

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
+55 −41
Open

Define syntax for escaping declarative config env var substitution #4375

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
e7f6017
Define syntax for escaping declarative config env var substitution
jack-berg Jan 17, 2025
92391d9
Convert examples to table, adopt marcalffs proposal
jack-berg Jan 24, 2025
760426a
typo
jack-berg Jan 24, 2025
20121c4
Add double quotes to output
jack-berg Jan 24, 2025
e50dbb6
original => input
jack-berg Jan 24, 2025
2d9633f
style
jack-berg Jan 24, 2025
c95c10c
Add changelog entry
jack-berg Jan 24, 2025
a26c3ff
Merge branch 'main' of https://github.com/open-telemetry/opentelemetr…
jack-berg Jan 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@ release.

### SDK Configuration

- Define syntax for escaping declarative configuration environment variable
references.
([#4375](https://github.com/open-telemetry/opentelemetry-specification/pull/4375))

### Common

### Supplementary Guidelines
Expand Down
92 changes: 51 additions & 41 deletions specification/configuration/data-model.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,29 @@ more non line break characters (i.e. any character except `\n`). If a referenced
environment variable is not defined and does not have a `DEFAULT_VALUE`, it MUST
be replaced with an empty value.

The `$` character is an escape sequence, such that `$$` in the input is
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: add changelog entry before merging

translated to a single `$` in the output. The resolved `$` from an escape
sequence MUST not be considered when matching input against the environment
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MUST NOT ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think (correct me if I'm wrong) this is a critical part of the mechanics:

Assuming API_KEY=123.

If the resolved $ is not considered when matching against environment variables, we have:

"$${API_KEY}" => "$" + resolveEnvVars("{API_KEY}") => "${API_KEY}"

If instead, the resolve $ is considered when matching against environment variables, we could have something like:

"$${API_KEY}" => "$" + "{API_KEY}" => resolveEnvVars("${API_KEY}") => "123".

In other words, without this, an parser might think that it can do the substitution in two separate passes. The first pass replacing $$ with $. The second pass replacing env var substitution references. A two pass approach produces incorrect results.

variable substitution regular expression. For example, `$${API_KEY}` resolves
to `${API_KEY}`, and the value of the `API_KEY` environment variable is NOT
substituted. See table below for more examples. In practice, this implies that
parsers consume the input from left to right, iteratively identifying the next
escape sequence, and matching the content since the prior escape sequence
against the environment variable substitution regular expression.

For example, the pseudocode for processing input `$${FOO} ${BAR} $${BAZ}`
where `FOO=a, BAR=b, BAZ=c` would resemble:

* Identify escape sequence `$$` at index 0. Perform substitution
against `input.substring(0, 0)=""` => `""` and append to output. Append `$`
to output. Current output: `"$"`.
* Identify escape sequence `$$` at index 15. Perform substitution
against `input.substring(0+2, 15)="{FOO} ${BAR} "` => `"{FOO} b "` and append
to output. Append `$` to output. Current output: `"${FOO} b $"`.
* Reach end of input without escape sequence. Perform substitution
against `input.substring(15+2, input.length)="{BAZ}"` => `"{BAZ}"` and append
to output. Return output: `"${FOO} b ${BAZ}"`.

When parsing a configuration file that contains a reference not matching
the references regular expression but does match the following PCRE2
regular expression, the parser MUST return an empty result (no partial
Expand All @@ -94,55 +117,42 @@ example, see references to `INVALID_MAP_VALUE` environment variable below.
It MUST NOT be possible to inject environment variable by environment variables.
For example, see references to `DO_NOT_REPLACE_ME` environment variable below.

For example, consider the following environment variables,
and [YAML](#yaml-file-format) configuration file:
The table below demonstrates environment variable substitution behavior for a
variety of inputs. Examples assume environment variables are set as follows:

```shell
export STRING_VALUE="value"
export BOOL_VALUE="true"
export INT_VALUE="1"
export FLOAT_VALUE="1.1"
export HEX_VALUE="0xdeadbeef" # A valid integer value written in hexadecimal
export HEX_VALUE="0xdeadbeef" # A valid integer value (i.e. 3735928559) written in hexadecimal
export INVALID_MAP_VALUE="value\nkey:value" # An invalid attempt to inject a map key into the YAML
export DO_NOT_REPLACE_ME="Never use this value" # An unused environment variable
export REPLACE_ME='${DO_NOT_REPLACE_ME}' # A valid replacement text, used verbatim, not replaced with "Never use this value"
```

```yaml
string_key: ${STRING_VALUE} # Valid reference to STRING_VALUE
env_string_key: ${env:STRING_VALUE} # Valid reference to STRING_VALUE
other_string_key: "${STRING_VALUE}" # Valid reference to STRING_VALUE inside double quotes
another_string_key: "${BOOL_VALUE}" # Valid reference to BOOL_VALUE inside double quotes
string_key_with_quoted_hex_value: "${HEX_VALUE}" # Valid reference to HEX_VALUE inside double quotes
yet_another_string_key: ${INVALID_MAP_VALUE} # Valid reference to INVALID_MAP_VALUE, but YAML structure from INVALID_MAP_VALUE MUST NOT be injected
bool_key: ${BOOL_VALUE} # Valid reference to BOOL_VALUE
int_key: ${INT_VALUE} # Valid reference to INT_VALUE
int_key_with_unquoted_hex_value: ${HEX_VALUE} # Valid reference to HEX_VALUE without quotes
float_key: ${FLOAT_VALUE} # Valid reference to FLOAT_VALUE
combo_string_key: foo ${STRING_VALUE} ${FLOAT_VALUE} # Valid reference to STRING_VALUE and FLOAT_VALUE
string_key_with_default: ${UNDEFINED_KEY:-fallback} # UNDEFINED_KEY is not defined but a default value is included
undefined_key: ${UNDEFINED_KEY} # Invalid reference, UNDEFINED_KEY is not defined and is replaced with ""
${STRING_VALUE}: value # Invalid reference, substitution is not valid in mapping keys and reference is ignored
recursive_key: ${REPLACE_ME} # Valid reference to REPLACE_ME
# invalid_identifier_key: ${STRING_VALUE:?error} # If uncommented, this is an invalid identifier, it would fail to parse
```

Environment variable substitution results in the following YAML:

```yaml
string_key: value # Interpreted as type string, tag URI tag:yaml.org,2002:str
env_string_key: value # Interpreted as type string, tag URI tag:yaml.org,2002:str
other_string_key: "value" # Interpreted as type string, tag URI tag:yaml.org,2002:str
another_string_key: "true" # Interpreted as type string, tag URI tag:yaml.org,2002:str
string_key_with_quoted_hex_value: "0xdeadbeef" # Interpreted as type string, tag URI tag:yaml.org,2002:str
yet_another_string_key: "value\nkey:value" # Interpreted as type string, tag URI tag:yaml.org,2002:str
bool_key: true # Interpreted as type bool, tag URI tag:yaml.org,2002:bool
int_key: 1 # Interpreted as type int, tag URI tag:yaml.org,2002:int
int_key_with_unquoted_hex_value: 3735928559 # Interpreted as type int, tag URI tag:yaml.org,2002:int
float_key: 1.1 # Interpreted as type float, tag URI tag:yaml.org,2002:float
combo_string_key: foo value 1.1 # Interpreted as type string, tag URI tag:yaml.org,2002:str
string_key_with_default: fallback # Interpreted as type string, tag URI tag:yaml.org,2002:str
undefined_key: # Interpreted as type null, tag URI tag:yaml.org,2002:null
${STRING_VALUE}: value # Interpreted as type string, tag URI tag:yaml.org,2002:str
recursive_key: ${DO_NOT_REPLACE_ME} # Interpreted as type string, tag URI tag:yaml.org,2002:str
```
| YAML - input | YAML - post substitution | Resolved Tag URI | Notes |
|-------------------------------------------|-----------------------------|---------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|
| `key: ${STRING_VALUE}` | `key: value` | `tag:yaml.org,2002:str` | YAML parser resolves to string |
| `key: ${BOOL_VALUE}` | `key: true` | `tag:yaml.org,2002:bool` | YAML parser resolves to true |
| `key: ${INT_VALUE}` | `key: 1` | `tag:yaml.org,2002:int` | YAML parser resolves to int |
| `key: ${FLOAT_VALUE}` | `key: 1.1` | `tag:yaml.org,2002:float` | YAML parser resolves to float |
| `key: ${HEX_VALUE}` | `key: 0xdeadbeef` | `tag:yaml.org,2002:int` | YAML parser resolves to int `3735928559` |
| `key: "${STRING_VALUE}"` | `key: "value"` | `tag:yaml.org,2002:str` | Double quoted to force coercion to string `"value"` |
| `key: "${BOOL_VALUE}"` | `key: "true"` | `tag:yaml.org,2002:str` | Double quoted to force coercion to string `"true"` |
| `key: "${INT_VALUE}"` | `key: "1"` | `tag:yaml.org,2002:str` | Double quoted to force coercion to string `"1"` |
| `key: "${FLOAT_VALUE}"` | `key: "1.1"` | `tag:yaml.org,2002:str` | Double quoted to force coercion to string `"1.1"` |
| `key: "${HEX_VALUE}"` | `key: "0xdeadbeef"` | `tag:yaml.org,2002:str` | Double quoted to force coercion to string `"0xdeadbeef"` |
| `key: ${env:STRING_VALUE}` | `key: value` | `tag:yaml.org,2002:str` | Alternative `env:` syntax |
| `key: ${INVALID_MAP_VALUE}` | `key: value\nkey:value` | `tag:yaml.org,2002:str` | Map structure resolves to string and _not_ expanded |
| `key: foo ${STRING_VALUE} ${FLOAT_VALUE}` | `key: foo value 1.1` | `tag:yaml.org,2002:str` | Multiple references are injected and resolved to string |
| `key: ${UNDEFINED_KEY}` | `key:` | `tag:yaml.org,2002:null` | Undefined env var is replaced with `""` and resolves to null |
| `key: ${UNDEFINED_KEY:-fallback}` | `key: fallback` | `tag:yaml.org,2002:str` | Undefined env var results in substitution of default value `fallback` |
| `${STRING_VALUE}: value` | `${STRING_VALUE}: value` | `tag:yaml.org,2002:str` | Usage of substitution syntax in keys is ignored |
| `key: ${REPLACE_ME}` | `key: ${DO_NOT_REPLACE_ME}` | `tag:yaml.org,2002:str` | Value of env var `REPLACE_ME` is `${DO_NOT_REPLACE_ME}`, and is _not_ substituted recursively |
| `key: ${STRING_VALUE:?error}` | n/a | n/a | Invalid substitution reference produces parse error |
| `key: $${STRING_VALUE}` | `key: ${STRING_VALUE}` | `tag:yaml.org,2002:str` | `$$` escape sequence is replaced with `$`, `{STRING_VALUE}` does not match substitution syntax |
| `key: $$${STRING_VALUE}` | `key: $value` | `tag:yaml.org,2002:str` | `$$` escape sequence is replaced with `$`, `${STRING_VALUE}` is replaced with `value` |
| `key: $$$${STRING_VALUE}` | `key: $${STRING_VALUE}` | `tag:yaml.org,2002:str` | `$$` escape sequence is replaced with `$`, `$$` escape sequence is replaced with `$`, `{STRING_VALUE}` does not match substitution syntax |
| `key: a $$ b` | `key: a $ b` | `tag:yaml.org,2002:str` | `$$` escape sequence is replaced with `$` |
| `key: a $ b` | `key: a $ b` | `tag:yaml.org,2002:str` | No escape sequence, no substitution references, value is left unchanged |
Loading