Struggles with integrating 1Password's op

[rsyring]

I've been trying to figure out how to get 1Password secrets integrated with mise (refs #1359). The biggest challenge I've encountered is that the op utility is rather slow, taking about 1s to return when asked for secrets. That delay on every prompt is pretty annoying.

Since direnv doesn't run at every prompt, I thought I had this solved with a pretty simple setup:

# .mise.toml
[env]
TF_VAR_wasabi_access_key = 'op://private/aws/access-key'
TF_VAR_wasabi_secret_key = 'op://private/aws/secret-key'

# .envrc
direnv_load op run --no-masking -- direnv dump
watch_file .mise.toml

This setup has the advantage of being usable with op run directly if someone prefers not to use direnv. The problem I encountered is that there is a conflict between direnv and mise which led to the environment variables not getting removed when leaving the directory. The secrets were no longer resolved but the environment variables (TF_VAR_*) remained in the current shell.

The benefit of direnv for this setup is that .envrc isn't executed on every prompt. If I need to refresh the secrets its easy with direnv reload.

Since mise has the stated intention of replacing direnv for any use case, it would be better to do:

# .mise.toml
[env]
TF_VAR_wasabi_access_key = 'op://private/aws/access-key'
TF_VAR_wasabi_secret_key = 'op://private/aws/secret-key'
_.source = 'op-run.sh'

But, in this case, for performance to be acceptable, I'd only want op-run.sh run the first time it was needed and/or on demand.

This may also also impact #1448.

One other thought regarding this: since we are dealing with secrets, I wouldn't want to see these values cached to the filesystem.

[andrewthauer]

I'm also using direnv with op to do similar things atm so I would find this useful.

I'm curious if there is a way to opt of out having the _mise_hook a precmd_function? I think if there was a way to activate mise as only a chpwd_function, this might resolve your issue. The alternative might be to avoid activating mise on shell startup and use eval $(mise hook-env) instead on demand.

[jdx]

you could experiment by running mise activate zsh > some_file then sourcing it in your zshrc with modifications

[jdx]

I'm not totally sure how we could resolve this since that would only work for zsh. Maybe what we could do is reuse previously resolved env vars if they've already been resolved and maybe offer a mise refresh command or something to get it to re-resolve them.

Gratefully, this would be feasible now that the env._ changes have been made since that logic is centralized and not part of config loading anymore.

[rsyring]

Maybe what we could do is reuse previously resolved env vars if they've already been resolved and maybe offer a mise refresh command or something to get it to re-resolve them.

This is exactly how it works (for me anyway) with direnv. If I have to adjust env vars after changing directories into the project, then I run direnv reload.

I'm assuming this would apply to scripts too? If so, I could see the possibility that some scripts people are running now are intended to be ran on every prompt.

Gratefully, this would be feasible now that the env._ changes have been made since that logic is centralized and not part of config loading anymore.

:)

[rsyring]

Maybe what we could do is reuse previously resolved env vars if they've already been resolved and maybe offer a mise refresh command or something to get it to re-resolve them.

Because these are secrets, I wouldn't want them cached to the file system. That means they'd have to stay in the shell. That would be a pretty big shift in architecture to what mise does now?

[jdx]

I'm not talking about caching to the file system, this would go into a different internal environment var like __MISE_DIFF. I think we'd need to track all of the current exported variables and what their source is—since if 2 config files export the same env var key that would require invalidation.

This is exactly how it works (for me anyway) with direnv.

Good to know, it's probably safe for us to do similar logic.

[jdx]

There shouldn't be any security risk here since the env vars already contain the secret's value. In fact we don't even need to track the value of the keys I don't think since they're already exported, we just need to track their source

[rsyring]

There shouldn't be any security risk here since the env vars already contain the secret's value. In fact we don't even need to track the value of the keys I don't think since they're already exported, we just need to track their source

Good to know. I didn't realize the shell environment variables were persistent between prompts.

[andrewthauer]

@rsyring - Correct me if I'm wrong, but wasn't your main concern around invoking a script that in turn triggers a 1password secret dump on each prompt/cmd?

I'm currently using op with direnv in some projects to trigger a script to set the secrets on cd change into the root project directory (via direnv). This then prompts for biometrics auth on every shell opening. It doesn't however prompt for auth on cd change within the project so it's not horrible.

Even on entry to the root can be a bit much and I've experimented and often just use a shell function ondemand when I need the secrets loaded, since many commands don't require them.

However, my point of this is that if we rely on the current _mise_hook to run mise hook-env this will occur on every cmd prompt (not just cd)? I don't think 1password will prompt each time for biometrics auth, but it will take a significant performance hit to run a source script that does the secrets dump.

So, I think the only reasonable approach would be to only do this on cd change into the project root. Back to my question about only running the _mise_hook on chpwd rather then precmd.

[rsyring]

@andrewthauer I may not be fully understanding your comments. The point of this issue was to address being able to run a script once (presumably on changing into a project directory) instead of at every prompt. I think you are saying the same thing.

We've also discussed just above being able to load on demand (e.g. mise reload, like direnv reload).

I guess another consideration would be to have the option to ONLY load on demand. For some of my projects, where I'm managing infrastructure, I always need secrets. For a others, I almost never do because I can do most of the development locally and only occasionally want to run against a live service. e.g. mise exec --keep-env secrets.sh

[andrewthauer]

Essentially, yes, I'm saying the same thing, but I guess I'm getting into the the technical details of how mise is working atm. When you eval $mise activate $shell) this is creating a hook that essentially calls mise hook-env on every cd change + every comment prompt. If you have something like env._.source which in turn is executing op to get secrets, this is going to be executed every command prompt since the prompt/cd change hook is calling mise hook-env.

So not activating and manually calling eval $(mise hook-env $shell) once on demand would avoid the extra cost of invoking the sourced script each prompt/cmd. However, you lose the benefit of mise auto activate in this case (if you care about that workflow).

What I was ultimately trying to say, which probably did a horrible job at, was that you can't really get your cake and eat it while using the mise activate in your zshrc, bashprofile without getting the performance hit on each prompt. Which isn't noticeable normally but would be for this sort of use case. However, if there was a way to say I only want to use the cd change hook (chpwd in zsh speak), then this hit would only occur on a cd change.

I'm not sure how mise would selectively only run the source script in some cases and not in others. For example if you direnv into a project directory with an .envrc file it will execute that. If that calls to op it runs it. However, if then you cd into a child directory it won't re-execute the .envrc file in the parent directory. Not sure this is the case with mise atm, but imo this would be desirable.

[jdx]

When you eval $mise activate $shell) this is creating a hook that essentially calls mise hook-env on every cd change + every comment prompt.

while accurate there is some short circuiting logic that might not be working correctly here. This should be preventing env._.source from running every time unless something isn't working right.

[andrewthauer]

TBH, I haven't tested this particular use case with mise yet. So if there is short circuiting logic then that might suffice along with the suggested on demand reload ability.

[andrewthauer]

So I had a go at porting my current op + direnv setup to mise. I think it can work as is, but ... there are at least a couple of major issues that make it not feasible atm (at least for me).

First I'll document my 2 current approaches for completeness.

Method 1: Direnv library helper

This combines a direnv re-usable function that uses the direnv_load & direnv dump which @rsyring is also using. It works relatively well, since when I cd into a project I get prompted by 1password for auth to resolve the secrets in the .env file that contains secrets refs then makes them available in my current shell. Subsequent command within the project are not slow as it doesn't seem to attempt to re-run the use_op_env in my .envrc file.

# ~/.config/direnv/lib/op.sh
use_op_env() {
   # implementation omitted to be brief, but essentially does a `direnv_load op run -- direnv dump`
}

# Then in a projects .envrc file
# ~/src/some-project/.envrc
...
use op_env some-account.1password.com

Method 2: On demand shell alias

This works well for either some shared global secrets and does not rely on direnv at all. It' also avoids getting the potentially annoying upfront prompt from 1password for auth or startup hit.

# ~/.local/bin/op-dotenv
# this essentially replaces the direnv_load part and does:
#   `cat <"$dotenv_file" | op inject` and then spits out the resulting resolved variables

# ~/.zshrc
alias ope='eval $(op-dotenv "$HOME"/.config/.global-env --export)'

# then at any point run this alias to get those secrets into your shell. op will prompt for auth the first time in the shell
ope

Usage in mise

In mise since I want to avoid usages of direnv I needed to adapt method 2 and came up with this ...

First the file to source which in turn calls the op-dotenv helper which invokes op and spits out the resolved env variables with export in front.

# ~/src/some-project/.config/scripts/op-env.sh
op-dotenv "$PWD"/.env --export

Then in the mise config

# ~/src/some-project/.mise.toml
[env]
_.source = "./config/scripts/op_env.sh"   # file above

NOTE: My setup could be adapted likely to work with variables defined in [env]. The op-dotenv would likely end up need to be run after the [env] variables are applied, then do something like env | op-dotenv which would in turn resolved any secret references in the env and export the resolved versions on top.

This works as expected, however, 2 issues I ran into:

1. It seems to invoke the sourced op_env.sh script after every command is executed. So, I think this confirms the short circuiting may not be working?

2. If my underlying op-dotenv executable script uses something that is managed by mise it gets caught in and endless loop. For example I have 2 versions of the op-dotenv I wrote. One in bash and one in deno. The bash one works fine, but the deno one infinity loops. My suspicion is this is because I use mise to manage my deno versions. I can file a separate issue for this.

So, in the end it seems definitely doable, but the 2 points above would need to be addressed before it can be widely used I think. The other thing that would be nice is a way to used "share" helpers like direnv through some common place to look up plugins (e.g. ~/.local/config/mise/lib).

[jdx]

I know this wouldn't be perfect but I wonder if somehow using the enter hook would solve this problem. You could have that in your global mise.toml which would effectively only run once. That can't export env vars though so idk how helpful it would be. Perhaps we could extend that somehow to allow for this.

Actually this made me realize that there is a bug using the global config, it never gets run for enter, but that's a seperate problem.

[jdx]

the specific issue where mise doesn't run if config files didn't change should be fixed now that hooks are out. As part of that I needed to make hook-env run anytime the directory changes.

[andrewthauer]

I tried revisiting this setup and got a little further. I tried 2 approaches:

1. using _.source = "some-script.sh that resolves secrets from 1password CLI, and then exports the resulting env vars. For some reason the exported env vars where not showing up in my parent shell, but I did get prompted by 1password. Would need to debug this more.

2. I created a quick POC of a lua plugin with a mise_env.lua hook, that calls out to 1password CLI and then modifies the path. This works and the resolve env vars show up in the parent shell.

However, both of these solutions currently suffer from the fact they get executed every single prompt (even non mise ones). I think twice actually, but not sure. This means it's a no go since the secrets resolution can take a few seconds.

With my current direnv solution it only happens the first cd into the directory containing the .envrc file. I can cd into child directories, etc. run commands and it won't try and re-run the file. I haven't looked, but I suspect direnv does some hashing of the current state and the .envrc file to determine if it needs to run or not when changing directories. Imo, something like this would be a hard requirement for me being able to use mise for the purposes of loading secrets or running longer processes to setup the env for a project.

@jdx - Am I missing anything currently built in that would help here? Or is this something new that would need to be incorporated into mise?

[jdx]

I think twice actually, but not sure.

this is likely shell specific but I did recently discover when changing directories we effectively call twice (once for cd and once for the prompt). I'm not sure I actually want to fix that because that code inevitably causes unexpected bugs whenever I touch it. hook-env should be fast on the second execution due to early exit though

I did recently add support for shell hooks which would be a partial, albeit ugly way to solve this. The problem with that is this is straight up executing shell code—mise won't be able to remove env vars added there once you leave. While you could in theory write a leave hook to manually clean up, I haven't implemented that hook yet and it's not trivial to add that one because reasons.

Ultimately it sounds like what you're looking for is the ability to use _.source (assuming it worked correctly) but with the ability to only run it on cd and not every prompt. btw there is an e2e test for this so it should be working. Maybe this could be configured like this:

[env]
_.source = {path = "some-script.sh", when="cd"}

As far as implementation, this would be a bit tricky since typically re-resolve this every time so there would need to be some caching in place. This would either be in the form of caching as a file (which is probably easier-implementation wise since we do similar things) or storing previous values in a distinct env var (as serialized data) like __MISE_ENV_SOURCE or something.

Alternatively, we could extend this to the hooks functionality as something like this:

[hooks]
enter = {source = "some-script.sh"}

Though I think this would be harder, since the hooks would somehow need to communicate the env var change in a way that it gets reset later. I think I don't like this idea.

[ncadou]

[env]
_.source = {path = "some-script.sh", when="cd"}

Interesting. This sourcing-at-every-prompt prevents me from switching to mise, as some projects I work on pull too many secrets from password-store to be practical with mise, and I was about to post my own suggestion when I found this discussion. Although maybe less of a concern, to fully emulate how smoothly direnv works in my case, I was also thinking about hot-reloading when changes are detected:

[env]
_.source = {path = "some-script.sh", when="cd", watch = true}

[syhol]

Just in-case you didn't know the current state of things @ncadou, the current recommended way of working with secrets in a password store is to use the password store cli to create a local json/yaml/toml file, then use sops/age to encrypt it. mise can read these sops/age files very fast with the file directive:

[env]
_.file = ".env.json"

You could automate this with a few lines in a mise task, and (if you wanted to) you could regenerate this file in an enter hook:

[env]
_.file = ".env.json"

[hooks]
enter = "mise run fetch-secrets"

[tasks.fetch-secrets]
run = '''
dcli inject --in .env.json.template --out .env.json
sops encrypt -i --age "~/.config/mise/age.txt" .env.json
'''

This is with the dashlane cli and you can imagine the .env.json.template looks something like:

{
  "AWS_ACCESS_KEY_ID": "{{ dl://7593DD31-5FA5-40F4-85B8-F5131FB6C617/password }}",
  "AWS_SECRET_ACCESS_KEY": "{{ dl://88C50F1F-E779-4DBE-92CD-63713F00E653/password }}"
}

[rsyring]

FWIW, I ended up realizing that setting environment variables was not always straight forward for a project. A decent number of my projects actually had groups of environment variables that needed to be set based on what I was trying to do. Different cloud environments, running Ansible vs Terraform for a project, etc. etc. That was true for more than just secrets as well, maybe API URLs changed, maybe I wanted to set AWS_PROFILE differently.

I also realized I often didn't want secrets loaded whenever I was in a project's directly b/c I wasn't going to do anything that used a secret. I.e. I was just going to be doing local development and would not be deploying anything.

I wanted to have the ability to define these profiles as well as integrate with secrets (we use 1Password) so I created env-config.

We still use mise for a lot including static environment variables but now use env-config for secrets and vars that are context/activity dependent.