Managing Elixir configs

2022-10-20 by Tomasz Kowal

elixir

Have you ever tried to deploy your Elixir/Phoenix application and the deployment failed due to a missing environment variable?

Or worse! The deployment succeeded, but the app crashed on the staging environment?

Or even worse! The environment variables were different in the staging environment and production, so you had to roll back?

I am going to talk about structuring your configs in such a way that minimizes differences between environments.

How do Elixir configs work?

I will not describe everything in detail, but I’ll highlight points essential to understanding the problem and proposed solution.

MIX_ENV and default environments

MIX_ENV describes the environment in which you apply configs. It is very flexible. You can define your environments according to your needs. However, the default generator provides you with three: dev, test and prod. The names look self-explanatory, but they can be misleading.

E.g. is your staging environment development or production? Should you create a new one like stag?

The consensus is that dev is your local development, test is for running mix test locally and in CI, and prod is everything that is deployed. That is the first important point; you will rarely need more than those three environments.

Compile-time vs runtime

Elixir evaluates compile-time configs during compile-time. That means:

• You can’t use your dependencies here because they are not compiled yet
• You can import_config here based on MIX_ENV.
• You can import_config based on MIX_ENV, so the default is to have config.exs, dev.exs, test.exs, and prod.exs.

Elixir evaluates runtime configs before booting the app (either with mix or as a release). That means:

• You can use your dependencies
• There is no MIX_ENV because we might be in a release
• There is config_env() with the same set of values, but you can’t import_config.
• Any config depending on the mix environment needs to live under if config_env() == ....

Order of evaluation

• First, config.env during the compilation
• Then [MIX_ENV].exs still during compilation
• Then runtime.exs when the application starts

Elixir “deep-merges” configurations. If a key occurs later during evaluation, it overrides the previous value. Unless both values are keyword lists, then Elixir merges them.

Assumptions

The need to control compilation with environment variables is infrequent. The only “compile-time environment variable” should be MIX_ENV. It would be best to read all other variables only in runtime.exs.

I’ll assume you have a deploy mechanism that prevents the deployment in case of an error during startup. If a variable is required to run, it is best to fail early. Use System.fetch_env!/1 instead System.get_env/1. Use System.get_env/2 only with meaningful default values that will work in production.

The problem with the default config setup

Our problem with configs generated by default is that they have big if config_env() == :prod in the runtime.exs. It is easy to configure your local and test environment, push changes, see the CI green and try to deploy without noticing it won’t work when deployed.

I want to use the same environment variables in development and production to automate checking if they are all set on remote servers.

Part1 of the solution - direnv

Direnv is an incredible tool that handles setting environment variables for you. You define the .envrc file with your exports, it asks you to confirm you like them (it is a security feature), and voilà, you are good to go.

With this, you can remove the nasty if config_env() == :prod. However, our tests would like to overwrite a handful of those values, e.g. database name or port.

Part2 of the solution - ConfigHelper

def fetch_env!(var) do
  value =
    with :test <- config_env(),
         {:ok, test_val} <- System.fetch_env("TEST_" <> key) do
      test_val
    else
      _ -> System.fetch_env!(key)
    end
end

This helper will behave like System.fetch_env! except in :test environment, it will first try to find TEST_ENV, and if it is not there, it will fall back to ENV.

We discovered that there is usually only a handful of “test overrides”, so it is OK to keep them in the .envrc alongside regular variables.

Benefits

With all necessary variables in .envrc and no conditional logic in runtime.exs, it is easy to compare local and deployed environments (dev and prod). We find it more maintainable.

Drawbacks

If someone unfamiliar with the convention enters the project without a proper introduction, they might be confused about all those TEST_ variables that are seemingly never read.

Should Phoenix adopt that technique?

I don’t think so. Needs vary, and the default configuration is not surprising. However, with a web app that only sticks to the basics, you can consider unifying environment variables across local and deployed environments with that trick.

Credits

I’d like to thank my colleague Henrik Sjööh who proposed that solution during our discussions at Bluecode