Requirements for a Configuration System
There are many different approaches to organize software configuration nowadays. Environment variables, config files in numerous formats (.json, .yaml, .ini) or CLI arguments and flags, just to name a few.
According to the Twelve-Factor Apps, configuration for cloud-native applications should only be stored in the environment. I agree with many ideas of the Twelve-Factor Apps, but I don’t think that environment configuration is the best approach for every use case out there.
One reason why I don’t like environment vars is that the environment can be set in many different ways, which could be extremely confusing. For example in a container deployment on Kubernetes we could set a variable:
- with the Dockerfile at image build time
- with the deployment descriptor of Kubernetes
- with a script in the container executed on startup
Additionally, environment variables have to be documented explicitly because they are not shown obviously like flags in the help text of your executables or config values in a default config file you could publish together with your application.
I think that environment configuration is important. But it doesn’t fulfill all of my requirements for a great configuration system. From my perspective a configuration system should:
- Be centralized
- It’s important to centralize the access to config in a single package of a project. A common flaw with env vars is that there’s no centralized access to the configuration. Thus all files in the project could possibly access an environment variable. This makes it hard to find all config variables and where they are referenced.
- Be well documented
- If a user or another developer in your project doesn’t know that something is configurable, he probably won’t use the configuration. A default config file in your project repository is a great way to document available config values and defaults.
- Be structured and commented
- I prefer .yaml config files because they allow me to structure my config in sections and describe config values in more detail with comments.
- Have default values but allow specific configurations
- Default values are great if you deploy your application in a multi-stage process. You could define common configuration as default values and parametrize the distinct values for each stage with env vars or config files
- Not allow too many options
- Allowing too many different configuration approaches leads to configuration spread all other the project
Configuration for Go web services
Now that I’ve stated what I expect of a configuration system, let’s see how I handle configuration for my Go web services. I’ll combine environment configuration with config files to get the best of both worlds.
🧐 There is no perfect config system for every use case. If you’re building a CLI application, CLI args and flags are definetly a better option to configure them.
Environment Variables
We start with a configuration using environment variables like recommend by the Twelve-Factor Apps.
I use the package github.com/kelseyhightower/envconfig to pass environment variables directly to a struct instance. The Process function of this package takes a prefix and a reference to a struct. It will lookup environment variables by the uppercase prefix + “_” + the uppercase field names of the struct. The split_words tag leads to env vars that are split with “_” on capital letters. For example:
- Name = APP_NAME
- DbUser = APP_DB_USER
For nested structs, you have to define the variable name explicitly with the envconfig tag.
package main
import (
"fmt"
"io/ioutil"
"gopkg.in/yaml.v2"
)
type AppConfig struct {
Name string
Port string
Db struct {
User string `envconfig:"DB_USER"`
Password string `envconfig:"DB_PASSWORD"`
Host string `envconfig:"DB_HOST"`
Port string `envconfig:"DB_PORT"`
}
}
func (config *AppConfig) ReadEnv() {
envconfig.Process("app", config)
}
func main() {
appConfig := &AppConfig{}
appConfig.ReadEnv()
fmt.Printf("%+v\n", *appConfig)
// in a shell: APP_NAME=test go run src/main.go
// Output {Name:test AppPort: DbUser: DbPassword: DbHost: DbPort:}
}
🤓 You can set a field in the struct as required with the tag `required:”true”`
Default Config YAML File
As we see above no one can know, how the Application is configured without looking at the source code. At least we could define default values with another tag on the struct and document the variable usage with comments but this would lead to very noisy code. To solve this problem we will extend the configuration with a default config .yaml file.
config.yaml
# The service will use this file for its default configuration. # Configuration can be changed by changing this file or by # setting environment variables. The variable names are # in the commands after each field. name: test # APP_NAME port: 8444 # APP_PORT db: host: localhost # DB_HOST port: 5432 # DB_PORT user: admin # DB_USER password: 1234 # DB_PASSWORD
Here’s the code to load the config into our application, we use the package gopkg.in/yaml.v2 to unmarshal the config.yaml file:
package main
import (
"fmt"
"io/ioutil"
"github.com/kelseyhightower/envconfig"
"gopkg.in/yaml.v2"
)
type AppConfig struct {
Name string `yaml:"name"`
Port string `yaml:"port"`
Db struct {
User string `yaml:"user" envconfig:"DB_USER"`
Password string `yaml:"password" envconfig:"DB_PASSWORD"`
Host string `yaml:"host" envconfig:"DB_HOST"`
Port string `yaml:"port" envconfig:"DB_PORT"`
}
}
func (config *AppConfig) ReadEnv() {
envconfig.Process("app", config)
}
func (config *AppConfig) ReadYaml(filepath string) {
yamlFile, err := ioutil.ReadFile(filepath)
if err != nil {
fmt.Printf("Error Reading Config file with path: %v\n", filepath)
}
yaml.Unmarshal(yamlFile, config)
}
func main() {
appConfig := &AppConfig{}
// Load default config file
appConfig.ReadYaml("config.yaml")
// Load specific configuration with env variables
appConfig.ReadEnv()
fmt.Printf("%+v\n", *appConfig)
// go run src/main.go
// {Name:test Port:8444 Db:{User:admin Password:1234 Host:localhost Port:5432}}
}
Now we have a structured default configuration and the possibility to set environment variables in our runtime environment to specify distinct config values for specific environments. Additionally, we have a great way to document the configuration with comments without polluting our source code.
Specific configuration with partial config file
Let’s assume we deploy our project with a multi-stage process with 3 stages dev, pre-prod, and prod. With the current solution, there are two ways to change the configuration to be stage-specific:
- Use one config file per stage
- Would result in many duplications and effort on configuration structure changes
- Using the default config file and specify all distinct values with env vars
- Is perfectly fine for only a few distinct values
- Environment variables can get verbose very fast, for example when using ConfigMaps in Kubernetes.
As the application grows and we have many distinct values across stages env configuration might be too confusing. We could use an additional partial config file, which includes only the distinct values. To do so, we need just a few more lines in our main.go:
type AppConfig struct {
Name string `yaml:"name"`
Port string `yaml:"port"`
// split word splits the env var name on uppercase letters with "_"
// the env var name is APP_CONFIG_FILE
ConfigFile string `split_word:"true"`
Db struct {
User string `yaml:"user" envconfig:"DB_USER"`
Password string `yaml:"password" envconfig:"DB_PASSWORD"`
Host string `yaml:"host" envconfig:"DB_HOST"`
Port string `yaml:"port" envconfig:"DB_PORT"`
}
}
....
func main() {
appConfig := &AppConfig{}
// Load default config file
appConfig.ReadYaml("config.yaml")
// Load specific configuration with env variables
appConfig.ReadEnv()
if appConfig.ConfigFile != "" {
appConfig.ReadYaml(appConfig.ConfigFile)
}
fmt.Printf("%+v\n", *appConfig)
}
Now we can create a partial config file:
prod-config.yaml
db: host: prod-db
We can specify the partial config file we want to include by providing the environment variable APP_CONFIG_FILE. Run the app with partial config prod-config.yaml:
APP_CONFIG_FILE=prod-config.yaml go run src/main.go
# Output: {Name:test Port:8444 ConfigFile:prod-config.yaml Db:{User:admin Password:1234 Host:prod-db Port:5432}}
With this approach config values will be used with the following priority:
- Partial Config File
- Environment Variables
- Default Config File
Conclusion
To sum this up let’s have a look at the requirements listed above again:
- Centralized✔
- There’s a single struct holding all configuration for the app
- Well Documented✔
- Parameters and Environment Variable Names and Usage can be documented in the comments of the default config.yaml file
- Structured and Commented✔
- YAML
- Default Values but allow customization✔
- App loads default config and allow customization by env variables or an additional config file
- Not allow too many Options✔
- No flags, command-line args, or different kind of supported config formats