How Not to Manage Server Configs

After working in software a few years, most people end up with a rich store of knowledge; unfortunately, this knowledge is mostly about how to do things wrong. I know lots of ways to not manage a 200-person government software project, having been on several such beasts.

4 years into my current project (my longest tenure on a single project in my 30 years of work life), I know lots of ways to not manage a server farm. In this case (unlike the huge government software projects) I am culpable; I setup the system myself, either directly, or through guidance and code reviews to my programmers.

My (broken) system is based on Git branches:

• core: default branch, should work out of the box on developer installations
  • foia: branch for the Freedom of Information Act version of the prodct
    • foia-dev-server: branch for the FOIA dev server
    • foia-test-server: branch for the FOIA test server
    • foia-customer: branch for one of our FOIA customers
      • foia-customer-dev-server: branch for the customer dev server
      • foia-customer-test-server: branch for the customer dev server
  • core-customer: branch for one of our core customers
    • core-customer-dev-server: branch for core customer dev server

And so on, and so on.

Commits to a branch flow to all downstream branches, so commits to core are applied to every branch; commits to foia are applied to the downstream foia-* branches; and so on.

Three facts make this a terrible system. Fact 1: the repository includes binary files (spreadsheets). Fact 2: my application updates many of these files at runtime. Fact 3: there is no hierarchy of configuration values; each branch is a complete copy of the entire configuration.

You may already see the obvious problem; such a system can’t be maintained automatically. The all-too-frequent merge conflicts cause me to spend much of my time caring and feeding the dev, test, and production systems. Merge conflicts in the spreadsheets are super-painful.

So what is to be done? How can I fix it?

First, I have to establish a separation of responsibilities, by setting up a hierarchy of configuration values. Default values (the core branch in my example Git branch tree) are maintained by the core application; values for the FOIA extension are maintained only by the FOIA version of the application; customer-specific values are maintained in only one place; and runtime changes are kept separate. When the application needs a configuration value, it looks first for runtime values; then for server-specific values; then for extension values; then for core values… Obviously this is done by a library so application code just looks up a key, like it does now.

Second, establish a uniform representation. Everything is YAML; if it’s not YAML, it’s not configuration. Replace our spreadsheets with YAML, then as the application starts, load the YAML data into the same structures as we load the spreadsheet into now. Get rid of all the Spring configuration files; build the Spring beans from the YAML structure.

In this way we should get automated, conflict-free system administration.