4.8 KiB
Work Time Tracker
IN-PROGRESS Write issue about config.edn simplification work
[2020-11-09 Mon 15:24]
Problem
we have a lot of duplication between our config.edn
files in
tenzin-config
.
This make it easy to be wrong while we copy (5 times) similar values.
We would like to make those configuration file a lot shorter and easier to
manage by only puting into them the field that are really different between
all environment.
A little bit of short-time history.
At first we relied on a single template that used different .json
entries.
Which is kind of duplicating the config.edn
by adding a layer of logic in
the middle.
After lot of confusions and bugs, we decided (both devs and ops) to duplicate
the config.edn and minimize the amount of templating in the process.
Abstract of previous discussions
If you look at the problem, there are plenty of different solutions to handle that. Here are what we thought about:
- Use a better templating system than jinja. The best in class in my
opinion (and by far) is dhall.
The issue with dhall is that it is still put a limitation about how we
generate the
config.edn
and this is also another new language to learn. - Use a Clojure project to handle
config.edn
templating/generation. Mainly re-write a dhall-like project in Clojure better suited for our need. - Use another
ConfigService
that would take care of some logic in Clojure. - Make every service to use better default values.
So typically some service will depend on some `tk-store` that are named
by something and generally the service is also responsible to know the
intended store to be used. Typically the service know that it should
rely on postgres and not redis nor RAM-only store.
The issue is the service will need to declare its expected configuration
to other services (typically to tk-stores). So we should move from
init
tostart
. Every of our service should declare its default config to a centralizedIROHConfigService
duringinit
phase. Then every service should initialize its context during thestart
phase during which we should use a function similar toget-in-config
but fromIROHConfigService
and not TKConfigService
.
From an architecture standpoint the conclusion was to prefer the 4th choice
(which would not be incompatible with the 3rd.)
Mainly the service start to take responsibility from some dependencies.
Also the IROHConfigService
could take care of potential configuration
conflict.
If two services want different value for the same field we should make the
configuration fail (unless the configuration is overwritten in config.edn
)
High level technical spec
So technically most (all?) our services have a init-context
function.
Generally this function use get-in-config
from TK ConfigService
.
Also the init-context
is called in the init
phase.
We should instead make all our service dependant on the new IROHConfigService
.
And during the init
phase the service should send its default
configuration to the IROHConfigService
.
Example:
(ns iroh.my-service
(:require [iroh.my-service.core :as core],,,))
(defprotocol MyServiceProt ,,,)
(tk/defservice
[[:IROHConfigService declare-default-conf get-in-conf]]
(init [this context]
(core/init-conf declare-default-conf))
(start [this context]
(into context
(core/start-context get-in-conf ,,,)))
,,,)
(ns iroh.my-service.core ,,,)
(def default-config
{:stores {"foo" {:type :postgres :conf {:table-name "foo"}}
"my-service-cache" {:type :redis :conf {:db 1}}}
:my-service {:default-timeout 3000}})
(defn init-conf
[declare-default-conf]
(declare-default-conf default-config))
(defn get-in-conf
[set-default-config
get-in-config path]
(or (get-in-config path) (get-in default-config path)))
(defn ^:always-validate start-context :- MyServiceContext
[get-in-conf ,,,]
(let [foo (get-in-conf [:my-service :default-timeout])]
))
So the IROHConfigService
should take care of doing a deep-merge-with
concat
on all default configurations and the config from ConfigService
.
It should also throw an exception in case of configuration conflict.
If two services do not agree on the value of some inner field.
Doing so, we should better separate the developer concerns from the ops concerns. Currently the ops are responsible to select the type of database, the default routes. While this should be configurable and at the same time not the matter of the ops.
Last word
This Epic is just here to keep track of the discussions about that recurring subject and make a proposal of solution. The objective would be to reach a consensus on a the best way to handle config.edn simplification and prevent duplication.