Merge pull request #1 from eSlider/release/v2

feat(go-config): multi-format codecs, envc CLI, fixtures, ASRs
This commit is contained in:
2026-05-02 14:18:23 +01:00
committed by GitHub
co-authored by GitHub
75 changed files with 2810 additions and 528 deletions
+3
View File
@@ -1,3 +1,6 @@
# IDE
.idea/
# Local environment
.env
+7 -3
View File
@@ -1,13 +1,17 @@
version: "2"
run:
timeout: 5m
tests: true
linters:
disable-all: true
formatters:
enable:
- errcheck
- gofmt
- goimports
linters:
enable:
- errcheck
- govet
- ineffassign
- revive
+16 -22
View File
@@ -1,44 +1,39 @@
# AGENTS.md — `go-env`
# AGENTS.md — `go-config`
This module is part of the eSlider `go-*` library standard (inventar ASR-0008).
## Purpose
Decode `os.Environ()` into Go structs. Extracted from
`produktor.io/ai-fabric/pkg/env` on 2026-04-24.
Convert **env**, **YAML**, **JSON**, and **INI** into nested `map[string]any` and Go structs (and back), with multi-source merging and the `envc` CLI.
## Public API surface
- `env.Unmarshal(dst any, opts ...Option) error`
- `env.UnmarshalPrefix(dst any, prefix string, opts ...Option) error`
- `env.AsMap() map[string]any`
- `env.AsMapPrefix(prefix string) map[string]any`
- Options: `WithTrim`, `WithWeaklyTyped`, `WithTagName`, `WithDecodeHook`
- Subpackages: `env`, `yaml`, `json`, `ini` — each exports `New`, `(*Codec).Map`, `Unmarshal`, `UnmarshalContext`, `Marshal`, `WriteTo`, and format-specific options.
- `cmd/envc` — binary `envc`: `convert`, `get`, `merge`.
- Internals under `internal/` are not stable API.
Breaking changes require a new major version tag (SemVer). Internal helpers
(`asMapFromEnviron`, `trimStringHook`, `insertPath`) are unexported and
may change without notice.
Breaking changes require a new major SemVer tag (or `/v2` module path if the policy changes).
## Testing policy
Follows the eSlider "no synthetic mocks" policy:
Follows the eSlider "no synthetic mocks" policy (see [.cursor/rules/no-synthetic-mocks.mdc](.cursor/rules/no-synthetic-mocks.mdc)):
- **Unit tests** (`env_test.go`): pure inputs only. We pass synthetic
`environ` slices to `asMapFromEnviron` — that is not a mock of anything
external; it's the normal way to test a pure function. Use `t.Setenv`
for `Unmarshal*` happy-path tests since Go's `os.Environ` lookup is
well-defined local behaviour, not a vendor protocol.
- **No `httptest`** — this library has no HTTP surface.
- **No integration-test build tag** — everything is in-process.
- **Unit tests** — pure inputs for merge, keymap, structconv, env flat-map helpers.
- **`httptest`** — allowed only to test **our** HTTP client behaviour in `internal/source` (not third-party API emulation).
- **Fixtures** — real files under `fixtures/`; `testfixtures` resolves paths from module root.
## Decisions
Architecture Significant Requirements: [docs/asr/README.md](docs/asr/README.md).
## Checklist before release
```sh
cd go-env
cd go-config
go mod tidy
go vet ./...
go test -race -count=1 ./...
golangci-lint run --timeout 5m # same preset as go-onlyoffice
golangci-lint run --timeout 5m
```
Bump `CHANGELOG.md`, tag `vX.Y.Z`, push.
@@ -46,5 +41,4 @@ Bump `CHANGELOG.md`, tag `vX.Y.Z`, push.
## Related
- `inventar/docs/asr/ASR-0008.md` — Go library module conventions
- `inventar/docs/asr/ASR-0008-ai-fabric-audit.md` — why this module exists
- `go-onlyoffice/AGENTS.md` — reference for the eSlider library template
+25 -2
View File
@@ -1,14 +1,37 @@
# Changelog
All notable changes to `go-env` are documented here.
All notable changes to `go-config` are documented here.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
## [0.2.0] - 2026-05-02
### Added
- Module rename to `github.com/eslider/go-config` with subpackages `env`, `yaml`, `json`, `ini`.
- Uniform `Codec` API: `New`, `Map`, `Unmarshal`, `UnmarshalContext`, `Marshal`, `WriteTo`.
- `internal/source` for bytes, readers, files, and HTTP(S) URLs with optional headers and custom `http.Client`.
- `internal/keymap` recursive key walk with default `LowerAlnum` normalizer.
- `internal/merge` deep merge: recursive maps, last-write-wins scalars, configurable slice replace vs concat.
- `internal/structconv` around `github.com/go-viper/mapstructure/v2`.
- CLI `envc` (`cmd/envc`): `convert`, `get`, `merge`.
- Fixtures under `fixtures/` for identity, merge, edge, and invalid parser cases.
- `testfixtures` helper for tests.
- Repo-local ASRs in `docs/asr/`.
### Changed
- **Breaking:** the v0.1.0 `github.com/eslider/go-env` API (`Unmarshal`, `UnmarshalPrefix`, `AsMap`, …) is not re-exported; use `env.New(...).Unmarshal(...)`.
### Dependencies
- `github.com/go-viper/mapstructure/v2`, `github.com/joho/godotenv`, `gopkg.in/yaml.v3`, `gopkg.in/ini.v1`.
## [0.1.0] - 2026-04-24
Initial release. Extracted from `produktor.io/ai-fabric/pkg/env` per
Initial release as `github.com/eslider/go-env`. Extracted from `produktor.io/ai-fabric/pkg/env` per
`inventar/docs/asr/ASR-0008-ai-fabric-audit.md`.
### Added
+95 -64
View File
@@ -1,97 +1,128 @@
# go-env
# go-config
[![Go Reference](https://pkg.go.dev/badge/github.com/eslider/go-env.svg)](https://pkg.go.dev/github.com/eslider/go-env)
[![Go Reference](https://pkg.go.dev/badge/github.com/eslider/go-config.svg)](https://pkg.go.dev/github.com/eslider/go-config)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Latest Release](https://img.shields.io/github/v/release/eSlider/go-config)](https://github.com/eSlider/go-config/releases/latest)
[![GitHub Stars](https://img.shields.io/github/stars/eSlider/go-config?style=social)](https://github.com/eSlider/go-config/stargazers)
Tiny, zero-ceremony library for decoding process environment variables into
Go structs.
Convert **env**, **YAML**, **JSON**, and **INI** to and from Go `map[string]any` and structs. Multi-source inputs merge with **deep map merge**: nested maps combine, **scalar leaves are last-write-wins**, and **slices** default to **replace** (opt-in **concat** via `WithSliceMerge`). Keys are normalized with a configurable **lower+alnum** rule so `sub-service`, `SUB_SERVICE`, and `SubService` line up across formats. Built on [go-viper/mapstructure/v2](https://github.com/go-viper/mapstructure).
- Uses `_` as a path separator: `SERVICE_HTTP_PORT` → `Service.HTTP.Port`.
- Weakly-typed by default (`"1"` → `int`, `"true"` → `bool`).
- Optional prefix filter with automatic stripping.
- Pluggable decode hooks via [`mitchellh/mapstructure`][mapstructure].
## Architecture
```mermaid
flowchart LR
Sources["Sources\nbytes reader file URL process env"]
Parser["Parser\ngodotenv yaml ini json"]
Norm["keymap.Walk\nNormalizer"]
MergeOp["merge.DeepMerge"]
Map["map string any"]
MS["structconv\nmapstructure v2"]
Struct["Go struct"]
Sources --> Parser --> Norm --> MergeOp --> Map
Map -->|Unmarshal| MS --> Struct
Struct -->|Marshal| MS --> Map
Map -->|WriteTo Marshal| Parser
```
## Hero example
```go
yamlCfg := yaml.New(yaml.WithURL("https://raw.githubusercontent.com/eSlider/mail-archive/refs/heads/master/docker-compose.yml"))
envCfg := env.New(
env.WithFile(".default.env"), // lowest priority
env.WithFile(".env"),
env.WithCurrentEnvironment(), // highest priority — process env wins
)
var svc MyService
_ = yamlCfg.Unmarshal(&svc)
_ = envCfg.Unmarshal(&svc) // later sources override earlier scalar leaves; maps recurse
```
- **Maps** merge recursively (sub-trees are combined, not replaced wholesale).
- **Scalar leaves**: last-write-wins when you list sources lowest → highest priority.
- **Slices**: `merge.Replace` by default; use `WithSliceMerge(merge.Concat)` to append.
Runnable offline variant: see `Example_hero_offline` in [example_hero_test.go](example_hero_test.go).
## Install
```sh
go get github.com/eslider/go-env
go get github.com/eslider/go-config
go install github.com/eslider/go-config/cmd/envc@latest
```
## Quick start
### 1. Single YAML file
```go
package main
c := yaml.New(yaml.WithFile("config.yaml"))
var cfg AppConfig
if err := c.Unmarshal(&cfg); err != nil { /* ... */ }
```
import (
"fmt"
### 2. JSON over HTTPS with a header
"github.com/eslider/go-env"
```go
c := json.New(
json.WithURL("https://api.example.com/v1/config.json"),
json.WithHTTPHeader("Authorization", "Bearer "+token),
)
type Config struct {
Service struct {
HTTP struct {
Port int
}
Key string
}
}
func main() {
var cfg Config
if err := env.Unmarshal(&cfg); err != nil {
panic(err)
}
fmt.Printf("%+v\n", cfg)
}
var cfg AppConfig
_ = c.Unmarshal(&cfg)
```
With a prefix:
### 3. Cross-format conversion (YAML → JSON)
```go
// Only looks at APP_* variables; strips the APP_ prefix before decoding.
_ = env.UnmarshalPrefix(&cfg, "APP_")
ctx := context.Background()
m, _ := yaml.New(yaml.WithFile("in.yaml")).Map(ctx)
b, _ := json.New().Marshal(m)
os.WriteFile("out.json", b, 0o644)
```
## API
## CLI: `envc`
| Function | Purpose |
|---|---|
| `Unmarshal(dst, opts...)` | Decode all env vars into `dst`. |
| `UnmarshalPrefix(dst, prefix, opts...)` | Same, but only vars starting with `prefix`. |
| `AsMap()` / `AsMapPrefix(prefix)` | Return the nested `map[string]any` used by the decoder (debugging, custom decoders). |
| Command | Purpose |
| --- | --- |
| `envc convert --from yaml --to env --input - --output -` | Convert stdin YAML to dotenv on stdout |
| `envc get --from yaml --path service.name config.yaml` | Print one path (dot-separated; segments normalized like codecs) |
| `envc merge --from yaml --to json --output out.json a.yaml b.yaml` | Deep-merge multiple YAML files, emit JSON |
### Options
## API (all codecs)
| Option | Default | Purpose |
|---|---|---|
| `WithTrim(bool)` | `true` | TrimSpace every string value. |
| `WithWeaklyTyped(bool)` | `true` | `mapstructure`'s weakly-typed coercion. |
| `WithTagName(string)` | `"mapstructure"` | Struct tag name for field overrides. |
| `WithDecodeHook(h)` | — | Append a `mapstructure.DecodeHookFunc` to the chain. |
| Method | Description |
| --- | --- |
| `New(opts...)` | Construct codec |
| `Map(ctx)` | Merged `map[string]any` |
| `Unmarshal(dst)` / `UnmarshalContext(ctx, dst)` | Decode into struct (or map) |
| `Marshal(src)` / `WriteTo(w, src)` | Encode struct or `map[string]any` |
## Semantics
Shared options (each subpackage): `WithBytes`, `WithReader`, `WithFile`, `WithURL`, `WithHTTPHeader`, `WithHTTPClient`, `WithKeyNormalizer`, `WithSliceMerge`, `WithTrim`, `WithWeaklyTyped`, `WithTagName`, `WithDecodeHook`.
- **Path collisions**: first write wins. If both `FOO=1` and `FOO=2` exist
in the environ, only `FOO=1` is kept. This matches the original
`ai-fabric/pkg/env` behaviour.
- **Case-insensitive keys**: all path components are lower-cased; struct
fields are matched via `mapstructure` which is also case-insensitive.
- **`_`-only delimiter**: there's no escape — if a variable legitimately
contains `_` inside a "leaf" name, you must restructure your struct to
match the nested layout.
`env` adds: `WithCurrentEnvironment`, `WithPrefix`.
## Status
## Cross-format mapping
Extracted from `produktor.io/ai-fabric` as part of the eSlider `go-*`
library standard (ASR-0008). Merges the best parts of three previously
divergent copies:
| Go | YAML | ENV |
| --- | --- | --- |
| `Service.SubService.Name` | `service.sub-service.name` | `SERVICE_SUBSERVICE_NAME` |
- `produktor.io/ai-fabric/pkg/env`
- `markets-platform/TP-general-code/pkg/system/env.go`
- the various `pkg/system/env.go` snapshots inside `var/agents/issue-*/`
INI uses dotted sections, e.g. `[service.subservice]` with `name=...`.
## Related libraries
| Module | Role |
| --- | --- |
| [go-matrix-bot](https://github.com/eSlider/go-matrix-bot) | Matrix bots |
| [go-onlyoffice](https://github.com/eSlider/go-onlyoffice) | OnlyOffice API |
| [go-ollama](https://github.com/eSlider/go-ollama) | Ollama client |
## Decisions
Repo-local ASRs: [docs/asr/README.md](docs/asr/README.md).
## License
MIT © Andriy Oblivantsev
[mapstructure]: https://github.com/mitchellh/mapstructure
+96
View File
@@ -0,0 +1,96 @@
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"io"
"github.com/eslider/go-config/env"
"github.com/eslider/go-config/ini"
"github.com/eslider/go-config/internal/bytesutil"
libjson "github.com/eslider/go-config/json"
"github.com/eslider/go-config/yaml"
yaml3 "gopkg.in/yaml.v3"
)
// RunConvert implements: envc convert --from F --to T [--input] [--output].
func RunConvert(args []string, stdin io.Reader, stdout, stderr io.Writer) int {
fs := flag.NewFlagSet("convert", flag.ContinueOnError)
fs.SetOutput(stderr)
from := fs.String("from", "", "source format: yaml|json|ini|env")
to := fs.String("to", "", "target format: yaml|json|ini|env")
input := fs.String("input", "-", "input path, URL, or - for stdin")
output := fs.String("output", "-", "output path or - for stdout")
if err := fs.Parse(args); err != nil {
return 2
}
if *from == "" || *to == "" {
_, _ = fmt.Fprintln(stderr, "convert: --from and --to are required")
return 2
}
ctx := context.Background()
src := openSource(*input, stdin)
b, err := bytesutil.ReadAll(ctx, src)
if err != nil {
_, _ = fmt.Fprintf(stderr, "convert: read: %v\n", err)
return 1
}
m, err := loadMapFromBytes(ctx, *from, b)
if err != nil {
_, _ = fmt.Fprintf(stderr, "convert: parse: %v\n", err)
return 1
}
outb, err := marshalMap(*to, m)
if err != nil {
_, _ = fmt.Fprintf(stderr, "convert: marshal: %v\n", err)
return 1
}
out, err := openOutput(*output, stdout)
if err != nil {
_, _ = fmt.Fprintf(stderr, "convert: output: %v\n", err)
return 1
}
defer func() { _ = out.Close() }()
if _, err := out.Write(outb); err != nil {
_, _ = fmt.Fprintf(stderr, "convert: write: %v\n", err)
return 1
}
if *output == "-" || *output == "" {
if len(outb) > 0 && outb[len(outb)-1] != '\n' {
_, _ = out.Write([]byte("\n"))
}
}
return 0
}
func loadMapFromBytes(ctx context.Context, format string, b []byte) (map[string]any, error) {
switch format {
case "yaml":
return yaml.New(yaml.WithBytes(b)).Map(ctx)
case "json":
return libjson.New(libjson.WithBytes(b)).Map(ctx)
case "ini":
return ini.New(ini.WithBytes(b)).Map(ctx)
case "env":
return env.New(env.WithBytes(b)).Map(ctx)
default:
return nil, fmt.Errorf("unknown format %q", format)
}
}
func marshalMap(format string, m map[string]any) ([]byte, error) {
switch format {
case "yaml":
return yaml3.Marshal(m)
case "json":
return json.MarshalIndent(m, "", " ")
case "ini":
return ini.New().Marshal(m)
case "env":
return env.New().Marshal(m)
default:
return nil, fmt.Errorf("unknown format %q", format)
}
}
+53
View File
@@ -0,0 +1,53 @@
package main
import (
"bytes"
"os"
"path/filepath"
"strings"
"testing"
)
func TestRunConvert_YAMLToJSON(t *testing.T) {
var stdout, stderr bytes.Buffer
stdin := strings.NewReader("a: 1\n")
code := RunConvert([]string{"--from", "yaml", "--to", "json", "--input", "-"}, stdin, &stdout, &stderr)
if code != 0 {
t.Fatalf("stderr: %s", stderr.String())
}
if !strings.Contains(stdout.String(), `"a"`) {
t.Fatalf("out: %s", stdout.String())
}
}
func TestRunGet(t *testing.T) {
var stdout, stderr bytes.Buffer
in := "service:\n name: x\n"
code := RunGet([]string{"--from", "yaml", "--path", "service.name", "-"}, strings.NewReader(in), &stdout, &stderr)
if code != 0 {
t.Fatalf("stderr: %s", stderr.String())
}
if strings.TrimSpace(stdout.String()) != "x" {
t.Fatalf("got %q", stdout.String())
}
}
func TestRunMerge_TwoYAMLFiles(t *testing.T) {
dir := t.TempDir()
f1 := filepath.Join(dir, "a.yaml")
f2 := filepath.Join(dir, "b.yaml")
if err := os.WriteFile(f1, []byte("k: 1\n"), 0o644); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(f2, []byte("k: 2\n"), 0o644); err != nil {
t.Fatal(err)
}
var stdout, stderr bytes.Buffer
code := RunMerge([]string{"--from", "yaml", "--to", "json", f1, f2}, strings.NewReader(""), &stdout, &stderr)
if code != 0 {
t.Fatalf("stderr: %s", stderr.String())
}
if !strings.Contains(stdout.String(), `"k"`) || !strings.Contains(stdout.String(), `2`) {
t.Fatalf("out: %s", stdout.String())
}
}
+3
View File
@@ -0,0 +1,3 @@
// Command envc converts, queries, and merges configuration between env, YAML,
// JSON, and INI formats.
package main
+83
View File
@@ -0,0 +1,83 @@
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"io"
"strings"
"github.com/eslider/go-config/internal/bytesutil"
"github.com/eslider/go-config/internal/keymap"
)
// RunGet implements: envc get --from F --path p [input].
func RunGet(args []string, stdin io.Reader, stdout, stderr io.Writer) int {
fs := flag.NewFlagSet("get", flag.ContinueOnError)
fs.SetOutput(stderr)
from := fs.String("from", "", "format: yaml|json|ini|env")
path := fs.String("path", "", "dot-separated path (e.g. service.subservice.name)")
if err := fs.Parse(args); err != nil {
return 2
}
if *from == "" || *path == "" {
_, _ = fmt.Fprintln(stderr, "get: --from and --path are required")
return 2
}
pos := fs.Args()
input := "-"
if len(pos) > 0 {
input = pos[0]
}
ctx := context.Background()
src := openSource(input, stdin)
b, err := bytesutil.ReadAll(ctx, src)
if err != nil {
_, _ = fmt.Fprintf(stderr, "get: read: %v\n", err)
return 1
}
m, err := loadMapFromBytes(ctx, *from, b)
if err != nil {
_, _ = fmt.Fprintf(stderr, "get: parse: %v\n", err)
return 1
}
v, err := walkPath(m, strings.Split(*path, "."))
if err != nil {
_, _ = fmt.Fprintf(stderr, "get: %v\n", err)
return 1
}
switch val := v.(type) {
case string, bool, float64, int, int64, nil:
_, _ = fmt.Fprintln(stdout, val)
default:
enc := json.NewEncoder(stdout)
enc.SetIndent("", " ")
if err := enc.Encode(val); err != nil {
_, _ = fmt.Fprintf(stderr, "get: encode: %v\n", err)
return 1
}
}
return 0
}
func walkPath(m map[string]any, parts []string) (any, error) {
var cur any = m
for i, p := range parts {
if p == "" {
continue
}
pk := keymap.LowerAlnum(p)
switch t := cur.(type) {
case map[string]any:
nx, ok := t[pk]
if !ok {
return nil, fmt.Errorf("missing key %q (normalized %q) at segment %d", p, pk, i)
}
cur = nx
default:
return nil, fmt.Errorf("not a map at segment %d", i)
}
}
return cur, nil
}
+28
View File
@@ -0,0 +1,28 @@
package main
import (
"fmt"
"os"
)
func main() {
stdin, stdout, stderr := os.Stdin, os.Stdout, os.Stderr
args := os.Args[1:]
if len(args) < 1 {
_, _ = fmt.Fprintln(stderr, "usage: envc <convert|get|merge> [flags]")
os.Exit(2)
}
var code int
switch args[0] {
case "convert":
code = RunConvert(args[1:], stdin, stdout, stderr)
case "get":
code = RunGet(args[1:], stdin, stdout, stderr)
case "merge":
code = RunMerge(args[1:], stdin, stdout, stderr)
default:
_, _ = fmt.Fprintf(stderr, "unknown command %q\n", args[0])
code = 2
}
os.Exit(code)
}
+80
View File
@@ -0,0 +1,80 @@
package main
import (
"context"
"flag"
"fmt"
"io"
"github.com/eslider/go-config/internal/bytesutil"
"github.com/eslider/go-config/internal/merge"
"github.com/eslider/go-config/internal/source"
)
// RunMerge implements: envc merge --from F --to T [--output] inputs...
func RunMerge(args []string, stdin io.Reader, stdout, stderr io.Writer) int {
fs := flag.NewFlagSet("merge", flag.ContinueOnError)
fs.SetOutput(stderr)
from := fs.String("from", "", "input format: yaml|json|ini|env")
to := fs.String("to", "", "output format: yaml|json|ini|env")
output := fs.String("output", "-", "output path or - for stdout")
if err := fs.Parse(args); err != nil {
return 2
}
if *from == "" || *to == "" {
_, _ = fmt.Fprintln(stderr, "merge: --from and --to are required")
return 2
}
inputs := fs.Args()
if len(inputs) == 0 {
_, _ = fmt.Fprintln(stderr, "merge: at least one input file or URL required")
return 2
}
ctx := context.Background()
acc := make(map[string]any)
for _, in := range inputs {
var src source.Source
if in == "-" {
b, err := io.ReadAll(stdin)
if err != nil {
_, _ = fmt.Fprintf(stderr, "merge: stdin: %v\n", err)
return 1
}
src = source.Bytes{Data: b, Name: "stdin"}
} else {
src = openSource(in, stdin)
}
b, err := bytesutil.ReadAll(ctx, src)
if err != nil {
_, _ = fmt.Fprintf(stderr, "merge: read %s: %v\n", in, err)
return 1
}
m, err := loadMapFromBytes(ctx, *from, b)
if err != nil {
_, _ = fmt.Fprintf(stderr, "merge: parse %s: %v\n", in, err)
return 1
}
merge.DeepMerge(acc, m)
}
outb, err := marshalMap(*to, acc)
if err != nil {
_, _ = fmt.Fprintf(stderr, "merge: marshal: %v\n", err)
return 1
}
out, err := openOutput(*output, stdout)
if err != nil {
_, _ = fmt.Fprintf(stderr, "merge: output: %v\n", err)
return 1
}
defer func() { _ = out.Close() }()
if _, err := out.Write(outb); err != nil {
_, _ = fmt.Fprintf(stderr, "merge: write: %v\n", err)
return 1
}
if *output == "-" || *output == "" {
if len(outb) > 0 && outb[len(outb)-1] != '\n' {
_, _ = out.Write([]byte("\n"))
}
}
return 0
}
+33
View File
@@ -0,0 +1,33 @@
package main
import (
"io"
"os"
"strings"
"github.com/eslider/go-config/internal/source"
)
func openSource(pathOrURL string, stdin io.Reader) source.Source {
if pathOrURL == "" || pathOrURL == "-" {
return source.Reader{R: stdin, Name: "stdin"}
}
if strings.HasPrefix(pathOrURL, "http://") || strings.HasPrefix(pathOrURL, "https://") {
return source.URL{Raw: pathOrURL}
}
return source.File{Path: pathOrURL}
}
type nopCloser struct{ io.Writer }
func (nopCloser) Close() error { return nil }
func openOutput(path string, stdout io.Writer) (io.WriteCloser, error) {
if path == "" || path == "-" {
if wc, ok := stdout.(io.WriteCloser); ok {
return wc, nil
}
return nopCloser{stdout}, nil
}
return os.Create(path)
}
+20 -25
View File
@@ -1,28 +1,23 @@
// Package env reads process environment variables and decodes them into
// Go structs using underscore-delimited paths.
// Package config is the module root for go-config (import path
// github.com/eslider/go-config). Use the subpackages [env], [yaml], [json], and
// [ini] for format-specific codecs and the [cmd/envc] command for CLI
// conversion.
//
// A variable named "SERVICE_HTTP_PORT" becomes the path Service.HTTP.Port
// (case-insensitive, '_' is a path separator). Values are decoded via
// github.com/mitchellh/mapstructure, so numeric, boolean and slice
// conversions happen automatically.
// Hero workflow (YAML + layered dotenv + process env):
//
// var cfg struct {
// Service struct {
// HTTP struct {
// Port int
// }
// Key string
// }
// }
// if err := env.Unmarshal(&cfg); err != nil { ... }
// yamlCfg := yaml.New(yaml.WithURL("https://example.com/config.yaml"))
// envCfg := env.New(
// env.WithFile(".default.env"),
// env.WithFile(".env"),
// env.WithCurrentEnvironment(),
// )
// var svc MyService
// _ = yamlCfg.Unmarshal(&svc)
// _ = envCfg.Unmarshal(&svc)
//
// UnmarshalPrefix ignores variables that don't start with the given
// prefix and strips it before building the path:
//
// _ = os.Setenv("APP_DB_HOST", "localhost")
// _ = env.UnmarshalPrefix(&cfg, "APP_")
// // cfg.Db.Host == "localhost"
//
// String values are TrimSpace-trimmed by default; pass WithTrim(false) to
// opt out.
package env
// [cmd/envc]: https://pkg.go.dev/github.com/eslider/go-config/cmd/envc
// [env]: https://pkg.go.dev/github.com/eslider/go-config/env
// [yaml]: https://pkg.go.dev/github.com/eslider/go-config/yaml
// [json]: https://pkg.go.dev/github.com/eslider/go-config/json
// [ini]: https://pkg.go.dev/github.com/eslider/go-config/ini
package config
@@ -0,0 +1,27 @@
# ASR-0001: Module rename and multi-package layout
## Context
The original `github.com/eslider/go-env` module only decoded process environment variables. The product scope expanded to YAML, JSON, INI, URLs, multi-source merging, and a CLI. The old name and single-package layout no longer matched the capability surface.
## Decision
1. The canonical module path is **`github.com/eslider/go-config`**.
2. Format-specific code lives in first-class subpackages: **`env/`**, **`yaml/`**, **`json/`**, **`ini/`**.
3. Shared building blocks live under **`internal/{source,keymap,merge,structconv,bytesutil}`**.
4. The CLI lives at **`cmd/envc/`** (one binary per inventar **ASR-0008** §2 — CLI inside the same module).
5. The GitHub repository is named **`eSlider/go-config`** (renamed from `go-env`).
## Consequences
- Consumers must change their `import` paths and `go get` target.
- CI, badges, and documentation reference `go-config` and `eslider/go-config`.
## Status
Accepted — 2026-05-02.
## References
- inventar ASR-0008 — Go library module conventions
- ASR-0002 — API clean break
@@ -0,0 +1,24 @@
# ASR-0002: Clean break from go-env v0.1.0
## Context
`go-env` v0.1.0 exposed package-level helpers (`Unmarshal`, `UnmarshalPrefix`, `AsMap`, …). The new design is `Codec`-centric with multiple sources per format.
## Decision
1. **No compatibility shims** in `go-config` re-exporting the old API.
2. v0.1.0 remains tagged on git history as `github.com/eslider/go-env@v0.1.0` for archival consumers.
3. Migration path is documented in `CHANGELOG.md` and the README.
## Consequences
- Single-call-site migrations must switch to `env.New(env.WithCurrentEnvironment(), …).Unmarshal(&cfg)`.
## Status
Accepted — 2026-05-02.
## References
- CHANGELOG `[0.2.0]`
- ASR-0001
@@ -0,0 +1,25 @@
# ASR-0003: Uniform codec API and Source abstraction
## Context
Users switch between YAML, JSON, INI, and env-file formats; a different function surface per format would increase cognitive load and test matrices.
## Decision
1. Every format package exports **`New(opts...) *Codec`** with **`Map`**, **`Unmarshal`**, **`UnmarshalContext`**, **`Marshal`**, **`WriteTo`**.
2. Bytes, readers, files, and URLs are modeled by **`internal/source.Source`** with **`Open(ctx) (io.ReadCloser, error)`**.
3. HTTP(S) sources support **`WithHTTPHeader`** and **`WithHTTPClient`**.
## Consequences
- Cross-format examples in the README stay structurally identical.
- URL behaviour is testable with `httptest` against our client only.
## Status
Accepted — 2026-05-02.
## References
- ASR-0005 — merge semantics across multiple sources
- ASR-0008 — httptest policy alignment
@@ -0,0 +1,23 @@
# ASR-0004: Configurable key normalization
## Context
Real-world configuration mixes `kebab-case`, `snake_case`, `SCREAMING_SNAKE`, and struct field names. Requiring `mapstructure` tags on every field is noisy.
## Decision
1. After parsing and merging, codecs run **`internal/keymap.Walk`** with a **`Normalizer`** (default **`LowerAlnum`**: lowercase + strip non `[a-z0-9]`).
2. Callers may set **`WithKeyNormalizer(nil)`** to disable walking, or supply a custom function.
## Consequences
- POSIX env keys without `-` still align with YAML keys that contain hyphens once normalized.
- CLI `get` applies the same normalizer to each path segment for consistency.
## Status
Accepted — 2026-05-02.
## References
- ASR-0003
@@ -0,0 +1,26 @@
# ASR-0005: Multi-source deep-merge semantics
## Context
Layered configuration (defaults file < local overrides < process environment) requires predictable composition rules.
## Decision
1. Sources are applied in **option order** (left → right). Each source parses to a `map[string]any`, then **`merge.DeepMerge`** folds them.
2. **Maps recurse** — nested keys are unioned; existing nested maps are not replaced wholesale by a sibling scalar.
3. **Scalar leaves** use **last-write-wins** (later source wins).
4. **Slices** default to **`merge.Replace`**; **`WithSliceMerge(merge.Concat)`** appends prior and new slice elements.
## Consequences
- Env files and `WithCurrentEnvironment()` compose when ordered **lowest → highest** priority.
- Tests lock behaviour via `fixtures/merge/*`.
## Status
Accepted — 2026-05-02.
## References
- ASR-0003
- `internal/merge`
@@ -0,0 +1,26 @@
# ASR-0006: Third-party library choices
## Context
We need battle-tested parsers and struct bridging without maintaining our own grammars.
## Decision
1. **Struct / map bridging:** `github.com/go-viper/mapstructure/v2` (maintained fork of `mitchellh/mapstructure`).
2. **YAML:** `gopkg.in/yaml.v3`.
3. **JSON:** `encoding/json` (stdlib).
4. **INI:** `gopkg.in/ini.v1` with `Loose` parsing for tolerant files.
5. **`.env` files:** `github.com/joho/godotenv` (`UnmarshalBytes`).
6. **CLI:** stdlib `flag` only (no Cobra).
## Consequences
- `go.mod` carries the above direct dependencies (plus transitive modules as resolved by MVS).
## Status
Accepted — 2026-05-02.
## References
- [go-viper/mapstructure](https://github.com/go-viper/mapstructure)
@@ -0,0 +1,24 @@
# ASR-0007: CLI `envc` — convert, get, merge
## Context
Operators need a small tool to translate configuration between formats and to inspect merged trees without writing Go.
## Decision
1. Binary name **`envc`**; module path **`github.com/eslider/go-config/cmd/envc`** (`go install …/cmd/envc@latest`).
2. Subcommands: **`convert`**, **`get`**, **`merge`** — each implemented as a **`Run*(args, stdin, stdout, stderr) int`** function for in-process tests.
3. **No** third-party CLI framework in v1 of the CLI.
## Consequences
- `cmd/envc` may grow flags; keep business logic in subpackages, not in `main` beyond wiring.
## Status
Accepted — 2026-05-02.
## References
- inventar ASR-0008 §2 — CLI placement
- ASR-0003
@@ -0,0 +1,25 @@
# ASR-0008: GitHub presentation and release flow
## Context
The repository was renamed from `eSlider/go-env` to `eSlider/go-config`. The v0.1.0 tag existed locally before the rename. Presentation should match other eSlider `go-*` libraries (e.g. `go-matrix-bot`).
## Decision
1. **v0.1.0 preservation:** push tag `v0.1.0` and create a GitHub **Release** with notes extracted from `CHANGELOG.md` before landing breaking work.
2. **Development branch:** `release/v2` carries the `go-config` implementation until merged to `main`.
3. **Rename:** `gh repo rename go-config` from `go-env`; local `origin` URL updated to `https://github.com/eSlider/go-config.git`.
4. **Metadata:** `gh repo edit` sets description, **`homepage`** `https://pkg.go.dev/github.com/eslider/go-config`, and **topics** (`go`, `golang`, `config`, `yaml`, `json`, `ini`, `env`, `dotenv`, `mapstructure`, `codec`, `encoder`, `decoder`, `cli`, `library`).
5. **README:** badges row, mermaid architecture diagram, hero snippet, quick starts, CLI table, API tables, related libraries, link to `docs/asr/`.
## Consequences
- Old `github.com/eslider/go-env` URLs redirect for a limited GitHub window; module path for v0.1.0 code remains the historical `go-env` import.
## Status
Accepted — 2026-05-02.
## References
- [go-matrix-bot](https://github.com/eSlider/go-matrix-bot) — README pattern reference
+14
View File
@@ -0,0 +1,14 @@
# Architecture Significant Requirements (ASRs)
Repo-local decisions for `go-config` (numbering starts at ASR-0001). Org-wide policies remain under `inventar/docs/asr/`.
| ID | Title |
| --- | --- |
| [ASR-0001](ASR-0001-module-rename-and-layout.md) | Module rename and multi-package layout |
| [ASR-0002](ASR-0002-clean-break-from-v0.1.0.md) | Clean break from go-env v0.1.0 |
| [ASR-0003](ASR-0003-uniform-codec-api-and-source-abstraction.md) | Uniform codec API and Source abstraction |
| [ASR-0004](ASR-0004-configurable-key-normalization.md) | Configurable key normalization |
| [ASR-0005](ASR-0005-multi-source-deep-merge-semantics.md) | Multi-source deep-merge semantics |
| [ASR-0006](ASR-0006-third-party-library-choices.md) | Third-party library choices |
| [ASR-0007](ASR-0007-cli-envc-convert-get-merge.md) | CLI `envc`: convert, get, merge |
| [ASR-0008](ASR-0008-github-presentation-and-release-flow.md) | GitHub presentation and release flow |
-139
View File
@@ -1,139 +0,0 @@
package env
import (
"fmt"
"os"
"reflect"
"strings"
"github.com/mitchellh/mapstructure"
)
// Unmarshal decodes all process environment variables into dst. The variable
// name is split by '_' and the resulting path is matched against dst's fields
// case-insensitively.
//
// dst must be a pointer to a struct (or a map that mapstructure can populate).
// See the package documentation for details and examples.
func Unmarshal(dst any, opts ...Option) error {
return UnmarshalPrefix(dst, "", opts...)
}
// UnmarshalPrefix is like Unmarshal but only considers environment variables
// that start with prefix. The prefix is stripped from each variable name
// before the path is built.
//
// An empty prefix is equivalent to Unmarshal.
func UnmarshalPrefix(dst any, prefix string, opts ...Option) error {
o := defaultOptions()
for _, f := range opts {
f(&o)
}
data := asMapFromEnviron(os.Environ(), prefix)
hooks := []mapstructure.DecodeHookFunc{}
if o.trim {
hooks = append(hooks, trimStringHook)
}
hooks = append(hooks, o.extraHooks...)
decoder, err := mapstructure.NewDecoder(&mapstructure.DecoderConfig{
Result: dst,
WeaklyTypedInput: o.weaklyTyped,
TagName: o.tagName,
DecodeHook: mapstructure.ComposeDecodeHookFunc(hooks...),
})
if err != nil {
return fmt.Errorf("env: configure decoder: %w", err)
}
if err := decoder.Decode(data); err != nil {
return fmt.Errorf("env: decode: %w", err)
}
return nil
}
// AsMap returns a nested map built from all current process environment
// variables, using '_' as path separator. Keys are lower-cased.
//
// It's primarily useful for debugging or for callers that want to plug
// their own decoder; normal code should prefer Unmarshal.
func AsMap() map[string]any {
return asMapFromEnviron(os.Environ(), "")
}
// AsMapPrefix is the prefix-scoped variant of AsMap. Variables whose names
// don't start with prefix are skipped; the prefix itself is stripped before
// the map is built.
func AsMapPrefix(prefix string) map[string]any {
return asMapFromEnviron(os.Environ(), prefix)
}
// asMapFromEnviron is the core building block. Exposed as an unexported
// function so tests can feed it a deterministic environ slice instead of
// mutating the real process env.
func asMapFromEnviron(environ []string, prefix string) map[string]any {
out := make(map[string]any)
for _, entry := range environ {
eq := strings.IndexByte(entry, '=')
if eq < 0 {
// Malformed entry — shouldn't happen on Unix but guard anyway.
continue
}
name, value := entry[:eq], entry[eq+1:]
if prefix != "" {
if !strings.HasPrefix(name, prefix) {
continue
}
name = name[len(prefix):]
if name == "" {
continue
}
}
insertPath(out, strings.Split(name, "_"), value)
}
return out
}
// insertPath walks path inside root, creating intermediate nested maps as
// needed, and stores value at the leaf.
//
// Collisions are resolved by preferring the first write: once a leaf value
// is set for a path, a later sibling with the same path prefix will NOT
// overwrite it. This matches the behaviour of the ai-fabric original so
// that tools migrating to go-env don't observe new surprises.
func insertPath(root map[string]any, path []string, value string) {
current := root
last := len(path) - 1
for i, raw := range path {
k := strings.ToLower(raw)
if i == last {
if _, exists := current[k]; exists {
return
}
current[k] = value
return
}
next, ok := current[k].(map[string]any)
if !ok {
next = make(map[string]any)
current[k] = next
}
current = next
}
}
func trimStringHook(from, to reflect.Type, data any) (any, error) {
if from.Kind() == reflect.String && to.Kind() == reflect.String {
if s, ok := data.(string); ok {
return strings.TrimSpace(s), nil
}
}
return data, nil
}
+172
View File
@@ -0,0 +1,172 @@
package env
import (
"bytes"
"context"
"fmt"
"io"
"net/http"
"sort"
"strings"
"github.com/eslider/go-config/internal/bytesutil"
"github.com/eslider/go-config/internal/keymap"
"github.com/eslider/go-config/internal/merge"
"github.com/eslider/go-config/internal/source"
"github.com/eslider/go-config/internal/structconv"
"github.com/joho/godotenv"
)
// Codec loads environment-style key/value data from multiple sources.
type Codec struct {
layers []func(context.Context) (map[string]string, error)
prefix string
normalizer keymap.Normalizer
sliceStrat merge.SliceStrategy
structOpts structconv.Options
mergeOpts []merge.Option
httpClient *http.Client
urlHeader http.Header
}
// New builds a Codec from options. Sources are merged left-to-right; later
// sources override earlier scalar leaves (see package merge).
func New(opts ...Option) *Codec {
c := &Codec{
normalizer: keymap.LowerAlnum,
sliceStrat: merge.Replace,
structOpts: structconv.Options{
TagName: "mapstructure",
WeaklyTyped: true,
Trim: true,
},
}
for _, o := range opts {
o(c)
}
c.mergeOpts = []merge.Option{merge.WithSliceStrategy(c.sliceStrat)}
return c
}
// Map returns the merged nested map after all sources are loaded and keys normalized.
func (c *Codec) Map(ctx context.Context) (map[string]any, error) {
if len(c.layers) == 0 {
return nil, fmt.Errorf("env: no sources configured")
}
acc := make(map[string]any)
for _, layer := range c.layers {
flat, err := layer(ctx)
if err != nil {
return nil, err
}
nested := nestedFromFlat(flat, c.prefix)
merge.DeepMerge(acc, nested, c.mergeOpts...)
}
if c.normalizer != nil {
keymap.Walk(acc, c.normalizer)
}
return acc, nil
}
// Unmarshal decodes using context.Background.
func (c *Codec) Unmarshal(dst any) error {
return c.UnmarshalContext(context.Background(), dst)
}
// UnmarshalContext decodes the merged map into dst using mapstructure.
func (c *Codec) UnmarshalContext(ctx context.Context, dst any) error {
m, err := c.Map(ctx)
if err != nil {
return err
}
return structconv.Decode(m, dst, c.structOpts)
}
// Marshal encodes src into dotenv-style bytes (KEY=value lines, sorted keys).
// src may be a struct or map[string]any.
func (c *Codec) Marshal(src any) ([]byte, error) {
m, err := encodeToMap(src)
if err != nil {
return nil, fmt.Errorf("env: marshal encode: %w", err)
}
flat := flattenMap(nil, m)
var buf bytes.Buffer
keys := make([]string, 0, len(flat))
for k := range flat {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
fmt.Fprintf(&buf, "%s=%s\n", k, flat[k])
}
return buf.Bytes(), nil
}
// WriteTo writes dotenv-style output to w.
func (c *Codec) WriteTo(w io.Writer, src any) (int64, error) {
b, err := c.Marshal(src)
if err != nil {
return 0, err
}
n, err := w.Write(b)
return int64(n), err
}
func flattenMap(prefix []string, m map[string]any) map[string]string {
out := make(map[string]string)
for k, v := range m {
path := append(prefix, k)
switch t := v.(type) {
case map[string]any:
for kk, vv := range flattenMap(path, t) {
out[kk] = vv
}
case []any:
parts := make([]string, 0, len(t))
for _, e := range t {
parts = append(parts, fmt.Sprint(e))
}
key := strings.Join(path, "_")
out[strings.ToUpper(key)] = strings.Join(parts, ",")
default:
key := strings.Join(path, "_")
out[strings.ToUpper(key)] = fmt.Sprint(t)
}
}
return out
}
func encodeToMap(src any) (map[string]any, error) {
if m, ok := src.(map[string]any); ok {
return m, nil
}
return structconv.Encode(src)
}
func flatFromEnviron(environ []string) map[string]string {
out := make(map[string]string)
for _, entry := range environ {
eq := strings.IndexByte(entry, '=')
if eq < 0 {
continue
}
out[entry[:eq]] = entry[eq+1:]
}
return out
}
func withSource(s source.Source, label string) func(*Codec) {
return func(c *Codec) {
c.layers = append(c.layers, func(ctx context.Context) (map[string]string, error) {
b, err := bytesutil.ReadAll(ctx, s)
if err != nil {
return nil, fmt.Errorf("env: read %s: %w", label, err)
}
m, err := godotenv.UnmarshalBytes(b)
if err != nil {
return nil, fmt.Errorf("env: parse %s: %w", label, err)
}
return m, nil
})
}
}
+125
View File
@@ -0,0 +1,125 @@
package env
import (
"context"
"testing"
"github.com/eslider/go-config/testfixtures"
)
type identityRoot struct {
Service svc `mapstructure:"service"`
}
type svc struct {
Name string
Subservice sub `mapstructure:"subservice"`
Database db `mapstructure:"database"`
}
type sub struct {
Name string
Key string
Timeout int
Enabled bool
}
type db struct {
URL string
PoolSize int `mapstructure:"poolsize"`
}
func TestCodec_IdentityEnvFixture(t *testing.T) {
c := New(WithBytes(testfixtures.Load(t, "identity", "service.env")))
var got identityRoot
if err := c.Unmarshal(&got); err != nil {
t.Fatal(err)
}
assertIdentity(t, got)
}
func assertIdentity(t *testing.T, got identityRoot) {
t.Helper()
if got.Service.Name != "my-service" {
t.Fatalf("Name=%q", got.Service.Name)
}
if got.Service.Subservice.Name != "abc" || got.Service.Subservice.Key != "abs" {
t.Fatalf("sub %+v", got.Service.Subservice)
}
if got.Service.Subservice.Timeout != 30 || !got.Service.Subservice.Enabled {
t.Fatalf("sub scalars %+v", got.Service.Subservice)
}
if got.Service.Database.URL != "postgres://localhost:5432/db" || got.Service.Database.PoolSize != 10 {
t.Fatalf("db %+v", got.Service.Database)
}
}
func TestCodec_MultiSourceLastWins(t *testing.T) {
c := New(
WithBytes(testfixtures.Load(t, "merge", "defaults.env")),
WithBytes(testfixtures.Load(t, "merge", "overlay.env")),
)
m, err := c.Map(context.Background())
if err != nil {
t.Fatal(err)
}
if got := m["service"].(map[string]any)["subservice"].(map[string]any)["name"]; got != "overlay-name" {
t.Fatalf("subservice.name = %v", got)
}
}
func TestCodec_WithCurrentEnvironment(t *testing.T) {
t.Setenv("SERVICE_KEY", "abc")
t.Setenv("SERVICE_PORT", "8080")
var cfg struct {
Service struct {
Key string `mapstructure:"key"`
Port int `mapstructure:"port"`
} `mapstructure:"service"`
}
c := New(WithCurrentEnvironment())
if err := c.Unmarshal(&cfg); err != nil {
t.Fatal(err)
}
if cfg.Service.Key != "abc" || cfg.Service.Port != 8080 {
t.Fatalf("%+v", cfg)
}
}
func TestCodec_WithTrimWeaklyTyped(t *testing.T) {
t.Setenv("NAME", " x ")
var cfg struct{ Name string }
c := New(WithCurrentEnvironment(), WithTrim(true))
if err := c.Unmarshal(&cfg); err != nil {
t.Fatal(err)
}
if cfg.Name != "x" {
t.Fatalf("%q", cfg.Name)
}
}
func TestCodec_WithWeaklyTypedOff(t *testing.T) {
t.Setenv("PORT", "8080")
var cfg struct{ Port int }
c := New(WithCurrentEnvironment(), WithWeaklyTyped(false))
if err := c.Unmarshal(&cfg); err == nil {
t.Fatal("expected error")
}
}
func TestCodec_WithTagName(t *testing.T) {
t.Setenv("DATABASE_URL", "postgres://x")
var cfg struct {
DB struct {
URL string `mapstructure:"url"`
} `mapstructure:"database" env:"database"`
}
c := New(WithCurrentEnvironment(), WithTagName("env"))
if err := c.Unmarshal(&cfg); err != nil {
t.Fatal(err)
}
if cfg.DB.URL != "postgres://x" {
t.Fatalf("%q", cfg.DB.URL)
}
}
Vendored
+14
View File
@@ -0,0 +1,14 @@
// Package env loads dotenv files and process environment variables into nested
// maps and Go structs. Multiple sources merge with later scalars overriding
// earlier ones; maps merge recursively.
//
// Example:
//
// c := env.New(
// env.WithFile(".default.env"),
// env.WithFile(".env"),
// env.WithCurrentEnvironment(),
// )
// var cfg MyConfig
// if err := c.Unmarshal(&cfg); err != nil { ... }
package env
+49
View File
@@ -0,0 +1,49 @@
package env
import (
"strings"
)
// insertPath walks path inside root, creating intermediate nested maps as
// needed, and stores value at the leaf. First write wins for a given leaf key
// within a single source map.
func insertPath(root map[string]any, path []string, value string) {
current := root
last := len(path) - 1
for i, raw := range path {
k := strings.ToLower(raw)
if i == last {
if _, exists := current[k]; exists {
return
}
current[k] = value
return
}
next, ok := current[k].(map[string]any)
if !ok {
next = make(map[string]any)
current[k] = next
}
current = next
}
}
func nestedFromFlat(flat map[string]string, prefix string) map[string]any {
out := make(map[string]any)
for k, v := range flat {
name := k
if prefix != "" {
if !strings.HasPrefix(name, prefix) {
continue
}
name = name[len(prefix):]
if name == "" {
continue
}
}
insertPath(out, strings.Split(name, "_"), v)
}
return out
}
+35
View File
@@ -0,0 +1,35 @@
package env
import (
"reflect"
"testing"
)
func TestNestedFromFlat(t *testing.T) {
got := nestedFromFlat(map[string]string{
"FOO": "bar",
"SERVICE_HTTP_PORT": "8080",
"SERVICE_KEY": "abc",
}, "")
want := map[string]any{
"foo": "bar",
"service": map[string]any{
"http": map[string]any{"port": "8080"},
"key": "abc",
},
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("got %#v want %#v", got, want)
}
}
func TestNestedFromFlat_FirstWriteWins(t *testing.T) {
flat := map[string]string{}
flat["A"] = "1"
got := nestedFromFlat(flat, "")
// Second insert for same key is ignored by insertPath within one build.
insertPath(got, []string{"a"}, "2")
if got["a"] != "1" {
t.Fatalf("got %v", got["a"])
}
}
+105
View File
@@ -0,0 +1,105 @@
package env
import (
"context"
"io"
"net/http"
"os"
"github.com/eslider/go-config/internal/keymap"
"github.com/eslider/go-config/internal/merge"
"github.com/eslider/go-config/internal/source"
"github.com/go-viper/mapstructure/v2"
)
// Option configures a Codec.
type Option func(*Codec)
// WithCurrentEnvironment appends the process environment as a source (read at Map time).
func WithCurrentEnvironment() Option {
return func(c *Codec) {
c.layers = append(c.layers, func(_ context.Context) (map[string]string, error) {
return flatFromEnviron(os.Environ()), nil
})
}
}
// WithFile appends a dotenv file path as a source.
func WithFile(path string) Option {
return withSource(source.File{Path: path}, path)
}
// WithBytes appends raw dotenv bytes as a source.
func WithBytes(b []byte) Option {
return withSource(source.Bytes{Data: b, Name: "bytes"}, "bytes")
}
// WithReader appends dotenv content from r (read fully on each Map call).
func WithReader(r io.Reader) Option {
return withSource(source.Reader{R: r, Name: "reader"}, "reader")
}
// WithURL appends an HTTP(S) URL returning dotenv content. Use WithHTTPHeader /
// WithHTTPClient before WithURL so they apply to this request.
func WithURL(raw string) Option {
return func(c *Codec) {
var hdr http.Header
if c.urlHeader != nil {
hdr = c.urlHeader.Clone()
}
s := source.URL{Raw: raw, Header: hdr, HTTPClient: c.httpClient}
withSource(s, raw)(c)
}
}
// WithHTTPClient sets the HTTP client used by subsequent WithURL sources.
func WithHTTPClient(client *http.Client) Option {
return func(c *Codec) { c.httpClient = client }
}
// WithHTTPHeader adds a header for subsequent WithURL sources.
func WithHTTPHeader(k, v string) Option {
return func(c *Codec) {
if c.urlHeader == nil {
c.urlHeader = make(http.Header)
}
c.urlHeader.Add(k, v)
}
}
// WithPrefix strips prefix from variable names before path splitting.
func WithPrefix(prefix string) Option {
return func(c *Codec) { c.prefix = prefix }
}
// WithKeyNormalizer sets key normalizer after merge (nil disables).
func WithKeyNormalizer(n keymap.Normalizer) Option {
return func(c *Codec) { c.normalizer = n }
}
// WithSliceMerge sets slice merge strategy when merging sources.
func WithSliceMerge(s merge.SliceStrategy) Option {
return func(c *Codec) { c.sliceStrat = s }
}
// WithTrim toggles struct string trim via mapstructure hook.
func WithTrim(enable bool) Option {
return func(c *Codec) { c.structOpts.Trim = enable }
}
// WithWeaklyTyped toggles weak typing for struct decode.
func WithWeaklyTyped(enable bool) Option {
return func(c *Codec) { c.structOpts.WeaklyTyped = enable }
}
// WithTagName sets the struct tag name for mapstructure.
func WithTagName(name string) Option {
return func(c *Codec) { c.structOpts.TagName = name }
}
// WithDecodeHook appends a decode hook.
func WithDecodeHook(h mapstructure.DecodeHookFunc) Option {
return func(c *Codec) {
c.structOpts.ExtraHooks = append(c.structOpts.ExtraHooks, h)
}
}
-222
View File
@@ -1,222 +0,0 @@
package env
import (
"reflect"
"testing"
)
// asMapFromEnviron is tested with a synthetic environ slice — this is NOT a
// mock of anything external. It's a pure-input unit test that isolates the
// core mapping logic from process-global state.
func TestAsMapFromEnviron_FlatAndNested(t *testing.T) {
got := asMapFromEnviron([]string{
"FOO=bar",
"SERVICE_HTTP_PORT=8080",
"SERVICE_KEY=abc",
}, "")
want := map[string]any{
"foo": "bar",
"service": map[string]any{
"http": map[string]any{"port": "8080"},
"key": "abc",
},
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("got %#v, want %#v", got, want)
}
}
func TestAsMapFromEnviron_PreservesValuesWithEquals(t *testing.T) {
got := asMapFromEnviron([]string{"TOKEN=a=b=c="}, "")
if got["token"] != "a=b=c=" {
t.Fatalf("expected full value preserved, got %#v", got["token"])
}
}
func TestAsMapFromEnviron_PrefixFilterAndStrip(t *testing.T) {
got := asMapFromEnviron([]string{
"APP_DB_HOST=localhost",
"APP_DB_PORT=5432",
"OTHER_THING=skip",
}, "APP_")
want := map[string]any{
"db": map[string]any{"host": "localhost", "port": "5432"},
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("got %#v, want %#v", got, want)
}
}
func TestAsMapFromEnviron_EmptyAndMalformedAreSkipped(t *testing.T) {
got := asMapFromEnviron([]string{"NOEQUALS", ""}, "")
if len(got) != 0 {
t.Fatalf("expected no entries, got %#v", got)
}
}
func TestAsMapFromEnviron_FirstWriteWinsOnCollision(t *testing.T) {
// This matches the ai-fabric original: once a leaf is set for a path,
// subsequent entries that would overwrite it are ignored.
got := asMapFromEnviron([]string{
"A=1",
"A=2",
}, "")
if got["a"] != "1" {
t.Fatalf("expected first-write-wins (a=1), got %#v", got["a"])
}
}
func TestAsMapFromEnviron_PrefixOnlyVariableIsSkipped(t *testing.T) {
// "APP_=xxx" with prefix "APP_" would strip to empty key — we skip it.
got := asMapFromEnviron([]string{"APP_=oops"}, "APP_")
if len(got) != 0 {
t.Fatalf("expected empty map, got %#v", got)
}
}
// ---------------------------------------------------------------------------
// Decoder / option tests
// ---------------------------------------------------------------------------
func TestUnmarshal_BasicStruct(t *testing.T) {
t.Setenv("SERVICE_KEY", "abc")
t.Setenv("SERVICE_PORT", "8080")
var cfg struct {
Service struct {
Key string
Port int
}
}
if err := Unmarshal(&cfg); err != nil {
t.Fatalf("unexpected error: %v", err)
}
if cfg.Service.Key != "abc" {
t.Errorf("Key = %q, want abc", cfg.Service.Key)
}
if cfg.Service.Port != 8080 {
t.Errorf("Port = %d, want 8080 (weakly-typed string->int)", cfg.Service.Port)
}
}
func TestUnmarshal_TrimsStringsByDefault(t *testing.T) {
t.Setenv("NAME", " hello ")
var cfg struct{ Name string }
if err := Unmarshal(&cfg); err != nil {
t.Fatalf("unexpected error: %v", err)
}
if cfg.Name != "hello" {
t.Fatalf("expected trimmed value, got %q", cfg.Name)
}
}
func TestUnmarshal_WithTrimOff(t *testing.T) {
t.Setenv("NAME", " keep me ")
var cfg struct{ Name string }
if err := Unmarshal(&cfg, WithTrim(false)); err != nil {
t.Fatalf("unexpected error: %v", err)
}
if cfg.Name != " keep me " {
t.Fatalf("expected untrimmed value, got %q", cfg.Name)
}
}
func TestUnmarshalPrefix_StripsAndFilters(t *testing.T) {
t.Setenv("APP_DB_HOST", "localhost")
t.Setenv("APP_DB_PORT", "5432")
t.Setenv("OTHER_DB_HOST", "should-be-ignored")
var cfg struct {
DB struct {
Host string
Port int
} `mapstructure:"db"`
}
if err := UnmarshalPrefix(&cfg, "APP_"); err != nil {
t.Fatalf("unexpected error: %v", err)
}
if cfg.DB.Host != "localhost" {
t.Errorf("Host = %q, want localhost", cfg.DB.Host)
}
if cfg.DB.Port != 5432 {
t.Errorf("Port = %d, want 5432", cfg.DB.Port)
}
}
func TestUnmarshal_WithTagName(t *testing.T) {
t.Setenv("DATABASE_URL", "postgres://x")
// The map is always nested by '_', so tags also reference the nested
// path, one component per struct level. Here we override the outer
// field name via a custom tag.
var cfg struct {
DB struct {
URL string
} `env:"database"`
}
if err := Unmarshal(&cfg, WithTagName("env")); err != nil {
t.Fatalf("unexpected error: %v", err)
}
if cfg.DB.URL != "postgres://x" {
t.Fatalf("URL = %q, want postgres://x", cfg.DB.URL)
}
}
func TestUnmarshal_WeaklyTypedOff(t *testing.T) {
t.Setenv("PORT", "8080")
var cfg struct{ Port int }
err := Unmarshal(&cfg, WithWeaklyTyped(false))
// With strict typing, "8080" (string) cannot decode into an int.
if err == nil {
t.Fatal("expected strict typing to reject string->int coercion")
}
}
func TestUnmarshal_SpecialCharactersSurvive(t *testing.T) {
// Regression: keys with special chars in their values must pass through
// unchanged (modulo TrimSpace).
t.Setenv("SPECIAL", "@!#$%^&*()")
var cfg struct{ Special string }
if err := Unmarshal(&cfg); err != nil {
t.Fatalf("unexpected error: %v", err)
}
if cfg.Special != "@!#$%^&*()" {
t.Fatalf("Special = %q, want @!#$%%^&*()", cfg.Special)
}
}
func TestUnmarshal_RejectsNonPointer(t *testing.T) {
var cfg struct{ A string }
err := Unmarshal(cfg) // no pointer
if err == nil {
t.Fatal("expected error when passing a non-pointer")
}
}
func TestAsMap_ExposesCurrentEnv(t *testing.T) {
t.Setenv("GO_ENV_ASMAP_PROBE", "hit")
m := AsMap()
goenv, ok := m["go"].(map[string]any)
if !ok {
t.Fatalf("expected nested map under 'go', got %T", m["go"])
}
env, ok := goenv["env"].(map[string]any)
if !ok {
t.Fatalf("expected nested map under 'go.env', got %T", goenv["env"])
}
asmap, ok := env["asmap"].(map[string]any)
if !ok {
t.Fatalf("expected nested map under 'go.env.asmap', got %T", env["asmap"])
}
if asmap["probe"] != "hit" {
t.Fatalf("probe = %v, want hit", asmap["probe"])
}
}
+20
View File
@@ -0,0 +1,20 @@
package config_test
import (
"fmt"
"github.com/eslider/go-config/env"
"github.com/eslider/go-config/yaml"
)
func Example_hero_offline() {
// README hero pattern using offline sources (no network, no .env files).
y := yaml.New(yaml.WithBytes([]byte(`service: { name: demo }`)))
e := env.New(env.WithBytes([]byte("SERVICE_PORT=8080")))
var m map[string]any
_ = y.Unmarshal(&m)
var flat map[string]any
_ = e.Unmarshal(&flat)
fmt.Println(m["service"].(map[string]any)["name"], flat["service"].(map[string]any)["port"])
// Output: demo 8080
}
+5
View File
@@ -0,0 +1,5 @@
a:
b:
c:
d:
e: leaf
View File
+4
View File
@@ -0,0 +1,4 @@
topLevel: 1
sub-Service: 2
snake_case: 3
CamelCase: 4
+7
View File
@@ -0,0 +1,7 @@
{
"flag": true,
"count": 42,
"ratio": 3.14,
"nothing": null,
"nested": [1, 2]
}
+7
View File
@@ -0,0 +1,7 @@
SERVICE_NAME=my-service
SERVICE_SUBSERVICE_NAME=abc
SERVICE_SUBSERVICE_KEY=abs
SERVICE_SUBSERVICE_TIMEOUT=30
SERVICE_SUBSERVICE_ENABLED=true
SERVICE_DATABASE_URL=postgres://localhost:5432/db
SERVICE_DATABASE_POOLSIZE=10
+12
View File
@@ -0,0 +1,12 @@
[service]
name = my-service
[service.subservice]
name = abc
key = abs
timeout = 30
enabled = true
[service.database]
url = postgres://localhost:5432/db
pool-size = 10
+15
View File
@@ -0,0 +1,15 @@
{
"service": {
"name": "my-service",
"sub-service": {
"name": "abc",
"key": "abs",
"timeout": 30,
"enabled": true
},
"database": {
"url": "postgres://localhost:5432/db",
"pool-size": 10
}
}
}
+10
View File
@@ -0,0 +1,10 @@
service:
name: my-service
sub-service:
name: abc
key: abs
timeout: 30
enabled: true
database:
url: postgres://localhost:5432/db
pool-size: 10
+3
View File
@@ -0,0 +1,3 @@
[sec]
a = 1
a = 2
+1
View File
@@ -0,0 +1 @@
{ not json
+1
View File
@@ -0,0 +1 @@
service: [unclosed
+7
View File
@@ -0,0 +1,7 @@
SERVICE_NAME=default-name
SERVICE_SUBSERVICE_NAME=default-sub
SERVICE_SUBSERVICE_KEY=default-key
SERVICE_SUBSERVICE_TIMEOUT=1
SERVICE_SUBSERVICE_ENABLED=false
SERVICE_DATABASE_URL=postgres://old
SERVICE_DATABASE_POOLSIZE=1
+13
View File
@@ -0,0 +1,13 @@
service:
name: default-name
sub-service:
name: default-sub
key: default-key
timeout: 1
enabled: false
hosts:
- a
- b
database:
url: postgres://old
pool-size: 1
+14
View File
@@ -0,0 +1,14 @@
service:
name: default-name
sub-service:
name: overlay-name
key: default-key
timeout: 1
enabled: false
hosts:
- a
- b
- x
database:
url: postgres://old
pool-size: 1
+12
View File
@@ -0,0 +1,12 @@
service:
name: default-name
sub-service:
name: overlay-name
key: default-key
timeout: 1
enabled: false
hosts:
- x
database:
url: postgres://old
pool-size: 1
+1
View File
@@ -0,0 +1 @@
SERVICE_SUBSERVICE_NAME=overlay-name
+5
View File
@@ -0,0 +1,5 @@
service:
sub-service:
name: overlay-name
hosts:
- x
+9 -2
View File
@@ -1,5 +1,12 @@
module github.com/eslider/go-env
module github.com/eslider/go-config
go 1.22.2
require github.com/mitchellh/mapstructure v1.5.0
require (
github.com/go-viper/mapstructure/v2 v2.5.0
github.com/joho/godotenv v1.5.1
gopkg.in/ini.v1 v1.67.0
gopkg.in/yaml.v3 v3.0.1
)
require github.com/stretchr/testify v1.11.1 // indirect
+16 -2
View File
@@ -1,2 +1,16 @@
github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY=
github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/go-viper/mapstructure/v2 v2.5.0 h1:vM5IJoUAy3d7zRSVtIwQgBj7BiWtMPfmPEgAXnvj1Ro=
github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0=
github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA=
gopkg.in/ini.v1 v1.67.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+163
View File
@@ -0,0 +1,163 @@
package ini
import (
"context"
"fmt"
"io"
"net/http"
"strings"
"github.com/eslider/go-config/internal/bytesutil"
"github.com/eslider/go-config/internal/keymap"
"github.com/eslider/go-config/internal/merge"
"github.com/eslider/go-config/internal/source"
"github.com/eslider/go-config/internal/structconv"
iniv1 "gopkg.in/ini.v1"
)
// Codec loads INI from multiple sources.
type Codec struct {
sources []source.Source
normalizer keymap.Normalizer
sliceStrat merge.SliceStrategy
structOpts structconv.Options
mergeOpts []merge.Option
httpClient *http.Client
urlHeader http.Header
}
// New creates an INI codec.
func New(opts ...Option) *Codec {
c := &Codec{
normalizer: keymap.LowerAlnum,
sliceStrat: merge.Replace,
structOpts: structconv.Options{
TagName: "mapstructure",
WeaklyTyped: true,
Trim: true,
},
}
for _, o := range opts {
o(c)
}
c.mergeOpts = []merge.Option{merge.WithSliceStrategy(c.sliceStrat)}
return c
}
// Map merges all INI sources into one map.
func (c *Codec) Map(ctx context.Context) (map[string]any, error) {
if len(c.sources) == 0 {
return nil, fmt.Errorf("ini: no sources")
}
acc := make(map[string]any)
for _, s := range c.sources {
b, err := bytesutil.ReadAll(ctx, s)
if err != nil {
return nil, fmt.Errorf("ini: read %s: %w", s.String(), err)
}
f, err := iniv1.LoadSources(iniv1.LoadOptions{Loose: true, Insensitive: false}, b)
if err != nil {
return nil, fmt.Errorf("ini: parse %s: %w", s.String(), err)
}
parsed := mapFromINIFile(f)
merge.DeepMerge(acc, parsed, c.mergeOpts...)
}
if c.normalizer != nil {
keymap.Walk(acc, c.normalizer)
}
return acc, nil
}
// Unmarshal decodes using context.Background.
func (c *Codec) Unmarshal(dst any) error {
return c.UnmarshalContext(context.Background(), dst)
}
// UnmarshalContext decodes merged INI into dst.
func (c *Codec) UnmarshalContext(ctx context.Context, dst any) error {
m, err := c.Map(ctx)
if err != nil {
return err
}
return structconv.Decode(m, dst, c.structOpts)
}
// Marshal encodes src to INI bytes (struct or map[string]any).
func (c *Codec) Marshal(src any) ([]byte, error) {
m, err := encodeToMap(src)
if err != nil {
return nil, fmt.Errorf("ini: encode: %w", err)
}
f := iniv1.Empty()
if err := emitMap(nil, m, f); err != nil {
return nil, err
}
var buf strings.Builder
if _, err := f.WriteTo(&buf); err != nil {
return nil, fmt.Errorf("ini: write: %w", err)
}
return []byte(buf.String()), nil
}
// WriteTo writes INI to w.
func (c *Codec) WriteTo(w io.Writer, src any) (int64, error) {
b, err := c.Marshal(src)
if err != nil {
return 0, err
}
n, err := w.Write(b)
return int64(n), err
}
func emitMap(prefix []string, m map[string]any, f *iniv1.File) error {
for k, v := range m {
switch t := v.(type) {
case map[string]any:
if err := emitMap(append(prefix, k), t, f); err != nil {
return err
}
case []any:
parts := make([]string, 0, len(t))
for _, e := range t {
parts = append(parts, fmt.Sprint(e))
}
secName := strings.Join(prefix, ".")
if secName == "" {
secName = iniv1.DefaultSection
}
sec, err := f.GetSection(secName)
if err != nil {
sec, err = f.NewSection(secName)
if err != nil {
return err
}
}
if _, err := sec.NewKey(k, strings.Join(parts, ",")); err != nil {
return err
}
default:
secName := strings.Join(prefix, ".")
if secName == "" {
secName = iniv1.DefaultSection
}
sec, err := f.GetSection(secName)
if err != nil {
sec, err = f.NewSection(secName)
if err != nil {
return err
}
}
if _, err := sec.NewKey(k, fmt.Sprint(t)); err != nil {
return err
}
}
}
return nil
}
func encodeToMap(src any) (map[string]any, error) {
if m, ok := src.(map[string]any); ok {
return m, nil
}
return structconv.Encode(src)
}
+20
View File
@@ -0,0 +1,20 @@
package ini
import (
"context"
"testing"
"github.com/eslider/go-config/testfixtures"
)
func TestCodec_IdentityINIFixture(t *testing.T) {
c := New(WithBytes(testfixtures.Load(t, "identity", "service.ini")))
m, err := c.Map(context.Background())
if err != nil {
t.Fatal(err)
}
svc := m["service"].(map[string]any)
if svc["name"] != "my-service" {
t.Fatalf("%#v", svc)
}
}
+3
View File
@@ -0,0 +1,3 @@
// Package ini loads INI configuration from bytes, files, or URLs into maps
// and structs. Section names may use dot notation for nesting.
package ini
+49
View File
@@ -0,0 +1,49 @@
package ini
import (
"strings"
iniv1 "gopkg.in/ini.v1"
)
func insertPath(root map[string]any, path []string, value string) {
current := root
last := len(path) - 1
for i, raw := range path {
k := strings.ToLower(raw)
if i == last {
if _, exists := current[k]; exists {
return
}
current[k] = value
return
}
next, ok := current[k].(map[string]any)
if !ok {
next = make(map[string]any)
current[k] = next
}
current = next
}
}
func mapFromINIFile(f *iniv1.File) map[string]any {
out := make(map[string]any)
for _, name := range f.SectionStrings() {
sec, err := f.GetSection(name)
if err != nil {
continue
}
if strings.EqualFold(name, iniv1.DefaultSection) {
for k, v := range sec.KeysHash() {
insertPath(out, []string{k}, v)
}
continue
}
path := strings.Split(name, ".")
for k, v := range sec.KeysHash() {
insertPath(out, append(append([]string{}, path...), k), v)
}
}
return out
}
+87
View File
@@ -0,0 +1,87 @@
package ini
import (
"io"
"net/http"
"github.com/eslider/go-config/internal/keymap"
"github.com/eslider/go-config/internal/merge"
"github.com/eslider/go-config/internal/source"
"github.com/go-viper/mapstructure/v2"
)
// Option configures a Codec.
type Option func(*Codec)
// WithBytes appends INI from bytes.
func WithBytes(b []byte) Option {
return func(c *Codec) { c.sources = append(c.sources, source.Bytes{Data: b, Name: "bytes"}) }
}
// WithReader appends INI from r.
func WithReader(r io.Reader) Option {
return func(c *Codec) { c.sources = append(c.sources, source.Reader{R: r, Name: "reader"}) }
}
// WithFile appends an INI file path.
func WithFile(path string) Option {
return func(c *Codec) { c.sources = append(c.sources, source.File{Path: path}) }
}
// WithURL appends INI from an HTTP(S) URL.
func WithURL(raw string) Option {
return func(c *Codec) {
var hdr http.Header
if c.urlHeader != nil {
hdr = c.urlHeader.Clone()
}
c.sources = append(c.sources, source.URL{Raw: raw, Header: hdr, HTTPClient: c.httpClient})
}
}
// WithHTTPClient sets the client for subsequent WithURL sources.
func WithHTTPClient(client *http.Client) Option {
return func(c *Codec) { c.httpClient = client }
}
// WithHTTPHeader adds a header for subsequent WithURL sources.
func WithHTTPHeader(k, v string) Option {
return func(c *Codec) {
if c.urlHeader == nil {
c.urlHeader = make(http.Header)
}
c.urlHeader.Add(k, v)
}
}
// WithKeyNormalizer sets key normalizer after merge (nil disables).
func WithKeyNormalizer(n keymap.Normalizer) Option {
return func(c *Codec) { c.normalizer = n }
}
// WithSliceMerge sets slice merge strategy across sources.
func WithSliceMerge(s merge.SliceStrategy) Option {
return func(c *Codec) { c.sliceStrat = s }
}
// WithTrim toggles string trim on struct decode.
func WithTrim(enable bool) Option {
return func(c *Codec) { c.structOpts.Trim = enable }
}
// WithWeaklyTyped toggles weak typing for struct decode.
func WithWeaklyTyped(enable bool) Option {
return func(c *Codec) { c.structOpts.WeaklyTyped = enable }
}
// WithTagName sets mapstructure tag name.
func WithTagName(name string) Option {
return func(c *Codec) { c.structOpts.TagName = name }
}
// WithDecodeHook appends a decode hook.
func WithDecodeHook(h mapstructure.DecodeHookFunc) Option {
return func(c *Codec) {
c.structOpts.ExtraHooks = append(c.structOpts.ExtraHooks, h)
}
}
+19
View File
@@ -0,0 +1,19 @@
// Package bytesutil reads entire sources into memory.
package bytesutil
import (
"context"
"io"
"github.com/eslider/go-config/internal/source"
)
// ReadAll reads and closes the stream opened by s.
func ReadAll(ctx context.Context, s source.Source) ([]byte, error) {
rc, err := s.Open(ctx)
if err != nil {
return nil, err
}
defer func() { _ = rc.Close() }()
return io.ReadAll(rc)
}
+56
View File
@@ -0,0 +1,56 @@
// Package keymap normalizes map keys recursively for cross-format matching.
package keymap
import "strings"
// Normalizer transforms a single path segment or key string.
type Normalizer func(string) string
// LowerAlnum lowercases and strips any rune that is not [a-z0-9].
func LowerAlnum(s string) string {
var b strings.Builder
b.Grow(len(s))
for _, r := range strings.ToLower(s) {
if r >= 'a' && r <= 'z' || r >= '0' && r <= '9' {
b.WriteRune(r)
}
}
return b.String()
}
// Identity returns s unchanged.
func Identity(s string) string { return s }
// Walk recursively rewrites every map key in place using n. Slices of maps are walked.
func Walk(m map[string]any, n Normalizer) {
if m == nil || n == nil {
return
}
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
for _, k := range keys {
v := m[k]
delete(m, k)
nk := n(k)
if nk == "" {
nk = k
}
switch vv := v.(type) {
case map[string]any:
Walk(vv, n)
m[nk] = vv
case []any:
for i := range vv {
if sm, ok := vv[i].(map[string]any); ok {
Walk(sm, n)
vv[i] = sm
}
}
m[nk] = vv
default:
m[nk] = v
}
}
}
+25
View File
@@ -0,0 +1,25 @@
package keymap
import (
"reflect"
"testing"
)
func TestLowerAlnum(t *testing.T) {
if got, want := LowerAlnum("sub-Service"), "subservice"; got != want {
t.Fatalf("got %q want %q", got, want)
}
}
func TestWalk(t *testing.T) {
m := map[string]any{
"Sub-Service": map[string]any{"Pool-Size": "10"},
}
Walk(m, LowerAlnum)
want := map[string]any{
"subservice": map[string]any{"poolsize": "10"},
}
if !reflect.DeepEqual(m, want) {
t.Fatalf("got %#v want %#v", m, want)
}
}
+77
View File
@@ -0,0 +1,77 @@
// Package merge provides deep map merging with configurable slice behaviour.
package merge
// SliceStrategy controls how two []any values are combined at the same key.
type SliceStrategy int
const (
// Replace overwrites the destination slice with the source slice.
Replace SliceStrategy = iota
// Concat appends source elements to the destination slice.
Concat
)
// Option configures DeepMerge.
type Option func(*config)
type config struct {
slice SliceStrategy
}
// WithSliceStrategy sets slice merge behaviour (default Replace).
func WithSliceStrategy(s SliceStrategy) Option {
return func(c *config) { c.slice = s }
}
// DeepMerge folds src into dst in place. Maps recurse; scalar leaves use
// last-write-wins (src overwrites dst). For slice values, behaviour follows cfg.slice.
func DeepMerge(dst, src map[string]any, opts ...Option) {
cfg := config{slice: Replace}
for _, o := range opts {
o(&cfg)
}
if dst == nil || src == nil {
return
}
for k, sv := range src {
dv, ok := dst[k]
if !ok {
dst[k] = cloneValue(sv)
continue
}
dm, dIsMap := dv.(map[string]any)
sm, sIsMap := sv.(map[string]any)
if dIsMap && sIsMap {
DeepMerge(dm, sm, opts...)
dst[k] = dm
continue
}
if dsa, dIsSlice := dv.([]any); dIsSlice {
if ssa, sIsSlice := sv.([]any); sIsSlice {
switch cfg.slice {
case Concat:
dst[k] = append(append([]any{}, dsa...), ssa...)
default:
dst[k] = append([]any{}, ssa...)
}
continue
}
}
dst[k] = cloneValue(sv)
}
}
func cloneValue(v any) any {
switch t := v.(type) {
case map[string]any:
out := make(map[string]any, len(t))
DeepMerge(out, t)
return out
case []any:
cp := make([]any, len(t))
copy(cp, t)
return cp
default:
return v
}
}
+52
View File
@@ -0,0 +1,52 @@
package merge
import (
"reflect"
"testing"
)
func TestDeepMerge_MapRecursion(t *testing.T) {
dst := map[string]any{
"a": map[string]any{"x": "1"},
}
src := map[string]any{
"a": map[string]any{"y": "2"},
}
DeepMerge(dst, src)
want := map[string]any{
"a": map[string]any{"x": "1", "y": "2"},
}
if !reflect.DeepEqual(dst, want) {
t.Fatalf("got %#v want %#v", dst, want)
}
}
func TestDeepMerge_ScalarLastWins(t *testing.T) {
dst := map[string]any{"k": "first"}
src := map[string]any{"k": "second"}
DeepMerge(dst, src)
if dst["k"] != "second" {
t.Fatalf("got %v", dst["k"])
}
}
func TestDeepMerge_SliceReplace(t *testing.T) {
dst := map[string]any{"s": []any{"a", "b"}}
src := map[string]any{"s": []any{"c"}}
DeepMerge(dst, src, WithSliceStrategy(Replace))
got := dst["s"].([]any)
if len(got) != 1 || got[0] != "c" {
t.Fatalf("got %#v", got)
}
}
func TestDeepMerge_SliceConcat(t *testing.T) {
dst := map[string]any{"s": []any{"a", "b"}}
src := map[string]any{"s": []any{"c"}}
DeepMerge(dst, src, WithSliceStrategy(Concat))
got := dst["s"].([]any)
want := []any{"a", "b", "c"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("got %#v want %#v", got, want)
}
}
+129
View File
@@ -0,0 +1,129 @@
// Package source provides pluggable configuration inputs (bytes, files, URLs).
package source
import (
"bytes"
"context"
"fmt"
"io"
"net/http"
"os"
"strings"
"time"
)
// Source opens a byte stream for reading. Callers must close the ReadCloser.
type Source interface {
Open(ctx context.Context) (io.ReadCloser, error)
String() string
}
// Bytes holds raw bytes in memory.
type Bytes struct {
Data []byte
Name string
}
// Open returns a ReadCloser over a copy of Data.
func (b Bytes) Open(_ context.Context) (io.ReadCloser, error) {
return io.NopCloser(bytes.NewReader(b.Data)), nil
}
// String returns a label for errors (defaults to "bytes").
func (b Bytes) String() string {
if b.Name != "" {
return b.Name
}
return "bytes"
}
// Reader wraps an io.Reader. The reader is consumed once per Open call.
type Reader struct {
R io.Reader
Name string
}
// Open returns the reader as a ReadCloser when possible, otherwise wraps r.R.
func (r Reader) Open(_ context.Context) (io.ReadCloser, error) {
rc, ok := r.R.(io.ReadCloser)
if ok {
return rc, nil
}
return io.NopCloser(r.R), nil
}
// String returns a label for errors (defaults to "reader").
func (r Reader) String() string {
if r.Name != "" {
return r.Name
}
return "reader"
}
// File opens a path on the local filesystem.
type File struct {
Path string
}
// Open opens the file path.
func (f File) Open(_ context.Context) (io.ReadCloser, error) {
return os.Open(f.Path)
}
// String returns the file path.
func (f File) String() string {
return f.Path
}
// URL fetches remote content over HTTP or HTTPS.
type URL struct {
Raw string
Header http.Header
HTTPClient *http.Client
}
func defaultClient() *http.Client {
return &http.Client{Timeout: 30 * time.Second}
}
// Open performs a GET request and returns the response body.
func (u URL) Open(ctx context.Context) (io.ReadCloser, error) {
client := u.HTTPClient
if client == nil {
client = defaultClient()
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u.Raw, nil)
if err != nil {
return nil, fmt.Errorf("source url: new request: %w", err)
}
if u.Header != nil {
req.Header = u.Header.Clone()
}
resp, err := client.Do(req)
if err != nil {
return nil, fmt.Errorf("source url %q: %w", u.Raw, err)
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
_ = resp.Body.Close()
return nil, fmt.Errorf("source url %q: status %s", u.Raw, resp.Status)
}
return resp.Body, nil
}
func (u URL) String() string {
return u.Raw
}
// JoinNames joins source labels for error messages.
func JoinNames(srcs []Source) string {
if len(srcs) == 0 {
return ""
}
parts := make([]string, 0, len(srcs))
for _, s := range srcs {
if s != nil {
parts = append(parts, s.String())
}
}
return strings.Join(parts, ", ")
}
+43
View File
@@ -0,0 +1,43 @@
package source
import (
"context"
"io"
"net/http"
"net/http/httptest"
"testing"
)
func TestURL_OK(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte("hello"))
}))
defer srv.Close()
u := URL{Raw: srv.URL}
rc, err := u.Open(context.Background())
if err != nil {
t.Fatal(err)
}
defer func() { _ = rc.Close() }()
b, err := io.ReadAll(rc)
if err != nil {
t.Fatal(err)
}
if string(b) != "hello" {
t.Fatalf("got %q", b)
}
}
func TestURL_StatusError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
u := URL{Raw: srv.URL}
_, err := u.Open(context.Background())
if err == nil {
t.Fatal("expected error")
}
}
+74
View File
@@ -0,0 +1,74 @@
// Package structconv wraps mapstructure for map<->struct conversion.
package structconv
import (
"fmt"
"reflect"
"strings"
"github.com/go-viper/mapstructure/v2"
)
// Options configures Decode and Encode.
type Options struct {
TagName string
WeaklyTyped bool
Trim bool
ExtraHooks []mapstructure.DecodeHookFunc
}
func trimStringHook(from, to reflect.Type, data any) (any, error) {
if from.Kind() == reflect.String && to.Kind() == reflect.String {
if s, ok := data.(string); ok {
return strings.TrimSpace(s), nil
}
}
return data, nil
}
func composeHooks(o Options) mapstructure.DecodeHookFunc {
var hooks []mapstructure.DecodeHookFunc
if o.Trim {
hooks = append(hooks, trimStringHook)
}
hooks = append(hooks, o.ExtraHooks...)
if len(hooks) == 0 {
return nil
}
return mapstructure.ComposeDecodeHookFunc(hooks...)
}
// Decode maps in into struct or map dst (dst must be pointer).
func Decode(in map[string]any, dst any, o Options) error {
tag := o.TagName
if tag == "" {
tag = "mapstructure"
}
cfg := &mapstructure.DecoderConfig{
Result: dst,
WeaklyTypedInput: o.WeaklyTyped,
TagName: tag,
DecodeHook: composeHooks(o),
}
dec, err := mapstructure.NewDecoder(cfg)
if err != nil {
return fmt.Errorf("structconv: decoder: %w", err)
}
if err := dec.Decode(in); err != nil {
return fmt.Errorf("structconv: decode: %w", err)
}
return nil
}
// Encode flattens src (struct or map) into a new map[string]any.
func Encode(src any) (map[string]any, error) {
out := make(map[string]any)
dec, err := mapstructure.NewDecoder(&mapstructure.DecoderConfig{Result: &out})
if err != nil {
return nil, fmt.Errorf("structconv: encoder: %w", err)
}
if err := dec.Decode(src); err != nil {
return nil, fmt.Errorf("structconv: encode: %w", err)
}
return out, nil
}
+28
View File
@@ -0,0 +1,28 @@
package structconv
import (
"fmt"
"testing"
)
func TestDecodeEncodeRoundTrip(t *testing.T) {
type nested struct {
V int `mapstructure:"v"`
}
type root struct {
A nested `mapstructure:"a"`
}
in := map[string]any{"a": map[string]any{"v": 7}}
var dst root
if err := Decode(in, &dst, Options{WeaklyTyped: true, Trim: true}); err != nil {
t.Fatal(err)
}
out, err := Encode(&dst)
if err != nil {
t.Fatal(err)
}
am := out["a"].(map[string]any)
if fmt.Sprint(am["v"]) != "7" {
t.Fatalf("%#v", out)
}
}
+114
View File
@@ -0,0 +1,114 @@
package json
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"github.com/eslider/go-config/internal/bytesutil"
"github.com/eslider/go-config/internal/keymap"
"github.com/eslider/go-config/internal/merge"
"github.com/eslider/go-config/internal/source"
"github.com/eslider/go-config/internal/structconv"
)
// Codec loads JSON from multiple sources.
type Codec struct {
sources []source.Source
normalizer keymap.Normalizer
sliceStrat merge.SliceStrategy
structOpts structconv.Options
mergeOpts []merge.Option
httpClient *http.Client
urlHeader http.Header
}
// New creates a JSON codec.
func New(opts ...Option) *Codec {
c := &Codec{
normalizer: keymap.LowerAlnum,
sliceStrat: merge.Replace,
structOpts: structconv.Options{
TagName: "mapstructure",
WeaklyTyped: true,
Trim: true,
},
}
for _, o := range opts {
o(c)
}
c.mergeOpts = []merge.Option{merge.WithSliceStrategy(c.sliceStrat)}
return c
}
// Map merges all JSON sources into one map.
func (c *Codec) Map(ctx context.Context) (map[string]any, error) {
if len(c.sources) == 0 {
return nil, fmt.Errorf("json: no sources")
}
acc := make(map[string]any)
for _, s := range c.sources {
b, err := bytesutil.ReadAll(ctx, s)
if err != nil {
return nil, fmt.Errorf("json: read %s: %w", s.String(), err)
}
var parsed map[string]any
if err := json.Unmarshal(b, &parsed); err != nil {
return nil, fmt.Errorf("json: parse %s: %w", s.String(), err)
}
if parsed == nil {
parsed = map[string]any{}
}
merge.DeepMerge(acc, parsed, c.mergeOpts...)
}
if c.normalizer != nil {
keymap.Walk(acc, c.normalizer)
}
return acc, nil
}
// Unmarshal decodes using context.Background.
func (c *Codec) Unmarshal(dst any) error {
return c.UnmarshalContext(context.Background(), dst)
}
// UnmarshalContext decodes merged JSON into dst.
func (c *Codec) UnmarshalContext(ctx context.Context, dst any) error {
m, err := c.Map(ctx)
if err != nil {
return err
}
return structconv.Decode(m, dst, c.structOpts)
}
// Marshal encodes src to JSON with indentation (struct or map[string]any).
func (c *Codec) Marshal(src any) ([]byte, error) {
m, err := encodeToMap(src)
if err != nil {
return nil, fmt.Errorf("json: encode: %w", err)
}
b, err := json.MarshalIndent(m, "", " ")
if err != nil {
return nil, fmt.Errorf("json: marshal: %w", err)
}
return b, nil
}
// WriteTo writes JSON to w.
func (c *Codec) WriteTo(w io.Writer, src any) (int64, error) {
b, err := c.Marshal(src)
if err != nil {
return 0, err
}
n, err := w.Write(b)
return int64(n), err
}
func encodeToMap(src any) (map[string]any, error) {
if m, ok := src.(map[string]any); ok {
return m, nil
}
return structconv.Encode(src)
}
+19
View File
@@ -0,0 +1,19 @@
package json
import (
"context"
"testing"
"github.com/eslider/go-config/testfixtures"
)
func TestCodec_IdentityJSONFixture(t *testing.T) {
c := New(WithBytes(testfixtures.Load(t, "identity", "service.json")))
m, err := c.Map(context.Background())
if err != nil {
t.Fatal(err)
}
if m["service"].(map[string]any)["name"] != "my-service" {
t.Fatalf("%#v", m)
}
}
+3
View File
@@ -0,0 +1,3 @@
// Package json loads JSON configuration from bytes, files, or URLs into maps
// and structs, with optional key normalization and multi-source merging.
package json
+87
View File
@@ -0,0 +1,87 @@
package json
import (
"io"
"net/http"
"github.com/eslider/go-config/internal/keymap"
"github.com/eslider/go-config/internal/merge"
"github.com/eslider/go-config/internal/source"
"github.com/go-viper/mapstructure/v2"
)
// Option configures a Codec.
type Option func(*Codec)
// WithBytes appends JSON from bytes.
func WithBytes(b []byte) Option {
return func(c *Codec) { c.sources = append(c.sources, source.Bytes{Data: b, Name: "bytes"}) }
}
// WithReader appends JSON from r.
func WithReader(r io.Reader) Option {
return func(c *Codec) { c.sources = append(c.sources, source.Reader{R: r, Name: "reader"}) }
}
// WithFile appends a JSON file path.
func WithFile(path string) Option {
return func(c *Codec) { c.sources = append(c.sources, source.File{Path: path}) }
}
// WithURL appends JSON from an HTTP(S) URL.
func WithURL(raw string) Option {
return func(c *Codec) {
var hdr http.Header
if c.urlHeader != nil {
hdr = c.urlHeader.Clone()
}
c.sources = append(c.sources, source.URL{Raw: raw, Header: hdr, HTTPClient: c.httpClient})
}
}
// WithHTTPClient sets the client for subsequent WithURL sources.
func WithHTTPClient(client *http.Client) Option {
return func(c *Codec) { c.httpClient = client }
}
// WithHTTPHeader adds a header for subsequent WithURL sources.
func WithHTTPHeader(k, v string) Option {
return func(c *Codec) {
if c.urlHeader == nil {
c.urlHeader = make(http.Header)
}
c.urlHeader.Add(k, v)
}
}
// WithKeyNormalizer sets key normalizer after merge (nil disables).
func WithKeyNormalizer(n keymap.Normalizer) Option {
return func(c *Codec) { c.normalizer = n }
}
// WithSliceMerge sets slice merge strategy across sources.
func WithSliceMerge(s merge.SliceStrategy) Option {
return func(c *Codec) { c.sliceStrat = s }
}
// WithTrim toggles string trim on struct decode.
func WithTrim(enable bool) Option {
return func(c *Codec) { c.structOpts.Trim = enable }
}
// WithWeaklyTyped toggles weak typing for struct decode.
func WithWeaklyTyped(enable bool) Option {
return func(c *Codec) { c.structOpts.WeaklyTyped = enable }
}
// WithTagName sets mapstructure tag name.
func WithTagName(name string) Option {
return func(c *Codec) { c.structOpts.TagName = name }
}
// WithDecodeHook appends a decode hook.
func WithDecodeHook(h mapstructure.DecodeHookFunc) Option {
return func(c *Codec) {
c.structOpts.ExtraHooks = append(c.structOpts.ExtraHooks, h)
}
}
-47
View File
@@ -1,47 +0,0 @@
package env
import "github.com/mitchellh/mapstructure"
// Option customises Unmarshal/UnmarshalPrefix behaviour.
type Option func(*options)
type options struct {
trim bool
weaklyTyped bool
tagName string
extraHooks []mapstructure.DecodeHookFunc
}
func defaultOptions() options {
return options{
trim: true,
weaklyTyped: true,
tagName: "mapstructure",
}
}
// WithTrim toggles automatic TrimSpace on string values. Default: true.
// Use WithTrim(false) when whitespace is semantically meaningful.
func WithTrim(enable bool) Option {
return func(o *options) { o.trim = enable }
}
// WithWeaklyTyped toggles mapstructure's WeaklyTypedInput. Default: true.
// When true, "1" decodes into int, "true" into bool, etc. Disable for
// strict string-only decoding.
func WithWeaklyTyped(enable bool) Option {
return func(o *options) { o.weaklyTyped = enable }
}
// WithTagName selects the struct tag mapstructure uses for field names.
// Default: "mapstructure". Set "env" to use `env:"FIELD_NAME"` tags.
func WithTagName(name string) Option {
return func(o *options) { o.tagName = name }
}
// WithDecodeHook appends a user hook to the decoder chain. Hooks run after
// the built-in trim hook (unless trimming is disabled) and in the order
// they're added.
func WithDecodeHook(h mapstructure.DecodeHookFunc) Option {
return func(o *options) { o.extraHooks = append(o.extraHooks, h) }
}
+46
View File
@@ -0,0 +1,46 @@
// Package testfixtures resolves paths to files under the module's fixtures/ directory.
package testfixtures
import (
"os"
"path/filepath"
"runtime"
"strings"
"testing"
)
func moduleRoot(t *testing.T) string {
t.Helper()
_, file, _, ok := runtime.Caller(0)
if !ok {
t.Fatal("runtime.Caller")
}
dir := filepath.Dir(file)
for {
data, err := os.ReadFile(filepath.Join(dir, "go.mod"))
if err == nil && strings.Contains(string(data), "module github.com/eslider/go-config") {
return dir
}
parent := filepath.Dir(dir)
if parent == dir {
t.Fatal("module root not found")
}
dir = parent
}
}
// Path joins repo fixtures/ with parts.
func Path(t *testing.T, parts ...string) string {
t.Helper()
return filepath.Join(append([]string{moduleRoot(t), "fixtures"}, parts...)...)
}
// Load reads a fixture file relative to fixtures/.
func Load(t *testing.T, parts ...string) []byte {
t.Helper()
b, err := os.ReadFile(Path(t, parts...))
if err != nil {
t.Fatal(err)
}
return b
}
+114
View File
@@ -0,0 +1,114 @@
package yaml
import (
"context"
"fmt"
"io"
"net/http"
"github.com/eslider/go-config/internal/bytesutil"
"github.com/eslider/go-config/internal/keymap"
"github.com/eslider/go-config/internal/merge"
"github.com/eslider/go-config/internal/source"
"github.com/eslider/go-config/internal/structconv"
yaml3 "gopkg.in/yaml.v3"
)
// Codec loads YAML from multiple sources.
type Codec struct {
sources []source.Source
normalizer keymap.Normalizer
sliceStrat merge.SliceStrategy
structOpts structconv.Options
mergeOpts []merge.Option
httpClient *http.Client
urlHeader http.Header
}
// New creates a YAML codec.
func New(opts ...Option) *Codec {
c := &Codec{
normalizer: keymap.LowerAlnum,
sliceStrat: merge.Replace,
structOpts: structconv.Options{
TagName: "mapstructure",
WeaklyTyped: true,
Trim: true,
},
}
for _, o := range opts {
o(c)
}
c.mergeOpts = []merge.Option{merge.WithSliceStrategy(c.sliceStrat)}
return c
}
// Map merges all YAML sources into one map.
func (c *Codec) Map(ctx context.Context) (map[string]any, error) {
if len(c.sources) == 0 {
return nil, fmt.Errorf("yaml: no sources")
}
acc := make(map[string]any)
for _, s := range c.sources {
b, err := bytesutil.ReadAll(ctx, s)
if err != nil {
return nil, fmt.Errorf("yaml: read %s: %w", s.String(), err)
}
var parsed map[string]any
if err := yaml3.Unmarshal(b, &parsed); err != nil {
return nil, fmt.Errorf("yaml: parse %s: %w", s.String(), err)
}
if parsed == nil {
parsed = map[string]any{}
}
merge.DeepMerge(acc, parsed, c.mergeOpts...)
}
if c.normalizer != nil {
keymap.Walk(acc, c.normalizer)
}
return acc, nil
}
// Unmarshal decodes using context.Background.
func (c *Codec) Unmarshal(dst any) error {
return c.UnmarshalContext(context.Background(), dst)
}
// UnmarshalContext decodes merged YAML into dst.
func (c *Codec) UnmarshalContext(ctx context.Context, dst any) error {
m, err := c.Map(ctx)
if err != nil {
return err
}
return structconv.Decode(m, dst, c.structOpts)
}
// Marshal encodes src to YAML bytes (struct or map[string]any).
func (c *Codec) Marshal(src any) ([]byte, error) {
m, err := encodeToMap(src)
if err != nil {
return nil, fmt.Errorf("yaml: encode: %w", err)
}
b, err := yaml3.Marshal(m)
if err != nil {
return nil, fmt.Errorf("yaml: marshal: %w", err)
}
return b, nil
}
// WriteTo writes YAML to w.
func (c *Codec) WriteTo(w io.Writer, src any) (int64, error) {
b, err := c.Marshal(src)
if err != nil {
return 0, err
}
n, err := w.Write(b)
return int64(n), err
}
func encodeToMap(src any) (map[string]any, error) {
if m, ok := src.(map[string]any); ok {
return m, nil
}
return structconv.Encode(src)
}
+78
View File
@@ -0,0 +1,78 @@
package yaml
import (
"context"
"reflect"
"testing"
"github.com/eslider/go-config/internal/keymap"
"github.com/eslider/go-config/internal/merge"
"github.com/eslider/go-config/testfixtures"
yaml3 "gopkg.in/yaml.v3"
)
func TestCodec_IdentityFixture(t *testing.T) {
c := New(WithBytes(testfixtures.Load(t, "identity", "service.yaml")))
m, err := c.Map(context.Background())
if err != nil {
t.Fatal(err)
}
exp, err := loadExpectedMap(t)
if err != nil {
t.Fatal(err)
}
keymap.Walk(exp, keymap.LowerAlnum)
if !mapsEqual(m, exp) {
t.Fatalf("map mismatch\n got: %#v\n exp: %#v", m, exp)
}
}
func TestCodec_MergeReplaceSlices(t *testing.T) {
c := New(
WithBytes(testfixtures.Load(t, "merge", "defaults.yaml")),
WithBytes(testfixtures.Load(t, "merge", "overlay.yaml")),
)
got, err := c.Map(context.Background())
if err != nil {
t.Fatal(err)
}
exp := map[string]any{}
if err := yaml3.Unmarshal(testfixtures.Load(t, "merge", "expected-replace.yaml"), &exp); err != nil {
t.Fatal(err)
}
keymap.Walk(exp, keymap.LowerAlnum)
if !mapsEqual(got, exp) {
t.Fatalf("merge mismatch\n got: %#v\n exp: %#v", got, exp)
}
}
func TestCodec_MergeConcatSlices(t *testing.T) {
c := New(
WithBytes(testfixtures.Load(t, "merge", "defaults.yaml")),
WithBytes(testfixtures.Load(t, "merge", "overlay.yaml")),
WithSliceMerge(merge.Concat),
)
got, err := c.Map(context.Background())
if err != nil {
t.Fatal(err)
}
exp := map[string]any{}
if err := yaml3.Unmarshal(testfixtures.Load(t, "merge", "expected-concat.yaml"), &exp); err != nil {
t.Fatal(err)
}
keymap.Walk(exp, keymap.LowerAlnum)
if !mapsEqual(got, exp) {
t.Fatalf("concat mismatch\n got: %#v\n exp: %#v", got, exp)
}
}
func loadExpectedMap(t *testing.T) (map[string]any, error) {
t.Helper()
var m map[string]any
err := yaml3.Unmarshal(testfixtures.Load(t, "identity", "service.yaml"), &m)
return m, err
}
func mapsEqual(a, b map[string]any) bool {
return reflect.DeepEqual(a, b)
}
+3
View File
@@ -0,0 +1,3 @@
// Package yaml loads YAML configuration from bytes, files, or URLs into maps
// and structs, with optional key normalization and multi-source merging.
package yaml
+16
View File
@@ -0,0 +1,16 @@
package yaml
import (
"context"
"testing"
"github.com/eslider/go-config/testfixtures"
)
func TestCodec_InvalidYAML(t *testing.T) {
c := New(WithBytes(testfixtures.Load(t, "invalid", "malformed.yaml")))
_, err := c.Map(context.Background())
if err == nil {
t.Fatal("expected parse error")
}
}
+87
View File
@@ -0,0 +1,87 @@
package yaml
import (
"io"
"net/http"
"github.com/eslider/go-config/internal/keymap"
"github.com/eslider/go-config/internal/merge"
"github.com/eslider/go-config/internal/source"
"github.com/go-viper/mapstructure/v2"
)
// Option configures a Codec.
type Option func(*Codec)
// WithBytes appends a YAML source from bytes.
func WithBytes(b []byte) Option {
return func(c *Codec) { c.sources = append(c.sources, source.Bytes{Data: b, Name: "bytes"}) }
}
// WithReader appends a YAML source from r.
func WithReader(r io.Reader) Option {
return func(c *Codec) { c.sources = append(c.sources, source.Reader{R: r, Name: "reader"}) }
}
// WithFile appends a YAML file path.
func WithFile(path string) Option {
return func(c *Codec) { c.sources = append(c.sources, source.File{Path: path}) }
}
// WithURL appends a YAML document at an HTTP(S) URL.
func WithURL(raw string) Option {
return func(c *Codec) {
var hdr http.Header
if c.urlHeader != nil {
hdr = c.urlHeader.Clone()
}
c.sources = append(c.sources, source.URL{Raw: raw, Header: hdr, HTTPClient: c.httpClient})
}
}
// WithHTTPClient sets the client for subsequent WithURL sources.
func WithHTTPClient(client *http.Client) Option {
return func(c *Codec) { c.httpClient = client }
}
// WithHTTPHeader adds a header for subsequent WithURL sources.
func WithHTTPHeader(k, v string) Option {
return func(c *Codec) {
if c.urlHeader == nil {
c.urlHeader = make(http.Header)
}
c.urlHeader.Add(k, v)
}
}
// WithKeyNormalizer sets key normalizer after merge (nil disables).
func WithKeyNormalizer(n keymap.Normalizer) Option {
return func(c *Codec) { c.normalizer = n }
}
// WithSliceMerge sets slice merge strategy across sources.
func WithSliceMerge(s merge.SliceStrategy) Option {
return func(c *Codec) { c.sliceStrat = s }
}
// WithTrim toggles string trim on struct decode.
func WithTrim(enable bool) Option {
return func(c *Codec) { c.structOpts.Trim = enable }
}
// WithWeaklyTyped toggles weak typing for struct decode.
func WithWeaklyTyped(enable bool) Option {
return func(c *Codec) { c.structOpts.WeaklyTyped = enable }
}
// WithTagName sets mapstructure tag name.
func WithTagName(name string) Option {
return func(c *Codec) { c.structOpts.TagName = name }
}
// WithDecodeHook appends a decode hook.
func WithDecodeHook(h mapstructure.DecodeHookFunc) Option {
return func(c *Codec) {
c.structOpts.ExtraHooks = append(c.structOpts.ExtraHooks, h)
}
}