Usage

Piny loads your YAML configuration file. It optionally validates data loaded from config file. Piny main logic is in a loader class. You can pass arguments in the loader class to change the way YAML file is parsed and validated.

Loaders

YamlLoader loader class is dedicated for use in Python applications. Based on PyYAML, it parses YAML files, (arguably) the most beautiful file format for configuration files!

Basic loader usage is the following.

1. Set your environment variables

2. Mark up your YAML configuration file with these env names:

db:
  login: user
  password: ${DB_PASSWORD}
mail:
  login: user
  password: ${MAIL_PASSWORD:-my_default_password}
sentry:
  dsn: ${VAR_NOT_SET}

3. In your app load config with Piny:

from piny import YamlLoader

config = YamlLoader(path="config.yaml").load()
print(config)
# {'db': {'login': 'user', 'password': 'my_db_password'},
# 'mail': {'login': 'user', 'password': 'my_default_password'},
# 'sentry': {'dsn': None}}

YamlStreamLoader class primary use is Piny CLI tool (see Command line utility). But it also can be used interchargably with YamlLoader whenever IO streams are used instead of file paths.

class piny.loaders.YamlLoader(path: str, *, matcher: ~typing.Type[~piny.matchers.Matcher] = <class 'piny.matchers.MatcherWithDefaults'>, validator: ~typing.Type[~piny.validators.Validator] | None = None, schema: ~typing.Any = None, **schema_params)

Bases: object

YAML configuration file loader

load(**params) → Any

Return Python object loaded (optionally validated) from the YAML-file

Parameters:

params – named arguments used as optional loading params in validation

class piny.loaders.YamlStreamLoader(stream: str | ~typing.IO[str], *, matcher: ~typing.Type[~piny.matchers.Matcher] = <class 'piny.matchers.MatcherWithDefaults'>, validator: ~typing.Type[~piny.validators.Validator] | None = None, schema: ~typing.Any = None, **schema_params)

Bases: YamlLoader

YAML configuration loader for IO streams, e.g. file objects or stdin

load(**params) → Any

Return Python object loaded (optionally validated) from the YAML-file

Parameters:

params – named arguments used as optional loading params in validation

Matchers

In the Loaders section we used Bash-style environment variables with defaults. You may want to discourage such envs in your project. This is where matchers come in handy. They apply a regular expression when parsing your YAML file that matches environment variables we want to interpolate.

By default MatcherWithDefaults is used. StrictMatcher is another matcher class used for plain vanilla envs with no default values support.

Both strict and default matchers return None value if environment variable matched is not set in the system.

Basic usage example is the following:

from piny import YamlLoader, StrictMatcher

config = YamlLoader(path="config.yaml", matcher=StrictMatcher).load()

class piny.matchers.Matcher(stream)

Bases: SafeLoader

Base class for matchers

Use this class only to derive new child classes

static constructor(loader, node)

matcher: Pattern[str] = re.compile('')

class piny.matchers.MatcherWithDefaults(stream)

Bases: Matcher

Expand an environment variable with its value

Forms supported: ${VAR}, ${VAR:-default} If value is not set and no default value given return None.

static constructor(loader, node)

matcher: Pattern[str] = re.compile('\\$\\{([a-zA-Z_$0-9]+)(:-.*)?\\}')

class piny.matchers.StrictMatcher(stream)

Bases: Matcher

Expand an environment variable of form ${VAR} with its value

If value is not set return None.

static constructor(loader, node)

matcher: Pattern[str] = re.compile('\\$\\{([^}^{^:]+)\\}')

Validators

Piny supports optional data validation using third-party libraries: Marshmallow, Pydantic, Trafaret.

In order to use data validation pass validator and schema arguments in the Loaders class. You may also initialize loader class with optional named arguments that will be passed to the validator’s schema. Additional loading arguments may be passed in load method invocation.

class piny.validators.MarshmallowValidator(schema: Any, **params)

Bases: Validator

Validator class for Marshmallow library

load(data: Dict[str, Any] | List[Any], **params)

Load data, return validated data or raise en error

class piny.validators.PydanticV2Validator(schema: Any, **params)

Bases: Validator

Validator class for Pydantic Version 2

load(data: Dict[str, Any] | List[Any], **params)

Load data, return validated data or raise en error

class piny.validators.PydanticValidator(schema: Any, **params)

Bases: Validator

Validator class for Pydantic Version 1

load(data: Dict[str, Any] | List[Any], **params)

Load data, return validated data or raise en error

class piny.validators.TrafaretValidator(schema: Any, **params)

Bases: Validator

Validator class for Trafaret library

load(data: Dict[str, Any] | List[Any], **params)

Load data, return validated data or raise en error

class piny.validators.Validator(schema: Any, **params)

Bases: ABC

Abstract base class for optional validator classes

Use only to derive new child classes, implement all abstract methods

abstract load(data: Dict[str, Any] | List[Any], **params)

Load data, return validated data or raise en error

Marshmallow validation example

import marshmallow as ma
from piny import MarshmallowValidator, StrictMatcher, YamlLoader


class DBSchema(ma.Schema):
    login = ma.fields.String(required=True)
    password = ma.fields.String()


class ConfigSchema(ma.Schema):
    db = ma.fields.Nested(DBSchema)


config = YamlLoader(
    path="database.yaml",
    matcher=StrictMatcher,
    validator=MarshmallowValidator,
    schema=ConfigSchema,
).load(many=False)

Pydantic validation example

from pydantic import BaseModel
from piny import PydanticV2Validator, StrictMatcher, YamlLoader

# Watch out!
# Pydantic V2 deprecated some model's methods:
# https://docs.pydantic.dev/2.0/migration/
#
# For Pydantic v2 use `PydanticV2Validator`
# For Pydantic v1 use `PydanticValidator`

class DBModel(BaseModel):
    login: str
    password: str


class ConfigModel(BaseModel):
    db: DBModel


config = YamlLoader(
    path="database.yaml",
    matcher=StrictMatcher,
    validator=PydanticV2Validator,
    schema=ConfigModel,
).load()

Trafaret validation example

import trafaret
from piny import TrafaretValidator, StrictMatcher, YamlLoader


DBSchema = trafaret.Dict(login=trafaret.String, password=trafaret.String)
ConfigSchema = trafaret.Dict(db=DBSchema)

config = YamlLoader(
    path="database.yaml",
    matcher=StrictMatcher,
    validator=TrafaretValidator,
    schema=ConfigSchema,
).load()

Exceptions

LoadingError is thrown when something goes wrong with reading or parsing a YAML file. ValidationError is a wrapper for exceptions raised by the libraries for optional data validation. Original exception can be accessed by origin attribute. It comes in handy when you need more than just an original exception message (e.g. a dictionary of validation errors).

Both exceptions inherit from the ConfigError.

exception piny.errors.ConfigError(origin: Exception | None = None, **context: Any)

Bases: PinyErrorMixin, Exception

Base class for Piny exceptions

exception piny.errors.LoadingError(origin: Exception | None = None, **context: Any)

Bases: ConfigError

Exception for reading or parsing configuration file errors

msg_template: str = 'Loading YAML file failed: {reason}'

class piny.errors.PinyErrorMixin(origin: Exception | None = None, **context: Any)

Bases: object

Mixin class to wrap and format original exception

msg_template: str

exception piny.errors.ValidationError(origin: Exception | None = None, **context: Any)

Bases: ConfigError

Exception for data validation errors

msg_template: str = 'Validation failed: {reason}'

Command line utility

Piny comes with CLI tool that substitutes the values of environment variables in input file or stdin and write result to an output file or stdout. Piny CLI utility is somewhat similar to GNU/gettext envsubst but works with files too.

piny

Substitute environment variables with their values.

Read INPUT, find environment variables in it, substitute them with their values and write to OUTPUT.

INPUT and OUTPUT can be files or standard input and output respectively. With no INPUT, or when INPUT is -, read standard input. With no OUTPUT, or when OUTPUT is -, write to standard output.

Examples:

piny input.yaml output.yaml

piny - output.yaml

piny input.yaml -

tail -n 12 input.yaml | piny > output.yaml

piny [OPTIONS] [INPUT] [OUTPUT]

Options

--strict, --no-strict

Enable or disable strict matcher

Arguments

INPUT

Optional argument

OUTPUT

Optional argument