Regression in customizing sources in pydantic V2

[ziima]

I used a customise_sources in pydantic V1 to pass a custom config file and recently discovered that this no longer works under pydantic V2.

MWE pydantic V1

In pydantic V1 following script works just fine:

from typing import Any, Dict, Optional, Tuple, cast

import yaml
from pydantic.v1 import BaseSettings, PrivateAttr
from pydantic.v1.env_settings import SettingsSourceCallable


def _config_file_settings(settings: 'Settings') -> Dict[str, Any]:
    if settings._config_file:
        with open(settings._config_file) as file:
            return cast(Dict[str, Any], yaml.safe_load(file))
    return {}


class Settings(BaseSettings):
    _config_file: Optional[str] = PrivateAttr(None)

    answer: int = 0

    class Config:
        @classmethod
        def customise_sources(cls, init_settings: SettingsSourceCallable, env_settings: SettingsSourceCallable,
                              file_secret_settings: SettingsSourceCallable) -> Tuple[SettingsSourceCallable, ...]:
            """Include config file to sources."""
            return (init_settings, env_settings, cast(SettingsSourceCallable, _config_file_settings),
                    file_secret_settings)

    def __init__(self, *args: Any, config_file: Optional[str] = None, **kwargs: Any):
        self._config_file = config_file
        super().__init__(*args, **kwargs)


if __name__ == "__main__":
    settings = Settings(config_file='/tmp/config.yml')
    print(repr(settings))  # >>> Settings(answer=42)

MWE pydantic V2

When upgraded to pydantic V2 the script doesn't work (and I don't see a way how to fix it)

from typing import Any, Dict, Optional, Tuple, Type, cast

import yaml
from pydantic import PrivateAttr
from pydantic_settings import BaseSettings, PydanticBaseSettingsSource


def _config_file_settings(settings: 'Settings') -> Dict[str, Any]:
    if settings._config_file:
        with open(settings._config_file) as file:
            return cast(Dict[str, Any], yaml.safe_load(file))
    return {}


class Settings(BaseSettings):
    _config_file: Optional[str] = PrivateAttr(None)

    answer: int = 0

    @classmethod
    def settings_customise_sources(
            cls,
            settings_cls: Type[BaseSettings],
            init_settings: PydanticBaseSettingsSource,
            env_settings: PydanticBaseSettingsSource,
            dotenv_settings: PydanticBaseSettingsSource,
            file_secret_settings: PydanticBaseSettingsSource
    ) -> Tuple[PydanticBaseSettingsSource, ...]:
        """Include config file to sources."""
        return (init_settings,
                env_settings,
                dotenv_settings,
                cast(PydanticBaseSettingsSource, _config_file_settings),
                file_secret_settings)

    def __init__(self, *args: Any, config_file: Optional[str] = None, **kwargs: Any):
        self._config_file = config_file  #>>> AttributeError: 'Settings' object has no attribute '__pydantic_private__'. Did you mean: '__pydantic_complete__'?
        super().__init__(*args, **kwargs)

    # # Alternatively:
    # def __init__(self, *args: Any, config_file: Optional[str] = None, **kwargs: Any):
    #     super().__init__(*args, **kwargs)  #>>> TypeError: _config_file_settings() missing 1 required positional argument: 'settings'
    #     self._config_file = config_file


if __name__ == "__main__":
    settings = Settings(config_file='/tmp/config.yml')
    print(repr(settings))

Analysis

I traced the changes back to #15 where settings sources have been remade, but settings instance was dropped from the callable. Yet the documentation about the return type of settings_customise_sources explicitely states:

Each callable should take an instance of the settings class as its sole argument and return a dict.

See https://docs.pydantic.dev/latest/concepts/pydantic_settings/#customise-settings-sources

It seems to me that some use cases were omitted in the sources refactoring and there is no reasonably simple way to restore the behavior under pydantic V2 API. I wouldn't mind to provide a PR, but currently I can't even tell how to handle the differences between V1 and V2.

[hramezani]

Thanks @ziima for reporting this.
I haven't checked your provided example in detail but it seems you are going to load configs from yaml file.

If so, pydantic-settings 2.2 added yaml settings source. please check the docs.

[stinovlas]

The problem with YamlConfigSettingsSource here is that you can't read from file with arbitrary name provided by user (let's say by a command line option). There's no way to pass that name into the source.

It was possible to set a private attribute on the BaseSettings. In V1, this instance was passed to custom config source and it could read the value. But now, in V2, the settings instance is not passed down to config source so there's no easy way to pass that value (apart from maybe creating some global variable).

If nothing else, documentation is clearly outdated. Also, I was quite surprised by the fact that if you provide multiple config files, it reads them all and merges the result (instead of just using the first one found). This should also be pointed out in the docs.

[hramezani]

The problem with YamlConfigSettingsSource here is that you can't read from file with arbitrary name provided by user (let's say by a command line option). There's no way to pass that name into the source.

You can implement your custom YamlConfigSettingsSource and override YamlConfigSettingsSource.__init__ and change the value of yaml_file_path there.

pydantic-settings/pydantic_settings/sources.py

Line 780 in 7e5731a

class YamlConfigSettingsSource(InitSettingsSource, ConfigFileSourceMixin):

If nothing else, documentation is clearly outdated. Also, I was quite surprised by the fact that if you provide multiple config files, it reads them all and merges the result (instead of just using the first one found). This should also be pointed out in the docs.

if you mean dotenv file it is already mentioned in Dotenv support section of doc

If you need to load multiple dotenv files, you can pass multiple file paths as a tuple or list. The files will be loaded in order, with each file overriding the previous one.

[stinovlas]

The problem with YamlConfigSettingsSource here is that you can't read from file with arbitrary name provided by user (let's say by a command line option). There's no way to pass that name into the source.

You can implement your custom YamlConfigSettingsSource and override YamlConfigSettingsSource.__init__ and change the value of yaml_file_path there.

Yes, but I need to instantiate this custom subclass of YamlConfigSettingsSource in classmethod settings_customise_sources. How do I get it there? The problem is not in the custom settings source, the problem is how to instantiate it with value that is not known until the user runs the code.

Again, in V1, I could use a PrivateAttr on BaseSettings subclass to pass this information down to settings source (because it received settings instance as an argument). In V2, I don't see any way except for some global antipatterns.

If nothing else, documentation is clearly outdated. Also, I was quite surprised by the fact that if you provide multiple config files, it reads them all and merges the result (instead of just using the first one found). This should also be pointed out in the docs.

if you mean dotenv file it is already mentioned in Dotenv support section of doc

I'm not talking about dotenv files. I'm refering to https://docs.pydantic.dev/latest/concepts/pydantic_settings/#customise-settings-sources

settings_customise_sources takes four callables as arguments and returns
any number of callables as a tuple. In turn these callables are called to build
the inputs to the fields of the settings class.

--> Each callable should take an instance of the settings class as its sole argument and return a dict. <--

And also https://docs.pydantic.dev/latest/concepts/pydantic_settings/#other-settings-source

You can also provide multiple files by providing a list of path:

toml_file = ['config.default.toml', 'config.custom.toml']

This doesn't mention anything about the way multiple files are handled.

[hramezani]

Yeah, you don't have access to the settings model instance in source classes and We can't change it in V2 but it will be an important thing to have in V3.

Regarding multiple files for yaml and toml, it is like dotenv file. files loaded from the list and each file's values override the previous one

[limotley]

+1 On this issue. Being able to parameterize the file source for yaml/toml/etc configurations feels like it should be a basic feature of these settings, and it's frustrating to have to work around it.

[mpkocher]

One possible workaround is to use a closure. It's not particularly pretty, but it will work.

import argparse
import sys
from typing import Type, Tuple
from argparse import ArgumentParser
from pathlib import Path

from pydantic_settings import (
    BaseSettings,
    SettingsConfigDict,
    PydanticBaseSettingsSource,
    YamlConfigSettingsSource,
)


def to_settings(yaml_file: Path):
    class Settings(BaseSettings):
        model_config = SettingsConfigDict(yaml_file=yaml_file)
        alpha: int
        beta: int

        @classmethod
        def settings_customise_sources(
            cls,
            settings_cls: Type[BaseSettings],
            init_settings: PydanticBaseSettingsSource,
            env_settings: PydanticBaseSettingsSource,
            dotenv_settings: PydanticBaseSettingsSource,
            file_secret_settings: PydanticBaseSettingsSource,
        ) -> Tuple[PydanticBaseSettingsSource, ...]:
            return (YamlConfigSettingsSource(settings_cls),)

    return Settings


def validate_path(sx: str) -> Path:
    px = Path(sx)
    if px.exists():
        return px
    raise argparse.ArgumentError(None, message=f"{sx} not a valid file path.")


def to_parser() -> ArgumentParser:
    p = ArgumentParser()
    p.add_argument(
        "-f", "--yaml-file", required=True, type=validate_path, help="Path to YAML file"
    )
    return p


def main(ax: list[str]) -> int:
    p = to_parser()
    pargs = p.parse_args(ax)
    klass = to_settings(pargs.yaml_file)
    settings = klass()
    print(settings)
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

In this example, I'm also validating at the argparse level to better communicate errors. Without this, the errors will be a generic Pydantic error (e.g., Field required ...) which isn't particularly clear in communicating the fundamental issue (the yaml file isn't found). Currently, pydantic-settings will ignore any file it can't find (see ConfigFileSourceMixin._read_files).

python example.py -f test.yml
alpha=1 beta=4

And on errors:

python example.py -f junkx.yml
usage: example.py [-h] -f YAML_FILE
example.py: error: junkx.yml not a valid file path.