Pydantic does not give precedence to os environment

Question

I am tyring to implement proper environment variable management across dev, prod and staging environments. Using Pydantic is critical since Pydantic is everywhere in the application.

Testing this application becomes challenging due to the setting of os-based environment variables under production and dev environements.

I have the following:

from pydantic import Field
from pydantic_settings import BaseSettings, SettingsConfigDict

from devtools import debug


class Config(BaseSettings):

    # Environment
    STAGE: str = Field('prod', validation_alias='STAGE')
    OTHER: str

    model_config = SettingsConfigDict(env_file='.test.env', env_file_encoding='utf-8', env_prefix='DEV_')


settings = Config()

debug(settings)

with a .test.env file that looks like this:

STAGE=test
DEV_OTHER="other"

and I set an OS environment variable as follows:

export STAGE=dev

giving the following response with printenv:

...
STAGE=dev
...

and then get the following output:

test.py:18 <module>
    settings: Config(
        STAGE='test',
        OTHER='other',
    ) 

In Pydantic, the OS environment variable will take precedence, but this is not happening here since the setting of environment variables in the dev environment is process-related.

This leads to unexpected behaviour with Pydantic Settings. Is there a workaround? I do not want to set STAGE in a bash file and restart the environment (which normally means restarting the computer) whenever I want to test behaviour under different environments.

Answer 1

The issue is not Pytdantic-related, but the visibility of environment variables and the respective processes in which they are created, as alluded to by juanpa.arrivillaga and lord_haffi. A variable created via export in a shell is only available to that shell and any subprocess of that shell. It needs to be persisted through adding to ~/.zshrc or similar before it will be available to other processes.