How to type-hint a truly optional parameter with overloads

Question

I have this function complete with type-hinting:

class HDF5DataTypes(Enum):
    SCALAR = "scalar"
    ARRAY = "array"
    UNKNOWN = "unknown"

@overload
def index_hdf5_to_value(file_or_group: Union[h5py.File, h5py.Group], indexes: List[str], expected_output_type: Literal[HDF5DataTypes.SCALAR]) -> np.float_:
    ...
@overload
def index_hdf5_to_value(file_or_group: Union[h5py.File, h5py.Group], indexes: List[str], expected_output_type: Literal[HDF5DataTypes.ARRAY]) -> npt.NDArray:
    ...
@overload
def index_hdf5_to_value(file_or_group: Union[h5py.File, h5py.Group], indexes: List[str], expected_output_type: Literal[HDF5DataTypes.UNKNOWN]) -> Union[npt.NDArray, np.float_]:
    ...
def index_hdf5_to_value(file_or_group: Union[h5py.File, h5py.Group], indexes: List[str], expected_output_type: HDF5DataTypes=HDF5DataTypes.UNKNOWN) -> Union[npt.NDArray, np.float_]:
    '''Given a file or group, returns the output of indexing the file or group with the indexes down until it gets to the dataset, at which point it gives back the value of the dataset (either the scalar or numpy array).
    '''
    dataset = index_hdf5(file_or_group, indexes, h5py.Dataset)
    if len(dataset.shape) == 0:
        if expected_output_type == HDF5DataTypes.ARRAY:
            raise ValueError(f"Expected output to be an array, but it was a scalar")
        return cast(np.float_, dataset[()])
    else:
        if expected_output_type == HDF5DataTypes.SCALAR:
            raise ValueError(f"Expected output to be a scalar, but it was an array")
        return cast(npt.NDArray, dataset[:])

However, when I go to call it with index_hdf5_to_value(hdf5_file, ["key1", "key2"]) I get the error No overloads for "index_hdf5_to_value" match the provided arguments Argument types: (File, list[str]).

In short, it's upset because I didn't provide a the third parameter.

Now, I could type-hint the third parameter as Optional, but then I worry that that hints that calling the function with index_hdf5_to_value(hdf5_file, ["key1", "key2"], None) is okay, which it is not.

How should I correctly type-hint this function in order to tell the user that the third parameter is optional, but cannot be set to None? (That is, it's optional, but not Optional.)

Answer 1

Despite the name, Optional doesn't mean "optional". It means "possibly None". Annotating the parameter with Optional wouldn't help - mypy still wouldn't let you omit the argument.

You need to provide an overload that actually accepts being called without an expected_output_type argument. This could be a separate overload:

@overload
def index_hdf5_to_value(
        file_or_group: Union[h5py.File, h5py.Group],
        indexes: List[str]) -> Union[npt.NDArray, np.float_]:
    ...

or you could add a default argument value to the HDF5DataTypes.UNKNOWN overload:

@overload
def index_hdf5_to_value(
        file_or_group: Union[h5py.File, h5py.Group],
        indexes: List[str],
        expected_output_type: Literal[HDF5DataTypes.UNKNOWN] = HDF5DataTypes.UNKNOWN
        ) -> Union[npt.NDArray, np.float_]:
    ...