Number of learnable parameters of MultiheadAttention
While testing (using PyTorch's MultiheadAttention), I noticed that increasing or decreasing the number of heads of the multi-head attention does not change the total number of learnable parameters of my model.
Is this behavior correct? And if so, why?
Shouldn't the number of heads affect the number of parameters the model can learn?
The standard implementation of multi-headed attention divides the model's dimensionality by the number of attention heads.
A model of dimensionality d with a single attention head would project embeddings to a single triplet of d-dimensional query, key and value tensors (each projection counting d2 parameters, excluding biases, for a total of 3d2).
A model of the same dimensionality with k attention heads would project embeddings to k triplets of d/k-dimensional query, key and value tensors (each projection counting d×d/k=d2/k parameters, excluding biases, for a total of 3kd2/k=3d2).
References:
From the original paper:
The Pytorch implementation you cited:
- matplotlib 3D scatter plot alpha varies when viewing different angles
- How to write very long string that conforms with PEP8 and prevent E501
- Getting Home Directory with pathlib
- how to avoid bot detection on websites using selenium python
- Python mock to create a fake object return a dictionary when any of its attributes are used
- Polars vs. Pandas: size and speed difference
- How to mock.patch a class imported in another module
- Python - error cannot determine truth value of Relational (Newton-Raphson)
- ProcessPoolExecutor logging fails to log inside function on Windows but not on Unix / Mac
- SQLAlchemy ORM Insert or Update when importing from JSON
- django managers vs proxy models
- Pytroch clamp for complex values
- For every identifier select only rows with largest order column
- truth value for Expr is ambiguous in with_columns ternary expansion on dates
- Remove equal characters from two python strings
- Python pyad module can't set UPN
- Macro VS Micro VS Weighted VS Samples F1 Score
- Printing a Tree data structure in Python
- How to fix/reset decreasing timestamps while preserving gaps in time-series data for CNN training?
- Test that module is NOT imported
- Pyserial module isn't installed on PATH
- Print a multiplication table in Python
- Python: ModuleNotFoundError: No module named 'xyz'
- Receiving Import Error: No Module named ***, but has __init__.py
- PyQt5 QProgressBar border radius issue
- URL-encoding and -decoding a string in Python
- Fastest way to find the smallest possible sum of the absolute differences of pairs within a single array?
- Flask: Update Code Reference for: current_app._get_current_object()
- Export Charts from Excel as images using Python
- Align yaxis label spanning two axes with yaxis labels of one axes in subplots