How Encoder passes Attention Matrix to Decoder in Tranformers 'Attention is all you need'?
I was reading the renowned paper 'Attention is all you need'. Though I am clear with most of the major concepts, got buggy with a few points
- How Encoder pass the attention matrix calculated using the input to Decoder? Like what I understood is it only passes the Key & Value matrix to the decoder
- From where do we get shifted output for the decoder while testing?
- As it is able to output just one token at a time, is this transformer run for multiple iterations to generate output sequence. If yes, then, how to know when to stop?
- Are weights trained in Multi-Head Attention in the decoder as it already gets Q,K & V from encoder & masked multi-head attention
Any help is appreciated
The Encoder passes the 'Attention' matrix calculated. This attention matrix is considered as the 'Key' & 'Value' matrix for the Decoder Multi-Head Attention module
Why do we need shifted output for testing? It is not required as when testing, we need to predict from token one for which 'BOS' (Beginning Of Sequence) token is considered as past token & hence automatically left-shifted
Yes, we need to iterate over & over predicting one token at a time. If the predicted token is 'EOS' (End Of Sequence), we stop
This isn't clear but looks like the Decoder's multi-head attention isn't trained
- StratifiedKFold vs KFold in scikit-learn
- Training a Keras model to identify leap years
- Ideas for Extracting Blade Tip Coordinates from masked Wind Turbine Image
- Macro VS Micro VS Weighted VS Samples F1 Score
- Doing PyWavelets calculation on GPU
- Training loss increases instead of decrease with epochs
- How to save a Dataset in multiple shards using `tf.data.Dataset.save`
- why explain logit as 'unscaled log probabililty' in sotfmax_cross_entropy_with_logits?
- What is the loss function used in Trainer from the Transformers library of Hugging Face?
- Using features extracted using a pretrained CNN as new features for an CNN/NN
- InvalidArgumentError: No DNN in stream executor while training a TensorFlow RetinaNet model on Google Colab
- ALS (Alternating Least Square) algorithm in multiple rankings for a user
- How does one set the pad token correctly (not to eos) during fine-tuning to avoid model not predicting EOS?
- How to create image of confusion matrix in Python
- Cross-validation with nb method
- GPU utilization almost always 0 during training Hugging Face Transformer
- The “Forward/Backward Passage Size” is too large for the pytorch model (Yolov3)
- How many images(minimum) should be there in each classes for training YOLO?
- Why do neural networks work so well?
- Will larger batch size make computation time less in machine learning?
- Why KL divergence is negative in Pytorch?
- Creating a voice identification system using machine learning
- Forward pass with all samples
- override pytorch Dataset efficiently
- fit method in sklearn
- Calibrating Probabilities in lightgbm or XGBoost
- Implementation of F1-score, IOU and Dice Score
- Is it ok to have the training history very similar to the validation history?
- How to understand Shapley value for binary classification problem?
- Stochastic Gradient Descent for Logistic Regression always returns a cost of Inf and weight vector never gets any closer