How to incorporate ELMo into the simple classification of AllenNLP Guide

Question

I am a beginner and not a native English speaker, so I may ask poor questions.Sorry!

I recently finished the official AllenNLP tutorial(https://guide.allennlp.org/training-and-prediction) and want to change the simple classifier's word embedding to ELMo.

Also, I want to make the architecture of the simple classifier more complex to increase its accuracy. I think I'm done with the implementation of the model.

simple_classifier.py

@Model.register("simple_classifier")
class SimpleClassifier(Model):
    def __init__(
        self, vocab: Vocabulary, embedder: TextFieldEmbedder, encoder: Seq2VecEncoder
    ):
        super().__init__(vocab)
        self.embedder = embedder
        self.encoder = encoder
        num_labels = vocab.get_vocab_size("labels")
        self.dropout = torch.nn.Dropout(p=0.2)
        self.relu = torch.nn.ReLU()
        self.layer1=torch.nn.Linear(encoder.get_output_dim(),512)
        self.layer2 = torch.nn.Linear(512, 128)
        self.layer3 = torch.nn.Linear(128, 50)
        self.layer4 = torch.nn.Linear(50, 10)
        self.classifier = torch.nn.Linear(10, num_labels)
        self.accuracy = CategoricalAccuracy()

    def forward(
        self, text: TextFieldTensors, label: torch.Tensor = None
    ) -> Dict[str, torch.Tensor]:
        # Shape: (batch_size, num_tokens, embedding_dim)
        embedded_text = self.embedder(text)
        # Shape: (batch_size, num_tokens)
        mask = util.get_text_field_mask(text)
        # Shape: (batch_size, encoding_dim)
        encoded_text = self.encoder(embedded_text, mask)
        x=self.relu(self.layer1(encoded_text))
        x=self.relu(self.layer2(x))
        x=self.relu(self.layer3(x))
        x=self.relu(self.layer4(x))
        # Shape: (batch_size, num_labels)
        logits = self.classifier(x)
        # Shape: (batch_size, num_labels)
        probs = torch.nn.functional.softmax(logits)
        # Shape: (1,)
        output = {"probs": probs}
        if label is not None:
            self.accuracy(logits, label)
            output["loss"] = torch.nn.functional.cross_entropy(logits, label)
        return output

    def get_metrics(self, reset: bool = False) -> Dict[str, float]:
        return {"accuracy": self.accuracy.get_metric(reset)}

But I have no idea how to change the configuration file. How do I change the following configuration file in the official tutorial to use ELMo?

my_text_classifier.jsonnet

{
    "dataset_reader" : {
        "type": "classification-tsv",
        "token_indexers": {
            "tokens": {
                "type": "single_id"
            }
        }
    },
    "train_data_path": "data/movie_review/train.tsv",
    "validation_data_path": "data/movie_review/dev.tsv",
    "model": {
        "type": "simple_classifier",
        "embedder": {
            "token_embedders": {
                "tokens": {
                    "type": "embedding",
                    "embedding_dim": 10
                }
            }
        },
        "encoder": {
            "type": "bag_of_embeddings",
            "embedding_dim": 10
        }
    },
    "data_loader": {
        "batch_size": 8,
        "shuffle": true
    },
    "trainer": {
        "optimizer": "adam",
        "num_epochs": 5
    }
}

I'm very happy if someone could help me.

Answer 1

Check out the way the BiDAF model uses ELMo: https://raw.githubusercontent.com/allenai/allennlp-models/main/training_config/rc/bidaf_elmo.jsonnet

You can steal some of the components of that config. You will need the token embedder under the name "elmo", and, I believe, both the token indexers under "tokens" and "elmo".

It should work without having to write any code.