I am building an LSTM that handles several parallel sequences, and I'm struggling to find any brainscript example that handles dynamic axes.
In my specific case, an example consists of a binary label and N sequences, where each sequence i has a fixed length (but may differ for j<>i). For example, sequence 1 is always length 1024, sequence 2 is length 4096, sequence 3 is length 1024. I am expressing these sequences by packing them in parallel in the CNTK text format:
0 |Label 1 |S1 0 |S2 1 |S3 0
0 |S1 1 |S2 1 |S3 1
... another 1021 rows
0 |S2 0
0 |S2 1
... another 3070 rows with only S2 defined
1 |Label 0 |S1 0 |S2 1 |S3 0
1 |S1 1 |S2 1 |S3 0
... another 1021 rows
1 |S2 1
1 |S2 0
... another 3070 rows with only S2 defined
2 |Label ...
and so on. I feel as though I've constructed examples like this in the past but I've been unable to track down any sample configs, or even any BS examples that specify dynamic axes. Is this approach doable?
The G2P example (...\Examples\SequenceToSequence\CMUDict\BrainScript\G2P.cntk) uses multiple dynamic axes. This is a snippet from this file:
# inputs and axes must be defined on top-scope level in order to get a clean node name from BrainScript.
inputAxis = DynamicAxis()
rawInput = Input (inputVocabDim, dynamicAxis=inputAxis, tag='feature')
rawLabels = Input (labelVocabDim, tag='label')
However, since in your case the axes all have the same length for each input, you may also want to consider to just put them into fixed-sized tensors. E.g instead of 1024 values, you would just have a single value of dimension 1024.
The choice depends on what you want to do with the sequences. Are you planning to run a recurrence over them? If so, you want to keep them as dynamic sequences. If they are just vectors that you plan to process with, say, big matrix products, you would rather want to keep them as static axes.