How does parameter sharing work in Efficient Neural Architecture Search (ENAS)

Question

ENAS implementation here

I'm trying to understand how the parameter sharing works in ENAS. The first two questions are there partially to answer the third main question.

1. Are all nodes only used ONCE during macro search?
2. For macro search, will all the nodes definitely link to its previous node?
3. How are the parameters shared? Does each operations have their own weights, which are always loaded when called? If this is the case, then which weight to update and memorize during training, assuming multiple instances of the same operation is used. Or are there weights for each unique connection, e.g. Node1 to Node3 (W13) has one weight set, Node2 to Node3 (W23) has another weight set. If so, then how does it handle cases when there are skip connections (e.g. Node1 and Node2 are concatenated, which are then passed to Node 3. Will it have W12-3?)?

Answer 1

I've gone through the code quite a few times, so I guess I'll answer these myself, in case anyone sees this in the future.

1. No, they can appear multiple times.
2. Yes.
3. Store the weights of all possible nodes in each layer, extract the weight from them when initializing new networks.