Hello Akshay! Glad you enjoyed.

The network chooses a subnetwork based on random initialization, when weights are in convenient places. If we were to take those subnetworks out and put them back in, presumably the network would not explore other networks because the subnetwork weights have already been trained (probably still 5% of the neurons would be used if the subnetwork is 5% of the network).

Perhaps I didn’t understand your comment correctly — let me know if I made a mistake.

Cheers!