pre-training

That means you can exploit this data for training the model, for example by unsupervised pre-training or with a pseudo-labeling approach.

This means there are far fewer parameters to learn in the early parts of the network, and so pre-training seems less necessary.

Geoff Hinton popularized pre-training nets with unsupervised latent-feature learning autoencoders in the 90s, making training nets much faster.

It turns out the unsupervised pre-training isn't even necessary, go figure...

This was back when pre-training was still strongly recommended and before Krizhevsky's ImageNet win (2012), which shifted the focus of the field away from unsupervised learning somewhat.

What I've taken from it is that deep neural network training is quite sensitive to initial starting point, hence the successes of pre-training.

Pretty sure it was Hinton's group, but they used greedy unsupervised pre-training to get a good initialization of the weights, then followed by supervised finetuning of said weights.

After all, it's only recently that we figured out how to efficiently train deep nets (either pre-training with Autoencoders or RBMs, or through Hessian-free optimization).

WRT the Glorot and Bengio paper, it’s true that there was a lot of excitement surrounding unsupervised pre-training of DNNs, but this mostly preceded the current wave of successes.

Second order methods, unsupervised pre-training, RBMs, graphical models, etc.

As I understand it, pre-training is very common.

The fact that these pre-training techniques seem to be effective if not necessary (at least, the people who I read swear by them) seems to indicate that, at least for some problems, this is a real issue.

They just use it for "fine tuning" (due to running pre-training first) instead of propagating error derivatives from randomly initialized weights.

Indeed, but I quote: "The pre-training uses stochastic binary units.

This pre-training allows it to do things like recognise common patterns in images, which the later training allows it to then associate with semantic ideas like "this is an apple", "this is a person", etc.

Pre-training takes 585.01 minutes, with an average of 13 minutes per epoch.