pretraining

What you are describing sounds like a technique called pretraining (in particular, greedy, layer-wise pretraining).

In the past, RBMs have been used for unsupervised pretraining of the model, however, nowadays, everyone uses supervised pretraining.

Is it just to show that they don't need pretraining to achieve good results or is there something deeper?

Apparently, all the most recent results in speech recognition just use standard backpropagation with no unsupervised pretraining.

More recently, people, especially the younger generation of deep learning researchers, tend to be skeptical of how much pretraining helps.

But according to my friends on the Google Brain team, unsupervised pretraining is now thought to be an irrelevant detour.

People tend to prefer, now, to just use those and a lot of computing power, rather than mess around with pretraining.

The pretraining learning rate is was 0.001 and the finetuning learning rate is 0.1.

Very interesting I was not aware unsupervised pretraining was a distant second to availability of data and Flops.

In 2006, Hinton introduced greedy layer-wise pretraining, which was intended to solve the problem of backpropagation getting stuck in poor local optima.

I recently implemented the more advanced hessian free optimization methods that don't require pretraining in my deep learning framework.

A question about the actual slides: why don't they use unsupervised pretraining (i.e.

That was a big deal, and I think pretty much eliminated greedy layerwise pretraining in the "we have plenty of data, but can't generalize well" case.