layer-wise

The model is handled as a DBN only for pre-training, and layer-wise pre-training does a similar job anyway.

Just a directed model where inference is attained by marginalization of the hidden binary nodes in a layer-wise manner and by using a naive mean field approximation.

Penalizing expert swaps doesn't seem like it would help much, because experts vary by layer and are picked layer-wise.

Also using layer-wise BP pre-training does bring performance close to DBN pre-training, with no use of DBNs paradigms at all.

You'd also be able to scale out your KV cache storage since that naturally splits layer-wise.

What you are describing sounds like a technique called pretraining (in particular, greedy, layer-wise pretraining).

In 2006, Hinton introduced greedy layer-wise pretraining, which was intended to solve the problem of backpropagation getting stuck in poor local optima.

They seem to tweak the pretraining with layer-wise back-propagation (instead of full MLP-as-DBN pre-training).

LLMs seem much too simple (just layer-wise pass-thru data flow) to be conscious.

You’re restricted to output logits only, with no access to attention patterns, intermediate activations, or layer-wise representations which are needed for proper knowledge transfer.

This would require layer-wise accumulation, likely via some form of tree reduction.

They expect to scale up to real LLMs by splitting a model layer-wise across several chips, which they can do without incurring any throughput penalty.

Layer-wise, the app is pretty far removed from request routing to GPU pools.