Package io.github.kirstenali.deepj.layers.transformer.norm

Class RMSNorm1D

java.lang.Object
io.github.kirstenali.deepj.layers.transformer.norm.RMSNorm1D
All Implemented Interfaces:
Layer, NormLayer, Trainable
public final class RMSNorm1D extends Object implements NormLayer
Root-Mean-Square Layer Normalisation — the normalisation used in Llama, Mistral, Qwen, and DeepSeek.

Differs from LayerNorm1D only in that it omits the mean-subtraction step, normalising purely by RMS and scaling by a learnable gamma (no beta).

Forward: 

   rms  = sqrt( mean(x²) + ε )         shape: [seqLen × 1]
   x̂   = x / rms                        shape: [seqLen × dim]
   out  = γ · x̂                          shape: [seqLen × dim]

Backward derivation (per row, feature index k): 

   g    = γ · dOut             (upstream scaled by gamma)
   dL/dx = ( g − x̂ · mean(g · x̂) ) / rms