Package io.github.kirstenali.deepj.layers.transformer

Class SwiGLULayer

java.lang.Object

io.github.kirstenali.deepj.layers.transformer.SwiGLULayer

All Implemented Interfaces:

Layer, Trainable

public final class SwiGLULayer
extends Object
implements Layer

SwiGLU feed-forward layer — the FFN used in Llama, Mistral, Qwen, and DeepSeek.

Unlike the standard two-projection MLP, SwiGLU uses three projections and a multiplicative gate:

   gate   = gateProj(x)              [seqLen × dFF]
   up     = upProj(x)                [seqLen × dFF]
   fused  = SiLU(gate) · up          [seqLen × dFF]   (element-wise)
   out    = downProj(fused)           [seqLen × dModel]
 

Backward (chain rule through the gated multiply):

   dFused     = downProj.backward(dOut)
   dSiluGate  = dFused · up           → fed into SiLU.backward → dGate
   dUp        = dFused · SiLU(gate)   → fed into upProj.backward
   dX         = gateProj.backward(dGate) + upProj.backward(dUp)
 

Used as the feed-forward component in LlamaTransformerBlock and DeepSeekTransformerBlock.

Constructor Summary

Constructors

Constructor	Description
SwiGLULayer(int dModel, int dFF, Random rnd)

Method Summary

Modifier and Type	Method	Description
Tensor	backward(Tensor dOut)
Tensor	forward(Tensor x)
List<Parameter>	parameters()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.github.kirstenali.deepj.training.Trainable

zeroGrad

Constructor Details

SwiGLULayer

public SwiGLULayer(int dModel,
 int dFF,
 Random rnd)

Parameters:

dModel - input and output dimension

dFF - intermediate (hidden) dimension — typically ≈ 8/3 × dModel for Llama style

rnd - random source for weight initialisation

Method Details

forward

public Tensor forward(Tensor x)

Specified by:

forward in interface Layer

backward

public Tensor backward(Tensor dOut)

Specified by:

backward in interface Layer

parameters

public List<Parameter> parameters()

Specified by:

parameters in interface Trainable