Package io.github.kirstenali.deepj.activations

Class SiLU

java.lang.Object

io.github.kirstenali.deepj.activations.SiLU

All Implemented Interfaces:

ActivationFunction

public final class SiLU
extends Object
implements ActivationFunction

Sigmoid Linear Unit (SiLU / Swish).

Forward: SiLU(x) = x · σ(x)

Backward: SiLU'(x) = σ(x) · (1 + x · (1 − σ(x)))

Used as the gate activation inside SwiGLULayer and is the standard FFN activation in Llama, Mistral, and Qwen.

Constructor Summary

Constructors

Constructor	Description
SiLU()

Method Summary

Modifier and Type	Method	Description
Tensor	backward(Tensor gradOutput)
Tensor	forward(Tensor input)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

SiLU

public SiLU()

Method Details

forward

public Tensor forward(Tensor input)

Specified by:

forward in interface ActivationFunction

backward

public Tensor backward(Tensor gradOutput)

Specified by:

backward in interface ActivationFunction