GELU

java.lang.Object

io.github.kirstenali.deepj.activations.GELU

All Implemented Interfaces:: ActivationFunction

public final class GELU extends Object implements ActivationFunction

Gaussian Error Linear Unit (GELU), using the tanh approximation popularized by GPT-2.

Forward: x * 0.5 * (1 + tanh(sqrt(2/pi)*(x + 0.044715*x^3))).

Constructor Summary

Constructors

Constructor

Description

GELU()
Method Summary

Modifier and Type

Method

Description

Tensor

backward(Tensor gradOutput)

Tensor

forward(Tensor input)

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details
- GELU
  
  public GELU()
Method Details
- forward
  
  public Tensor forward(Tensor input)
  
  Specified by:
  
  forward in interface ActivationFunction
- backward
  
  public Tensor backward(Tensor gradOutput)
  
  Specified by:
  
  backward in interface ActivationFunction