Package io.github.kirstenali.deepj.inference

Class InferenceContext

java.lang.Object

io.github.kirstenali.deepj.inference.InferenceContext

All Implemented Interfaces:

AutoCloseable

public final class InferenceContext
extends Object
implements AutoCloseable

Cache slots follow logical attention visits, so each repeated Loop pass has its own state.

Constructor Summary

Constructors

Constructor	Description
InferenceContext(int capacity)
InferenceContext(int capacity, boolean rolling)

Method Summary

Modifier and Type	Method	Description
Tensor	attend(Object owner, Tensor query, Tensor key, Tensor value, int queryHeads, int keyValueHeads, float scale)
void	begin(int offset)
void	close()
void	finish()
int	position()
Tensor	rotate(RotaryEmbedding rotary, Tensor input, int length, int heads)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

InferenceContext

public InferenceContext(int capacity)

InferenceContext

public InferenceContext(int capacity,
 boolean rolling)

Method Details

position

public int position()

begin

public void begin(int offset)

rotate

public Tensor rotate(RotaryEmbedding rotary,
 Tensor input,
 int length,
 int heads)

attend

public Tensor attend(Object owner,
 Tensor query,
 Tensor key,
 Tensor value,
 int queryHeads,
 int keyValueHeads,
 float scale)

finish

public void finish()

close

public void close()

Specified by:

close in interface AutoCloseable