com.plexpt.chatgpt.util

Class TokensUtil

java.lang.Object

com.plexpt.chatgpt.util.TokensUtil

public class TokensUtil
extends Object

Field Summary

Fields

Modifier and Type	Field and Description
static com.knuddels.jtokkit.api.Encoding	encoding
static com.knuddels.jtokkit.api.Encoding	encoding4o
static com.knuddels.jtokkit.api.EncodingRegistry	registry

Constructor Summary

Constructors

Constructor and Description
TokensUtil()

Method Summary

Modifier and Type	Method and Description
static int	countTextTokens(String text) 计算text信息的tokens
static int	countTextTokens(String text, String model) 计算text信息的tokens
static com.knuddels.jtokkit.api.ModelType	getModelTypeByName(String name) 获取modelType
static int	tokens(List<Message> messages, String model) 通过模型名称计算messages获取编码数组 参考官方的处理逻辑： https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
static int	tokens(String modelName, List<Message> messages) 计算tokens

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

registry

public static com.knuddels.jtokkit.api.EncodingRegistry registry

encoding

public static com.knuddels.jtokkit.api.Encoding encoding

encoding4o

public static com.knuddels.jtokkit.api.Encoding encoding4o

Constructor Detail

TokensUtil

public TokensUtil()

Method Detail

countTextTokens

public static int countTextTokens(String text)

计算text信息的tokens

Parameters:

text -

Returns:

countTextTokens

public static int countTextTokens(String text,
                                  String model)

计算text信息的tokens

Parameters:

text -

Returns:

getModelTypeByName

public static com.knuddels.jtokkit.api.ModelType getModelTypeByName(String name)

获取modelType

Parameters:

name -

Returns:

tokens

public static int tokens(List<Message> messages,
                         String model)

通过模型名称计算messages获取编码数组 参考官方的处理逻辑： https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb

Parameters:

messages - 消息体

Returns:

tokens

public static int tokens(String modelName,
                         List<Message> messages)

计算tokens

Parameters:

modelName - 模型名称

messages - 消息列表

Returns:

计算出的tokens数量