How to clear Map entry after certain time of inactivity

Question

I'm working on a chatbot powered by OpenAI. I'm using the new gpt-3.5-turbo model with ChatCompletion requests. I've already come up with a way for the bot to remember conversations for each individual user using a HashMap and it all works. The bot is able to remember all the previous responses and generate a new response using the history as context. However, the more responses in the query, the more tokens that it'll use, increasing costs. I want make it where if there is a certain amount of inactivity (like 2 minutes of not talking to the bot), it'll clear the history and start a fresh new conversation to save on tokens. What would be the best way to accomplish this?

Here is my code:

private static OpenAiService service;
    
private static Map<String, List<ChatMessage>> conversations = new HashMap<String, List<ChatMessage>>();
            
public static void initializeOpenAi() {
    service = new OpenAiService(Settings.OPENAI_ACCESS_KEY);
}

public static String generateChatResponse(User user, String prompt) {
    List<ChatMessage> conversation;
        
    if (conversations.containsKey(user.getId())) {
        conversation = conversations.get(user.getId());
    } else {
        conversation = new ArrayList<ChatMessage>();
    }
        
    if (conversation.size() < 1) {
        ChatMessage system = new ChatMessage();
        system.setRole("system");
        system.setContent("Omnis is a chatbot with sarcasm");
        conversation.add(system);
    }
        
    ChatMessage userPrompt = new ChatMessage();
    userPrompt.setRole("user");
    userPrompt.setContent(prompt);
    conversation.add(userPrompt);
        
    ChatCompletionRequest chatRequest = ChatCompletionRequest.builder()
            .model("gpt-3.5-turbo")
            .maxTokens(1000)
            .temperature(0.9)
            .topP(0.3)
            .frequencyPenalty(0.9)
            .presencePenalty(0.0)
            .messages(conversation)
            .build();
        
    ChatCompletionResult chatResult = null;
        
    try {
        chatResult = service.createChatCompletion(chatRequest);
    } catch (Exception e) {
        System.out.println("An OpenAI request failed!");
            
        if (e.getMessage().contains("timeout")) {
            return "Your request timed out. Please try again after a breif wait for try a different request.";
        }
            
        if (e.getMessage().contains("Rate limit")) {
            return "Rate limit for chat requests has been reached!";
        }
            
        return "Something went wrong with your request. Cause of error is " + e.getMessage();
    }
        
    if (chatResult != null) {
        System.out.println("Created chat completion request that used " + chatResult.getUsage().getTotalTokens() + " tokens with " + chatResult.getModel());
    }
        
    String response = chatResult.getChoices().get(0).getMessage().getContent();
        
    ChatMessage assistantPrompt = new ChatMessage();
    assistantPrompt.setRole("assistant");
    assistantPrompt.setContent(response);
    conversation.add(assistantPrompt);
        
    conversations.put(user.getId(), conversation);
        
    return response;
}

The "generateChatResonse" method gets called each time a message is sent by a user. It get's the user's ID and pulls the conversation from the HashMap if it exists, if not it creates a new conversation.

I don't know what to try.

I don't want to clear the entire HashMap, just the entry associated with the inactive user.

Answer 1

I've added Guava to my project and replaced the Map with a LoadingCache object. It takes the same type of parameters and allows me to easily set the time before expiration.

private static LoadingCache<String, List<ChatMessage>> conversations;

public static void initializeOpenAi() {
    service = new OpenAiService(Settings.OPENAI_ACCESS_KEY);
        
    conversations = CacheBuilder.newBuilder()
            .maximumSize(10000)
            .expireAfterWrite(2, TimeUnit.MINUTES)
            .build(
                new CacheLoader<String, List<ChatMessage>>() {
                    @Override
                    public List<ChatMessage> load(String key) throws Exception {
                        return conversations.get(key);
                    }
                });
}

public static String generateChatResponse(User user, String prompt) {
    List<ChatMessage> conversation = conversations.getIfPresent(user.getId());
        
    if (conversation == null) {
        conversation = new ArrayList<ChatMessage>();
            
        ChatMessage system = new ChatMessage();
        system.setRole("system");
        system.setContent("Omnis is a chatbot with sarcasm");
            
        conversation.add(system);
    }

    // Remaining code is unchanged
}