Search code examples
node.jsartificial-intelligenceserverlessvercelchatgpt-api

How can I stream in a Vercel Serverless function? It's working in local, but not once deployed

I'm testing some ChatGPT functionalities, and found out how to stream the responses as if they were typed in real-time.

While I'm able to reproduce this correctly in my local, for some reason when I deploy this, the response only shows once the full message is loaded.

Vercel documentation states this:

Vercel supports streaming for Serverless Functions, Edge Functions, and React Server Components in Next.js projects.

I've only been able to find documentation about how to do it to stream for Edge Functions, not for Serverless.

Here you can compare how it works for local, but not for deploy: enter image description here

This is the repo: https://github.com/andna/errorsrepo with this being the specific handler:

const { Configuration, OpenAIApi } = require("openai");

const configuration = new Configuration({
    apiKey: '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',
    //this key was deleted after video recording
    //replace with your own free API KEY obtained here: https://platform.openai.com/account/api-keys
});
const openai = new OpenAIApi(configuration);

export default async function handler(req, res) {

    try {
        const completion = await openai.createChatCompletion({
            model: "gpt-3.5-turbo",
            stream: true,
            messages: [
                { role: "system", content: "You are an AI." },
                { role: "user", content: "how are you?" }],
        }, { responseType: 'stream' });

        completion.data.on('data', data => {
            const lines = data.toString().split('\n').filter(line => line.trim() !== '');

            res.write("<!DOCTYPE html><html><body>");

            for (const line of lines) {
                const message = line.replace(/^data: /, '');
                if (message === '[DONE]') {
                    res.end();
                    return; // Stream finished
                }
                try {
                    const parsed = JSON.parse(message);
                    const content = parsed.choices[0].delta.content;
                    if(content){
                        res.write(content);
                    }
                    ;
                } catch(error) {
                    console.error('Could not JSON parse stream message', message, error);
                }
            }


        });
    } catch (error) {
        console.error('An error occurred during OpenAI request', error);
    }



}

Any help how to make it work in prod too?

Solution

  • As noted by @AcclaimHosting, there are many differences.

    Ended up following this example with Edge Functions instead of serverless: https://vercel.com/blog/gpt-3-app-next-js-vercel-edge-functions#the-frontend