Copy blob from one storage account to another using Azure Function written in Java

Question

I am writing an Azure function in java to copy blobs between storage accounts within same tenant without first loading it into memory. I came across this post which suggested to use AzCopy to do that. I downloaded AzCopy binary from this location and added it to the function root directory before creating a docker image. Below is my code to invoke azcopy binary in java, this runs fine locally but does nothing when function deployed in Kubernetes. Not sure if it is the issue with binary or if it is the environment since function is deployed in kubernets not function app.

The source and destination are urls with temporary SAS tokens.

ProcessBuilder processBuilder = new ProcessBuilder(azCopyBinaryLocation, "copy", source, destination);
    Process process = processBuilder.start();

    InputStream is = process.getInputStream();
    InputStreamReader isr = new InputStreamReader(is);
    BufferedReader br = new BufferedReader(isr);
    String line;

    StringBuilder sb = new StringBuilder();

    while ((line = br.readLine()) != null) {
        sb.append("\n");
        sb.append(line);
    }

    context.getLogger().info("Logs: "+sb.toString()); //print logs in console

I am not sure if I am on right path like should I use azcopy or try to use java sdk instead, for SDK I am not sure if blob apis load the content into memory before copy to another storage account, if thats the case it would be slow and memory intensive which I want to avoid at all cost. I am looking for an option that uses azure backbone just like azcopy.

Answer 1

Issue was AZCOPY_LOG_LOCATION and AZCOPY_JOB_PLAN_LOCATION was missing from environment variables, setting the required variables through docker ENV value fixed it.

Final docker file looks like this

# Use a base image with the necessary dependencies
FROM mcr.microsoft.com/azure-functions/java:4-java17

# Install dependencies for azcopy
RUN apt-get update && apt-get install -y wget

# Download and install azcopy
RUN wget https://aka.ms/downloadazcopy-v10-linux -O azcopy.tar.gz && \
    tar -xvf azcopy.tar.gz && \
    cp ./azcopy_linux_amd64_*/azcopy /usr/bin/ && \
    chmod +x /usr/bin/azcopy && \
    rm -rf azcopy.tar.gz azcopy_linux_amd64_*

COPY ./target ./package

RUN mkdir -p /home/site/wwwroot && \
    cd /package/azure-functions/ && \
    cd $(ls -d */|head -n 1) && \
    cp -a . /home/site/wwwroot && \
    groupadd appuser -g 2000 && \
    useradd -r -M -s /sbin/nologin -g appuser -c appuser appuser -u 1000 && \
    mkdir -p /home/appuser/.local && \
    chown -R appuser:appuser /azure-functions-host && \
    chown -R appuser:appuser /home/appuser/.local

ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
    AzureFunctionsJobHost__Logging__Console__IsEnabled=true \
    ASPNETCORE_URLS=http://+:8080 \
    FUNCTIONS_WORKER_RUNTIME=java \
    JAVA_OPTS="--add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/jdk.internal.misc=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED"

RUN mkdir -p /home/site/wwwroot/log_files && \
    chown -R appuser:appuser /home/site/wwwroot/log_files && \
    chmod 777 /home/site/wwwroot/log_files

ENV AZCOPY_LOG_LOCATION=/home/site/wwwroot/log_files
ENV AZCOPY_JOB_PLAN_LOCATION=/home/site/wwwroot/log_files

# Switch to the user
USER appuser

# Expose the port
EXPOSE 8080

and java code

File azCopyPath = new File("/usr/bin/azcopy");

ProcessBuilder processBuilder = new ProcessBuilder(azCopyPath.getAbsolutePath(), "copy", sourceFileSasUrl, destinationFileSasUrl);
Process process = processBuilder.start();