How can I make AR video to always face user's camera with marker vuforia native android

Question

I am working on marker AR with native android vuforia what I am trying to do is to move my video object according to camera (it should always face camera) I tried following but its not working but sometime its working but its not persistence

public void renderFrame(State state, float[] projectionMatrix) {

        mSampleAppRenderer.renderVideoBackground();
        GLES20.glEnable(GLES20.GL_DEPTH_TEST);

        isTracking = false;

        for (int tIdx = 0; tIdx < state.getNumTrackableResults(); tIdx++) {
            TrackableResult trackableResult = state.getTrackableResult(tIdx);
            ImageTarget imageTarget = (ImageTarget) trackableResult
                    .getTrackable();
            imageTarget.startExtendedTracking();

            isTracking = true;

            float[] modelViewMatrixVideo = Tool.convertPose2GLMatrix(
                    trackableResult.getPose()).getData();
            float[] modelViewProjectionVideo = new float[16];



            Matrix44F invTranspMV = SampleMath.Matrix44FTranspose(SampleMath.Matrix44FInverse(Tool.convertPose2GLMatrix(trackableResult.getPose())));


            Matrix.translateM(modelViewMatrixVideo, 0, 0f, 0f, 1f);
            Matrix.rotateM(modelViewMatrixVideo, 0, (float) Math.toDegrees(Math.asin(-invTranspMV.getData()[6])), 0.0f, 0.f, 1.0f);


            Matrix.multiplyMM(modelViewProjectionVideo, 0,
                    projectionMatrix, 0, modelViewMatrixVideo, 0);


            GLES20.glEnable(GLES20.GL_BLEND);
            GLES20.glBlendFunc(GLES20.GL_SRC_ALPHA, GLES20.GL_ONE_MINUS_SRC_ALPHA);
            GLES20.glUseProgram(videoPlaybackShaderID);


            GLES20.glVertexAttribPointer(videoPlaybackVertexHandle, 3,
                    GLES20.GL_FLOAT, false, 0, quadVertices);

            GLES20.glVertexAttribPointer(videoPlaybackTexCoordHandle,
                    2, GLES20.GL_FLOAT, false, 0,
                    fillBuffer(videoQuadTextureCoordsTransformedStones));

            GLES20.glEnableVertexAttribArray(videoPlaybackVertexHandle);
            GLES20.glEnableVertexAttribArray(videoPlaybackTexCoordHandle);

            GLES20.glActiveTexture(GLES20.GL_TEXTURE0);


            GLES20.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES,
                    videoPlaybackTextureID);
            GLES20.glUniformMatrix4fv(videoPlaybackMVPMatrixHandle, 1,
                    false, modelViewProjectionVideo, 0);

            GLES20.glDrawElements(GLES20.GL_TRIANGLES, NUM_QUAD_INDEX,
                    GLES20.GL_UNSIGNED_SHORT, quadIndices);

            GLES20.glDisableVertexAttribArray(videoPlaybackVertexHandle);
            GLES20.glDisableVertexAttribArray(videoPlaybackTexCoordHandle);

            GLES20.glUseProgram(0);
            GLES20.glDisable(GLES20.GL_BLEND);

            SampleUtils.checkGLError("VideoPlayback renderFrame");
        }

        GLES20.glDisable(GLES20.GL_DEPTH_TEST);

        Renderer.getInstance().end();


    }

I have tried above so far and its working sometime but not properly Please help me i am trying to do this from a week.

Answer 1

I am trying to do is to move my video object according to camera (it should always face camera)

If you want that a object faces the camera (Billboarding), then you have to use a model matrix which is the inverse view matrix, but without the translation part. Use Matrix44FInverse, to geht the inverse matrix of a Matrix44F:

public void renderFrame(State state, float[] projectionMatrix) {

    .....

    // get the view matrix and set translation part to (0, 0, 0)
    float[] tempViewMat = Tool.convertPose2GLMatrix(trackableResult.getPose()).getData();
    tempViewMat[12] = 0;
    tempViewMat[13] = 0;
    tempViewMat[14] = 0;

    // create the billboard matrix 
    Matrix44F billboardMatrix = new Matrix44F();
    billboardMatrix.setData(tempViewMat);
    billboardMatrix = SampleMath.Matrix44FInverse(billboardMatrix);

    // calculate the model view projection matrix

    float[] viewMatrixVideo = Tool.convertPose2GLMatrix(trackableResult.getPose()).getData();

    float[] modelViewVideo = new float[16];
    Matrix.multiplyMM(modelViewVideo, 0, viewMatrixVideo, 0, billboardMatrix.getData(), 0);

    float[] modelViewProjectionVideo = new float[16];
    Matrix.multiplyMM(modelViewProjectionVideo, 0, projectionMatrix, 0, modelViewVideo, 0);

    .....
}

---

I used your code but my AR is rotating in every direction but i want some different thing I have one video that I want to place it vertical on marker and if user move left or right only then i want to rotate it to face camera so that i will see my AR video same that is vertical from any view

What you want to do is to fix the upwards direction, but to orientate the normal vector to the line of sight.

The line of sight is the inverse Z-axis of the view space in a Right-Handed Coordinate System.

Matrix44F inverse_view = SampleMath.Matrix44FInverse(
    Tool.convertPose2GLMatrix(trackableResult.getPose()));

// line of sight  
Vec3F los = new Vec3F(-inverse_view[8], -inverse_view[9], -inverse_view[10] );

Eiterh the Y-Axis (0, 1, 0) or the Z-Axis (0, 1, 0) of the model has to be the Z-axis of the orientation matrix. The X-Axis is the cross product (Vec3FCross) the line of sight and the Z-axis of the orientation matrix.

e.g.

Vec3F z_axis = new Vec3F(0, 0, 1);
Vec3F x_axis = Vec3FNormalize(Vec3FCross(los, z_axis));
Vec3F y_axis = Vec3FCross(z_axis, x_axis);

float[] orientationMatrix = new float[]{
    x_axis.getData()[0], x_axis.getData()[1], x_axis.getData()[2], 0,
    y_axis.getData()[0], y_axis.getData()[1], y_axis.getData()[2], 0,
    z_axis.getData()[0], z_axis.getData()[1], z_axis.getData()[2], 0,
    0,                   0,                   0,                   1
};

// calculate the model view projection matrix

float[] viewMatrixVideo = Tool.convertPose2GLMatrix(trackableResult.getPose()).getData();

float[] modelViewVideo = new float[16];
Matrix.multiplyMM(modelViewVideo, 0, viewMatrixVideo, 0, orientationMatrix, 0);

float[] modelViewProjectionVideo = new float[16];
Matrix.multiplyMM(modelViewProjectionVideo, 0, projectionMatrix, 0, modelViewVideo, 0);