Search code examples
javascripttensorflow2.0pose-estimation

TENSORFLOW.JS 3D Pose Estimation is not working

Ive done a lot of Pose Estimation in Python using OpenCV and Mediapipe a year ago. I recently read an the Tensorflow Blog(https://blog.tensorflow.org/2021/08/3d-pose-detection-with-mediapipe-blazepose-ghum-tfjs.html) about the possibility of 3d Pose estimation. This got me triggered, so i learned a bit of JavaScript to start my journey. After 3 days, i realised that my code doest work, Javascript gives out no errors, though nothing works. I followed the steps closely with a mate who has used JavaScript for a month now, Though he couldnt help me, cause he never worked with AI. I never worked with it, so i got no idea how await works in JS. MAy be a problem lol

    <body>
        <div id="app"></div>
        <video id="video" controls><source src="vid.mp4" type="video/mp4"/></video>
        <script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/pose-detection"></script>
        <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-core"></script>
        <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-converter"></script>
        <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-backend-webgl"></script>
        <script src="https://cdn.jsdelivr.net/npm/@mediapipe/pose"></script>
    <script>
        const model = poseDetection.SupportedModels.BlazePose;
        const detectorConfig = {
          runtime: 'tfjs',
          modelType: 'full'
        };
        const detector = await poseDetection.createDetector(model, detectorConfig);
        const video = document.getElementById('video');
        const poses = await detector.estimatePoses(video);
        document.getElementById("app").innerHTML = str(poses[0].keypoints3D);
    </script>
    </body>

My Goal is to "print" the array of detected poses to the screen so i can see it worked. Any ideas?

######################################################################### [EDIT]

    import * as poseDetection from '@tensorflow-models/pose-detection';
    import '@mediapipe/pose';
    
    async function estimatePosesOfVideo(videoelement) {
        const model = poseDetection.SupportedModels.BlazePose;
        const detectorConfig = {runtime:'mediapipe',modelType:'full'};
        const detector = await poseDetection.createDetector(model, detectorConfig);
        const poses = await detector.estimatePoses(videoelement);
        return poses
    }
    
    const videoelement = document.getElementById('video');
    const poses = estimatePosesOfVideo(videoelement);
    
    console.log(poses)

enter image description here

Ive got no idea what i can do about those.

Solution

  • I would recommend console.log for testing purposes. This will print what you entered to the Inspector (Crtl+Shift+I). Alternatively you could document.innerText to '''print''' something out.

    Response to edit

    Firstly, your asynchronous function estimatePosesOfVideo returns a Promise as do all async functions. When printing the result to the console you are instead printing the Promise object. I would advise to use the .then of the promise object instead.

    Regarding the other issues, it is difficult to give any meaningful reply as I lack information as to in which line your error occurs. Please add that information as a comment within the code fence or by another way.