I can't see Videos in Processing, I only hear the audio from it

Question

I'm trying to play one of two videos based on sensor from Arduino (for a University project)

The issue is that I can't see any of the videos in Processing but I only hear the audio.

I rendered the videos using the H264 codec and they are in Full HD. I am using Processing 4.1.1.

This is my attempt at playing back video based on sensor values:

import processing.serial.*;
import processing.video.*;

Movie movie1, movie2;
boolean playing = false;
char previousIn;




Serial mySerial;

String myString = null;
int nl = 10;
float myVal;

void setup() {

  size(1000, 1000);
  frameRate(25);



  String myPort =Serial.list()[2];
  mySerial = new Serial(this, myPort, 9600);
  movie1 = new Movie(this, "Final_Clips_HD.mp4");
  movie2 = new Movie(this, "Final_Clips_HD_2.mp4");
}




void movieEvent(Movie movie) {
  movie.read();
}

void draw() {


  while (mySerial.available() > 0) {
    myString = mySerial.readStringUntil(nl);

    if (myString != null) {

      myVal = float(myString);
      println(myVal);




      if (myVal > 10 && myVal < 20) {
        movie1.play();
        movie2.stop();
        myVal = previousIn;
        if (myVal == previousIn) {
          movie1.jump(0);
        }
      }
      image(movie1, 0, 0, width, height);


      //fill(0,255,0);
      if (myVal < 10 && myVal > 0) {
        movie2.play();
        movie1.stop();
        myVal = previousIn;
        if (myVal == previousIn) {
          movie2.jump(0);
        }
      }
      image(movie2, 0, 0, width, height);
    }
  }
}

Answer 1

You've got all the "ingredients" right, you might just need to shuffle them a bit. Well done on using H264 with Processing (I've seen previous errors with other video codecs) (and checking for null, etc.)

Currenty the movies are played only if there is a serial connection, a formatted string (sensor value) is sent and that value is within a range.

I would re-organise the code slightly so it can display a movie frame anytime (whatever that movie may be), then simply change the reference to the movie which is currently updated (based on the float value from serial).

(Additionally I'd also check for errors just in case the arduino isn't connected, is busy open in Serial Monitor, etc. Another minor tip is that yuo can use bufferUntil() once in setup and after that, just readString() from serialEvent()).

Here's a revised version of your code (with comments):

import processing.serial.*;
import processing.video.*;

Movie movie1, movie2;
Movie currentMovie;
boolean playing = false;

Serial mySerial;

// sensor values (current and previous)
float myVal;
float previousIn;

void setup() {

  size(1000, 1000);
  frameRate(25);

  String myPort = Serial.list()[2];
  // try to open the serial connection, but print error message if it fails
  try{
    mySerial = new Serial(this, myPort, 9600);
    // tell arduino once to buffer until new line character is found
    mySerial.bufferUntil('\n');
  }catch(Exception e){
    println("error opening serial connection to " + myPort);
    e.printStackTrace();
  }

  // load movies
  movie1 = new Movie(this, "Final_Clips_HD.mp4");
  movie2 = new Movie(this, "Final_Clips_HD_2.mp4");
  // reference one of the movies for updating
  currentMovie = movie1;
}




void movieEvent(Movie movie) {
  movie.read();
}

void draw() {
  // render the current movie
  image(currentMovie, 0, 0);
}

void serialEvent(Serial mySerial){
  String myString = mySerial.readString();
  if (myString != null) {
    myVal = float(myString);
    println(myVal);
    
    if (myVal > 10 && myVal < 20) {
      movie1.play();
      movie2.stop();
      myVal = previousIn;
      if (myVal == previousIn) {
        movie1.jump(0);
      }
      // update reference to the currently playing movie
      currentMovie = movie1;
    }
    
    if (myVal < 10 && myVal > 0) {
      movie2.play();
      movie1.stop();
      myVal = previousIn;
      if (myVal == previousIn) {
        movie2.jump(0);
      }
      // update reference to the currently playing movie
      currentMovie = movie2;
    }
    
  }
}

Note that the above code isn't tested, so may not be 100% accurate: hopefully it illustrates the logic.