Storing motion vectors from calculated optical flow in a practical way which enables reconstruction of subsequent frames from initial keyframes

Question

I am trying to store the motion detected from optical flow for frames in a video sequence and then use these stored motion vectors in order to predict the already known frames using just the first frame as a reference. I am currently using two processing sketches - the first sketch draws a motion vector for every pixel grid (each of width and height 10 pixels). This is done for every frame in the video sequence. The vector is only drawn in a grid if there is sufficient motion detected. The second sketch aims to reconstruct the video frames crudely from just the initial frame of the video sequence combined with information about the motion vectors got from the first sketch.

My approach so far is as follows: I am able to determine the size, position and direction of each motion vector drawn in the first sketch from four variables. By creating four arrays (two for the motion vector's x and y coordinate and another two for its length in the x and y direction), every time a motion vector is drawn I can append each of the four variables to the arrays mentioned above. This is done for each pixel grid throughout an entire frame where the vector is drawn and for each frame in the sequence - via for loops. Once the arrays are full, I can then save them to a text file as a list of strings. I then load these strings from the text file into the second sketch, along with the first frame of the video sequence. I load the strings into variables within a while loop in the draw function and convert them back into floats. I increment a variable by one each time the draw function is called - this moves on to the next frame (I used a specific number as a separator in my text-files which appears at the end of every frame - the loop searches for this number and then increments the variable by one, thus breaking the while loop and the draw function is called again for the subsequent frame). For each frame, I can draw 10 by 10 pixel boxes and move then by the parameters got from the text files in the first sketch. My problem is simply this: How do I draw the motion of a particular frame without letting what I've have blitted to the screen in the previous frame affect what will be drawn for the next frame. My only way of getting my 10 by 10 pixel box is by using the get() function which gets pixels that are already drawn to the screen.

Apologies for the length and complexity of my question. Any tips would be very much appreciated! I will add the code for the second sketch. I can also add the first sketch if required, but it's rather long and a lot of it is not my own. Here is the second sketch:

      import processing.video.*;

      Movie video;

      PImage [] naturalMovie = new PImage [0];
      String xlengths [];
      String ylengths [];
      String xpositions [];
      String ypositions [];
      int a = 0;
      int c = 0;
      int d = 0;
      int p;
      int gs = 10;

      void setup(){

        size(640, 480, JAVA2D); 

       xlengths = loadStrings("xlengths.txt");
       ylengths = loadStrings("ylengths.txt");
       xpositions = loadStrings("xpositions.txt");
       ypositions = loadStrings("ypositions.txt");

        video = new Movie(this, "sample1.mov");
        video.play(); 
        rectMode(CENTER);
      }

      void movieEvent(Movie m) {
        m.read();
        PImage f = createImage(m.width, m.height, ARGB);
       f.set(0, 0, m);
       f.resize(width, height);
          naturalMovie = (PImage []) append(naturalMovie, f); 
          println("naturalMovie length: " + naturalMovie.length);
        p = naturalMovie.length - 1;
      }

      void draw() {
        if(naturalMovie.length >= p && p > 0){

      if (c == 0){
          image(naturalMovie[0], 0, 0);
      }
        d = c;
      while (c == d && c < xlengths.length){
        float u, v, x0, y0;
        u = float(xlengths[a]);
        v = float(ylengths[a]);
        x0 = float(xpositions[a]);
        y0 = float(ypositions[a]);

        if (u != 1.0E-19){
        //stroke(255,255,255);
        //line(x0,y0,x0+u,y0+v);
        PImage box;
        box = get(int(x0-gs/2), int(y0 - gs/2), gs, gs);
        image(box, x0-gs/2 +u, y0 - gs/2 +v, gs, gs);

        if (a < xlengths.length - 1){
        a += 1;
        }
        }

        else if (u == 1.0E-19){

          if (a < xlengths.length - 1){
          c += 1;
          a += 1;
          }
        }

      }
                }
              }

Answer 1

Word to the wise: most people aren't going to read that wall of text. Try to "dumb down" your posts so they get to the details right away, without any extra information. You'll also be better off if you post an MCVE instead of only giving us half your code. Note that this does not mean posting your entire project. Instead, start over with a blank sketch and only create the most basic code required to show the problem. Don't include any of your movie logic, and hardcode as much as possible. We should be able to copy and paste your code onto our own machines to run it and see the problem.

All of that being said, I think I understand what you're asking.

How do I draw the motion of a particular frame without letting what I've have blitted to the screen in the previous frame affect what will be drawn for the next frame. My only way of getting my 10 by 10 pixel box is by using the get() function which gets pixels that are already drawn to the screen.

Separate your program into a view and a model. Right now you're using the screen (the view) to store all of your information, which is going to cause you headaches. Instead, store the state of your program into a set of variables (the model). For you, this might just be a bunch of PVector instances.

Let's say I have an ArrayList<PVector> that holds the current position of all of my vectors:

ArrayList<PVector> currentPositions = new ArrayList<PVector>();

void setup() {
  size(500, 500);
  for (int i = 0; i < 100; i++) {
    currentPositions.add(new PVector(random(width), random(height)));
  }
}

void draw(){
 background(0);
 for(PVector vector : currentPositions){
   ellipse(vector.x, vector.y, 10, 10);
 }
}

Notice that I'm just hardcoding their positions to be random. This is what your MCVE should do as well. And then in the draw() function, I'm simply drawing each vector. This is like drawing a single frame for you.

Now that we have that, we can create a nextFrame() function that moves the vectors based on the ArrayList (our model) and not what's drawn on the screen!

void nextFrame(){
  for(PVector vector : currentPositions){
    vector.x += random(-2, 2);
    vector.y += random(-2, 2);
  }
}

Again, I'm just hardcoding a random movement, but you would be reading these from your file. Then we just call the nextFrame() function as the last line in the draw() function:

If you're still having trouble, I highly recommend posting an MCVE similar to mine and posting a new question. Good luck.