I'm tracking an object moving within a fixed-size box using a standard 2D CV (constant velocity) dynamic model as it moves through free space in the center of the box. But when it hits the edges of the box, its movement is more complex. Billiard-ball mechanics would work as a first approximation, but that model seems quite different than the CV model and I'm not sure how to incorporate it into the update step of the Kalman filter.
Is bounded motion (object in a box) a good use case for Interacting Multiple Model (IMM) filters, combining both the CV and billiard models? If so, will such a filter predict future collisions if I ask it to predict several tens of time steps into the future? Or is there a way to alter the CV filter state variables when a collision occurs such that subsequent update and predict steps continue to track the motion of the object (albeit in its new, reflected direction)?
A Kalman filter is only optimal when operating on linear systems, but almost no systems are linear, and almost all Kalman Filters are used on non-linear systems via some approximation like the EKF (Extended Kalman Filter) or UCF (Unscented Kalman Filter).
So your intuition is good, but your specific case has a problem: When using a KF with a non-linear system, there is an assumption that over a very short period of time (the time between updates) you can approximate your system as a bunch of straight lines.
The EKF does this with the Jacobian. When you replace F (the linear matrix) with f() the function (which takes into account your bounce), you easily fix your update step. But the real heart of the KF is the error propagation. To make that work, the EKF uses the Jacobian to find the slope of the line tangent to your nonlinear function f() at your estimated point. So it's okay if your f() curves all around, because at the update step we zoom waaay in and approximate it as a straight line (sort of like drawing a circle with many straight line segments). The catch in your case is that f() has to be continuously differentiable function. When you zoom in on the point of the bounce there's an inflection point where the ball reverses direction. No matter how short your timestep, no matter how close you come to the point of the bounce, any estimate of your motion as a straight line either continues into the bumper or "bounces" early by moving away.
How will this fail? When you estimate your measurements (via Hx or h(x)) and compare with the real measurements z, you will find your object has a slightly wrong position. Let's say the object is farther from the bumper than expected. Did it hit the bumper too soon? Or is it behind, and hitting too late? The filter doesn't know, it just assumes based on which linear approximation you chose for F above. If your linearization point is before the bounce then the filter will move the object towards the bumper and increase its estimated speed. If the linearization is after (and the velocity term's contribution to position has flipped) it will move the ball closer and decrease the estimated speed.