I have a huge 3D (x,y,z) data set with (x,y) being the inputs and (z) is the output. Now the dataset is very large, and I need to use that information in real time with minimal delay.
Therefore, indexing/look-up table might seem slow. So my thought is to interpolate the dataset and in real time, instead of look-up table, I calcualte the value. So I don't have to store the original dataset but instead I can store the coefficients, which hopefully would be of smaller size than the original data set.
I used the scipy.interpolate.RectBivariateSpline
to perform interpolation. And I was able to fit the data and also obtain coefficients. But I am not sure how to reconstruct the interpolation function from the coefficients.
I want to emphesize that the interpolation function will only be evaluated at input (x,y). Generalization is not of concern here.
from scipy import interpolate
import numpy as np
x = np.arange(1,500)
y = np.arange(2,200)
X,Y = np.meshgrid(x,y)
z = np.sin(X+Y).T
a = interpolate.RectBivariateSpline(x,y,z)
# print(len(a.get_coeffs()))
# coefficients can be obtained by a.get_coeffs()
# I want to have the following
# f = construct_spline_from_coefficient(a.get_coeffs())
# z = f(x_old, y_old)
Another approach I had in mind is use deep neural network. Can anyone shed some light here? Is this an over-kill?
The solution is in the scipy official doc (link).
bisplrep
function (rep stands for representation) to obtain the interpoltaion output tck
(see the docstring for bisplrep
).tck
is an array and can be stored in a file.bisplev
(ev stands for evaluation) to constrcut an function.For using nueral network at interpolation see this state-of-the-art (paper) Training Neural Networks for and by Interpolation.