how to scale and by which factor to scale dctmtx coefficients from float to get following integer values:
float dctmtx:
( (0.3536 0.3536 0.3536 0.3536 0.3536 0.3536 0.3536 0.3536),
(0.4904 0.4157 0.2778 0.0975 -0.0975 -0.2778 -0.4157 -0.4904),
(0.4619 0.1913 -0.1913 -0.4619 -0.4619 -0.1913 0.1913 0.4619),
(0.4157 -0.0975 -0.4904 -0.2778 0.2778 0.4904 0.0975 -0.4157),
(0.3536 -0.3536 -0.3536 0.3536 0.3536 -0.3536 -0.3536 0.3536),
(0.2778 -0.4904 0.0975 0.4157 -0.4157 -0.0975 0.4904 -0.2778),
(0.1913 -0.4619 0.4619 -0.1913 -0.1913 0.4619 -0.4619 0.1913),
(0.0975 -0.2778 0.4157 -0.4904 0.4904 -0.4157 0.2778 -0.0975)
)
integer dctmtx:
(( 125, 122, 115, 103, 88, 69, 47, 24 ),
( 125, 103, 47, -24, -88, -122, -115, -69 ),
( 125, 69, -47, -122, -88, 24, 115, 103 ),
( 125, 24, -115, -69, 88, 103, -47, -122 ),
( 125, -24, -115, 69, 88, -103, -47, 122 ),
( 125, -69, -47, 122, -88, -24, 115, -103 ),
( 125, -103, 47, 24, -88, 122, -115, 69 ),
( 125, -122, 115, -103, 88, -69, 47, -24 )
);
If you read up on the discrete cosine transform, you will find that the basic coefficient is
cos(pi*i*(2*j+1)/16), i,j=0..7
Then the first table consists of these values scaled by 0.5, except for the first row/column, which is scaled by 0.25*sqrt(2)=1/sqrt(8). Which is the correct way to obtain an orthogonal matrix. The square sum of the first column is 8, of the others 4.
The second table is the rounded results when multiplying the cosine values with 125, uniformly. Here one has to take care to properly rescale the vector when using the transpose matrix to compute the inverse transform.
First table reproduced, except for the first column:
> [[ Cos(pi*i*(2*j+1)/16)/2 : i in [0..7] ]: j in [0..7] ];
[
[ 0.5, 0.49039264, 0.46193977, 0.41573481, 0.35355339, 0.27778512, 0.19134172, 0.09754516 ],
[ 0.5, 0.41573481, 0.19134172, -0.09754516, -0.35355339, -0.49039264, -0.46193977, -0.27778512 ],
[ 0.5, 0.27778512, -0.19134172, -0.49039264, -0.35355339, 0.09754516, 0.46193977, 0.41573481 ],
[ 0.5, 0.09754516, -0.46193977, -0.27778512, 0.35355339, 0.41573481, -0.19134172, -0.49039264 ],
[ 0.5, -0.09754516, -0.46193977, 0.27778512, 0.35355339, -0.41573481, -0.19134172, 0.49039264 ],
[ 0.5, -0.27778512, -0.19134172, 0.49039264, -0.35355339, -0.09754516, 0.46193977, -0.41573481 ],
[ 0.5, -0.41573481, 0.19134172, 0.09754516, -0.35355339, 0.49039264, -0.46193977, 0.27778512 ],
[ 0.5, -0.49039264, 0.46193977, -0.41573481, 0.35355339, -0.27778512, 0.19134172, -0.09754516 ]
]
Second table, before integer rounding
> [[ Cos( pi*i*(2*j+1)/16 ) *125 : i in [0..7] ]: j in [0..7] ];
[
[ 125, 122.5982, 115.4849, 103.9337, 88.3883, 69.4463, 47.8354, 24.3863 ],
[ 125, 103.9337, 47.8354, -24.3863, -88.3883, -122.5982, -115.4849, -69.4463 ],
[ 125, 69.4463, -47.8354, -122.5982, -88.3883, 24.3863, 115.4849, 103.9337 ],
[ 125, 24.3863, -115.4849, -69.4463, 88.3883, 103.9337, -47.8354, -122.5982 ],
[ 125, -24.3863, -115.4849, 69.4463, 88.3883, -103.9337, -47.8354, 122.5982 ],
[ 125, -69.4463, -47.8354, 122.5982, -88.3883, -24.3863, 115.4849, -103.9337 ],
[ 125, -103.9337, 47.8354, 24.3863, -88.3883, 122.5982, -115.4849, 69.4463 ],
[ 125, -122.5982, 115.4849, -103.9337, 88.3883, -69.4463, 47.8354, -24.3863 ]
]