RTreeDistWithin( x Double precision , y Double precision , radius Double precision )
This function retrieves from an R*Tree Spatial Index any entity whose MBR intersect the square circumscribed on the given circle. Is there any analog for 3 and more dimensional data?
There is no three or n-dimensional implementation of any R*Tree related function in SpatiaLite. According to the documentation:
(...) an R*Tree defines a tree-like structure based on rectangles (the R in R*Tree stands exactly for Rectangle).
In general, spatial databases and spatial indices are designed to manage spatial data (understanding spatial data as geographical data), so the interesting coordinates are X and Y (or longitude/latitude). Z (or altitude) is much less important and is not taken into account when working with spatial indices.
So, in short, Spatialite is not designed to work with n-dimensional data and you can't define n-dimensional (n > 2) R*Tree spatial indices in SpatiaLite. You may need to think what are you trying to do with SpatiaLite because maybe this is not the right tool to achieve your goals.
As a workaround, if you still want to work with spatial databases to store your data, you can think of your n-dimensional data in terms of it's projection in 2-dimensional planes. Each projection will be stored in a table with a spatial index (you will need n-1 projections to manage n-dimensional data).