I'm rendering a huge 3D volume using WebGL and the ray casting algorithm is implemented in the shader(glsl). The volume is created from bio-image stacks. What I want to do is to keep zooming in and out smoothly when rendering this 3D volume. But the images stacks are high resolution, to get a real-time performance I need to use Octree. Do you have any suggestions how I can implement it?
I don't think you'll get an Octree on GPU with just WebGL - it supports just OpenGL ES 2.0 so it may be too little to implement some decent Octree.
However, there're some tricks that may help with your problem. In general you want to speed up the rendering during zooming/movement but keep your target quality when possible. You may implement some 'low def' quality and use it during camera movement. The simplest would be to just use a bigger ray casting step, you may also prepare some lower resolution versions of your images.
Then, when you see that user starts to move/zoom the view you can turn your 'low def' rendering on to get more fps. During this movement the user won't notice much loss in quality on moving image and when he stops changing the camera you render the scene once more - but in your target quality.
Let's say your typical rendering takes 1 second and the 'low def' takes 0.1 second. With this approach you can get interactive camera movement at ~10fps and when you stop moving, then you get the final image just after 1 second.
I know it's not ideal but may serve you quite well, I saw few bigger gfx applications use that trick.
Some slightly different but similar approach is to render one of every X pixels during interactive movement. In some 3D modelling apps, when you change the view/move some object and have a raytraced preview enabled you may notice it renders just a part of all pixels, maintaining interactivity. When you stop the movement it renders the rest of the pixels.