How to compute a realistic morphology between panoramic photos?

I had an idea today, but not sure how to implement it.

A 360 degree panoramic photo is a photo whch allows you to rotate and view in all directions as I'm sure you know. ie: http://www.vinod.ucoz.com/flexnavigator/Panoroma.swf

If you have two 360 degree panoramic photographs which were taken only 1 meter apart, one photo taken at point A and one at point B, then the two photos are very similar. Then, compute a morphology between the two photos and store this as a movie. If you play back the morphology slowly and view it in the 360 degree viewer, then it will look as if the camera moved from point A to point B.

If you take three 360 degree panoramic photographs, and each point A, B, C is in a triangle viewed from above then you can compute a morphology at any point between A, B and C. If you dynamically compute the morphology between photographs, then it is not necessary to store this as a movie. You can have a function which returns the photograph at any point between A, B and C.

If you take a LOT of 360 degree photos outdoors, then use this method, you can create a computer program to allow a person to move from one photo to the next and it should appear as if the viewer is moving smoothly and realistically between camrea points. As long as you know their position and have millions of photos.

The problem is how to compute a morphology between the photos that gives a realistic transformation between one photo and another. How could this be done? I looked at SURF for inspiration. http://www.youtube.com/watch?v=0Y4tAVO7Nno

Does anyone have any other suggestion how to compute a releastic morphology between the photos so that someone could walk through the panoramic photos?

This is not possible without some extremely intelligent software to approximate hidden geometry. For example what would the algorithm put between these two walls: