SerialSebTechEd: SeaDragon and PhotoSynth
This is a private session I ended up going to because of my craving for orange juice..
PhotoSynth is in Silverlight, but it's not clear which version it will be shipped with. It's a multi scale system, the size of objects is not important. The maximum info needed at any moment is whatever is on the screen. It supports pictures, text or even a fractal.
Another demo is an application of the scaling with an issue of the guardian, zooming out to show al pages and zoom in into a corner to see a very close text that you normally would need a microscope to see on a real printed animation.
This works with the hardware of the macine because the GPU is used. [EDIT: I thought silverlight was only software, or is there something more in the pieline?]
Another demo is a bunch of photos from flicker of Notre Dames, and generate a 3d reconstruction from that picture, and where all the cameras were positioned. You can then do a photo to photo navigation.
It's all about large navigations of photos and 3d, and SeaDragon became Photosynth. The system starts from images, extract features, then those feature get matched, and the matching turns into a 3d scene.
There's some work that ha been done in the summer with NASA. A picture of the shuttle display recrangles that are features: scale, orientation and corners.
Descriptors are informative about what the content is, but minimally informative about resolution, point of view, cropping, etc. If we extract the same features for a number of images, we manage to match them. All the features get in a feature index, look for nearest neighbors, and that gives matches between several pictures. Now that you know that two points on two pictures match and you do that across pictures, you can find out where the camera would have been and where the point would have been.Each point in the 3d visualization is one of these match features.
The system used to take several days, and we're getting ready to release in the spring community photosynth which will let a single user capture a 3d environment on his own. You start being able to capture a real part of the world. What it will allow is for users to make their own synths and share them, and link them together. When you link it you can navigate seamlessly from one synth to the next. We're going to see overlapping synths.
We've also been exploring public collections.
One of the secrets in photosynth is that it's almost 3d. That Synth was done yesterday after 5 mnutes of crawling. Some of these transitions are less than ideal ones. If you notice, the reason is because all these photos are projected on flat planes. It doesn't correspond to the geometry of that place. It's in between 2d and 3d. Something that a lot of peole have asked us to do real 3d reconstruction. There's been a flurry of research around it, and some of our colleagues have done is fabulous. There's an experimental viewer made oer the summer, with a project with the BBC. It's reconstructed 3d, we're not projecting on flat planes anymore we're projecting onto 3d models reconstructed from these pictures.
Another demo used in Live Labs is very close to the birds eye morphine in virtual earth. In virtual earth there are models of the city, photogrammetry, gps, etc. In this case we don't use any models, they come from the photos.There are still some errors but we're not finished with
We're working on integration with Virtual Earth because it relates to the interplay between top down and bottom up representations.