Post on 10-Feb-2017
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
Department of Informatics | Technische Universität München
FachgebietAugmented Reality
Panoramic Image Mosaics
Dimitar Dimitrov
Heung-Yeung Shum and Richard SzeliskiMicrosoft Research
15.01.2007
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics> Motivation
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics> Motivation
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics> Motivation
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Motion Parallax
Image blending
Image warping
Image registration
Jacobian, Hessian
Planar Homography
Image intensity gradient
Terms that should be known:
Panoramic Image Mosaics
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Motion parallax: The motion parallax is the effect of
closer objects moving by faster than more distant ones
“apparent motion” of an object against a distant background because of a perspective shift.
change of angular position of two stationary points relative to each other as seen by an observer, due to the motion of an observer
The key to depth perception
Motion Parallax Image warping Image blending Image resampling Jacobian, Hessian Planar Homography Image intensity
gradient
Perspective meaning of motion parallax
Principle of depth perception
> Terms that should be known
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Distorting the image through some kind of filter.
Distorted city scene Warped versions of a face image Deforming a mesh. Assigned
to image this gives the warp
> Terms that should be known
Image warping: Motion Parallax Image warping Image blending Image resampling Jacobian, Hessian Planar Homography Image intensity
gradient
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Image blending is the process of calculation intensities of two pixels of overlapped images with some function
> Terms that should be known
Image blending: Motion Parallax Image warping Image blending Image resampling Jacobian, Hessian Planar Homography Image intensity
gradient
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
In graphics software, the resample command is used to increase or decrease the size and/or resolution of a bitmap-based image
Resampling an image usually results in a loss of image quality because pixels must either be interpolated or thown out
Resampling an image when rotated.
> Terms that should be known
Image resampling: Motion Parallax Image warping Image blending Image resampling Jacobian, Hessian Planar Homography Image intensity
gradient
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Jacobian matrix – the derivative of multivariate function. The Jacobian matrix is the matrix of all first-order partial derivatives of a vector-valued function. Its importance lies in the
fact that it represents the best linear approximation to a differentiable function near a given point.
Hessian matrix – the matrix of f is the matrix of second partial derivatives evaluated at x.
> Terms that should be known
Jacobian, Hessian: Motion Parallax Image warping Image blending Image resampling Jacobian, Hessian Planar Homography Image intensity
gradient
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Matematically homography is defined as a relation between two figures, such that to any point in one figure corresponds one and only one point in the other, and vise versa.
In the field of Computer Vision, a homography is defined in 2 dimensional space as a mapping between a point on a ground plane from as seen from one camera, to the same point on the ground plane as seen from a second camera.
Different transformations in image space
Basic homography
matrix.Homogeneous 2d coordinates
Planar homography:> Terms that should be known
Motion Parallax Image warping Image blending Image resampling Jacobian, Hessian Planar
Homography Image intensity
gradient
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
The intensity gradient of image is
Original image X-GradientY-Gradient
Image gradient:> Terms that should be known
Motion Parallax Image warping Image blending Image resampling Jacobian, Hessian Planar Homography Image intensity
gradient
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Image registration is the process of finding a homography between two images (reference and target) of a scene and spatially align them.
> Terms that should be known > Image registration process
What is image registration:
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Building point correspondences through extracting feature points from which a homography is estimated
Iteratively comparing intensity pattern of the images, search of minimum of the intensity error, and so finding the homography.
Once a homography is estimated warping and resampling of the target image is made. Bilinear pixel resampling is used in this paper.
Image with extracted feature points used for correspondence
Edges and lines detected with Moravec corner detector
> Terms that should be known > Image registration process
How is the registration performed:
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Measuring similarity between images by means of intensity difference between all overlapping pixels of the two images
Defining a function which gives the amount of error between the compared images under the current transformation
Minimizing this error function – finding at which transformation the images have minimum differences in intensity. Minimizing the error function by taking it’s derivative to null with any gradient descent algorithm.
Problem - Algorithms for searching minimum of a function often find local minimums, not global.
Principle of intensity-based approach to image registration:> Terms that should be known > Image registration process
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
1 1 3
2 2 4
Registering two identical images by iterative intensity error minimization
Searching the best alignment of the patch to the image by intensity error minimization
> Terms that should be known > Image registration process
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Local minima problem by intensity-based image registration
> Terms that should be known > Image registration process
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Constructing Cylindrical or spherical panorama: Images from camera on a fixed tripod with
fixed focal length and field of view.
World transformation to cylindrical or spherical coordinates.
Translation of the images with each’s panning angle, recovered through estimation of intensity difference between the images.
Limitations: Tilt angle problems. Tilting towards north
or south pole, requires knowing the focal length
Estimating the focal length can be quite. Inaccurate.
> Cylindrical and spherical panoramas – example
Cylindrical and spherical panorama
Warping an image into cylindrical
coordinates
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Warping the images into cylindrical (spherical) coordinates: Converting plane coordinates into cylindrical
Converting plane coordinates into spherical
panning angle
scanline
> Cylindrical and spherical panoramas – example
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
The error function estimates the squared error over each pixel i between the images for a given incremental δt
Linearizing the error function about I1
> Cylindrical and spherical panoramas – example
Defining the intensity minimization function:
Coordinate vector
Coordinate vector with shift
The incremental value
Intensity error
Image gradient
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Solving the linear least-squares for the linearized equation about the incremental δt
When solved about δt we can now update our initial estimate t with t = t + δt
Blending images with simple feathering algorithm
> Cylindrical and spherical panoramas – example
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Replace cylindrical and spherical coordinates with rotational matrices (and focal length) for each image. Performing the registration in the input image’s coordinate system.
develop a global optimization technique to find the optimal overall registration. compute local motion estimates (block-based optical flow) between pairs of
overlapping images, and use these estimates to warp each input image so as to reduce the ghosting.
Create a panorama without using of special hardware, and remove the need for pure panning motion with no motion parallax.
Paint a full view panoramic mosaic with a simple hand-held camera or camcorder. accumulated misregistration errors, which are always present in any large image
mosaic. any deviations from the pure parallax-free motion model may result in local
misregistrations, which are visible as a loss of detail or multiple images (ghosting).
Problems to solve:
Idea:
> Schum – Szelski Algorithm
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Creating small mosaic using planar projective motion model and then calculating the rough focal length from it.
Sequentially assemble complete panoramic image using rotational motion model and patch-based alignment
Block adjustment (global alignment) modifies each image’s transformand focal length to minimize global error. Removes large inconsistensies, gaps
Deghosting (local alignment) reduces any local missregistration errors
Environment mapping. The mosaic is stored like a collection of images with their associated parameters or mapped over a polyhedron
Processing flow:
> Schum – Szelski Algorithm
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Representing the mosaic as collection of images with associated geometrical transformations
Hierarchical motion estimation framework Pyramid constructor Motion estimation Image warping Coarse-to-fine refinement
Sequential mosaic construction. Finding the best alignment of image with
the mosaic, constructed from all the previous images. Thus estimating only incremental motion.
Warp/register/update loop
Image warping Using parametric motion model for
calculating warped image coordinates
> Schum – Szelski Algorithm
Alignment Framework:
Image pyramid used for coarse-to-finerefinement
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Given two images from the same viewpoint containing a projection of a 3d point in space a planar perspective transformation exist between them
Warping image into another using planar perspective motion model
> 8–parameter motion model
8-parameter perspective transformations (homographies):
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Warping image into another using planar perspective motion model
> 8–parameter motion model
Rewriting the matrix equation into function from x and y
Homography equationwith homogenous coordinates
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Iterative equation
> 8–parameter motion model
Rewriting the coordinates function according the iteration equasion
Where D is the deformation matrix
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Intensity error function with the iterative equation integrated.
> 8–parameter motion model
First order taylor expansion over bivariate function
Intensity error between the images under current transformation
The Jacobian matrix
Image gradient at xi
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Basic least-squares problem through normal equations
Basic linear least-squares equation
The hessian from image gradients
Gradient-weightend intensity error
Solving the least-squares problem for the intensity error function with normal equations
> 8–parameter motion model
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Limitations: Has more free parameters than
necessary thus slow convergence
Gets easily stuck in local minima when the initial misregistration is big
> 8–parameter motion model
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics> 3-d rotations and zooms
From the fact the camera and the scene are in common coordinate system follows that the movement of the camera between consequtive images could be described by translation scaling and rotation.
Given a focal length and assuming no translational motion, one must estimate only 3 more parameters
3-d rotations and zooms:
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics> 3-d rotations and zooms
Point in 3-d space
Translation, Scale and Rotation matrices
Recovering 3-d direction from 2-d projection coordinates
Point in 2-d space Homogeneous coordinates
Perspective transformation exists described by
Rotational matrix equation
Inverse equation.Assuming no translation
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Perspective projection between two images
Rewriting the equation as parameter based. The Parameter is the angular velocities vector
> 3-d rotations and zooms
The parameter matrix presented in rotational model
Rodriguez formula for constructing rotation matrix from tree angles
Angular velocities vector. The unknown in the equation.
Cross product operator for the rotation matrix with the tree angles
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Building the iterative parameter estimation equation
> 3-d rotations and zooms
Equivalent to the 8-parameter method iterative equation
Defining the Jacobian for the rotational model parameter estimation algorithm
The deformation matrix for the rotational model
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Significantly faster convergence caused by only 3 degrees of freedom of the parameter matrix
The same problems as by 8-parameter motion estimation exist, stucking in local minima
> 3-d rotations and zooms
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics> Estimating focal length
The correspondence between the two matrices
From the orthonormality of the rotation matrix
Orthogonal: all row (column) vectors have the same norm
Normal: all row (column) vector have dot product equal to null
Estimating f0 (analog f1)
Estimating the focal length from the initial mosaic:
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Always present due to errors
Closing by registering the first image at the end – the quotient of the parameter matrices gives the error coefiticent
Converting the error into angle and update the focal length
Registering the sequence with the new focal length
> Estimating focal length
Closing the gap in the panorama:
Example for gap in a panorama by registering the first image at the end
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Computational effort for taking a single gradient step consisist of wapring (resampling) the target image, computing the local intensity errors and gradients and accumulation of A and b. Very computationaly expensive Such precisenes is not requried
Approximation Divide the image into little patches (8 x 8)
Approximate the Jacobian Assume that the Jacobian is constant for each patch Evaluate it only once at patch center Calculate A and b with this approximation
Image divided into patches
Patch-based Alignment:
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Block matching technique Search the minimum intensity
difference between target patch and reference image in a given neighbourhood
Framework Divide the image into patches Estimate the best integral shift
for each patch which will best align the two image
Calculate a displacement and use it to calculate new patch hessian and gradient matrices
Limitations Very computationaly expensive Apply only on the coarsest level
of the refinement image pyramid
Searching for the best integral shift by matching patch with a neighborhood
> Patch-based alignment
Local Search:
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Motivation Big mosaics show significant
amount of error due to unaviodable rotational parameters focal length estimation errors.
The accumulated error causes visible image gaps and artifacts.
Bundle block adjustment technique Optimizes globally the
transformation parameters.
Panoramic Image Mosaics
Block adjustment is a feature-based technique. Establishing point
correspondences Usind local seach algorithm for
correcting the accumulated error
Bundle block adjustment of a feature point over tree
overlapping images
Block Adjustment (Global Alignment):
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics> Block Adjustment > Point correspondences
Dividing the image into patches (16 x 16)
Filter the patches Only patches which have interesting gradient
map Corners, Edges and Lines All patches with uniform texturing are
excluded
Use the patches as feature points with Local Search algorithm to build the corespondences
Point correspondences by rotation
Establishing the point correspondences:
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics> Block Adjustment > Framework
If no error in parameter estimation the points of the l corresponds to the point in k given by
Problem – not the case in practice
Instead run Local Search over two overlapping images under the current transformations
Local search result ujl is added to the patch center xjk to find the real position within the reference image
This equation rarely maps the points correctly in practice
Convertion into l’s coordinate system gives the correct position of the patch
Converting from k into l coordinate system
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics> Block Adjustment > Framework
Estimating the homography matrix as to minimize the difference in screen coordinates for all the images
Problem – the gradients with respect to motion parameters are complicated
Solution – Minimizing distance between ray directions of corresponding points usint rotational representation Minimizing the distance to the center
Correct the Mk as to minimize the distance
Converting from k into l coordinate system
The perfect match is in the centre so the bundle of rays must converge to that point
Minimizing the distance between every two images
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics> Block Adjustment
Calculating the bundle ray direction Used in the deghosting algorithm
Direction of 3d bundle ray from 2d point in image l
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Motivation Addresses problems, which cannot be solved
with the global alignment, due to deviations from the perfect rotational (projective) model used.
Radial distortion Moving objects Motion parallax
Framework Computing the optical flow over the whole
mosaic to find local missregistrations.
Using inverse mapping technique to localy warp images as to neutralize the optical feld
Panoramic Image Mosaics> Deghosting
Calculates the projection of the bundle ray vector into image kThat represents the perfect position of the feature point
Defines a motion vector between the perfect and the current position
Deghosting (Local Alignment):
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics> Deghosting > Optical Field examples
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
The algorithms presented in this paper provide a efficient and reliable way of constructing full-view panoramic image mosaics without the need of special hardware equipment.
Conclution:
CAMP-AR | Department of Informatics | Technische Universität München |
Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de
FachgebietAugmented Reality
Panoramic Image Mosaics
Shum and Szelski – Panoramic Image Mosaics Kristin Branson – Feature-based Mosaic
Construction Daniel Sylora – Image registration I Alper Yilmaz – CAP5415 Computer Vision
References: