Panoramic image mosaics

Chair for Computer Aided Medical Procedures & Augmented Reality | campar.in.tum.de

Department of Informatics | Technische Universität München

FachgebietAugmented Reality

Panoramic Image Mosaics

Dimitar Dimitrov

Heung-Yeung Shum and Richard SzeliskiMicrosoft Research

15.01.2007

CAMP-AR | Department of Informatics | Technische Universität München |



Panoramic Image Mosaics> Motivation




Motion Parallax

Image blending

Image warping

Image registration

Jacobian, Hessian

Planar Homography

Image intensity gradient

Terms that should be known:






Motion parallax: The motion parallax is the effect of

closer objects moving by faster than more distant ones

“apparent motion” of an object against a distant background because of a perspective shift.

change of angular position of two stationary points relative to each other as seen by an observer, due to the motion of an observer

The key to depth perception

Motion Parallax Image warping Image blending Image resampling Jacobian, Hessian Planar Homography Image intensity

gradient

Perspective meaning of motion parallax

Principle of depth perception

> Terms that should be known





Distorting the image through some kind of filter.

Distorted city scene Warped versions of a face image Deforming a mesh. Assigned

to image this gives the warp


Image warping: Motion Parallax Image warping Image blending Image resampling Jacobian, Hessian Planar Homography Image intensity

gradient





Image blending is the process of calculation intensities of two pixels of overlapped images with some function


Image blending: Motion Parallax Image warping Image blending Image resampling Jacobian, Hessian Planar Homography Image intensity

gradient





In graphics software, the resample command is used to increase or decrease the size and/or resolution of a bitmap-based image

Resampling an image usually results in a loss of image quality because pixels must either be interpolated or thown out

Resampling an image when rotated.


Image resampling: Motion Parallax Image warping Image blending Image resampling Jacobian, Hessian Planar Homography Image intensity

gradient





Jacobian matrix – the derivative of multivariate function. The Jacobian matrix is the matrix of all first-order partial derivatives of a vector-valued function. Its importance lies in the

fact that it represents the best linear approximation to a differentiable function near a given point.

Hessian matrix – the matrix of f is the matrix of second partial derivatives evaluated at x.


Jacobian, Hessian: Motion Parallax Image warping Image blending Image resampling Jacobian, Hessian Planar Homography Image intensity

gradient





Matematically homography is defined as a relation between two figures, such that to any point in one figure corresponds one and only one point in the other, and vise versa.

In the field of Computer Vision, a homography is defined in 2 dimensional space as a mapping between a point on a ground plane from as seen from one camera, to the same point on the ground plane as seen from a second camera.

Different transformations in image space

Basic homography

matrix.Homogeneous 2d coordinates

Planar homography:> Terms that should be known

Motion Parallax Image warping Image blending Image resampling Jacobian, Hessian Planar

Homography Image intensity

gradient





The intensity gradient of image is

Original image X-GradientY-Gradient

Image gradient:> Terms that should be known

Motion Parallax Image warping Image blending Image resampling Jacobian, Hessian Planar Homography Image intensity

gradient





Image registration is the process of finding a homography between two images (reference and target) of a scene and spatially align them.

> Terms that should be known > Image registration process

What is image registration:





Building point correspondences through extracting feature points from which a homography is estimated

Iteratively comparing intensity pattern of the images, search of minimum of the intensity error, and so finding the homography.

Once a homography is estimated warping and resampling of the target image is made. Bilinear pixel resampling is used in this paper.

Image with extracted feature points used for correspondence

Edges and lines detected with Moravec corner detector


How is the registration performed:





Measuring similarity between images by means of intensity difference between all overlapping pixels of the two images

Defining a function which gives the amount of error between the compared images under the current transformation

Minimizing this error function – finding at which transformation the images have minimum differences in intensity. Minimizing the error function by taking it’s derivative to null with any gradient descent algorithm.

Problem - Algorithms for searching minimum of a function often find local minimums, not global.

Principle of intensity-based approach to image registration:> Terms that should be known > Image registration process





1 1 3

2 2 4

Registering two identical images by iterative intensity error minimization

Searching the best alignment of the patch to the image by intensity error minimization






Local minima problem by intensity-based image registration






Constructing Cylindrical or spherical panorama: Images from camera on a fixed tripod with

fixed focal length and field of view.

World transformation to cylindrical or spherical coordinates.

Translation of the images with each’s panning angle, recovered through estimation of intensity difference between the images.

Limitations: Tilt angle problems. Tilting towards north

or south pole, requires knowing the focal length

Estimating the focal length can be quite. Inaccurate.

> Cylindrical and spherical panoramas – example

Cylindrical and spherical panorama

Warping an image into cylindrical

coordinates





Warping the images into cylindrical (spherical) coordinates: Converting plane coordinates into cylindrical

Converting plane coordinates into spherical

panning angle

scanline






The error function estimates the squared error over each pixel i between the images for a given incremental δt

Linearizing the error function about I1


Defining the intensity minimization function:

Coordinate vector

Coordinate vector with shift

The incremental value

Intensity error

Image gradient





Solving the linear least-squares for the linearized equation about the incremental δt

When solved about δt we can now update our initial estimate t with t = t + δt

Blending images with simple feathering algorithm






Replace cylindrical and spherical coordinates with rotational matrices (and focal length) for each image. Performing the registration in the input image’s coordinate system.

develop a global optimization technique to find the optimal overall registration. compute local motion estimates (block-based optical flow) between pairs of

overlapping images, and use these estimates to warp each input image so as to reduce the ghosting.

Create a panorama without using of special hardware, and remove the need for pure panning motion with no motion parallax.

Paint a full view panoramic mosaic with a simple hand-held camera or camcorder. accumulated misregistration errors, which are always present in any large image

mosaic. any deviations from the pure parallax-free motion model may result in local

misregistrations, which are visible as a loss of detail or multiple images (ghosting).

Problems to solve:

Idea:

> Schum – Szelski Algorithm





Creating small mosaic using planar projective motion model and then calculating the rough focal length from it.

Sequentially assemble complete panoramic image using rotational motion model and patch-based alignment

Block adjustment (global alignment) modifies each image’s transformand focal length to minimize global error. Removes large inconsistensies, gaps

Deghosting (local alignment) reduces any local missregistration errors

Environment mapping. The mosaic is stored like a collection of images with their associated parameters or mapped over a polyhedron

Processing flow:






Representing the mosaic as collection of images with associated geometrical transformations

Hierarchical motion estimation framework Pyramid constructor Motion estimation Image warping Coarse-to-fine refinement

Sequential mosaic construction. Finding the best alignment of image with

the mosaic, constructed from all the previous images. Thus estimating only incremental motion.

Warp/register/update loop

Image warping Using parametric motion model for

calculating warped image coordinates


Alignment Framework:

Image pyramid used for coarse-to-finerefinement





Given two images from the same viewpoint containing a projection of a 3d point in space a planar perspective transformation exist between them

Warping image into another using planar perspective motion model

> 8–parameter motion model

8-parameter perspective transformations (homographies):





Warping image into another using planar perspective motion model


Rewriting the matrix equation into function from x and y

Homography equationwith homogenous coordinates





Iterative equation


Rewriting the coordinates function according the iteration equasion

Where D is the deformation matrix





Intensity error function with the iterative equation integrated.


First order taylor expansion over bivariate function

Intensity error between the images under current transformation

The Jacobian matrix

Image gradient at xi





Basic least-squares problem through normal equations

Basic linear least-squares equation

The hessian from image gradients

Gradient-weightend intensity error

Solving the least-squares problem for the intensity error function with normal equations






Limitations: Has more free parameters than

necessary thus slow convergence

Gets easily stuck in local minima when the initial misregistration is big





Panoramic Image Mosaics> 3-d rotations and zooms

From the fact the camera and the scene are in common coordinate system follows that the movement of the camera between consequtive images could be described by translation scaling and rotation.

Given a focal length and assuming no translational motion, one must estimate only 3 more parameters

3-d rotations and zooms:




Panoramic Image Mosaics> 3-d rotations and zooms

Point in 3-d space

Translation, Scale and Rotation matrices

Recovering 3-d direction from 2-d projection coordinates

Point in 2-d space Homogeneous coordinates

Perspective transformation exists described by

Rotational matrix equation

Inverse equation.Assuming no translation





Perspective projection between two images

Rewriting the equation as parameter based. The Parameter is the angular velocities vector

> 3-d rotations and zooms

The parameter matrix presented in rotational model

Rodriguez formula for constructing rotation matrix from tree angles

Angular velocities vector. The unknown in the equation.

Cross product operator for the rotation matrix with the tree angles





Building the iterative parameter estimation equation


Equivalent to the 8-parameter method iterative equation

Defining the Jacobian for the rotational model parameter estimation algorithm

The deformation matrix for the rotational model





Significantly faster convergence caused by only 3 degrees of freedom of the parameter matrix

The same problems as by 8-parameter motion estimation exist, stucking in local minima





Panoramic Image Mosaics> Estimating focal length

The correspondence between the two matrices

From the orthonormality of the rotation matrix

Orthogonal: all row (column) vectors have the same norm

Normal: all row (column) vector have dot product equal to null

Estimating f0 (analog f1)

Estimating the focal length from the initial mosaic:





Always present due to errors

Closing by registering the first image at the end – the quotient of the parameter matrices gives the error coefiticent

Converting the error into angle and update the focal length

Registering the sequence with the new focal length

> Estimating focal length

Closing the gap in the panorama:

Example for gap in a panorama by registering the first image at the end





Computational effort for taking a single gradient step consisist of wapring (resampling) the target image, computing the local intensity errors and gradients and accumulation of A and b. Very computationaly expensive Such precisenes is not requried

Approximation Divide the image into little patches (8 x 8)

Approximate the Jacobian Assume that the Jacobian is constant for each patch Evaluate it only once at patch center Calculate A and b with this approximation

Image divided into patches

Patch-based Alignment:





Block matching technique Search the minimum intensity

difference between target patch and reference image in a given neighbourhood

Framework Divide the image into patches Estimate the best integral shift

for each patch which will best align the two image

Calculate a displacement and use it to calculate new patch hessian and gradient matrices

Limitations Very computationaly expensive Apply only on the coarsest level

of the refinement image pyramid

Searching for the best integral shift by matching patch with a neighborhood

> Patch-based alignment

Local Search:




Motivation Big mosaics show significant

amount of error due to unaviodable rotational parameters focal length estimation errors.

The accumulated error causes visible image gaps and artifacts.

Bundle block adjustment technique Optimizes globally the

transformation parameters.


Block adjustment is a feature-based technique. Establishing point

correspondences Usind local seach algorithm for

correcting the accumulated error

Bundle block adjustment of a feature point over tree

overlapping images

Block Adjustment (Global Alignment):




Panoramic Image Mosaics> Block Adjustment > Point correspondences

Dividing the image into patches (16 x 16)

Filter the patches Only patches which have interesting gradient

map Corners, Edges and Lines All patches with uniform texturing are

excluded

Use the patches as feature points with Local Search algorithm to build the corespondences

Point correspondences by rotation

Establishing the point correspondences:




Panoramic Image Mosaics> Block Adjustment > Framework

If no error in parameter estimation the points of the l corresponds to the point in k given by

Problem – not the case in practice

Instead run Local Search over two overlapping images under the current transformations

Local search result ujl is added to the patch center xjk to find the real position within the reference image

This equation rarely maps the points correctly in practice

Convertion into l’s coordinate system gives the correct position of the patch

Converting from k into l coordinate system




Panoramic Image Mosaics> Block Adjustment > Framework

Estimating the homography matrix as to minimize the difference in screen coordinates for all the images

Problem – the gradients with respect to motion parameters are complicated

Solution – Minimizing distance between ray directions of corresponding points usint rotational representation Minimizing the distance to the center

Correct the Mk as to minimize the distance

Converting from k into l coordinate system

The perfect match is in the centre so the bundle of rays must converge to that point

Minimizing the distance between every two images




Panoramic Image Mosaics> Block Adjustment

Calculating the bundle ray direction Used in the deghosting algorithm

Direction of 3d bundle ray from 2d point in image l




Motivation Addresses problems, which cannot be solved

with the global alignment, due to deviations from the perfect rotational (projective) model used.

Radial distortion Moving objects Motion parallax

Framework Computing the optical flow over the whole

mosaic to find local missregistrations.

Using inverse mapping technique to localy warp images as to neutralize the optical feld

Panoramic Image Mosaics> Deghosting

Calculates the projection of the bundle ray vector into image kThat represents the perfect position of the feature point

Defines a motion vector between the perfect and the current position

Deghosting (Local Alignment):




Panoramic Image Mosaics> Deghosting > Optical Field examples





The algorithms presented in this paper provide a efficient and reliable way of constructing full-view panoramic image mosaics without the need of special hardware equipment.

Conclution:





Shum and Szelski – Panoramic Image Mosaics Kristin Branson – Feature-based Mosaic

Construction Daniel Sylora – Image registration I Alper Yilmaz – CAP5415 Computer Vision

References:

Panoramic image mosaics

Science

Transcript of Panoramic image mosaics