Saturday, April 1, 2006

Image mosaic - Stitch

Download Source (License: GNU GPL)
First published: April 14, 2002

This project aims at designing software for creating image mosaic. Image registration/mosaic is a technique for creating images which cannot be created by a single frame of the camera, for example satellite     imagery. The user gives a series of pictures with overlapping regions and the software returns one large image with all the pictures merged as accurately as possible

Tools used for the project
  • Intel openCV library for C
  • MATLAB for calculating affine transformation matrix
  • GTK library for GUI
  • Linux/Unix Platform
  • GCC compiler for C

  • The following are some of the assumptions we made when we solved the problem.
  • The input images given by the user should have at least 10% overlap for good results
  • The user selects the corresponding points in the two images fairly accurately for good results
  • If the user does not get good results in the first time then user needs to try again with new points for better accuracy
  • The user selects 2-3 points that are not close to each other for better results
  • The brightness of the images given by the user should not differ too much
  • The user gives the names of the images on command line in the order of the sequential order of overlapping    

Our method of solving the problem
Program Flowchart is as follows:

Program usage at command line is as follows

Usage: stitch <num of images> <image names..> [options]

-b Enable Averaging on common area upon merging, note that this reduces quality if more than two images: Default Disabled

-i Disable Interpolation in merged image: Default Enabled

-e Disable Edge Blurring of merged image (For 2 Images only): Default Enabled

-z If given second image come on top of first: Deafult (Goes Behind)

-F Specify output filename, it MUST end with .jpg: Default is output.jpg

Example: stitch 2 picA.jpg picB.jpg -F merged.jpg -b -e -i

1. The user inputs on the command line command like the following: stitch 4 a.jpg b.jpg c.jpg d.jpg
2. Program then loads the images
3. User is presented with the first two images with suggested good points marked
4. User then selects the corresponding points after which MATLAB is used to calculate the affine transformation matrix

Where x1, y1.... xn, yn are points in the first image and x'1, y'1... x'n, y'n are the corresponding points in the second image. MATLAB is used to calculate the values of a11, a12, a13, a21, a22, a23.

5. Then the affine transformation is applied to the second image and the two images are merged. During merging depending on the user's choice the features are applied (section 5)
6. If more than two images are there then the process is repeated.

The following features are available in our program. Some features can be selected at the command line.
Color and Size - The input images given by the user can be of any size and our implementation can candle color images as well

Good features detection - The Stitch program uses the cvGoodFeatureToTrack to show the users         some points in both images simultaneously which are good features. This helps the user in getting a better idea of good corresponding points in the two images. The points are marked in green color

Manual Point Selection - Our Implementation of the program enables manual selection of points by the user. Users are asked to select corresponding points on two images shown on our user friendly GUI. To minimize the error in points selected we take 9 close points for every selection the user makes for our affine transformation calculation.

Interpolation - After putting the second image into the final image some pixels are missing because of the affine transformation as more than one pixel of the original image may transform to one pixel in the transformed image. We use a weighted bilinear interpolation method to fill up the color values at these points. For every missing pixel we take the closest neighbors and assign a weighted average color value             to the missing point based on the neighbors' color values. So the farthest the neighbor from the missing pixel the least will be its contribution in the average value. The -i switch can disable this option

Bilinear interpolation method used is illustrated as follows:

Stitching any number of Images - Stitch can be used to merge any number of overlapping images. The program loops until it creates the image mosaic of the specified number of pictures. The first two images are merged and the third is merged with the result of the these and so on

Averaging overlap Area
The -b switch can select this option and overlapping area of the images is then blurred. However we noticed it does not give pleasing results especially if more than two images are used

Blur Edge (First two images only) - Sometimes when two images are merged the boundary of the two images look obviously stitched. In some cases it helps to blur the image boundary for the two images to give a smoother view of the registration and to make it less obvious to the user that the images were indeed          stitched. This works for most cases but may not be proper where the image stitching is so perfect that there is no need to give blurring effect. So we decided to put this feature as an option which can be disabled by the user. The -e switch can disable this option and it blurs the area of join for the first two images. In             the current implementation it only works for the first two images.

A window size of 3x3 is used for taking the average the edge where the two images overlap.

Sample Results
The following are some of our best results.

Alps - France (4 Images)

Alps - France (2 Images)

Bath - UK (5 Images)

Granada - Spain (3 Images)

Graphics Lab, NUS (5 Images)

Westminster, London (2 Images)

LT27, NUS (3 Images)

S16, NUS (4 Images)

S17, NUS (2 Images)

Kent Ridge, NUS (2 Images)

Stonehenge, UK (2 Images)

Lake District, UK (2 Images)

University Cultural Center, NUS (4 Images)