October 2023
I shot the pictures with my Pixel 4 camera which shoots pictures at 2268x4032 with a 12.2 megapixel camera. This image size is too large so I downsampled it to a size of 900x1600. I tried using the same aperature and exposure settings for each photo.
To do a projective transformation, w want to recover a 3x3 homography matrix, H, from one set of points to another. We have 8 unknowns, which means this can be solved with just 4 points. However, this might be prone to noise so we can use more points and create an overdetermined system. We can solve this using Least Squares to find H.
Since we have found a way to get the homography matrix, we can do this neet trick of "rectification," i.e. warping the plane as if the object was upfront and right before you (front-parallel). We have one image, so we need to choose a set of correspondance points that we know the correct shape of. A good example are the corners of a box, which we know should be square or rectangular. Then, I just have to choose corresponding points of a similar sized square and compute the homography between these set of points. Finally, just warp the image to make it front-parallel.
Now we have the tools to create a mosaic of multiple images that covers a wider perception field than a single image. My procedure was to find the correspondance points between a series of images that covers around 40-70% of another image. I would have a left, middle, and right image. The middle image would be the image that I would make the center naturally. I pick 4 correspondance points for each pair of homography. One for the left-middle pair. One for the right-middle pair. We can get the homographies for these pairs and compute the warped image. Then, we have a warped image of the same size that need to stiched together. I stitched each photo each with the middle image. Then I combined those two stiched photos into one photo, lining up 1 specific point that exists in each stiched photo. Next, I had to blend the image. I just did a simple weighted average of the images. I find the area that overlaps and average the pixels and keep the rest of the pixels the same. Unfortunately, I do get aliasing with a black line that shows up between each photo, but other than that it blends well.
This was really cool. I've taken some panaramas before and never really thought about how it was made. I also didn't think I could just change the perspective of any image.
We want to find keypoints in our image. We do this by using a harris corner detector, where for each point we give it a value between 0 and 1 for its harris cornerness. We convolve it with a 3x3 kernel that will find each point's harrisness. After this, we take points above a certain harris threshold that we set as our harris corners. Below you can see the harris corners for two images with a threshold of 0.2.
There are a lot of harris corners and many of them are clustered closely together. To get a sparser set of key points, we will use a technique known as ANMS. ANMS will make an ordering of our points based on their harrisness and distance from other points, as seen below. We take the first N points in our ANMS ordering as our keypoints. I set N to be 250 points.
Now when we want to match our points, and we want to take into consideration our surroundings. We will capture a 40x40 patch and then downsample to a normalized 8x8 patch. This will act as a blurred descriptor that will capture the most of the feature with less noise.
For feature matching, we will take a point set from one image and find the nearest neighbors. To do this, we will find the sum of squared differences of their feature descriptors, then we will find the two points that result in the lowest error and that will be our nearest neighbor. We will also find our 2nd nearest neighbor. We have a thresold to determine which pairs of neighbors are good matches. I decided to take any matches where the SSD1/SSD2 was less than a thresold of 1.15 and 1.5 for my left image matches and right image matches respectively. Below, we see the result of our prelimary feature matching.
But, we still have bad matches. We will use RANSAC to find the best homographic transformation between our sets of points. We will do this in a random algorithm with N steps, where on each step we choose 4 random point correspondences, compute their homographic matrix, and then find how many other points properly transformed to their corresponding point in the other image (inliers). We will take the transformation that produces the most amount of points that have been mapped correctly. If we choose a large enough N, this will usually give a good homography.
Here's my automatic stitching final results! Left is the manual alignment. Right is the automatic alignment. The automatic alignment is clearly better for the mosaic of the 3rd floor of way west.
I learned how to do homographies and it was so cool to automate the key points. It was cool seeing how different images from different rotations could be stiched together using homographies.