Project 3 / Camera Calibration and Fundamental Matrix Estimation with RANSAC

0, Project description

The purpose of this project is to get the camera projection matrix which maps the 3D world coordinates to image coordinates, and the fundamental matrix using in stereo vision that can transform the coordinates from one camera to those of another. Also, this project will use RANSAC algorithm to estimate the fundamental matrix and thus performing a matching between two images.

  1. Estimation of projection matrix and camera center
  2. Estimation of fundamental matrix of base image pair.(with Extra normalization)
  3. Examples using RANSAC.

1, Estimation of projection matrix and camera center

The estimation of projection matrix is calculation of the relationship between world coordinates and image coordinates. It takes points in 2D images and points in 3D images, returning a projection matrix M that indicates the relationship of those points in two sets.


[height, width] = size(Points_2D);
A = zeros(height * 2, 11);
Y = zeros(height * 2, 1);
for i = 1:height
    A(i*2 - 1, 1:3) = Points_3D(i,:);
    A(i*2 - 1, 4) = 1;
    A(i*2, 5:7) = Points_3D(i,:);
    A(i*2, 8) = 1;
    uProducts = [-Points_2D(i, 1)*Points_3D(i,1), -Points_2D(i, 1)*Points_3D(i,2), -Points_2D(i, 1)*Points_3D(i,3)];
    vProducts = [-Points_2D(i, 2)*Points_3D(i,1), -Points_2D(i, 2)*Points_3D(i,2), -Points_2D(i, 2)*Points_3D(i,3)];
    A(i * 2 - 1, 9:11) = uProducts;
    A(i * 2, 9:11) = vProducts;
    Y(i * 2 - 1, 1) = Points_2D(i, 1);
    Y(i * 2, 1) = Points_2D(i, 2);
end
M = A\Y;
M = [M;1];
M = reshape(M, [], 3)';

The resulting matrix M can be further used in calculating the camera center as . The projection matrix consists of a 3x3 matrix Q and a 4th column m_4. Using the formula , the camera center can be caculated as following:


M4 = M(:, 4);
Q = M(:, 1:3);
Center = -(Q\M4);

Here is the results from my implementations:

Estimation of project matrix relating 2D image points and 3D world coordinates.

Estimation of camera center.

2, Estimation of fundamental matrix of base image pair.(with Extra normalization)

In this estimation, the fundamental matrix relating points in two images in stereo are calculated using the following formula. Here, u and v means the corresponding 2D points in first image, and u', v' means the ones in second images. Therefore, the matrix A containing the left side from eight points algorithm can be setup. Also, the right part matrix B is simply a column vector with all values of -1.


for i = 1:height
    A(i, :) = [Points_a(i,1)*Points_b(i,1), Points_a(i,2)*Points_b(i,1), Points_b(i, 1),
    Points_a(i,1)*Points_b(i,2), Points_a(i,2)*Points_b(i,2), Points_b(i,2), Points_a(i,1), Points_a(i,2)];
end
B = -ones(height, 1);

Also, in order to get a more accurate result, the algorithm is modified to a normalized version using the formula: , where s is the scalue factor, and c_u and c_v are the mean coordinates correspondingly. In addition, the fundamental matrix is obtained by scaling the matrices by the transformation matrix: 


A = zeros(height, 8);
Points_a_cu = mean(Points_a(:,1));
Points_b_cu = mean(Points_b(:,1));
Points_a_cv = mean(Points_a(:,2));
Points_b_cv = mean(Points_b(:,2));
scaleA = 1/(std(Points_a(:,1) - Points_a_cu) + std(Points_a(:,2) - Points_a_cv));
scaleB = 1/(std(Points_b(:,1) - Points_b_cu) + std(Points_b(:,2) - Points_b_cv));
firstVectorA = [scaleA, 0, 0; 0, scaleA, 0; 0, 0, 1];
firstVectorB = [scaleB, 0, 0; 0, scaleB, 0; 0, 0, 1];
secondVectorA = [1, 0, -Points_a_cu; 0, 1, -Points_a_cv; 0, 0, 1];
secondVectorB = [1, 0, -Points_b_cu; 0, 1, -Points_b_cv; 0, 0, 1];
T_a = firstVectorA * secondVectorA;
T_b = firstVectorB * secondVectorB;

F_matrix = T_b' * F_matrix * T_a;

Here is the results from my codes:

Image a

Image b

3, Examples using RANSAC.

The third part of the project is to implementing RANSAC algorithm to build up a relatively accurate fundamental matrix from the points collected from SIFT matching. The basic RANSAC algorithm is:


for i = 1:iteration
    currInliers_a = [];
    currInliers_b = [];
    sample_a = zeros(sampleSize, 2);
    sample_b = zeros(sampleSize, 2);
    randoms = randsample(height, sampleSize);
    for j = 1:sampleSize
        sample_a(j,:) = matches_a(randoms(j),:);
        sample_b(j,:) = matches_b(randoms(j),:);
    end
    currentMatrix = estimate_fundamental_matrix(sample_a, sample_b);
    inliers = 0;
    k = 1;
    for j = 1:height
        currentA = [matches_a(j,:), 1];
        currentB = [matches_b(j,:), 1];
        product = abs(currentB * currentMatrix * currentA');
        if product < threshold
            inliers = inliers + 1;
            currInliers_a(k, :) = currentA(1:2);
            currInliers_b(k, :) = currentB(1:2);
            k = k + 1;
        end
    end
    if inliers > bestInliers
        bestInliers = inliers;
        Best_Fmatrix = currentMatrix;
        inliers_a = currInliers_a;
        inliers_b = currInliers_b;
    end
end

In my implementation, I randomly chose a sample size of 8 points in each images to estimate the fundamental matrix using the method completed in part II. After getting the fundamental matrix, apply it to the original sample set, then count the number of inliers within threshold. The formulat I used to count for the inliers is the product of points from image a with fundamental matrix and transpose of points from image b. If the product is smaller than the threshold 0,01, which is close enough to be 0, the matching points can be considered as inliers. Finally, pick up the one model with most inliers. Using RANSAC algorithm greatly enhances the accuracy of feature matching, providing sets of relatively reliable matching points. The following table shows the feature matching of two images and epipolar lines drawn according to the fundamental matrix found:

The matching features

The epipolar lines of image a with calculated fundamental matrix

The epipolar lines of image b with calculated fundamental matrix