Image matching with AffNet and HardNet

We had a task to find best matching image to the given query image.

First we tried classic image matching with SIFT descriptors and Flann based matcher in OpenCV. Results were completely wrong. Then Bag-of-Words …

Finally we have learnt about combination of AffNet and HardNet from these slides.

AffNet is a novel method for learning local affine-covariant regions with
the hard negative-constant loss which outperform s the state-of-the-art in Bag-of-Words image retrieval and wide baseline stereo.

AffNet training (from original paper)

HardNet is a novel compact learned feature descriptor showing state-of-art superior performance relating to both hand-crafted and learned descriptors on standard matching and retrieval benchmarks and it is fast to compute on a GPU (authors paper). It is publicly available on github.

Here is an excellent example of matching images captured with extreme affine transformation using AffNet

Here authors describe the approach in more detail in a paper. Good presentation about HardNet.

Clone repository from github:

git clone https://github.com/ducha-aiki/affnet.git

Then move to the folder.

Start Jupyter :

jupyter notebook

and open notebook SIFT-AffNet-HardNet-kornia-matching.ipynb in folder examples. First install all required libraries. Create a cell in the top of notebook and run

!pip install kornia pydegensac extract_patches

If you have OpenCV version 4 you can get error with function SIFT_create: “The function/feature is not implemented This algorithm is patented and is excluded in this configuration;”. SIRF and SURF are no longer available in opencv > 3.4.2.16. Install older version of OpenCV:

pip install opencv-python==3.4.2.16
pip install opencv-contrib-python==3.4.2.16

There is three pipelines:

• DoG-AffNet-OriNet-HardNet (Using AffNetJIT, OriNetJIT, extracting SIFT keypoints, HardNet descriptors, )
• DoG-Affine-OriNet-HardNet
• DoG-OriNet-HardNet

When using DoG-AffNet-OriNet-HardNet pipeline for AffNet matching on images provided by authors we get following result

30.0 inliers found

Quite good for these images.

For pipeline DoG-AffNet-OriNet-HardNet

18.0 inliers found

And for pipelines DoG-OriNet-HardNet

25.0 inliers found

When we tested AffNet on our images of indoor scene matches were excited without any adjusting parameters. No retraining of AffNet is necessary here.

To integrate the AffNet you need only to install required libraries and download pretrained model weights

wget https://github.com/ducha-aiki/affnet/raw/master/convertJIT/AffNetJIT.pt
wget https://github.com/ducha-aiki/affnet/raw/master/convertJIT/OriNetJIT.pt
wget https://github.com/ducha-aiki/affnet/raw/master/test-graf/img1.png
wget https://github.com/ducha-aiki/affnet/raw/master/test-graf/img6.png
wget https://github.com/ducha-aiki/affnet/raw/master/test-graf/H1to6p

Convert Jupyter notebook to python script

jupyter nbconvert SIFT-AffNet-HardNet-kornia-matching.ipynb --to python

CPU is used by PyTorch on default

dev = torch.device('cpu')

You can switch to GPU commenting line above and uncommenting following line

#dev = torch.device('cuda')

That’s it. Good luck in solving your problems with matching images.