Image matching with AffNet and HardNet

We had a task to find best matching image to the given query image.

First we tried classic image matching with SIFT descriptors and Flann based matcher in OpenCV. Results were completely wrong. Then Bag-of-Words …

Finally we have learnt about combination of AffNet and HardNet from these slides.

AffNet is a novel method for learning local affine-covariant regions with
the hard negative-constant loss which outperform s the state-of-the-art in Bag-of-Words image retrieval and wide baseline stereo.

Image for post
Image for post
AffNet training (from original paper)

HardNet is a novel compact learned feature descriptor showing state-of-art superior performance relating to both hand-crafted and learned descriptors on standard matching and retrieval benchmarks and it is fast to compute on a GPU (authors paper). It is publicly available on github.

Here is an excellent example of matching images captured with extreme affine transformation using AffNet

Image for post
Image for post

Here authors describe the approach in more detail in a paper. Good presentation about HardNet.

Clone repository from github:

Then move to the folder.

Start Jupyter :

and open notebook SIFT-AffNet-HardNet-kornia-matching.ipynb in folder examples. First install all required libraries. Create a cell in the top of notebook and run

If you have OpenCV version 4 you can get error with function SIFT_create: “The function/feature is not implemented This algorithm is patented and is excluded in this configuration;”. SIRF and SURF are no longer available in opencv > 3.4.2.16. Install older version of OpenCV:

There is three pipelines:

  • DoG-AffNet-OriNet-HardNet (Using AffNetJIT, OriNetJIT, extracting SIFT keypoints, HardNet descriptors, )
  • DoG-Affine-OriNet-HardNet
  • DoG-OriNet-HardNet

When using DoG-AffNet-OriNet-HardNet pipeline for AffNet matching on images provided by authors we get following result

Image for post
Image for post

Quite good for these images.

For pipeline DoG-AffNet-OriNet-HardNet

Image for post
Image for post

And for pipelines DoG-OriNet-HardNet

Image for post
Image for post

When we tested AffNet on our images of indoor scene matches were excited without any adjusting parameters. No retraining of AffNet is necessary here.

To integrate the AffNet you need only to install required libraries and download pretrained model weights

Convert Jupyter notebook to python script

CPU is used by PyTorch on default

You can switch to GPU commenting line above and uncommenting following line

That’s it. Good luck in solving your problems with matching images.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store