Virtual Event: ECMWF-ESA Workshop on Machine Learning for Earth System Observation and Prediction

With the unprecedented advances in the satellite technology, recent years have witnessed a significant increase in the volume of remote sensing (RS) image archives (Demir and Bruzzone 2016). Thus, the development of efficient and accurate content based image retrieval (CBIR) systems in massive archives of RS images is a growing research interest in RS. CBIR aims to search for RS images of the similar information content within a large archive with respect to a query image. To this end, CBIR systems are defined based on two main steps: i) image description step (which characterizes the spatial and spectral information content of RS images); and ii) image retrieval step (which evaluates the similarity among the considered descriptors and then retrieve images similar to a query image in the order of similarity).

Due to the significant growth of RS image archives, an image search and retrieval through linear scan (which exhaustively compares the query image with each image in the archive) is computationally expensive and thus impractical. This problem is also known as large-scale CBIR problem. In large-scale CBIR, the storage of the data is also challenging as RS image contents are often represented in high-dimensional features. Accordingly, in addition to the scalability problem, the storage of the image descriptors also becomes a critical bottleneck. To address these problems, approximate nearest neighbour (ANN) search has attracted extensive research attention in RS. In particular, hashing based ANN search schemes have become a cutting-edge research topic for large-scale RS image retrieval due to their high efficiency in both storage cost and search /retrieval speed. Hashing methods encode highdimensional image descriptors into a low-dimensional Hamming space where the image descriptors are represented by binary hash codes. By this way, the (approximate) nearest neighbours among the images can be efficiently identified based on the Hamming distance with simple bit-wise operations. In addition, the binary codes can significantly reduce the amount of memory required for storing the content of images. Traditional hashing-based RS CBIR systems initially extract hand-crafted image descriptors and then generate hash functions that map the original high-dimensional representations into low-dimensional binary codes, such that the similarity to the original space can be well preserved (Fernandez-Beltran et al. 2020). Thus, descriptor generation and hashing processes are independently applied, resulting in sub-optimal hash codes. Success of DNNs in image feature learning has inspired research on developing DL based hashing methods, which can simultaneously learn the image representation and the hash function with proper loss functions.

This paper aims at presenting recent advances in CBIR systems in RS for fast and accurate information discovery from massive data archives. Initially, we analyse the limitations of the traditional CBIR systems that rely on the hand-crafted RS image descriptors applied to exhaustive search and retrieval problems. Then, we focus our attention on the advances in RS CBIR systems for which the DL models are at the forefront. In particular, we present the theoretical properties of the deep hashing based CBIR systems that have high time-efficient search capability within huge data archives (Roy et al. 2020). A particular attention is given to the metric learning and the graph structure driven deep hashing networks for scalable and accurate content-based indexing and retrieval of RS images. Finally, the most promising research directions in RS CBIR are discussed together with the description of the BigEarthNet (which is a new large-scale Sentinel-2 multispectral benchmark archive introduced to advance deep learning studies in RS) (Sumbul et al. 2019).

REFERENCES

Demir, B., Bruzzone, L., 2016. Hashing Based Scalable Remote Sensing Image Search and Retrieval in Large Archives. IEEE Transactions on Geoscience and Remote Sensing 54 (2):892-904.

Fernandez-Beltran, R., Demir, B., Pla, F. and Plaza, A., 2020. Unsupervised Remote Sensing Image Retrieval using Probabilistic Latent Semantic Hashing. IEEE Geoscience and Remote Sensing Letters, doi: 10.1109/LGRS.2020.2969491.

Roy, S., Sangineto, E., Demir, B., Sebe, N., 2020. Metric-Learning based Deep Hashing Network for Content Based Retrieval of Remote Sensing Images. IEEE Geoscience and Remote Sensing Letters, in press.

Sumbul, G., Charfuelan, M., Demir, B., Markl, V., 2019. BIGEARTHNET: A Large-Scale Benchmark Archive for Remote Sensing Image Understanding. IEEE International Conference on Geoscience and Remote Sensing Symposium, pp. 5901-5904, Yokohama, Japan.