Cloud and cloud shadow detection for optical satellite imagery - Features, algorithms, validation, and prospects
Li, Z., Shen, H., Weng, Q., Zhang, Y., Dou, P., Zhang, L. (2022). Cloud and cloud shadow detection for optical satellite imagery: Features, algorithms, validation, and prospects. ISPRS Journal of Photogrammetry and Remote Sensing, 188, 89–108. [HTML] [PDF] [Project]
Abstract
The presence of clouds prevents optical satellite imaging systems from obtaining useful Earth observation information and negatively affects the processing and application of optical satellite images. Therefore, the detection of clouds and their accompanying shadows is an essential step in preprocessing optical satellite images and has emerged as a popular research topic in recent decades due to the interest in image time series analysis and remote sensing data mining. This review first analyzes the trends of the field, summarizes the progress and achievements in the cloud and cloud shadow detection methods in terms of features, algorithms, and validation of results, and then discusses existing problems, and provides our prospects at the end. We aim at identifying the emerging research trends and opportunities, while providing guidance for selecting the most suitable methods for coping with cloud contaminated problems faced by optical satellite images, an extremely important issue for remote sensing of cloudy and rainy areas. In the future, expected improvements in accuracy and generalizability, the combination of physical models and deep learning, as well as artificial intelligence and online big data processing platforms will be able to further promote processing efficiency and facilitate applications of image time series. In addition, this review collects the latest open-source tools and datasets for cloud and cloud shadow detection and launches an online project (Open Satellite Image Cloud Detection Resources, i.e., OpenSICDR) to share the latest research outputs (https://github.com/dr-lizhiwei/OpenSICDR).
1. Introduction
Over the past decades, with the release of massive amounts of optical satellite data and the continuous introduction of new data sources, many studies have focused on cloud and cloud shadow detection (CCS) detection for images taken by different sensors, and many CCS detection methods have been developed. Although the CCS detection literature has been reviewed in previous studies, they mainly focus on the categorization of methodologies, cloud measuring equipment, and forms of CCS detection results. To further conduct a systematic summary of the current achievements and challenges in this field, this study comprehensively reviews the CCS detection literature from the features, algorithms, and validation perspectives as shown in Fig. 1. Different CCS detection methods are reviewed separately from both the features and algorithms perspectives given that algorithms of different categories may achieve CCS detection by using the same type of features and because algorithms of the same category may also accept different types of features. This review also summarizes the main types of CCS detection results and their validation.
2. Literature analysis
We have conducted a literature survey with Scopus (www.scopus.com) on the topic of cloud and cloud shadow detection for optical satellite imagery, and the search keywords include cloud/cloud shadow AND detection/masking/extraction/screening/identification, which are limited to title, keywords, and abstract of the journal articles in English. Based on search results as of December 1, 2021, a total of 1425 journal papers were returned from the Scopus database. Eventually, there are 504 papers selected for the literature analysis after manually excluding irrelevant articles. Noted that articles related to cloud classification which further classify clouds into different types according to cloud phase and altitude are not discussed and included in the literature survey. Besides, to ensure the high quality of the papers used for literature analysis, only articles published in formal journals were selected, and each paper was manually checked and confirmed to make sure that it fits the topic of this review. As shown in Fig. 2, the survey result suggests that there is a general increasing trend in the number of published papers and their citations over the past 37 years from 1985 to 2021, especially in the latest decade, which indicates extensive attention was paid to this field recently. Besides, the major countries/region, institutions, and journals in the field are listed in Fig. 2-5.
Moreover, the number of papers with different types of satellite images was counted to determine the major types of images that occurred in these 504 papers. The top twelve types of images occurring in >10 papers are selected and shown in Fig. 6, which are sorted by the number of papers that are selected and counted according to whether the satellite keyword occurs in the title, keywords, and abstract of the paper. Fig. 6 also shows the distributions of the number of papers relevant to each type of image over the past each 5–6 years periods from 1985 to 2021.
Literature analysis was also conducted on different types of CCS detection algorithms over different periods, the results shown in Fig. 7 confirmed that physical-rule algorithms benefiting from their simplicity and efficiency are the most popular type of CCS detection algorithms, which have been heavily studied at different periods. In addition, the results shown in Fig. 7 also indicated that DL-based methods as a branch of ML-based algorithms have gained much attention in recent years, benefiting from the great progress of DL in tasks such as image classification and the release of a large number of open-source datasets. Moreover, temporal-changes based algorithms have gradually received more attention in the past periods owing to the increase in satellite data sources and the widespread use of image time series. Notably, variational-model based algorithms have been developed in recent years, and it can be expected that such types of algorithms will have broader application potentials for integrated cloud detection and removal in multi-temporal images.
3. Problems and prospects
Although the current CCS detection algorithms for optical satellite images have witnessed much progress, some problems remain unsolved or need to be better resolved. This review summarizes the current problems faced in CCS detection and prospects in terms of accuracy, generalizability, efficiency, and image time series processing and applications.
1) Common problems and accuracy improvement for CCS detection
The results of thin cloud detection have a significant impact on the accuracy of different cloud detection algorithms. Large-area thin clouds that are similar to haze or translucent clouds in the transition region between thick clouds and the clear sky surface have large variability due to different underlying surfaces, hence impeding an accurate detection. Moreover, high bright surfaces, including snow/ice, built-up areas, and bright water bodies, can be easily misidentified in cloud detection due to their spectral properties similar to clouds, especially in areas with large-area snow/ice, given the difficulty in extracting identifiable features from the central region of large clouds and bright surfaces but with slight differences in the edge regions. Moreover, given the varying geometric characteristics of clouds in an image, the limited block size of the input image and the receptive field of the deep model, CNN-based algorithms cannot easily separate large-area clouds from non-cloud bright surfaces completely and comprehensively. To address these problems, the combination of object-oriented image analysis (OBIA) and deep models with spatial attention mechanisms and the introduction of geographical information and additional auxiliary data may benefit the detection of large-area clouds and their discrimination from snow. In addition, constructing individual models for different local scenes and learning scene adaptive models for cloud detection in different complex surfaces may also help boost accuracy.
In addition, there is still much room for improvement in cloud shadow detection. For example, the cloud shadow accuracy of the widely used Fmask algorithm for Landsat images is only about 70%. Cloud shadow detection has received less attention than cloud detection given that cloud shadows usually account for only a small percentage compared with clouds in images. On the one hand, cloud shadows are easily confused with dark and low-reflectivity targets in images, such as terrain shadows and water bodies, thereby challenging the accurate detection of cloud shadow, especially for high spatial resolution images in which the features of cloud shadow are not sufficiently significant. The cloud shadow detection accuracy is also limited by the accuracy of cloud detection and the errors generated by the physical-rule based models when matching clouds with their shadows. On the other hand, the overall area of cloud shadows in the image is usually much smaller than that of clouds due to the fact that most cloud shadows are obscured by clouds influenced by satellite viewing and solar angles, which may lead to sample categories imbalance and reduce the accuracy of cloud shadow detection for machine-learning based algorithms to some degree. The insufficient cloud shadow training samples in DL-based algorithms is one factor that leads to poor cloud shadow detection accuracy. In this regard, given the category imbalance between clouds and cloud shadows, introducing small sample transfer learning will help improve cloud shadow detection accuracy. Moreover, with the aid of multi-source auxiliary data (e.g., DEM), terrain shadows in images can be predicted and used to reduce the misidentification of cloud shadows.
2) Combination of physical model and deep learning for CCS detection and removal
Physical-rule and DL-based algorithms have been widely and individually developed for CCS detection, which can be essentially categorized as model-driven and data-driven methods given their characteristics and advantages, respectively. The model-driven (i.e. physical-rule based and variational-model based) methods are usually definite and are construed based on the empirical assumptions, which may be difficult to cope with complex land cover conditions and the solving efficiency of variational models needs to be improved. On the contrary, while the data-driven (i.e. traditional machine-learning based and DL-based) methods have more strong feature representation capabilities than model-driven methods, their performances rely heavily on training data. Considering that the acquisition of large-scale CCS training samples is time-consuming and inefficient as there are many different types of images, the combination of model-driven and data-driven methods will be promising for CCS detection, and boosting model performance and efficiency while reducing the need for training samples. In terms of features and data, the output of the physical model can be used as the prior knowledge and input features of the deep model, and the physical model such as atmospheric scattering law can also be used to stimulate cloud samples and to meet the sample requirements of the deep model to some degree. In terms of methods, model-driven (i.e. variational model) and data-driven (i.e. deep learning) methods can be combined in different forms, their coupling will combine their strengths and benefit the improvements of accuracy and efficiency simultaneously. Therefore, the combination of the physical model and DL, especially the coupling of model-driven and data-driven methods, are promising for CCS detection and worths further exploration in the future.
In addition, the commonly called cloud detection usually does not include the detection of thin clouds like haze, instead, direct dehazing and thin cloud removal is performed to reduce the effect of haze and thin clouds. Considering that both haze and clouds are degrading factors affecting the quality of optical satellite images, the processing of clouds usually consists of three parts, including dehazing/thin cloud removal, cloud detection, and thick cloud removal, which are usually performed independently or partially coupled. In the future, a promising consideration is to combine physical models and data-driven methods to achieve a joint estimation of cloud thickness for both thin and thick clouds, and conduct an integrated thin/thick cloud removal to improve the image quality and usability. In particular, physical models can be used to simulate a large number of hazy and cloud-covered image samples for model training of data-driven methods, such as deep learning, to construct an integrated model for simultaneous thin and thick cloud removal in images. With the rapidly growing amount of optical satellite images, such a combination and method will be able to provide strong support for efficient cloud processing of massive images.
3) Development of a unified framework for CCS detection for multi-sensor images
Due to the differences in band settings and spectral responses of satellite images of different sensors, most of the current CCS detection algorithms have been developed and applied to specific types of images, thereby limiting the applicability of CCS detection algorithms for multi-sensor images. Given the increasing number of satellite image data sources, developing different algorithms for images of previously or newly launched satellites becomes inefficient. Therefore, a unified framework should be developed to boost the generalizability of CCS detection for images of multiple sensors. In this regard, analyzing common spectral characteristics and mining the invariant spatial features and even temporal difference features of images will benefit the development of a unified framework for the CCS detection of multi-sensor images. Although several recent studies have attempted to achieve multi-sensor CSS detection based on DL, the application of current DL-based CCS detection algorithms cannot be effectively extended to other types of images due to the large sample requirements for model training and the unique characteristics of images from different sensors. With the continuous advancements of DL techniques, algorithms supported by transferring learning techniques, such as domain adaptation, have promising applications in addressing large sample requirements, in which models trained on the source domain can also be applied to the target domain without much accuracy reduction, and thus achieve the goal of CCS detection for multi-sensor images. In addition, simulating cloudy images based on clear images of multiple sensors through advanced image synthesis techniques, such as GAN, can generate a large number of training samples with very little human effort. Such an approach can be used to meet the training requirements of multi-sensor models, and provide a new scheme for the construction of a unified framework for CCS detection in multi-sensor images.
4) AI-enabled online cloud processing for large-scale optical satellite data
With the release of a large amount of optical satellite data, the dense processing requirements motivate us to develop more effective online cloud detection algorithms. Online processing platforms, such as Google Earth Engine (GEE), Amazon Web Services, and Microsoft Azure, provide opportunities to quickly access and process big satellite data online to support their wide range of applications. In this case, online cloud detection algorithms are crucial to achieving a near-real-time pre-processing of optical satellite images. Recent studies have investigated cloud detection in online GEE platforms and achieved satisfactory results. In the future, the further combination of online processing platforms and artificial intelligence (AI) technologies will provide a new paradigm for the processing of large-scale satellite data, including but not limited to cloud detection, and enable the high-precision and efficient application of large-scale Earth observation data.
5) Integration of cloud detection/removal with image time series analysis
Image time series play an important role in the long-term monitoring of the Earth surface, however, cloud coverage causes the problem of information missing in image time series. In this regard, cloud detection and removal are both essential for the composition of clean and seamless images of desired areas, especially cloudy areas and other areas with abundant rainfall. Given that multi-temporal cloud removal usually utilizes auxiliary information from the adjacent temporal images of the same areas to reconstruct the cloud-covered areas, cloud detection and cloud removal are treated as individual processes in most previous studies. Therefore, the integrated multi-temporal cloud detection and removal will improve processing efficiency and reduce error accumulation. On the one hand, the temporal features in image time series benefit the accuracy improvement of CCS detection, especially in complex and bright land surfaces. On the other hand, image time series, which provide longer and more continuous observation information, also benefit the accurate reconstruction of cloud-covered areas. In terms of the integration of cloud detection and cloud removal, time series models and variational models have demonstrated their application potentials, and more advanced techniques, such as DL, warrant further exploration.
4. Conclusions
CCS detection is an essential step in satellite image preprocessing and applications and has remained an important topic in the field of optical remote sensing. This review has examined the trends in the field through a literature survey, reviewed the CCS detection studies from the features, algorithms, and validation aspects and summarized the existing problems, and provided our prospects for future development. It can be concluded that the combination of spectral, spatial, temporal, and multi-source features from auxiliary data has been proved effective to alleviate the commission and omission problems of CCS. In terms of using only the information from the images themselves, spectral-spatial features and spectral-spatial-temporal features are the two most promising types of features that enable high-precision CCS detection. Particularly, DL-based methods have been demonstrated their superiority over other types of algorithms in CCS detection with the support of sufficient training data. Collocated LiDAR/radar data and ground-based camera data were alternative sources for validation, as the acquisition of manually labeled masks was time-consuming. Further improvement in CCS detection algorithms can be made by improving accuracy in complex conditions, and by boosting the cross-sensor generalizability by developing a unified framework. Further research is warranted in investigating the combination of physical model and deep learning, exploring the integration of thin/thick cloud detection/removal, and developing AI-enabled online processing methods for large-scale satellite data, which will facilitate efficient processing and fine applications of optical satellite imagery.
Enjoy Reading This Article?
Here are some more articles you might like to read next: