Crop Type Mapping from Satellite Image Time Series using Deep Learning
Sina Mohammadi is a PhD student in the department of Earth Observation Science. (Co)Promotors are prof.dr.ir. A. Stein and dr. M. Belgiu from the faculty ITC.
Accurate and timely information about crop types plays a crucial role in various agriculture-related applications. Applying crop type mapping methods to data from new target study areas often faces several challenges. To achieve good classification results, these methods require a large volume of labeled data from the target study areas for training. Unfortunately, obtaining labeled samples of cultivated crops is time-consuming and expensive. Therefore, the development of methods capable of leveraging the potential of the available samples (labeled/unlabeled) is required. This dissertation presents deep learning-based methods to address three identified challenges by exploiting the information that can be derived from the available samples.
First, most deep learning-based crop type mapping methods use the cross-entropy loss, while several studies showed its reduced generalization performance. Moreover, these methods rely on output supervision, neglecting the potential benefits of supervising intermediate layers of deep neural networks. This research aims to address these challenges by exploiting the source domain data to extract more discriminative features. The study proposes to supervise intermediate layers of a 3D Fully Convolutional Neural Network (FCN) by employing two middle supervision methods: 1) Cross-entropy loss Middle Supervision (CE-MidS) that aims to improve feature learning by applying the cross-entropy loss to middle layers; 2) a novel middle supervision method, namely Supervised Contrastive loss Middle Supervision (SupCon-MidS) that applies Supervised Contrastive loss to middle layers. This method pulls together features belonging to the same class in embedding space, while pushing apart features from different classes. Furthermore, this study investigated the effectiveness of two output supervision methods, namely F1 loss and Intersection Over Union (IOU) loss, as replacements for the cross-entropy loss. Our experiments on identifying different crops from Landsat image time series in the U.S. showed that the optimal configuration of the method, referred to as IOU+SupCon-MidS, performed better than the state-of-the-art methods. The mIOU scores increased by 3.5% and 0.5% on average across different years and different regions, respectively. Adding SupCon-MidS to the output supervision methods improved mIOU scores by 1.2% and 7.6% on average across different years and different regions, respectively. The results show that proper supervision of deep neural networks plays a significant role in improving crop type mapping performance.
Second, deep learning methods have achieved promising crop type mapping results. However, they require a large volume of labeled samples for training. This limitation hinders their application in numerous regions across the globe, where the availability of labeled crop samples is scarce. Therefore, this thesis develops methods capable of exploiting labeled data from label-rich environments to classify crops in label-scarce environments using only a few labeled samples per class. To this aim, I adapted and evaluated eight Few-shot Learning (FSL) methods for mapping infrequent crops cultivated in selected study areas from France and a large diversity of crops from a complex agricultural area in Ghana. To enable realistic evaluation of FSL methods on unlabeled sets from the target domain data, i.e., query sets, the Dirichlet distribution was used to model the class proportions as random variables. This study demonstrates that Transductive Information Maximization based upon α-divergence (α-TIM) performed better than the competing methods, including Dynamic Time Warping (DTW). α-TIM achieved a macro F1-score of 59.6% in Ghana in a 24-way 20-shot setting and a macro F1-score of 75.9% in a 7-way 20-shot setting in France, outperforming the second best-performing methods by 2.7% and 5.7%, respectively. Moreover, α-TIM outperformed a baseline deep learning model, highlighting the benefits of effectively integrating the query sets into the learning process.
Third, crop type mapping methods often face significant challenges in cross-regional and cross-time scenarios with large discrepancies between temporal-spectral characteristics of crops from different regions and years. Unsupervised domain adaptation (UDA) methods have been employed to mitigate the domain shift between the source and target domains. Since these methods require source domain data during the adaptation phase, they demand substantial computational resources and data storage, especially when large labeled crop type mapping source datasets are available. This leads to increased energy consumption and financial costs. To address this limitation, this research develops a source-free UDA method for cross-regional and cross-time crop type mapping. It mitigates the domain shift by leveraging mutual information loss. The diversity and discriminability terms in the loss function are balanced by means of a novel unsupervised weighting strategy based upon mean confidence scores of the predicted categories. Our experiments on mapping corn and soybean from Landsat image time series in the U.S. demonstrate that the adapted models using different backbone networks outperform their non-adapted counterparts. With CNN, Transformer, and LSTM backbone networks, this adaptation method increased the macro F1 scores by 12.9%, 7.1%, and 5.8% on average for cross-time tests and by 20.1%, 12.5%, and 8.8% on average for cross-regional tests, respectively. This shows that the method is modular and flexible in employing various backbone networks. Our experiments on mapping the same classes using Sentinel-2 image times series in France demonstrated the effectiveness of our method in a different country and for a different sensor data. Moreover, in within-season experiments, the adapted models performed better than the non-adapted models in the vast majority of weeks. These results and their comparison to those obtained by the other investigated UDA methods demonstrate the efficiency of our proposed method for both end-of-season and within-season crop type mapping tasks.
To summarize, this dissertation presents deep learning-based methods that enhance crop type mapping performance by exploiting the information provided by the available samples. This study is important to the scientific endeavors concerned with crop type mapping where there is no access to large labeled sample sets in the target study areas.
