PhD thesis

With the monitoring, we aim at being able to keep track of the model's performance over its ODD. In our case, we trained a model on a training dataset and we deploy it over IGN orthoimages covering all France. Therefore, our ODD, which we call the mapping area is France.

The standard way of evaluating a model's accuracy is to compare its prediction against ground truth labels. Over France, we do not have such ground truth labels so instead we have to rely on the closest available data, which is the Registre national d'installations (RNI). The RNI indicates for each city the total number of installations and the cumulated installed capacity.

To monitor the accuracy of the registry with the RNI, we rely on unsupervised model evaluation methods [7]. These approaches consist in automatically comparing the outputs of the model with an external data source. In our case, we aggregate the detections of our detection model to obtain an estimation of the installed capacity and the number of installations. We then compare these estimations with the records of the RNI. We called this method the downstream task accuracy (DTA) as it enables to evaluate a model's accuracy according to metrics that are relevant for the downstream or final users: for instance, we can compute the average error in the estimation of the aggregated installed capacity, and derive metrics indicating whether the model locally overestimates or underestimates the aggregated installed capacity.

In [8], we leveraged the DTA to show that the performance of the model varies greatly during its deployment. We were also able to quantify the performance drop encountered by rooftop PV mapping algorithms which was highlighted by earlier works [9,10]. Deploying a model based on [3] and fine-tuned over France using the BDAPPV dataset [11], we documented a 30 percentage point accuuracy drop when shifting from the training dataset to the mapping area. The main question that arises is to understand why such performance drop occurs.

The aim of this part is to assess whether the model detects PV panels for correct reasons. By correct, we mean that the model should detect PV panels by relying on relevant components on the image and not spurious factors (e.g. rooftop or nearby objects such as pools). To carry out this assessment, we initially relied on a class of feature attribution method called the Gradient Class Activation Map (GradCAM, [12]). This method highlights the areas on the input image that contributed the most to a model's decision.

The GradCAM was sufficient to reveal that the model does not rely on spurious factor and focuses where PV panels are located. However, it failed to explain false positives and false negatives. A thourought analysis of these failure cases led us to suppose that they were caused by the fact that the model relies on different scales, and that depending on whether patterns are present at different scales false predictions can arise.

In other words, to better understand why false prediction arise, we need to assess what does the model see on the input image and not only where it looks, as traditional feature attribution methods do. Towards this end, we introduced the wavelet scale attribution method (WCAM).

The WCAM builds on the sensitivity analysis method of [13]. This method consists in sampling random perturbation masks. In the original method, these masks are applied to the input image. The sensitivity of the model to the masks' perturbation is then recorded through the variation in the predicted probability. If the occluded area are not important for the prediction, the variation of the predicted probability is small, and if the variation is high, then it means that the occluded area was important. Importance is evaluated using Sobol's sensitivity analysis.

To highlight the important scales, we perturb the dyadic wavelet transform of the image instead of the image itself. The dyadic wavelet decomposition [14] consists in decomposing an image into different scales. As the wavelet transform is invertible, we can reconstruct a perturbed image from its perturbed wavelet transform. The model is evaluated on the perturbed image, but this time we can traceback the importance of the perturbations in the wavelet domain instead of the image domain.

The outcome of our approach is a heatmap in the wavelet domain. This representation highlights the important scales in the model's prediction. An interesting feature for the application to PV panel classification, the WCAM revealed that for a single spatial location (highlighted in standard feature attribution methods) the model relies on different scales. These scales correspond to structural elements such as details within the PV modules or the gridded pattern.

We used the WCAM to carry out analyses for explaining the false predictions. We highlighted the fact that false positives tend to arise when gridded pattern appear on the input image. On the other hand, in [15] we studied how the acquisition conditions disrupted some frequency ranges (which corresponded to scales) and could lead to false negatives.

The analysis of the model's decision with the WCAM showed that the classification model relies on different scales. While dealing with the false positives is difficult, we can at least focus on the false negatives. In [15], we showed that the varying acquisition conditions contributed significantly in altering the model's performance during its real-life deployment.

To improve the robustness of the model to variying acquisition conditions, we introduced a new data augmentation technique, the wavelet perturbation. This method consists in altering the wavelet transform of the image to force the model to rely on a wide range of scales. Our aim is that if a scale is disrupted by the acquisition conditions, the model has learned to rely on other, less perturbed scales to make its prediction.

We compared our wavelet perturbation with other popular data augmentation methods aiming at improving the robustness of the model to image corruptions. We evaluated our method against AugMix [16], RandAugment [17] and AutoAugment [18] and achieved state-of-the-art results. Our benchmark dataset is BDAPPV [11]: we train a ResNet [19] model on Google images and evaluate its accuracy on IGN images depicting the same PV panel. This benchmark is a natural case study for varying acquisition conditions.

In addition to the robustness to varying acquisition conditions, we introduced PyPVRoof [20], a Python package for extracting the characteristics of rooftop PV systems. The motivation for introducing this package was that existing works lacked standardization: each method had its data requirements and the various method introduced, such as in [21,22,23] couldn't be compared. Our aim was to systematically compare existing approaches and to design a set of method that could be used in different cases in terms of data availability. Practitioners can use PyPVRoof to extract the tilt and azimuth angles, surface, installed capacity and localization of rooftop PV systems even if they have no other data than a geolocalized polygon, if they have 3D LiDAR data or access to limited information on PV systems.

The last step towards building DeepPVMapper was to provide a comprehensive benchmark of classification and segmentation models, evaluate the gains brought by the two-step approach introduced by DeepSolar and bring several improvements to the pipeline to minimize false detections and speed-up the computations. DeepPVMapper improves over existing works by being more robust to varying acquisition conditions, less prone to false detections and more flexible than previous works to extract the characteristics of PV systems as it accomodates for different cases of complementary data availability (e.g., 3D LiDAR data).

1. Malof, J. M., Bradbury, K., Collins, L. M., & Newell, R. G. (2016). Automatic detection of solar photovoltaic arrays in high resolution aerial imagery. Applied energy, 183, 229-240.
2. Kausika, B. B., Nijmeijer, D., Reimerink, I., Brouwer, P., & Liem, V. (2021). GeoAI for detection of solar photovoltaic installations in the Netherlands. Energy and AI, 6, 100111.
3. Mayer, K., Rausch, B., Arlt, M. L., Gust, G., Wang, Z., Neumann, D., & Rajagopal, R. (2022). 3D-PV-Locator: Large-scale detection of rooftop-mounted photovoltaic systems in 3D. Applied Energy, 310, 118469.
4. Yu, J., Wang, Z., Majumdar, A., & Rajagopal, R. (2018). DeepSolar: A machine learning framework to efficiently construct a solar deployment database in the United States. Joule, 2(12), 2605-2617.
5. Saint-Drenan, Y. M., Bofinger, S., Fritz, R., Vogt, S., Good, G. H., & Dobschinski, J. (2015). An empirical approach to parameterizing photovoltaic plants for power forecasting and simulation. Solar Energy, 120, 479-493.
6. Boisnard, F., Boumazouza, R., Ducoffe, M., Fel, T., Glize, E., Hervier, L., ... & Vigouroux, D. (2023). Guidelines to explain machine learning algorithms.
7. Zhang, H., Fritts, J. E., & Goldman, S. A. (2008). Image segmentation evaluation: A survey of unsupervised methods. Computer vision and image understanding, 110(2), 260-280.
8. Kasmi, G., Dubus, L., Blanc, P., & Saint-Drenan, Y. M. (2022). Towards unsupervised assessment with open-source data of the accuracy of deep learning-based distributed PV mapping. In MACLEAN: MAChine Learning for EArth ObservatioN Workshop co-located with the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML/PKDD 2022)
9. De Jong, T., Bromuri, S., Chang, X., Debusschere, M., Rosenski, N., Schartner, C., ... & Curier, L. (2020). Monitoring spatial sustainable development: semi-automated analysis of satellite and aerial images for energy transition and sustainability indicators. arXiv preprint arXiv:2009.05738.
10. Malof, J. M., Li, B., Huang, B., Bradbury, K., & Stretslov, A. (2019). Mapping solar array location, size, and capacity using deep learning and overhead imagery. Preprint at https://arxiv. org/abs/1902.10895.
11. Kasmi, G., Saint-Drenan, Y. M., Trebosc, D., Jolivet, R., Leloux, J., Sarr, B., & Dubus, L. (2023). A crowdsourced dataset of aerial images with annotated solar photovoltaic arrays and installation metadata. Scientific Data 10(1), 59.
12. Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision (pp. 618-626).
13. Fel, T., Cadène, R., Chalvidal, M., Cord, M., Vigouroux, D., & Serre, T. (2021). Look at the variance! efficient black-box explanations with sobol-based sensitivity analysis. Advances in neural information processing systems, 34, 26005-26014.
14. Mallat, S. G. (1989). A theory for multiresolution signal decomposition: the wavelet representation. IEEE transactions on pattern analysis and machine intelligence, 11(7), 674-693.
15. Kasmi, G., Dubus, L., Saint-Drenan, Y. M., & Blanc, P. (2023). Can We Reliably Improve the Robustness to Image Acquisition of Remote Sensing of PV Systems?. In Tackling Climate Change with Machine Learning workshop at NeurIPS 2023.
16. Hendrycks, D., Mu, N., Cubuk, E. D., Zoph, B., Gilmer, J., & Lakshminarayanan, B. (2019). Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv:1912.02781.
17. Cubuk, E. D., Zoph, B., Shlens, J., & Le, Q. V. (2020). Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops (pp. 702-703).
18. Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., & Le, Q. V. (2018). Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501.
19. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).
20. Trémenbert, Y., Kasmi, G., Dubus, L., Saint-Drenan, Y. M., & Blanc, P. (2023). PyPVRoof: a Python package for extracting the characteristics of rooftop PV installations using remote sensing data. arXiv preprint arXiv:2309.07143.
21. Edun, A. S., Perry, K., Harley, J. B., & Deline, C. (2021). Unsupervised azimuth estimation of solar arrays in low-resolution satellite imagery through semantic segmentation and Hough transform. Applied Energy, 298, 117273.
22. So, B., Nezin, C., Kaimal, V., Keene, S., Collins, L., Bradbury, K., & Malof, J. M. (2017). Estimating the electricity generation capacity of solar photovoltaic arrays using only color aerial imagery. In 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS) (pp. 1603-1606). IEEE.
23. Rausch, B., Mayer, K., Arlt, M. L., Gust, G., Staudt, P., Weinhardt, C., ... & Rajagopal, R. (2020). An enriched automated PV registry: Combining image recognition and 3D building data. arXiv preprint arXiv:2012.03690.
24. Dobos, A. P. (2014). PVWatts version 5 manual (No. NREL/TP-6A20-62641). National Renewable Energy Lab.(NREL), Golden, CO (United States).
25. Qu, Z., Oumbe, A., Blanc, P., Espinar, B., Gesell, G., Gschwind, B., ... & Wald, L. (2017). Fast radiative transfer parameterisation for assessing the surface solar irradiance: The Heliosat-4 method. Meteorologische Zeitschrift, 26(1), 33-57.