Which aspects of the images presented to citizen scientists are important when considering their accuracy?
Particularly in phase 1, poor image quality significantly affected the results — this was most obvious in images collected from El Miedo on Santa Fe Island, which was also coincidentally the colony with the highest density and smallest iguanas within that phase and is thus the primary reason for the higher values of iguana undercounting here. This colony was surveyed as part of our pilot phase (January 2020), where our drone protocols were not optimal in terms of altitude and image overlap, producing comparably lower-quality images. Moreover, in this initial phase, we added a watermark to each image on Zooniverse, which negatively influenced the visibility of objects. Our image collection procedure has since significantly improved; this impact is evidenced by the results obtained in phases 2 and 3. For this reason, the two later phases are more representative of the approach and thus are used here to validate the method.
In addition to image quality, the general predominance of ‘blank’ images within the dataset was also an issue. This scarcity of images with iguanas undoubtedly reduced volunteer opportunities to ‘learn by doing’. This issue is common among projects where focal objects are rare, too small, or too similar to the background and may have contributed to the false negative error rate, particularly in phase 1 where the density of iguanas on one of the surveyed islands — San Cristobal — is extremely low. Undercounting is also common when several individuals are present in the image — generally, the more individuals per image, the lower the agreement between CS and the experts. However, our statistical analyses showed that volunteers can also count accurately when several iguanas are present, except in phase 1, where image quality was a compounding factor.
To address this question, we must consider both the participation rate of the volunteers — to assess whether analysing a dataset within a reasonable timeframe is likely — and the accuracy of the volunteer-generated data. In our cumulative participation curve, we experienced a daily classification rate of 200-9800; the higher-end numbers being in response to promotional work, in keeping with results from other projects. Crowdsourcing undoubtedly offers the opportunity to analyse large datasets with relatively little expert input, but without active promotion and engagement, it can take considerably longer to complete the analysis. Each phase of our project took between 5 and 14 months to complete. This timeframe was influenced by the size of the phase dataset, with an overall average of almost 1900 images being fully analysed each month (phase 1: 2031; phase 2: 1819; phase 3: 1740). This length of time was manageable for our purposes, and there are straightforward ways to speed up the process via promotion if time is pressing.
After applying the best-performing aggregation method to the CS data and omitting suboptimal data from the project’s pilot phase, we found that CS-counts were 91% and 92% accurate when compared to those of experts. This meets the criteria we defined as ‘accurate’ in terms of counts, and thus, we find the approach suitable for counting marine iguanas. Although expert counts in phases 2 and 3 are not significantly higher than those from the CS data, a tendency for volunteers to undercount is still evident. This indicates the need to calculate and apply a correction factor if CS inputs are used to estimate the population size.
Since CS projects rely entirely on volunteer inputs, it is important to consider the volunteer experience and factors motivating participants. A study by Aceves-Bueno et al. found that participants who received economic recognition outperformed those who did not. Whilst this type of ‘reward’ may be helpful, our experience here indicates that recognition for work undertaken is not a prerequisite for involvement. Our finding matches those projects where many participating volunteers remained anonymous (unregistered). Volunteers in these types of projects seem not to expect external rewards but may rather be motivated by the intrinsic desire to contribute to science/conservation, engage with researchers, and be involved in scientific discussion. Another pattern noted in previous work, confirmed in our project, is that most participants contribute few classifications, while few volunteers contribute many. In our case, we found several volunteers who classified thousands of images, including one who looked at all images within each phase. This finding was confirmed by the volunteers’ replies to our survey, where most users responding estimated themselves to have contributed up to 100 classifications. This finding makes a strong case for using multiple independent classifications for each image; this allows a dataset to be rapidly analysed even when each user’s contribution is small.
It is also important to consider factors that reduce volunteer motivation; in our case, the infrequency of marine iguanas in the images seems to have been important, as reported by the volunteers (supplementary Fig. S6). Interestingly, other researchers have found that such ‘blank’ images motivated the volunteers to keep looking for the target, and when blanks were removed, their participation time decreased; though it is worth noting that this study did not explicitly test the number of classifications made in relation to the proportion of blank images. In our dataset, where 90% of our images were blank, volunteers contributing only a few classifications may not have seen any iguanas and may have also spent a large amount of time attempting to distinguish these objects from a visually similar substrate.
Image-based datasets for wildlife monitoring are increasingly used, in great part due to technological advances that have made devices — such as camera traps and drones — more affordable and more suitable for such work. However, although these approaches can reduce survey time, the effort required for image analysis can constitute a considerable burden for projects with limited resources. Online citizen science projects — which involve the collaboration of the general public to analyse images remotely — are helping to resolve this issue, offering key advantages such as increased cost-effectiveness and a significant reduction in workload.
Despite the historical debate regarding the accuracy of citizen science-generated dataonline CS is now recognized as an important approach for large-scale ecological research. Multiple studies have found that researchers can obtain accurate volunteer data by properly aggregating multiple independent responses for one subject (e.g. an image or audio file). In parallel, researchers may also use CS to engage and educate the public on themes related to science and conservation. Studies have shown that citizen scientists participating in nature-focused projects tend to develop positive environmental attitudes. Therefore, involving the public may benefit the project and help the scientific community, aid in public engagement and education, and increase interest in topics related to biodiversity and the environment.
One big draw of the CS approach is that by crowdsourcing, researchers can meet their aims in less time and/or expand their aims past what would be possible using more traditional approaches. A review across 17 CS projects estimated that the CS approach allowed analysis to be completed on average within 2.4 years per project, as opposed to the 37 years estimated if experts classified the data.
Large-scale monitoring in the Galápagos is logistically extremely challenging and is, therefore, rare. For most of the species, the majority of studies focus on just a few colonies. For the marine iguana, surveying the whole range of the species is only realistic when new approaches — such as drone-based surveying — are applied. Still, analysis of the large datasets generated remains a significant obstacle to the completion of this work. Here, we confirm that aerial images have the potential to provide reliable data from volunteers with little training, indicating a reasonable approach to alleviate the analysis bottleneck.
Moreover, the images can address numerous questions about other taxa and the environment. Apart from the tasks related to the marine iguanas, we also asked the volunteers to classify other species and detect plastic objects; this is data we could easily collect alongside our tasks, which will be made available to other researchers as a contribution to their work. Combining an image-based method for surveying remote areas with crowdsourced analysis via online CS can be a valuable approach in collecting and analysing large datasets.
Our results validate the use of the citizen science approach to accurately identify and count marine iguanas from aerial images. However, there is a tendency to undercount the number of iguanas. Our next step is to continue our analysis, which includes images from the most populated colonies, to identify a correction factor that will allow CS inputs to be used for accurate population-size estimates of marine iguanas. This is possible because experts have already validated the counting of iguanas from aerial imagery against traditional ground-based approaches. This work is an essential contribution to our overall goal of addressing the population-size data-gap that currently hampers the effective conservation of this species.
In future work, we expect to use our images to analyse reproduction dynamics, and potentially habitat characteristics, using a CS approach. We are interested to see whether volunteers can reliably identify certain aspects of the colonies, such as the presence of leks and males with breeding colouration; these data will be helpful to address the dearth of information regarding marine iguana breeding activity across the archipelago.
Our next major goal is to use machine learning (ML) to analyse drone imagery. As with several other projects CS-input is being used to train Artificial Intelligence for pattern recognition and minimize training time for the computers. We also expect to use ML to filter data, enabling us to remove blank images from our CS datasets and focus this human effort on the most important images. We envisage that this will improve volunteer participation and decrease the running time of the online project. By combining CS with ML, we aim to create a semi-automated pipeline capable of finding and counting marine iguanas and other biologically relevant objects in drone images, significantly reducing the effort needed to undertake such work.
We collected the aerial images used for this project using commercial drones (DJI Mavic 2 Pro), flown from land and boats along the rocky coastline of several marine iguana colonies in the Galápagos Archipelago during three successive field seasons from 2020 to 2022 (Fig. 3). With the images, we created orthomosaics (2D-georeferenced maps) using the software Agisoft Metashape; for full details on image collection and analysis, see Varela-Jaramillo et al.. We ‘sliced’ the orthomosaics using Adobe Photoshop to create individual images of 1,000 × 1,000 pixels on average (resulting image size: up to 1 MB). We did this in an attempt to standardize the size of the individual iguanas in the images to aid recognition, as this depends on the height at which we flew the drones (20-30 m altitude), which varied in response to the body size of each subspecies of marine iguanas monitored.
We created our project on the Zooniverse platform in English, with translations available in Spanish, French, and German. Our workflow for the project requires volunteers to complete three tasks for each image classification (Fig. 4): (1) identify presence or absence of marine iguanas in the image; (2) mark individuals of marine iguanas, distinguishing adult males and reproductive groups (leks) from ‘others’ (females, sub-adult males, and juveniles) when possible. The category ‘partial iguana’ was an addition made to avoid double-counting of occasional individuals that were bisected at the edge of the image during slicing (this was explicitly explained to the volunteers via the training on Zooniverse); and (3) identify and count individuals of cohabiting species, which included sea lions (Zalophus wollebaeki), crabs (Grapsus grapsus), Green Turtles (Chelonia mydas), sea birds, plants, and algae (various species), as well as plastic objects (bottles and fishing gear). We provide a tutorial and a ‘Field Guide’ for species identification. A message board for discussion is enabled where volunteers interact with researchers and each other regarding image classification; this also provides a space for ongoing discussion on Galápagos wildlife and conservation matters.
To date, we have launched five phases of the Zooniverse project, each following an annual field trip undertaken between 2020 and 2023, during which we surveyed all islands within the mating season of the marine iguana, when the iguanas aggregate in breeding territories. For this study, we analysed the first three complete phases. Analysis of phase 1 began in August 2020 and comprised 24,373 images from San Cristobal and Santa Fe islands surveyed as a pilot project in January 2020. The three subspecies covered have medium to large body sizes and relatively low densities. Phase 2 — launched in February 2022 — included 9,097 images from Española and Floreana islands, collected in January 2021. This phase featured the more colourful and abundant “Christmas iguana”, which has a medium to large body size and is renowned for the turquoise colouration displayed by males during the mating season. Phase 3 was launched in July 2022 with 24,368 images from the northern islands of Genovesa, Marchena, and Pinta, surveyed in December 2021. Marine iguanas on these islands are rare, small-bodied, highly cryptic against the rocky substrate, and males appear to lack the colouration seen on other islands in the pairing season. Two sites from Phase 2 were also included here.
We circulated each image among a predefined number of independent volunteers, whose input is referred to as a classification. We required 20 classifications per image for phases 1 and 2, and 30 classifications for phase 3. The extra 10 classifications in the latter phase were intended to test whether more classifications per image would improve CS accuracy.
To increase volunteer participation, we undertook promotional activities including press releases, newsletters, inviting inputs from schools, universities and companies via webinars, social media and blog posts, as well as participating in the Citizen Science Month event promoted by SciStarter (https://scistarter.org/iguanas-from-above) and being a featured Zooniverse project. We also ran a competition to award the best classifiers in phase 2 to motivate the volunteers.
We used RStudio version 2023.09.1 + 494 and Python version 3.10.13 with the library Pandas version 2.1.2 for data frame management, ensuing analyses and plotting of results. We downloaded CS-classifications from the Zooniverse platform and used the Panoptes Aggregation python package (See Code Availability for Panoptes script) to extract and summarise (‘aggregate’) data from Task 1 (marine iguana presence/absence – question-type data) and to extract data from Task 2 (marine iguana counts – point-type data). We randomly selected around 5-10% of the images per phase as a Gold-Standard (GS) dataset; these images were chosen to cover a range of challenges, including a variety of iguana body sizes and colouration, both high- and low-density colonies, and from various fieldwork years (since quality of images improved throughout the project). This included 4,345 images from 30 colonies selected from 7 major islands: 2,733 images from phase 1, 456 images from phase 2 and 1,156 images from phase 3. Three people from the research team (henceforth referred to as ‘experts’) analysed the GS datasets and generated consensus results (henceforth called ‘expert’ data) for presence/absence, number of iguanas in the image, and a judgement on image quality (simply “good” or “bad”, Fig. 5a). Image quality judgement was based on the sharpness of camera focus, light levels within the image, image blur due to camera movement, image ‘smear’ due to mosaicking artefacts, and complexity of the substrate on which the animals occur. The expert consensus count was obtained using a two-step process. First, two experts analysed the images independently, then compared results. In cases of disagreement, images were discussed in person to attempt an agreement over whether an object in question was an iguana or not. If an agreement was not reached, the images were then analysed by a third expert. Of the ‘uncertain’ iguanas, only those that were then confirmed by the third expert were accepted as correct. From 4,345 GS images, experts disagreed in 33 images (0.8%), in most cases this was due to an “iguana” marked by only one expert, and therefore these uncertain iguanas were not included in the consensus. We then compared this ‘expert data’ to the CS data for each phase, and for all images together.
In the online task, each volunteer is asked whether marine iguanas are present (‘yes’) or absent (‘no’) in the image; therefore, for each image in our dataset, we obtained 20 (phase 1 & 2) or 30 (phase 3) answers. A commonly used approach for analysing results where multiple volunteers give input is the ‘simple plurality algorithm’ (also known as ‘majority vote’). This is where the answer selected by 50% plus one of the volunteers for any given task is accepted for this image. For example, in our case, where 20 volunteers classified an image and 11 or more selected ‘yes’ to indicate presence of an iguana, the result would be ‘iguana(s) present’. We sought a minimum agreement level of > 95% with the expert classifications. We were interested in testing the performance of the ‘majority vote’ approach, while specifically seeking to test whether a smaller number of volunteers could also give accurate results. For this purpose, we calculated the agreement between CS and expert 11 times for each image, determining an agreement level for the case where one of 20 volunteers selected ‘yes’, then two of 20, and so on up to 11 of 20. The aim was to find a ‘minimum threshold’ for the smallest number of volunteers indicating iguana presence in (95%+) agreement with the experts (Fig. 5b; see Code Availability for Minimum Threshold Search script).
We also analysed if removing anonymous (i.e. users not logged into the platform) inputs increased volunteer accuracy in identifying marine iguanas to explore whether project loyalty and volunteer experience might affect the reliability of inputs. Likewise, we tested how removing inputs from volunteers with 10 or fewer classifications affected accuracy, since we hypothesised that these infrequent participants have less experience at the task and thus may be less skilled.
We analysed whether, after selecting yes for marine iguana presence, volunteers were able to detect all the marine iguanas present in the image. In this task, volunteers added marks to the image where they detected the iguanas (Fig. 4). To analyse these outputs, we first selected all images where iguanas were present (obtained using the ‘minimum threshold’ rule) and aggregated the volunteer counts for each (Fig. 5c). We calculated two statistical metrics to aggregate these counts: the median and the mode. The median searches for the value in the middle of an ordered data sample, while the mode seeks the most frequently repeated value within the sample, thus eliminating the outlier effect.
Additionally, we tested the Density-Based Spatial Clustering of Applications with Noise algorithm (DBSCAN) and the Hierarchical Density-Based Spatial Clustering of Applications with Noise algorithm (HDBSCAN) to collate the volunteer annotation marks into spatial clusters of points within the image. The number of clusters then represents the number of iguanas present in the image (see supplementary Fig. S7 and Text S1 for further details on aggregating methods and Code Availability for Clustering script). We excluded data for the ‘partial iguana’ category (i.e. those bisected by the image) since they were very rare and thus, we lack sufficient data for analysis (e.g. only 11 cases in phase 1). From these methods, we investigated volunteer accuracy by comparing counts between experts and CS for each image. We summed all GS aggregated counts by the method used, and compared these to the expert counts, calculating a percentage of agreement (‘accuracy’), as well as the number of iguanas missed (undercounted) and over-counted. We also looked for: the number of images where volunteer counts were in 100% agreement with expert counts; where volunteers counted fewer iguanas than the experts; and where CS counts were higher than experts.
We statistically explored — using generalised linear models with a quasi-poisson error matrix and Chi-square tests — how similar the counts in the GS images were between the experts and the volunteers (comparing median, mode, and HDBSCAN). DBSCAN was excluded here due to poor initial results that were related to higher percentages of overcounting. Subsequently, we performed pairwise comparisons using the Estimated Marginal Means R package (emmeans) to test which of the volunteer counting methods produced results most similar to the expert counts (Fig. 5d). The emmeans package allows pairwise comparisons from generalised linear models on non-normally distributed data, such as the counts obtained in this study. Further, a logistic regression was performed to find the CS aggregation method which best fits the expert data.
We also investigated the effect of three factors on volunteer counts using the same statistical analyses. These were: phase analysed; image quality (see GS methods for criteria); and the number of iguanas present on the image (three categories were defined based on data distribution: Low: 1-5, medium: 6-10, and high: >10 marine iguanas; supplementary Fig. S8; see Code Availability for Statistical Comparisons script).
To better understand the volunteers’ experiences, we undertook a short online survey. These included questions about the number of images classified, factors affecting their motivation to participate, and differences they perceived between the phases (see supplementary Text S2 for details). These results were explored to better understand the relationship between volunteer perceptions and our results and were also used to improve our ongoing CS work.
