Background: Zoology’s dark matter comprises hyperdiverse, poorly known taxa that are numerically dominant but largely unstudied, even in temperate regions where charismatic taxa are well understood. Dark taxa are everywhere, but high diversity, abundance, and small size have historically stymied their study. We demonstrate how entomological dark matter can be elucidated using high-throughput DNA barcoding (“megabarcoding”). We reveal the high abundance and diversity of scuttle flies (Diptera: Phoridae) in Sweden using 31,800 specimens from 37 sites across four seasonal periods. We investigate the number of scuttle fly species in Sweden and the environmental factors driving community changes across time and space. Results: Swedish scuttle fly diversity is much higher than previously known, with 549 putative specie) detected, compared to 374 previously recorded species. Hierarchical Modelling of Species Communities reveals that scuttle fly communities are highly structured by latitude and strongly driven by climatic factors. Large dissimilarities between sites and seasons are driven by turnover rather than nestedness. Climate change is predicted to significantly affect the 47% of species that show significant responses to mean annual temperature. Results were robust regardless of whether haplotype diversity or species-proxies were used as response variables. Additionally, species-level models of common taxa adequately predict overall species richness. Conclusions: Understanding the bulk of the diversity around us is imperative during an era of biodiversity change. We show that dark insect taxa can be efficiently characterised and surveyed with megabarcoding. Undersampling of rare taxa and choice of operational taxonomic units do not alter the main ecological inferences, making it an opportune time to tackle zoology’s dark matter.

New, rapid, accurate, scalable, and cost-effective species discovery and delimitation methods are needed for tackling “dark taxa,” here defined as groups for which <<10%% of all species are described and the estimated diversity exceeds 1,000 species. Species delimitation for these taxa should be based on multiple data sources (“integrative taxonomy”) but collecting multiple types of data risks impeding a discovery process that is already too slow. We here develop large-scale integrative taxonomy (LIT), an explicit method where preliminary species hypotheses are generated based on inexpensive data that can be obtained quickly and cost-effectively. These hypotheses are then evaluated based on a more expensive type of “validation data” that is only obtained for specimens selected based on objective criteria applied to the preliminary species hypotheses. We here use this approach to sort 18,000 scuttle flies (Diptera: Phoridae) into 315 preliminary species hypotheses based on next-generation sequencing barcode (313 bp) clusters (using objective clustering [OC] with a 3%% threshold). These clusters are then evaluated with morphology as the validation data. We develop quantitative indicators for predicting which barcode clusters are likely to be incongruent with morphospecies by randomly selecting 100 clusters for in-depth validation with morphology. A linear model demonstrates that the best predictors for incongruence between barcode clusters and morphology are maximum p-distance within the cluster and a newly proposed index that measures cluster stability across different clustering thresholds. A test of these indicators using the 215 remaining clusters reveals that these predictors correctly identify all clusters that are incongruent with morphology. In our study, all morphospecies are true or disjoint subsets of the initial barcode clusters so that all incongruence can be eliminated by varying clustering thresholds. This leads to a discussion of when a third data source is needed to resolve incongruent grouping statements. The morphological validation step in our study involved 1,039 specimens (5.8%% of the total). The formal LIT protocol we propose would only have required the study of 915 (5.1%%⁠: 2.5 specimens per species), as we show that clusters without signatures of incongruence can be validated by only studying two specimens representing the most divergent haplotypes. To test the generality of our results across different barcode clustering techniques, we establish that the levels of incongruence are similar across OC, Automatic Barcode Gap Discovery (ABGD), Poisson Tree Processes (PTP), and Refined Single Linkage (RESL) (used by Barcode of Life Data System to assign Barcode Index Numbers [BINs]). OC and ABGD achieved a maximum congruence score with the morphology of 89%% while PTP was slightly less effective (84%%⁠). RESL could only be tested for a subset of the specimens because the algorithm is not public. BINs based on 277 of the original 1,714 haplotypes were 86%% congruent with morphology while the values were 89%% for OC, 74%% for PTP, and 72%% for ABGD.

Most of the unknown animal biodiversity on earth is in groups of invertebrates that are hyperdiverse and abundant, yet poorly known (“dark taxa”). The study of these organisms requires a multi-faceted approach and methodologies designed to tackle large numbers of species and specimens. The scuttle flies (Diptera: Phoridae) are a classic example of a dark taxon and the focus of this thesis. Paper I is a molecular phylogeny of the phorid genus Megaselia based on one nuclear (28S rDNA) and three mitochondrial (ND1, COI and 16S) markers from 145 species of Nordic Megaselia. Molecular data was analysed with Bayesian analysis, maximum likelihood, and parsimony methods. Based on these results, and supporting morphological data, we divide Megaselia into 22 informal species groups, 20 of which fall into a monophyletic “core Megaselia”. We discuss implications for the future circumscription of Megaselia and associated genera. Paper II presents a pipeline for rapid and cost-effective species discovery using the Oxford Nanopore mobile sequencing technology MinION. This paper reveals the presence of ca. 650 species of Phoridae from a single Malaise trap placed in Kibale National Park, Uganda. Based on our data, we estimate that the phorid fauna of the Afrotropical region could be as high as 100 000 species: this figure dwarfs previous diversity estimates. The implications for species discovery and description are discussed, and a new species (Megaselia sepsioides sp. nov.) is described. Paper III outlines a large-scale integrative approach to species discovery and delimitation in hyperdiverse groups, exemplified using a dataset of 18 000 phorid flies from Sweden. COI minibarcodes (313 bp) were obtained for all specimens and classified into putative species using different clustering methods (objective clustering, Poisson tree process, automatic barcode gap discovery and refined single linkage). No clustering method was accurate enough to use for species delimitation without confirmation from additional data. We found that the stability of a cluster to change across genetic-distance thresholds and the genetic variation within a cluster both accurately predict clusters where morphology is likely to be incongruent with barcode data. With molecular clustering integrated with morphological validation, we found that we could examine less than 5% of specimens and still delimit all species fully and accurately. Paper IV addresses questions about the scuttle fly fauna of Sweden with data from 32 000 scuttle flies from 37 sites and 4 time periods. We estimate that the total Swedish fauna contains 652-713 (based on Chao 1 or CNE estimates, respectively) species of scuttle flies, 1.5 times the 372 species currently documented from Sweden. Ordination techniques show that scuttle fly communities are organized in a gradient across Sweden, which is well correlated with plant hardiness zones defined by the Swedish Horticultural Society. Hierarchical modelling of species communities (HMSC) reveals that phorid community composition is largely determined by climatic and temporal variables, but much of the variance remains unexplained by the models we explored. Comparison of our phorid data with that of species more commonly utilised for biodiversity assessments revealed that phorids may allow more fine-scaled analysis as they may exist in smaller ranges, and that they additionally may give unique patterns of distribution that are unlike those seen in other taxa.

The genus Megaselia Rondani (Diptera: Phoridae) is one of the largest in the animal kingdom, with nearly 1700 described species and many remaining to be discovered. Work on this group is notoriously challenging due to the extreme species diversity, poor knowledge of higher‐level relationships and lack of molecular data. In this paper, we present the largest study to date of Megaselia relationships based on molecular data from one nuclear (28S rDNA) and three mitochondrial (ND1, COI and 16S) markers for 175 Nordic specimens representing 145 species of Megaselia, plus outgroups. Based on phylogenetic analyses of these data, we propose 22 informal Megaselia species groups, all of which match well‐supported terminal clades. Relationships among these groups, and between them and several isolated species, remain largely uncertain. Of the 22, 20 species groups fall into a moderately well‐supported monophyletic clade of ‘core Megaselia’. Two species groups, the spinigera and ruficornis groups, fall outside of core Megaselia, as does the single representative of Myriophora, a genus that is included in Megaselia by some specialists. Here, we explore the morphology of these molecular species groups to aid future studies, and we discuss the implications of our findings for the generic circumscription of Megaselia. Hopefully, our results can aid further characterization of subgroups within the enormous Megaselia radiation and among its closest relatives, thus facilitating future work on this challenging but fascinating group of small flies.

The Swedish Malaise Trap Project (SMTP) is one of the most ambitious insect inventories ever attempted. The project was designed to target poorly known insect groups across a diverse range of habitats in Sweden. The field campaign involved the deployment of 73 Malaise traps at 55 localities across the country for three years (2003-2006). Over the past 15 years, the collected material has been hand sorted by trained technicians into over 300 taxonomic fractions suitable for expert attention. The resulting collection is a tremendous asset for entomologists around the world, especially as we now face a desperate need for baseline data to evaluate phenomena like insect decline and climate change. Here, we describe the history, organisation, methodology and logistics of the SMTP, focusing on the rationale for the decisions taken and the lessons learned along the way. The SMTP represents one of the early instances of community science applied to large-scale inventory work, with a heavy reliance on volunteers in both the field and the laboratory. We give estimates of both staff effort and volunteer effort involved. The project has been funded by the Swedish Taxonomy Initiative; in total, the inventory has cost less than 30 million SEK (approximately 3.1 million USD). Based on a subset of the samples, we characterise the size and taxonomic composition of the SMTP material. Several different extrapolation methods suggest that the material comprises around 20 million specimens in total. The material is dominated by Diptera (75% of the specimens) and Hymenoptera (15% of specimens). Amongst the Diptera, the dominant groups are Chironomidae (37% of specimens), Sciaridae (15%), Phoridae (13%), Cecidomyiidae (9.5%) and Mycetophilidae (9.4%). Within Hymenoptera, the major groups are Ichneumonidae (44% of specimens), Diaprioidea (19%), Braconidae (9.6%), Platygastroidea (8.5%) and Chalcidoidea (7.9%). The taxonomic composition varies with latitude and season. Several Diptera and Hymenoptera groups are more common in non-summer samples (collected from September to April) and in the North, while others show the opposite pattern. About 1% of the total material has been processed and identified by experts so far. This material represents over 4,000 species. One third of these had not been recorded from Sweden before and almost 700 of them are new to science. These results reveal the large amounts of taxonomic work still needed on Palaearctic insect faunas. Based on the SMTP experiences, we discuss aspects of planning and conducting future large-scale insect inventory projects using mainly traditional approaches in relation to more recent approaches that rely on molecular techniques.

Background The phorid fly genus Megaselia Rondani is a large, poorly-known taxon whose species are found worldwide. New information A new species of Megaselia Rondani, M. simunorum, is described from both urban and rural sites in southern California. With a large area of white colour on the posterior part of the abdominal dorsum, it closely resembles the much more common species M. sulphurizona, but M. simunorum has distinctly thicker ventral setae on the abdomen and a differently-shaped white spot.

Background: More than 80% of all animal species remain unknown to science. Most of these species live in the tropics and belong to animal taxa that combine small body size with high specimen abundance and large species richness. For such clades, using morphology for species discovery is slow because large numbers of specimens must be sorted based on detailed microscopic investigations. Fortunately, species discovery could be greatly accelerated if DNA sequences could be used for sorting specimens to species. Morphological verification of such “molecular operational taxonomic units” (mOTUs) could then be based on dissection of a small subset of specimens. However, this approach requires cost-effective and low-tech DNA barcoding techniques because well-equipped, well-funded molecular laboratories are not readily available in many biodiverse countries.

Results: We here document how MinION sequencing can be used for large-scale species discovery in a specimen- and species-rich taxon like the hyperdiverse fly family Phoridae (Diptera). We sequenced 7059 specimens collected in a single Malaise trap in Kibale National Park, Uganda, over the short period of 8 weeks. We discovered > 650 species which exceeds the number of phorid species currently described for the entire Afrotropical region. The barcodes were obtained using an improved low-cost MinION pipeline that increased the barcoding capacity sevenfold from 500 to 3500 barcodes per flowcell. This was achieved by adopting 1D sequencing, resequencing weak amplicons on a used flowcell, and improving demultiplexing. Comparison with Illumina data revealed that the MinION barcodes were very accurate (99.99% accuracy, 0.46% Ns) and thus yielded very similar species units (match ratio 0.991). Morphological examination of 100 mOTUs also confirmed good congruence with morphology (93% of mOTUs; > 99% of specimens) and revealed that 90% of the putative species belong to the neglected, megadiverse genus Megaselia. We demonstrate for one Megaselia species how the molecular data can guide the description of a new species (Megaselia sepsioides sp. nov.).

Conclusions: We document that one field site in Africa can be home to an estimated 1000 species of phorids and speculate that the Afrotropical diversity could exceed 200,000 species. We furthermore conclude that low-cost MinION sequencers are very suitable for reliable, rapid, and large-scale species discovery in hyperdiverse taxa. MinION sequencing could quickly reveal the extent of the unknown diversity and is especially suitable for biodiverse countries with limited access to capital-intensive sequencing facilities.

The urban heat island effect is a worldwide phenomenon that has been linked to species distributions and abundances in cities. However, effects of urban heat on biotic communities are nearly impossible to disentangle from effects of land cover in most cases because hotter urban sites also have less vegetation and more impervious surfaces than cooler sites within cities. We sampled phorid flies, one of the largest, most biologically diverse families of true flies (Insecta: Diptera: Phoridae), at 30 sites distributed within the central Los Angeles Basin, where we found that temperature and the density of urban land cover are decoupled. Abundance, richness, and community composition of phorids inside urban Los Angeles were most parsimoniously accounted for by mean air temperature in the week preceding sampling. Sites with intermediate mean temperatures had more phorid fly individuals and higher richness. Communities were more even at urban sites with lower minimum temperatures and sites located further away from natural areas, suggesting that communities separated from natural source populations may be more homogenized. Species composition was best explained by minimum temperature. Inasmuch as warmer areas within cities can predict future effects of climate change, phorid fly communities are likely to shift nonlinearly under future climates in more natural areas. Exhaustive surveys of biotic communities within cities, such as the one we describe here, can provide baselines for determining the effects of urban and global climate warming as they intensify.