Abstract EN:

The assessment and characterization of the gut microbiome has become a focus of research in the area of human autoimmune diseases. Many diseases such as obesity, inflammatory bowel (IBD), lean or beses twins, colorectal cancers and so on (Qin et al. 2010; Turnbaugh et al. 2009) have already been found to be associated with changes in the human microbiome. To investigate these relationships, quantitative metagenomics (QM) studies based on sequencing data could be performed. Understanding the role of the microbiome in human health and how it can be modulated is becoming increasingly relevant for precision medicine and for the medical management of chronic diseases. Results from such QM studies which report the organisms present in the samples and profile their abundances, will be used for continuous analyses. The terms microbiome and microbiota are used indistinctly to describe the community of microorganisms that live in a given environment. The development of high-throughput DNA sequencing technologies has boosted microbiome research through the study of microbial genomes allowing a more precise quantification of microbial and functional abundance. However, microbiome data analysis is challenging because it involves high-dimensional structured multivariate sparse data and because of its compositional structure of microbiome data. The data preprocessing is typically implemented as a pipeline (workflow) with third-party software that each process input files and produces output files. The pipelines are often deep, with ten or more tools, which could be very diverse from different languages such as R, Python, Perl etc. and integrated into different frameworks (Leipzig 2017) such as Galaxy, Apache Taverna, Toil etc. The challenges with existing approaches is that they are not always efficient with very large datasets in terms of scalability for individual tools in a metagenomics pipeline and their execution speed also has not met the expectations of the bioinformaticians. To date, more and more data are captured or generated from many different research areas such as Physics, Climatology, Sociology, Remote sensing or Management as well as bioinformatics. Indeed, Big Data Analytics (BDA) describes the unprecedented growth of data generated and collected from all kinds of data sources as mentioned above. This growth could be in the volume of data, in the speed of data moving in/out or in the speed of analyzing data which depends on high-performance computing (HPC) technologies. In the past few decades since the invention of the computer, HPC has contributed significantly to our quality of life - driving scientific innovation, enhancing engineering design and consumer goods manufacturing, as well as strengthening national and international security. This has been recognised and emphasised by both government and industry, with major ongoing investments in areas encompassing weather forecasting, scientific research and development as well as drug design and healthcare outcomes. In many ways, those two worlds (HPC and big data) are slowly, but surely converging. They are the keys to overcome limitations of bioinformatics analysis in general and quantitative metagenomics analysis in particular. Within the scope of this thesis, we contributed a novel bioinformatics framework and pipeline called QMSpy which helped bioinformaticians overcome limitations related to HPC and big data domains in the context of quantitative metagenomics. QMSpy tackles two challenges introduced by large scale NGS data: (i) sequencing data alignment - a computation intensive task and (ii) quantify metagenomics objects - a memory intensive task. By leveraging the powerful distributed computing engine (Apache Spark), in combination with the workflow management of big data processing (Hortonwork Data Platform), QMSpy allows us not only to bypass [...]