This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision Next revision Both sides next revision | ||
|
teaching:mfe:is [2016/04/01 16:30] svsummer |
teaching:mfe:is [2019/05/13 12:27] mahmsakr [Visualizing spatiotemporal data] |
||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ====== MFE 2016-2017 : Web and Information Systems ====== | + | ====== MFE 2019-2020 : Web and Information Systems ====== |
| ===== Introduction ===== | ===== Introduction ===== | ||
| Line 18: | Line 18: | ||
| ===== Master Thesis in Collaboration with Euranova ===== | ===== Master Thesis in Collaboration with Euranova ===== | ||
| - | Our laboratory performs collaborative research with Euranova R&D (http://euranova.eu/). The list of subjects proposed for this year by Euranova can be found | + | Our laboratory performs collaborative research with Euranova R&D (http://euranova.eu/). The list of subjects proposed for this year by Euranova can be found [[https://research.euranova.eu/wp-content/uploads/proposals-thesis-2019.pdf|here]]. |
| - | {{:teaching:mfe:master_thesis_euranova_2015.pdf|here}} | + | |
| These subject include topics on distributed graph processing, processing big data using Map/Reduce, cloud computing, and social networks. | These subject include topics on distributed graph processing, processing big data using Map/Reduce, cloud computing, and social networks. | ||
| Line 25: | Line 25: | ||
| * Contact : [[ezimanyi@ulb.ac.be|Esteban Zimanyi]] | * Contact : [[ezimanyi@ulb.ac.be|Esteban Zimanyi]] | ||
| - | ===== Complex Event Processing in Apache Spark and Apache Storm ===== | ||
| - | The master thesis is put forward in the context of the SPICES "Scalable Processing and mIning of Complex Events for Security-analytics" research project, funded by Innoviris. | ||
| - | Within this project, our lab is developping a declarative language for Complex Event Processing (CEP for short). The goal in Complex Event Processing is to derive pre-defined patterns in a stream of raw events. Raw events are typically sensor readings (such as "password incorrect for user X trying to log in on machine Y" or "file transfer from machine X to machine Y"). The goal of CEP is then to correlate these events into complex events. For example, repeated failed login attempts by X to Y should trigger a complex event "password cracking warning" that refers to all failed login attempts. | + | ===== Dynamic Query Processing on GPU Accelerators ===== |
| - | The objective of this master thesis is to build an interpreter/compiler for this declarative CEP language that targets the distributed computing frameworks Apache Spark and/or Apache Storm as backends. Getting aquaintend with these technologies is part of the master thesis objective. | + | This master thesis is put forward in the context of the DFAQ Research Project: "Dyanmic Processing of Frequently Asked Queries", funded by the Wiener-Anspach foundation. |
| + | |||
| + | Within this project, our lab is hence developing novel ways for processing "fast Big Data", i.e., processing of analytical queries where the underlying data is constantly being updated. The analytics problems envisioned cover wide areas of computer science and include database aggregate queries, probabilistic inference, matrix chain computation, and building statistical models. | ||
| + | |||
| + | The objective of this master thesis is to build upon the novel dynamic processing algorithms being developed in the lab, and complement these algorithms by proposing dynamic evaluation algorithms that execute on modern GPU architectures, thereby exploiting their massive parallel processing capabilities. | ||
| + | |||
| + | Since our current development is done in the Scala programming language, prospective students should either know Scala, or being willing to learn it within the context of the master thesis. | ||
| + | |||
| + | |||
| + | **Validation of the approach** Validation of master thesis' work should be done on two levels: | ||
| + | * a theoretical level; by proposing and discussing alternative ways to do incremental computation on GPU architectures, and comparing these from a theoretical complexity viewpoint | ||
| + | * an experimental level; by proposing a benchmark collection of CEP queries that can be used to test the obtained versions of the interpreter/compiler, and report on the experimentally observed performance on this benchmark. | ||
| - | **Validation of the approach** Validation of the proposed interpreter/compiler should be done on two levels: | ||
| - | * a theoretical level; by comparing the generated Spark/Storm processors to a processor based on "Incremental computation" that is being developped at the lab | ||
| - | * an experimental level; by proposing a benchmark collection of CEP queries that can be used to test the obtained interpreter/compiler, and report on the experimentally observed performance on this benchmark. | ||
| **Deliverables** of the master thesis project | **Deliverables** of the master thesis project | ||
| - | * An overview of the processing models of Spark and Storm | + | * An overview of query processing on GPUs |
| - | * A definition of the declarative CEP language under consideration | + | * A definition of the analytics queries under consideration |
| - | * A description of the interpretation/compilation algorithm | + | * A description of different possible dynamic evaluation algorithms for the analytical queries on GPU architectures. |
| - | * A theoretical comparison of this algorithm wrt an incremental evaluation algorithm. | + | * A theoretical comparison of these possibilities |
| - | * The interpreter/compiler itself (software artifact) | + | * The implementaiton of the evaluation algorithm(s) (as an interpreter/compiler) |
| - | * A benchmark set of CEP queries and associated data sets for the experimental validation | + | * A benchmark set of queries and associated data sets for the experimental validation |
| * An experimental validation of the compiler, and analysis of the results. | * An experimental validation of the compiler, and analysis of the results. | ||
| - | **Interested?** | + | |
| - | * Contact : [[stijn.vansummeren@ulb.ac.be|Stijn Vansummeren]] | + | **Interested?** Contact : [[svsummer@ulb.ac.be|Stijn Vansummeren]] |
| **Status**: available | **Status**: available | ||
| + | ===== Multi-query Optimization in Spark ===== | ||
| + | |||
| + | Distributed computing platforms such as Hadoop and Spark focus on addressing the following challenges in large systems: (1) latency, (2) scalability, and (3) fault tolerance. Dedicating computing resources for each application executed by Spark can lead to a waste of resources. Unified distributed file systems such as Alluxio has provided a platform for computing results among simultaneously running applications. However, it is up to the developers to decide on what to share. | ||
| + | |||
| + | The objective of this master thesis is to optimize various applications running on a Spark platform, optimize their execution plans by autonomously finding sharing opportunities, namely finding the RDDs that can be shared among these applications, and computing these shared plans once instead of multiple times for each query. | ||
| + | |||
| + | **Deliverables** of the master thesis project | ||
| + | * An overview of the Apache Spark architecture. | ||
| + | * Develop a performance model for queries executed by Spark. | ||
| + | * An implementation that optimizes queries executed by Spark and identify sharing opportunities. | ||
| + | * An experimental validation of the developed system. | ||
| + | |||
| + | **Interested?** Contact : [[svsummer@ulb.ac.be|Stijn Vansummeren]] | ||
| + | |||
| + | **Status**: available | ||
| ===== Graph Indexing for Fast Subgraph Isomorphism Testing ===== | ===== Graph Indexing for Fast Subgraph Isomorphism Testing ===== | ||
| Line 62: | Line 84: | ||
| **Status**: available | **Status**: available | ||
| - | ===== A Scala-based runtime and compiler for Distributed Datalog ===== | ||
| - | Datalog is a fundamental query language in datamanagement based on logic programming. It essentially extends select-from-where SQL queries with recursion. There is a recent trend in data management research to use datalog to specify distributed applications, most notably on the web, as well as do inference on the semantic web. The goal of this thesis is to engineer a basic **distributed datalog system**, i.e., a system that is capable of compiling & running distributed datalog queries. The system should be implemented in the Scala programming language. Learning Scala is part of the master thesis project. | + | =====Extending SPARQL for Spatio-temporal Data Support===== |
| - | The system should: | + | [[http://www.w3.org/TR/rdf-sparql-query/|SPARQL]] is the W3C standard language to query RDF data over the semantic web. Although syntactically similar to SQL, SPARQL is based on graph matching. In addition, SPARQL is aimed, basically, to query alphanumerical data. |
| - | * incorporate recently proposed worst-case join algorithms (i.e., the [[http://arxiv.org/abs/1210.0481|leapfrog trie join]]) | + | Therefore, a proposal to extend SPARQL to support spatial data, called [[http://www.opengeospatial.org/projects/groups/geosparqlswg/|GeoSPARQL]], has been presented to the Open Geospatial Consortium. |
| - | * employ known local datalog optimizations (such as magic sets and QSQ) | + | |
| + | In this thesis we propose to (1) perform an analysis of the current proposal for GeoSPARQL; (2) a study of current implementations of SPARQL that support spatial data; (3) implement simple extensions for SPARQL to support spatial data, and use these language in real-world use cases. | ||
| + | |||
| - | **Validation of the approach** The thesis should propose a benchmark collection of datalog queries and associated data workloads that be used to test the obtained system, and measure key performance characteristics (elasticity of the system; memory frootprint; overall running time, ...) | + | * Contact: [[ezimanyi@ulb.ac.be|Esteban Zimányi]] |
| - | **Required reading**: | ||
| - | * Datalog and Recursive Query Processing - Foundations and trends in query processing. | ||
| - | * LogicBlox, Platform and Language: A Tutorial (Todd J. Green, Molham Aref, and Grigoris Karvounarakis) | ||
| - | * Dedalus: Datalog in Time and Space (Peter Alvaro, William R. Marczak, Neil Conway, Joseph M. Hellerstein, David Maier, and Russell Sears) | ||
| - | * Declarative Networking (Loo et al). For the distributed evaluation strategy. | ||
| - | * Parallel processing of recursive queries in distributed architectures (VLDB 1989) | ||
| - | * Evaluating recursive queries in distributed databases (IEEE trans knowledge and data engieneering, 1993) | ||
| - | **Deliverables**: | + | ====== MFE 2019-2020 : Spatiotemporal Databases ====== |
| - | * Semantics of datalog; overview of known optimization strategies (document) | + | Moving object databases (MOD) are database systems that can store and manage moving object data. A moving object is a value that changes over time. It can be spatial (e.g., a car driving on the road network), or non-spatial (e.g., the temperature in Brussels). Using a variety of sensors, the changing values of moving objects can be recorded in digital formats. A MOD, then, helps storing and querying such data. A couple of prototypes have also been proposed, some of which are still active in terms of new releases. Yet, a mainstream system is by far still missing. Existing prototypes are merely research. By mainstream we mean that the development builds on widely accepted tools, that are actively being maintained and developed. A mainstream system would exploit the functionality of these tools, and would maximize the reuse of their ecosystems. As a result, it becomes more closer to end users, and easily adopted in the industry. |
| - | * Description of the leapfrog trie join (document) | + | |
| - | * Datalog system (software artifact) | + | In our group, we are building MobilityDB, a mainstream MOD. It builds on PostGIS, which is a spatial database extension of PostgreSQL. MobilityDB extends the type system of PostgreSQL and PostGIS with ADTs for representing moving object data. It defines, for instance, the tfloat for representing a time dependant float, and the tgeompoint for representing a time dependant geometry point. MobilityDB types are well integrated into the platform, to achieve maximal reusability, hence a mainstream development. For instance, the tfloat builds on the PostgreSQL double precision type, and the tgeompoint build on the PostGIS geometry(point) type. Similarly MobilityDB builds on existing operations, indexing, and optimization framework. |
| - | * Experimental analysis of developped system on a number of use cases (document) | + | |
| + | This is all made accessible via the SQL query interface. Currently MobilityDB is quite rich in terms of types and functions. It can answer sophisticated queries in SQL. The first beta version has been released as open source April 2019 (https://github.com/ULB-CoDE-WIT/MobilityDB). | ||
| + | |||
| + | The following thesis ideas contribute to different parts of MobilityDB. They all constitute innovative development, mixing both research and development. They hence will help developing the student skills in: | ||
| + | * Understanding the theory and the implementation of moving object databases. | ||
| + | * Understanding the architecture of extensible databases, in this case PostgreSQL. | ||
| + | * Writing open source software. | ||
| + | |||
| + | |||
| + | =====JDBC driver for Trajectories===== | ||
| + | An important, and still missing, piece of MobilityDB is Java JDBC driver, that will allow Java programs to establish connections with MobilityDB, and store and retrieve data. This thesis is about developing such a driver. As all other components of PostgreSQL, its JDBC driver is also extensible. This documentation gives a good explanation of the driver and the way it can be extended: | ||
| + | https://jdbc.postgresql.org/documentation/head/index.html | ||
| + | It is also helpful to look at the driver extension for PostGIS: | ||
| + | https://github.com/postgis/postgis-java | ||
| + | |||
| + | As MobilityDB build on top of PostGIS, the Java driver will need to do the same, and build on top of the PostGIS driver. Mainly the driver will need to provide Java classes to represent all the types of MobilityDB, and access the basic properties. | ||
| **Interested?** | **Interested?** | ||
| - | * Contact : [[stijn.vansummeren@ulb.ac.be|Stijn Vansummeren]] | + | * Contact : [[ezimanyi@ulb.ac.be|Esteban Zimanyi]] |
| **Status**: available | **Status**: available | ||
| + | =====Mobility data exchange standards===== | ||
| + | Data exchange standards allow different software systems to integrate together. Such standards are essential in the domain of mobility. Consider for example the case of public transportation. Different vehicles (tram, metro, bus) come from different vendors, and are hence equipped with different location tracking sensors. The tracking software behind these vehicle use different data formats. These software systems need to push real time information to different apps. To support the passengers, for example, there must be a mobile or a Web app to check the vehicle schedules and to calculate routes. This information shall also be open to other transport service providers and to routing apps. This is how google maps, for instance, is able to provide end to end route plans that span different means of transport. | ||
| + | The goal of this thesis is to survey the available mobility data exchange standards, and to implement in MobilityDB import/export functions for the relevant ones. Examples for these standards are: | ||
| + | * GTFS static, https://developers.google.com/transit/gtfs/ | ||
| + | * GTFS realtime, https://developers.google.com/transit/gtfs-realtime/ | ||
| + | * NeTEx static, http://netex-cen.eu/ | ||
| + | * SIRI, http://www.transmodel-cen.eu/standards/siri/ | ||
| + | * More standards can be found on http://www.transmodel-cen.eu/category/standards/ | ||
| - | =====Publishing and Using Spatio-temporal Data on the Semantic Web===== | ||
| - | [[http://www.w3c.org/|RDF]] is the [[http://www.w3c.org/|W3C]] proposed framework for representing information | + | **Interested?** |
| - | in the Web. Basically, information in RDF is represented as a set of triples of the form (subject,predicate,object). RDF syntax is based on directed labeled graphs, where URIs are used as node labels and edge labels. The [[http://linkeddata.org/|Linked Open Data]] (LOD) initiative is aimed at extending the Web by means of publishing various open datasets as RDF, setting RDF links between data items from different data sources. Many companies and government agencies are moving towards publishing data following the LOD initiative. | + | * Contact : [[ezimanyi@ulb.ac.be|Esteban Zimanyi]] |
| - | In order to do this, the original data must be transformed into Linked Open Data. Although most of these data are alphanumerical, most of the time they contained a spatial or spatio-temporal component, that must also be transformed. This can be exploited | + | |
| - | by application providers, that can build attractive and useful applications, in particular, for devices like mobile phones, tablets, etc. | + | |
| - | The goals of this thesis are: (1) study the existing proposals for mapping spatio-temporal data into LOD; (2) apply this mapping to a real-world case study (as was the case for the [[http://www.oscb.be/|Open Semantic Cloud for Brussels]] project; (3) Based on the produced mapping, and using existing applications like the [[http://linkedgeodata.org/|Linked Geo Data project]], build applications that make use of LOD for example, to find out which cultural events are taking place at a given time at a given location. | + | **Status**: available |
| - | + | ||
| - | * Contact: [[ezimanyi@ulb.ac.be|Esteban Zimányi]] | + | =====Visualizing spatiotemporal data===== |
| + | Data visualization is essential for understanding and presenting it. starting with the temporal point, which is the database representation of a moving point object. Typically, it is visualized in a movie style, as a point that moves over a background map. The numerical attributes of this temporal point, such as the speed, are temporal floats. These can be visualized as function curves from the time t to the value v. | ||
| - | =====Extending SPARQL for Spatio-temporal Data Support===== | + | The goal of this thesis is to develop a visualization tool for the MobilityDB temporal types. The architecture of this tool should be innovative, so that it will be easy to extend it with more temporal types in the future. should be This tool should be integrated as an extension of a mainstream visualization software. A good candidate is QGIS (https://www.qgis.org/en/site/). The choice is however left open as part of the survey. |
| - | [[http://www.w3.org/TR/rdf-sparql-query/|SPARQL]] is the W3C standard language to query RDF data over the semantic web. Although syntactically similar to SQL, SPARQL is based on graph matching. In addition, SPARQL is aimed, basically, to query alphanumerical data. | ||
| - | Therefore, a proposal to extend SPARQL to support spatial data, called [[http://www.opengeospatial.org/projects/groups/geosparqlswg/|GeoSPARQL]], has been presented to the Open Geospatial Consortium. | ||
| - | |||
| - | In this thesis we propose to (1) perform an analysis of the current proposal for GeoSPARQL; (2) a study of current implementations of SPARQL that support spatial data; (3) implement simple extensions for SPARQL to support spatial data, and use these language in real-world use cases. | ||
| - | |||
| - | * Contact: [[ezimanyi@ulb.ac.be|Esteban Zimányi]] | + | **Interested?** |
| - | + | * Contact : [[ezimanyi@ulb.ac.be|Esteban Zimanyi]] | |
| + | |||
| + | **Status**: available | ||