Deep Learning in Graph Domains for Sensorised Environments

Abstract

As our society moves swiftly towards an era where technology seamlessly integrates into our daily lives, our homes and cities are becoming increasingly sensorised. This change is fueled by advancements in artificial intelligence that facilitate harnessing the potential of smart environments. The main focus of this thesis is to investigate how Graph Neural Networks (GNNs) can be effectively applied to these environments, with a focus on those where humans and robots share the space. In these scenarios, integrating and exploiting data from multiple sources and analysing interactions between individuals, objects, sensors and robots is paramount. As the literature shows, GNNs have advantageous properties to process this kind of data when compared to more established deep learning approaches. This thesis presents a range of methods and applications in sensorised environments that leverage GNNs’ properties. The main contributions span applications in three main fields: human-aware robot navigation, human pose estimation, and the generation of traffic images. For human-aware navigation, this thesis proposes a model capable of estimating the level of discomfort caused by a robot’s presence among people and objects, considering not only the entities themselves but also the interactions happening. This model is later improved to yield discomfort maps that can be used as cost maps for motion planning. In the domain of human pose estimation, two different solutions are presented: a model capable of estimating the position and orientation of the people in the environment, and a multi-camera and multi-person 3D human full pose estimator. This last model, which does not require a labelled dataset for training, can be used for tracking people and feed their poses into the aforementioned cost map generator, as seen in the experimentation of this thesis. These works exhibit superior results in terms of precision, accuracy, and time efficiency when compared to similar state-of-the-art works. Finally, in the field of image generation, the thesis explores an application within the context of smart cities: generating realistic traffic images conditioned with graphs. This work leverages the strengths of GNNs when working with semantic data. The model can generate realistic images based on the properties of the items expected in them –namely their position, size and colour– and global properties such as the time of day. GNNs can be time-inefficient due to the added complexity of dealing with heterogeneously structured data. Consequently, the success of the applications presented in this thesis is the result of the effective integration of this networks, often in conjunction with other well-known approaches. One notable example is the fusion of convolutional networks with GNNs, which in this thesis leads to more efficient image generation when compared to pure GNN architectures. These methods constitute the central contribution of this thesis, as they allow GNNs to fully exploit their potential while mitigating inefficiencies.