Activity Recognition in Residential Spaces with Internet of Things Devices and Thermal Imaging


In this paper, we design algorithms for indoor activity recognition and 3D thermal model generation using thermal images, RGB images, captured from external sensors, and the internet of things setup. Indoor activity recognition deals with two sub-problems: Human activity and household activity recognition. Household activity recognition includes the recognition of electrical appliances and their heat radiation with the help of thermal images. A FLIR ONE PRO camera is used to capture RGB-thermal image pairs for a scene. Duration and pattern of activities are also determined using an iterative algorithm, to explore kitchen safety situations. For more accurate monitoring of hazardous events such as stove gas leakage, a 3D reconstruction approach is proposed to determine the temperature of all points in the 3D space of a scene. The 3D thermal model is obtained using the stereo RGB and thermal images for a particular scene. Accurate results are observed for activity detection, and a significant improvement in the temperature estimation is recorded in the 3D thermal model compared to the 2D thermal image. Results from this research can find applications in home automation, heat automation in smart homes, and energy management in residential spaces.

Publication DOI:
Divisions: College of Engineering & Physical Sciences
Additional Information: © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// 4.0/). Funding: This research was partly funded by the Natural Sciences and Engineering Research Council of Canada.
Uncontrolled Keywords: 3D scene reconstruction,3D thermal model,Activity recognition,Internet of Things (IoT),Thermal images,Analytical Chemistry,Biochemistry,Atomic and Molecular Physics, and Optics,Instrumentation,Electrical and Electronic Engineering
Publication ISSN: 1424-8220
Last Modified: 14 May 2024 07:21
Date Deposited: 02 Feb 2021 12:53
Full Text Link:
Related URLs: https://www.mdp ... 4-8220/21/3/988 (Publisher URL)
http://www.scop ... tnerID=8YFLogxK (Scopus URL)
PURE Output Type: Article
Published Date: 2021-02-02
Accepted Date: 2021-01-21
Authors: Naik, Kshirasagar
Pandit, Tejas
Naik, Nitin (ORCID Profile 0000-0002-0659-9646)
Shah, Parth



Version: Published Version

License: Creative Commons Attribution

| Preview

Export / Share Citation


Additional statistics for this record