Automated provenance collection of runtime model evolution to enable explanation

Abstract

Context: New techniques exist to build large complex systems that perform autonomous decisionmaking. These systems may present emergent behaviours at runtime that were unforeseeable at design time, which may need to be understood. Data collected at runtime can be used to understand the system’s behaviour. However, ‘collecting runtime data’ usually leaves developers deciding what data to log and how. Objective: To develop a systematic approach to collecting runtime data. Furthermore, the approach should utilise automated techniques to minimise design-time costs while providing a consistent, reliable and robust implementation. Finally, the approach should maintain the runtime performance of a system. Method: Develop, implement and evaluate an approach to collecting runtime data based on Model-Driven Engineering practices combined with provenance-based techniques. A series of case studies evaluate an implementation using two different target systems. The collected runtime data is analysed to verify that it contains indicators of observable system behaviours or can ‘explain’ the causes of a system fault. Results: The experiments show that the proposed approach to collecting runtime data requires some extra effort. However, the system’s rate of execution can be considered minimally changed. Conclusions: Systematically collecting runtime data from a system to describe the changes and causes of changes to its runtime model provides insights into a system’s behaviour. The system’s design-time costs are managed via reusable and automated coding practices. Similarly, runtime costs are mitigated by adjusting the level of abstraction at which data is collected. Furthermore, data is stored using a representation that permits irrelevant data to be deleted.