Training‐free few‐shot construction tool and material detection using pre‐trained vision‐language model

Abstract

Direct visual understanding of construction entities, such as tools and materials (T&M), underpin construction management and resource scheduling. Traditional supervised learning methods suffer from high annotation cost, severe computational demands, and limited datasets. In contrast, training‐free approaches offer an effective alternative well‐suited for construction scenarios constrained by data scarcity and limited resources. Besides, vision‐language models (VLMs) can directly learn image semantics through natural language supervision and also demonstrate strong zero‐shot detection capabilities without requiring retraining. Existing methods often exhibit limited image–text semantic alignment in construction scenarios, which restricts their effectiveness in construction tasks. Therefore, there is an urgent need for approaches that can enhance cross‐modal understanding in such domain‐specific contexts. To address this challenge, this paper proposes a training‐free, knowledge‐enhanced VLM to recognize T&M in construction tasks. The proposed approach leverages image matching and image–text knowledge alignment strategies, thereby utilizing the training‐free nature of existing VLMs while benefiting from enhanced performance brought by knowledge integration. This method offers a novel solution for construction management and robotic collaboration tasks that are traditionally constrained by data and computational resource dependencies.