Many factors lead to an incident for workers at construction sites. They were exposed to a different type of hazardous such as fall from scaffolding, electric shock, and hit by a crane. Yet, at the moment, we are still lacking a solution to mitigate such incidents by using image detection and machine learning algorithm with a cost-effective and real-time solution. Hence, this paper presents a hazardous detection model at a construction site by using image detection to ensure worker safety at a construction site. This experiment was conducted by using the Faster Region-based Convolutional Neural Networks (R-CNN) algorithm embedded in TensorFlow, 6000 images for training dataset from the MIT Places Database (from Scene Recognition), and 600 anonymous dataset images from construction sites for testing. Based on the experiment conducted, the model can detect possible hazardous incident at the construction site with a more than 70% accuracy rate.