Hardhats play an essential role in protecting construction individuals from accidents. However, wearing hardhats is not strictly enforced among workers due to all kinds of reasons. To enhance construction sites safety, the majority of existing works monitor the presence and proper use of hardhats through multi-stage data processing, which come with limitations on adaption and generalizability. In this paper, a one-stage system based on convolutional neural network is proposed to automatically monitor whether construction personnel are wearing hardhats and identify the corresponding colors. To facilitate the study, this work constructs a new and publicly available hardhat wearing detection benchmark dataset, which consists of 3174 images covering various on-site conditions. Then, features from different layers with different scales are fused discriminately by the proposed reverse progressive attention to generate a new feature pyramid, which will be fed into the Single Shot Multibox Detector (SSD) to predict the final detection results. The proposed system is trained by an end-to-end scheme. The experimental results demonstrate that the proposed system is effective under all kinds of on-site conditions, which can achieve 83.89% mAP (mean average precision) with the input size 512 × 512.