As there are a large number of fatalities recorded in the construction sites, it is considered as one of the hazardous occupations in the world when compared to any other sector. This article intends to study the factors that lead to accidents in construction sites, hazards, and safety management through previous research. It is noted that the majority of the construction site accidents are due to unsafe acts by the workers. Previous studies focused on behavior based safety and safety management in constructional sites but still, the influential factor for the accident is not reported. Hence from the study, it is concluded that prioritizing the hazardous task through risk assessment and allocating the right person to the right job may reduce accidents drastically. Furthermore, determining safety performance in the site and identifying the probable human error by various techniques in each task can assist the worker in altering their behaviors.