Unsafe behavior as a direct cause accounts for some 80% of accidents in the construction industry. While numerous studies have investigated the factors that influence unsafe behavior, most of them have approached the problem from a statistical standpoint, which is unable to provide causality patterns of how unsafe behavior forms. This paper illuminates the causality patterns obtained from a series of near-miss fall incidents through a practical study using a video camera system to record and analyze the unsafe behavior of workers. A total of 1347 cases of near-miss incidents were obtained in two weeks. Association Rules method was employed to verify the revealed causality patterns of workers’ unsafe behavior, and a risk matrix was developed based on the indicators of rules to classify workers’ risk level. Results indicate that habits, motivation, perceived behavioral control, and subjective norms are the strongest contributing factors to unsafe behavior. While results introduce a new research perspective on the concept of construction safety behavior, the practical insights gained hereby enable construction managers to better analyze the combined negative effect of causal factors on workers’ safety behavior.