Realizing safety improvements in construction site layout planning (CSLP) is vitally important to construction project safety management. Unlike previous studies in which the safety objective is built without detailed risk factors analysis, this study transforms CSLP into a multi-objective optimization (MOO) problem with designing two safety objective functions due to facility safety relationships (potential risks arising from interaction flows) and geographic safety relationship (potential risks arising from hazardous sources) from the holistic interpretation of interaction relationship connecting temporary facilities. Besides, a supplementary cost reduction objective function was also derived as cost is a critical barrier against safety improvement. Subsequently, a tri-objective ant colony optimization based model was developed to solve MOO problem. Finally, a case study is used to verify the proposed model. The study enriches safety implications by considering onsite safety issues from interaction relationship and enhances site safety of CSLP in the pre-construction stage.