The majority of construction projects, especially large ones, experience time delays, cost overruns, productivity loss, and/or accidents. This is particularly so in case of congested and disorganized sites that contain obstacles that affect workers’ productivity and safety. Effective site layout planning, therefore, is one of the most important project management tasks, and has a significant impact on all aspects of construction, including safety, productivity, site operations, and ultimately time and cost. Site layout planning is a complex process that determines the best location for the needed site facilities (e.g., workshops, storage areas, equipment, etc.) needed to execute the project, so that productivity and safety are optimized. Despite the many simulation and optimization models in the literature for site layout planning, they mostly consider the site location without the low-level details of the workers’ movements within site, particularly around site obstacles. This research aims at developing a construction site-layout planning framework that uses Agent-Based Modelling and Simulation (ABMS) technology to perform a micro-level analysis of workers’ movements and behaviors on site, to study the impact on site productivity and safety. For practicality, this research considers variety of productivity-hindering and safety-hindering obstacles on site. The model also considers two types of workers’ behaviors in their movement around site obstacles: avoider, and aggressive. Given any site layout with any number of resources of different behaviors, the ABMS simulation quantifies the site overall productivity and accident/injury potential. To optimize the site layout, the framework integrates an optimization procedure that determines the optimum site layout that maximizes productivity and safety. A sensitivity analysis is also incorporated to examine the impact of obstacle type and workers’ behavioral characteristics. The results of two case studies prove that the framework is a valuable tool for analyzing and assessing site productivity and safety, and for providing decision support for project managers in establishing site regulations and rewards for positive workers’ behaviors. This research is expected to help construction companies deliver projects with less time and cost, and help to reduce accidents on complex sites.