Concrete bridge surface damage detection is crucial for bridge maintenance. However, existing machine vision-based methods suffer from low detection efficiency and accuracy when dealing with small-scale damages and complex backgrounds. In this paper, a novel detection network based on YOLOv5 is proposed. By optimizing the YOLOv5 backbone network and introducing a global attention mechanism and a multi-scale pyramid spatial pooling structure, the detection accuracy and efficiency are effectively improved, especially in the detection of small-scale damages in complex backgrounds. Experimental results show that the average detection accuracy of the improved model is increased by 4.1% compared to the original network structure. The detection performance of small-scale damages such as holes and complex backgrounds is superior to YOLOv7 and YOLOv5. Compared with YOLOv7, the proposed method achieves a 2.3% increase in average detection accuracy and a 30% improvement in detection speed.