In order to improve the segmentation accuracy of single stage instance segmentation and improve the small target detection, an improved YOLACTR algorithm is proposed based on YOLACT algorithm. The algorithm first uses the combination of CNN and Transformer to design a new head prediction network to further extract features, and uses two-way attention to correlate the mask information of the same instance and distinguish the mask features between different instances. It pays attention to the correlation information around the feature points, making the prediction of the detection box more accurate. Then the mask branch is formed by the combination of multi-level up sampling module and the designed CS attention module, which integrates a variety of different scale information, and uses CS attention module to pay attention to different scale information. On the MS COCO, compared with YOLACT algorithm, YOLACTR algorithm improves the detection accuracy of Box and Mask by 7.4% and 2.9% respectively, and improves the detection accuracy of small targets by 18.9% and 13.5% respectively. Experiments show that YOLACTR algorithm can improve the accuracy of detection, segmentation and classification in multi-target complex scenes, and improve the problems of missed detection and false detection of small targets and overlapping targets.