Abstract: To address the issues of missed detection and false alarms resulting from ambiguous fish features and insufficient prior information in the dataset for fish detection in fishways, this paper proposes an object detection algorithm founded on decision fusion between coarse-grained and fine-grained YOLO models. The method first enhances the coarse-grained YOLO model by embedding a coordinate attention module into the backbone layers and an adaptive spatial feature fusion module in the feature aggregation part to fuse features from different scale layers, thereby improving the model's detection capability for arbitrary fish species. Finally, the detection outcomes from the refined coarse-grained YOLO model and fine-grained YOLO model are filtered by confidence scores to identify boxes that need fusion. These boxes are then weighted fused based on their confidence values. This allows the proposed approach to attain a lower miss rate and false alarm rate for out-of-distribution detection on unknown and blurry fish. Evaluated on real-world fishway fish datasets, The proposed method achieves 98.59% accuracy and 94.19% recall for unknown fish, which are 9.25% and 11.21% higher than the confidence-based out-of-distribution detection method, and 6.42% and 3.69% higher than the energy-based out-of-distribution detection method, respectively. The recognition accuracy of fuzzy targets reaches 95.45% and the recall rate reaches 91.8%, which is 16.63% and 18.58% higher than the confidence-based out-of-distribution detection method, and 11.27% and 1.74% higher than the energy-based out-of-distribution detection method, respectively. The research findings have valuable implications for fish detection in fishways.