Reinforcement Learning for Cell Segmentation in Histopathological and Fluorescent Microscopy

Automated semantic cell segmentation is crucial in biomedical research, underpinning cell morphometrics and facilitating disease diagnosis, particularly in cancer.

Despite its significance, the practical application of deep learning in cell segmentation encounters challenges, notably in generalizing across various cell types and experimental conditions. Furthermore, the hesitancy to incorporate automated solutions in clinical settings stems from the widespread lack of clear and interpretable explanations.

We advocate for a transformation in tackling the cell segmentation challenge, transitioning from a supervised deep learning framework to a decision-centric approach. In alignment with human learning principles, Reinforcement Learning guides the annotation of cells as actions driven by visual inputs. Learning is then achieved indirectly through the utilization of reward signals.

We seamlessly integrate computer vision and reinforcement learning to harness the generalization capabilities of our system, resulting in a policy that provides visibility into the decision-making process during cell annotation. Our aim is to cultivate trust in our segmentation algorithm, believing that this approach represents a promising direction for overcoming limitations in clinical applications.

Our research introduces advanced policy-gradient-based reinforcement learning algorithms that transform the supervised cell segmentation problem into a decisionmaking challenge. The resulting three-dimensional cell segmentation not only surpasses traditional deep learning methods in generalization but also provides humancomprehensible insights into the decision-making process. To enhance adaptability, we incorporate interactive context, leveraging a dynamic method for optimizing a point transformer-based three-dimensional cell segmentation. Drawing inspiration from neuroplasticity, we represent medical images using a graph structure and optimize it with a self-attention mechanism. This enables accurate cell segmentation across multiple magnification levels. Our work addresses the critical need for more robust and interpretable cell segmentation methods, contributing significantly to biomedical research and clinical