Improving Object Detection and Instance Segmentation of Eukaryotic Cells Using Semantic Aware Data Augmentation



Invention Summary:

Machine Learning and Artificial Intelligence (AI)-Driven analysis have provided great advancements in microscopic imaging in the Biomedical field. The efficacy of many of these systems relies on the availability of many well-defined and empirically annotated images to serve as training data to teach such advanced neural networks how to properly identify and analyze target pictures. Manual generation of such images is time labor prohibitive, and many attempts have been made to artificially generate images realistic enough to train such models. However, many of the current systems used for such purposes like Generative Adversarial Networks lack key capabilities such as creating images with labeled masks, limited generated resolutions, and a need for their own large training datasets.

Rutgers Researchers have created a novel algorithm for generating artificial images to serve as training data for AI-driven imaging networks using semantically aware data augmentation. This system uses nuclei to discriminate between cells and relies on augmentation of annotated natural images to generate a large quantity of authentic yet simplified artificial images (e.g., immunological synapse images for immunotherapy research and clinical development) to use in AI image program training. This method of augmentation preserves key features of the natural image profile while also allowing for mass generation of highly authentic images, thus overcoming many of the shortcomings of current training data generation techniques. This program is compatible with a variety of color spectrums and can greatly improve the accuracy of eukaryotic cell detection and structure discrimination.

Market Applications:

  • Novel method for generation of AI-training images.
  • Can improve existing machine learning algorithms.
  • Enables AI imaging systems to accurately analyze wider variety of cell types (e.g., CAR-modified immune cells) and structures.


  • Greatly improves object and cell detection.
  • Adaptable to any cell type or structure.
  • Generates authentic, labeled training images

Intellectual Property & Development Status: Provisional filed, patent pending. Available for licensing and/or search collaboration. For any business development and other collaborative partnerships contact:


Patent Information:
Shemaila Sultana
Assistant Director
Rutgers, The State University of New Jersey