Project Description
Our work involves developing an image processing pipeline for automatically classifying and quantifying the dynamics of chloroplasts, stromules, and the plant’s cytoskeleton to better understand the function, relationships, and movement characteristics of these intracellular structures. Our current pipeline consists of fast, automatic segmentation of microscopy images, active contour-based tracking, and unsupervised movement classification based on a U-Net, a convolutional neural network (CNN) for segmentation, and Computer vision methods for tracking.
For the CAREERS project, we propose to undertake and complete a transformer based pipeline, more specifically, TransUNet to leverage the power of CNNs and transformers for the image segmentation task. The proposed TransUNet involves CNN-Transformer Hybrid Encoder, Patch Embedding and Cascaded Upsampler. For tracking, we propose to use the tracking-by-attention paradigm which not only applies attention for data association but jointly performs tracking and detection, using TrackFormer, an end-to-end trainable MOT (multi-object tracking) approach based on an encoder-decoder Transformer architecture.