Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentation

Abstract 

3D spatial information is known to be beneficial to the semantic segmentation task. Most existing methods take 3D spatial data as an additional input, leading to a two-stream segmentation network that processes RGB and 3D spatial information separately. This solution greatly increases the inference time and severely limits its scope for real-time applications. To solve this problem, we propose Spatial information guided Convolution (S-Conv), which allows efficient RGB feature and 3D spatial information integration. S-Conv is competent to infer the sampling offset of the convolution kernel guided by the 3D spatial information, helping the convolutional layer adjust the receptive field and adapt to geometric transformations. S-Conv also incorporates geometric information into the feature learning process by generating spatially adaptive convolutional weights. The capability of perceiving geometry is largely enhanced without much affecting the amount of parameters and computational cost. We further embed S-Conv into a semantic segmentation network, called Spatial information Guided convolutional Network (SGNet), resulting in real-time inference and state-of-the-art performance on NYUDv2 and SUNRGBD datasets.

Paper

• Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentation, Lin-Zhuo Chen, Zheng Lin, Ziqin Wang, Yong-Liang Yang, Ming-Ming Cheng*, IEEE TIP, 2021. [pdf|code|project| bib ]

If you find our work is helpful, please cite

@article{21TIP-SGNet,   
  author={Lin-Zhuo Chen and Zheng Lin and Ziqin Wang and Yong-Liang Yang and Ming-Ming Cheng},   
  journal={IEEE Transactions on Image Processing},    
  title={Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentation},    
  year={2021},   
  volume={},   
  number={},   
  pages={1-1},   
  doi={10.1109/TIP.2021.3049332} 
}

Contact

If you have any questions, please leave a comments in this project page: https://mmcheng.net/sgnet/ The authors will try to reply your question ASAP.