Structure-Preserving Neural Style Transfer

Abstract

State-of-the-art neural style transfer methods have demonstrated amazing results by training feed-forward convolutional neural networks or using an iterative optimization strategy. The image representation used in these methods, which contains two components: style representation and content representation, is typically based on high-level features extracted from pre-trained classification networks. Because the classification networks are originally designed for object recognition, the extracted features often focus on the central object and neglect other details. As a result, the style textures tend to scatter over the stylized outputs and disrupt the content structures. To address this issue, we present a novel image stylization method that involves an additional structure representation. Our structure representation, which considers two factors: i) the global structure represented by the depth map and ii) the local structure details represented by the image edges, effectively reflects the spatial distribution of all the components in an image and the structure of dominant objects respectively. Experimental results demonstrate that our method achieves impressive visual effects, which is particularly significant when processing images sensitive to structure distortion, e.g., images containing multiple objects potentially at different depths, or dominant objects with clear structures.

Paper

1. Structure-Preserving Neural Style Transfer, Ming-Ming Cheng#*, Xiao-Chang Liu#, Jie Wang, Shao-Ping Lu, Yu-Kun Lai, Paul L. Rosin, IEEE TIP, 29:909-920, 2020. [pdf | bib | project | code]