Material Reconstruction from Single Image Combining Neural Networks with Singular Value Decomposition

doi:10.16182/j.issn1004731x.joss.25-0834

Abstract

Abstract:

The tabulated BRDFs (bidirectional reflectance distribution function) can realistically reproduce the surface appearance of objects. However, due to their high-dimensional characteristics and the fact that a single planar image contains limited reflectance information and small differences, methods for estimating tabulated BRDFs typically require complex equipment or the capture of multiple images. To address this issue, a method is proposed for reconstructing material properties from a single image by combining neural networks with singular value decomposition. The singular value decomposition is introduced to compress the material into a lower-dimensional space. The task of solving the tabulated BRDFs is simplified to estimating the weight vectors of material bases. By combining a deep convolutional neural network, a mapping relationship between the single planar image and the weight vector is established. A logarithmic relative loss function is designed to suppress high dynamic range within the material. A planar sample dataset is constructed to train the network model. A two-sided training method is proposed to avoid ringing at the highlights. Experiments results on both synthetic and real data show that the proposed method outperforms previous methods.

Key words: tabulated BRDF, neural networks, singular value decomposition, high dynamic range, material reconstruction

CLC Number:

TP391.9

Li Zhiqiang, Shen Xukun, Hu Yong, Zhou Xueyang, Chen Yifan. Material Reconstruction from Single Image Combining Neural Networks with Singular Value Decomposition[J]. Journal of System Simulation, 2026, 38(1): 189-199.

Figures/Tables 8

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Table 1

Fig. 7

References 30

[1]	Dana K J, van Ginneken Bram, Nayar S K, et al. Reflectance and Texture of Real-world Surfaces[J]. ACM Transactions on Graphics, 1999, 18(1): 1-34.
[2]	Lawrence J, Ben-Artzi A, Decoro C, et al. Inverse Shade Trees for Non-parametric Material Representation and Editing[J]. ACM Transactions on Graphics, 2006, 25(3): 735-745.
[3]	Matusik W. A Data-driven Reflectance Model[D]. Cambridge: Massachusetts Institute of Technology, 2003.
[4]	Jannik Boll Nielsen, Jensen H W, Ramamoorthi R. On Optimal, Minimal BRDF Sampling for Reflectance Acquisition[J]. ACM Transactions on Graphics, 2015, 34(6): 186.
[5]	Xu Zexiang, Jannik Boll Nielsen, Yu Jiyang, et al. Minimal BRDF Sampling for Two-shot Near-field Reflectance Acquisition[J]. ACM Transactions on Graphics, 2016, 35(6): 188.
[6]	Asselin Louis-Philippe, Laurendeau Denis, Lalonde Jean-François. Deep SVBRDF Estimation on Real Materials[C]//2020 International Conference on 3D Vision (3DV). Piscataway: IEEE, 2020: 1157-1166.
[7]	Guo Jie, Lai Shuichang, Tao Chengzhi, et al. Highlight-aware Two-stream Network for Single-image SVBRDF Acquisition[J]. ACM Transactions on Graphics, 2021, 40(4): 123.
[8]	Ye Wenjie, Li Xiao, Dong Yue, et al. Single Image Surface Appearance Modeling with Self-augmented CNNs and Inexact Supervision[J]. Computer Graphics Forum, 2018, 37(7): 201-211.
[9]	Chen Zhe, Nobuhara Shohei, Nishino Ko. Invertible Neural BRDF for Object Inverse Rendering[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(12): 9380-9395.
[10]	Santo Hiroaki, Samejima Masaki, Sugano Yusuke, et al. Deep Photometric Stereo Networks for Determining Surface Normal and Reflectances[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(1): 114-128.
[11]	Ren Peiran, Wang Jiaping, Snyder J, et al. Pocket Reflectometry[J]. ACM Transactions on Graphics, 2011, 30(4): 45.
[12]	Lin Y, Peers P, Ghosh A. On-site Example-based Material Appearance Acquisition[J]. Computer Graphics Forum, 2019, 38(4): 15-25.
[13]	Hui Zhuo, Sunkavalli K, Lee J Y, et al. Reflectance Capture Using Univariate Sampling of BRDFs[C]//2017 IEEE International Conference on Computer Vision (ICCV). Piscataway: IEEE, 2017: 5372-5380.
[14]	Deschaintre Valentin, Aittala Miika, Durand Fredo, et al. Single-IMAGE SVBRDF Capture with a Rendering-aware Deep Network[J]. ACM Transactions on Graphics, 2018, 37(4): 128.
[15]	Li Zhengqin, Sunkavalli K, Chandraker M. Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Image[C]//Computer Vision – ECCV 2018. Cham: Springer International Publishing, 2018: 74-90.
[16]	Wen Tao, Wang Beibei, Zhang Lei, et al. SVBRDF Recovery from a Single Image with Highlights Using a Pre-trained Generative Adversarial Network[J]. Computer Graphics Forum, 2022, 41(6): 110-123.
[17]	Zhao Yezi, Wang Beibei, Xu Yanning, et al. Joint SVBRDF Recovery and Synthesis from a Single Image Using an Unsupervised Generative Adversarial Network[C]//Eurographics Symposium on Rendering - DL-only Track. Geneva: The Eurographics Association, 2020: 53-66.
[18]	Deschaintre Valentin, Aittala M, Durand F, et al. Flexible SVBRDF Capture with a Multi-image Deep Network[J]. Computer Graphics Forum, 2019, 38(4): 1-13.
[19]	Gao Duan, Li Xiao, Dong Yue, et al. Deep Inverse Rendering for High-resolution SVBRDF Estimation from an Arbitrary Number of Images[J]. ACM Transactions on Graphics, 2019, 38(4): 134.
[20]	Guo Yu, Smith C, Hašan Miloš, et al. MaterialGAN: Reflectance Capture Using a Generative SVBRDF Model[J]. ACM Transactions on Graphics, 2020, 39(6): 254.
[21]	Li Zhiqiang, Shen Xukun, Hu Yong, et al. High-resolution SVBRDF Estimation Based on Deep Inverse Rendering from Two-shot Images[J]. The Visual Computer, 2023, 39(10): 4609-4622.
[22]	Deschaintre Valentin, Drettakis G, Bousseau A. Guided Fine-tuning for Large-scale Material Transfer[J]. Computer Graphics Forum, 2020, 39(4): 91-105.
[23]	Zhou Xilong, Kalantari N K. Adversarial Single-image SVBRDF Estimation with Hybrid Training[J]. Computer Graphics Forum, 2021, 40(2): 315-325.
[24]	Goodfellow I, Pouget-Abadie Jean, Mirza Mehdi, et al. Generative Adversarial Networks[J]. Communications of the ACM, 2020, 63(11): 139-144.
[25]	Zhang Xiuming, Srinivasan P P, Deng Boyang, et al. NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination[J]. ACM Transactions on Graphics, 2021, 40(6): 237.
[26]	Zhou Shuyao, Zhu Tianqian, Shi Kanle, et al. Review of Light Field Technologies[J]. Visual Computing for Industry, Biomedicine, and Art, 2021, 4(1): 29.
[27]	Matusik W, Pfister H, Brand M, et al. Efficient Isotropic BRDF Measurement[C]//Proceedings of the 14th Eurographics Workshop on Rendering. Goslar: Eurographics Association, 2003: 241-247.
[28]	He Kaiming, Zhang Xiangyu, Ren Shaoqing, et al. Deep Residual Learning for Image Recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Piscataway: IEEE, 2016: 770-778.
[29]	Hu Bingyang, Guo Jie, Chen Yanjun, et al. DeepBRDF: A Deep Representation for Manipulating Measured BRDF[J]. Computer Graphics Forum, 2020, 39(2): 157-166.
[30]	Zhang R, Isola P, Efros A A, et al. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 586-595.

方法	RMSE(BRDF)	RMSE(渲染球体)	LPIPS(渲染球体)
本文	0.169 86	0.018 82	0.014 00
文献[4]1次采样	0.542 77	0.118 63	0.118 29
文献[4]2次采样	0.391 95	0.036 14	0.031 63
文献[4]5次采样	0.207 61	0.035 43	0.024 16
文献[14]		0.056 38	0.045 14
文献[23]		0.055 48	0.049 34