Unsupervised 3D reconstruction method based on multi-view propagation

基于多视图传播的无监督三维重建方法

¹ School of Automation, Northwestern Polytechnical University, Xi'an 710072, China
² School of Cyber Engineering, Xidian University, Xi'an 710071, China

Received: 17 March 2023

Abstract

In this paper, an end-to-end deep learning framework for reconstructing 3D models by computing depth maps from multiple views is proposed. An unsupervised 3D reconstruction method based on multi-view propagation is introduced, which addresses the issues of large GPU memory consumption caused by most current research methods using 3D convolution for 3D cost volume regularization and regression to obtain the initial depth map, as well as the difficulty in obtaining true depth values in supervised methods due to device limitations. The method is inspired by the Patchmatch algorithm, and the depth is divided into n layers within the depth range to obtain depth hypotheses through multi-view propagation. What's more, a multi-metric loss function is constructed based on luminosity consistency, structural similarity, and depth smoothness between multiple views to serve as a supervisory signal for learning depth predictions in the network. The experimental results show our proposed method has a very competitive performance and generalization on the DTU, Tanks & Temples and our self-made dataset; Specifically, it is at least 1.7 times faster and requires more than 75% less memory than the method that utilizes 3D cost volume regularization.

摘要

提出一种端到端的深度学习框架, 从多视图中计算深度图从而重建出三维模型。针对目前大多数研究方法通过3D卷积实现3D成本体积正则化并回归得到初始深度图带来巨大的GPU内存消耗, 以及由于设备受限导致有监督的方法中深度图真值难以获取的问题, 提出一种多视图传播的无监督三维重建方法。该方法借鉴Patchmatch算法思想, 在深度范围内将深度划分n层, 通过多视图传播得到深度假设, 并利用多个视图之间的光度一致性、结构相似性和深度平滑度构建多指标损失函数, 作为网络中学习深度预测的监督信号。实验表明, 文中提出的方法在DTU、Tanks & Temples和自制数据集上的性能和泛化性非常有竞争力, 比采用3D成本体积正则化的方法快1.7倍以上, 内存使用量减少75%。