Abstract

Learning-based structure-from-motion methods such as ACE-Zero have demonstrated strong performance in estimating camera poses and scene coordinates from unordered image collections without requiring ground truth supervision. However, the lack of global and multi-view consistency constraints in ACE-Zero can lead to pose drift and misalignment, particularly in complex or ambiguous scenes. In this work, we propose a hybrid framework that integrates pose graph optimization (PGO) into ACE-Zero to refine camera poses and suppress incorrect refinements. We construct pose graphs directly from ACE-Zero outputs by extracting relative pose constraints from predicted scene coordinates. Furthermore, we introduce an uncertainty-aware optimization strategy by estimating confidence scores using geometric priors, including epipolar and optical flow consistencies across views. Our approach improves the robustness and accuracy of pose estimation, demonstrating that global geometric reasoning can effectively complement learning-based inference in structure-from-motion.

Method

ACE-Zero reconstructs a scene from unposed images by alternating between training a scene coordinate regressor and registering and refining camera poses. Its pose refinement is local: each pose is updated from information in its own image only, so errors can accumulate into drift. We add a global step to every ACE-Zero iteration.

1. Pose graph from predicted scene coordinates. ACE-Zero provides no pairwise constraints, so we derive them from its own predictions. For an image pair, each scene coordinate predicted in one image is projected into the other; if the scene coordinate predicted at that pixel lies within a threshold τ of the original point, the pixel pair is kept as a match. A relative pose is estimated from these matches with PnP and RANSAC and becomes an edge of the pose graph.
2. Uncertainty-aware edges. Scene coordinate predictions are noisy and come without uncertainty estimates. We score matches with two unsupervised geometric priors: the epipolar error, the distance of a reprojected point to its epipolar line, and the optical flow error, the distance between the reprojected point and the location predicted by RAFT optical flow. Their weighted sum, σ = α₁e_epi + α₂e_flow, is aggregated per edge and sets its information matrix Ω = I / (σ² + ε), so unreliable constraints have less influence.
3. Progressive global optimization. The weighted pose graph is optimized with Levenberg–Marquardt and a Huber loss at every ACE-Zero iteration, and the refined poses carry into the next iteration. The consistency threshold τ is tightened from 0.2 m to 0.05 m as the predictions improve.

The PGO step takes about 30 s per iteration, compared to about 150 s for an ACE-Zero iteration, but it roughly halves the number of iterations needed, so a full reconstruction takes less time than ACE-Zero.

Results

We follow ACE-Zero and measure pose quality by novel view synthesis: each method's poses are used to train a Nerfacto model, and PSNR is computed on held-out views. COLMAP poses serve as the pseudo ground truth reference.

Bold marks the best result excluding COLMAP. COLMAP uses its default settings, except on the long Tanks and Temples videos, where it uses the settings recommended for large collections (on 7-Scenes these reach 21.8 dB). The paper reports per-scene results, VGGT-SLAM and DROID-SLAM baselines, and reconstruction times.

Ablations

Code

The code builds on ACE-Zero. The paper's results were produced with an internal pose graph optimizer that cannot be released, so the repository provides a re-implementation based on GTSAM, together with an extended pipeline and scripts to reproduce its results. It is distributed under the ACE-Zero license for non-commercial use.

Paper and Supplementary Material

T. H. E. Tse, J. Peng, A. Yao.
Learning Scene Coordinate Reconstruction from Unposed Images via Pose Graph Optimization.
In CVPR, 2026.
(CVF Open Access, supplementary, conference page)

BibTeX

@InProceedings{Tse_2026_CVPR,
    author    = {Tse, Tze Ho Elden and Peng, Jizong and Yao, Angela},
    title     = {Learning Scene Coordinate Reconstruction from Unposed Images via Pose Graph Optimization},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2026},
    pages     = {21816-21825}
}

[bibtex.txt]

Acknowledgements

This work builds on ACE-Zero by Brachmann et al. This template was originally made by Phillip Isola and Richard Zhang for a colorful ECCV project; the code can be found here.