iNeRF

iNeRF

Motivation

why invert???

How to obtain an initial pose estimation?

given an image, find the translation and rotation of a camera relative to a 3D object or scene.

Setting

no mesh object model

one image

using gradient descent to minimize the residual

Experiment

how different batch sizes of rays affect iNeRF on a synthetic dataset

We then show that for complex real-world scenes from the LLFF dataset, iNeRF can improve NeRF by estimating the camera poses of novel images and using these images as additional training data for NeRF

how to sample rays during pose refinement for iNeRF to collect informative gradients

Finally, we show iNeRF can perform category-level object pose estimation, including object instances not seen during training, with RGB images by inverting a NeRF model inferred from a single view

INeRF formulation

Gradient-Based SE(3) Optimization

3.Self-supervised NeRF with iNeRF

NeRF: T, I —> theta

iNeRF: I, theta —> T

click to edit

2）use iNeRF to take in additional unknown-pose observed images and solve for estimated poses

1）train a NeRF given a set of training
RGB images with known camera poses

click to edit

Sampling Rays

click to edit

3) add 2) to 1) to do a semi-supervised learning

click to edit

Experiments

LLFF Dataset

Sim2Real Cars

Synthetic dataset

For each scene, choose 5 test images and generate 5 different pose initializations???

click to edit

results: report pose error with less than 5 degrees or 5cm

click to edit

ShapeNet-SRN Cars

click to edit

Dataset Details:
1) using the "car" classes of ShapeNet introduced by Stizmann including 3514 cars.

2) split the 3514 cars into training, validation and test set.

3) for objects in the test set, we render an image in an archimedean spiral and then select another image within certain offset.