Toward recovering shape and motion of 3D curves from multi-view image sequences

Toward Recovering Shape and Motion of3D Curves

from Multi-View Image Sequences

Rodrigo L.Carceroni Kiriakos N.Kutulakos

Computer Science Department Computer Science Department&

University of Rochester Department of Dermatology

Rochester,NY14627University of Rochester

Rochester,NY14627

Abstract

We introduce a framework for recovering the3D shape

and motion of unknown,arbitrarily-moving curves from two

or more image sequences acquired simultaneously from dis-

tinct points in space.We use this framework to(1)iden-

tify ambiguities in the multi-view recovery of(rigid or non-

rigid)3D motion for arbitrary curves,and(2)identify a

novel spatio-temporal constraint that couples the problems

of3D shape and3D motion recovery in the multi-view case.

We show that this constraint leads to a simple hypothesize-

and-test algorithm for estimating3D curve shape and mo-

tion simultaneously.Experiments performed with synthetic

data suggest that,in addition to recovering3D curve mo-

tion,our approach yields shape estimates of higher accuracy

than those obtained when stereo analysis alone is applied to

a multi-view sequence.

1Introduction

A fundamental problem in computer vision is to recover the

3D shape and motion of unknown dynamic scenes from se-

quences of images.While this problem has received consid-

erable attention(e.g.,[1–6]),existing approaches to3D mo-

tion estimation of unknown scenes assume that all images

are acquired from a single viewpoint.Unfortunately,the sin-

gle viewpoint assumption puts strong constraints on both the

types of motion that can be recovered(e.g.,rigid[2–6],artic-

ulated[7],parametric[8],isometric[9])and the scenes that

can be analyzed(e.g.,known3D shape[7,10],known dy-

namics[11,12],availability of distinguished feature points

[2–7]).As a result,little is currently known about how to re-

cover the3D shape and motion of unknown scenes simulta-

neously viewed from two or more distinct viewpoints,about

the constraints and ambiguities that this problem embodies,

and about the algorithms required to solve it.

In this paper,we answer these questions through a ge-

ometrical analysis of the multi-view shape and motion re-

(a)(b)

(c)

Figure 1:Motion constraints from a single image.(a)The projected displacement of point can be decomposed into two

orthogonal components,and ,tangent and normal to the projected curve at ,respectively.Only

is recoverable from the curve’s time-varying projection [9].(b)The 3D component of ’s displacement is completely determined by ’s 3D position,its projection ,the image displacement ,and the camera’s projection matrix.(c)The knowledge of constrains the point to a plane,called the Motion Constraint Plane.

ple views of a moving curve is almost always suf?cient to uniquely determine the curve’s3D shape at every time.1 As a result,two types of complementary constrains are avail-able to compute’s3D shape and motion at a point:

Stereo constraints:must project to,.

Motion constraints:must lie on’s Multi-View Motion Constraint Line.

The dependence of the Multi-View Motion Constraint Line on’s3D position suggests that the above stereo and motion constraints are coupled.Here we exploit this cou-pling to estimate shape and motion simultaneously by en-suring the joint satisfaction of both the(spatial)stereo con-straints on and and the(spatio-temporal)motion con-straints on and.When these constraints are combined, they over-determine the3D coordinates of and.

A key question that must therefore be addressed to com-pute shape and motion is how to jointly satisfy these con-straints.We answer this question by formulating constraint satisfaction as the joint solution of a set of linear func-tional minimization problems.Solving these problems cor-responds to?nding points and that minimize their least-squares distance from appropriately-chosen3D lines that represent the stereo and motion constraints at.

3.1Enforcing Stereo Constraints

Intuitively,the point should be as close as possible to the optical rays through’s projections,,.We can therefore formulate enforcement of all stereo constraints at as the problem of?nding a3D point that minimizes its perpendicular distance to these rays in a least-squares sense. This gives rise to the error functional

(1)

where is the unit vector parallel to the optical ray through and is the vector-product operator.If we denote the antisymmetric vector-product matrix for by,mini-mization of this error functional is equivalent to solving the linear system

.. ...

.(2)

in which’s3D coordinates are the unknowns.

2See the Appendix for a derivation.

a priori estimate of the cameras’known epipolar geome-

try to establish initial stereo correspondences between the

curve’s multiple https://www.360docs.net/doc/e0909399.html,ing these correspondences

as a starting point,the algorithm uses Eq.(5)to compute the

3D shape and motion of each point on the curve,indepen-

dently.

The key idea of the algorithm is to use a hypothesize-and-

test approach to determine,for each given3D point on the

curve,the two recoverable components of’s in?nitesimal

displacement,.More speci?cally,the algorithm conjec-

tures multiple temporal correspondences for,constructs

an over-constrained system of the form shown in Eq.(5)for

each hypothesized correspondence,and then selects the cor-

respondence that minimizes the residual in the system’s so-

lution.Observation2tells us that the displacement,,de-

rived from this correspondence determines the’s true in-

stantaneous velocity up to a1D ambiguity along the curve’s

tangent.

Multi-View Shape&Motion Recovery Algorithm

Step1:Choose a reference viewpoint,,and let be’s

projection in the reference image at time.

Step2:Use the epipolar geometry of each camera to estab-

lish’s correspondences,,in the remaining

images at time.

Step3:For every curve point in the reference image at

time that is near:

https://www.360docs.net/doc/e0909399.html,e the epipolar geometry of each camera to es-

tablish’s correspondences,,in the re-

maining images at time;

b.solve the over-constrained system of Eq.(5)to com-

pute and,and let be the residual left by this

solution.3

Step4.Return the pair with the smallest.

5Experimental Results

In order to determine the applicability of the Multi-View

Shape&Motion Recovery Algorithm,we performed pre-

liminary experiments with synthetic data.The goal was to

test the hypothesis that besides providing3D motion esti-

mates(which are not readily computable by stereo analysis

alone)our formulation can also be used to improve3D shape

computations.To test this hypothesis,we compared the ac-

curacy of shape estimates obtained by stereo analysis alone

(Section3.1)to those obtained by stereo-motion analysis

(Section3.3)on randomly-generated multi-view sequences

for which ground truth was available.

4i.e.,by solving Eq.(2)for the correspondences established in

Step2of the Multi-View Shape&Motion Recovery Algorithm.

mbox

Cam1

Cam2

Cam3

0.20.5125102050

Camera position imprecision (mm)

102030405060

(a)

(b)(c)

Figure 2:Experimental evaluation.(a)Snapshots at the ?rst instant of a multi-view sequence used in the experiments.(b)Effects of camera calibration errors on reconstruction accuracy.Camera calibration errors denote the norm of the random displacement vectors added to the ground-truth position of the input cameras.In these experiments,a displacement of 4mm parallel to the image plane corresponds to an image error of approximately one pixel.The reconstruction error is the RMS distance between the reconstructed curve points and the sphere’s true surface.These errors were averaged over 10randomly-generated trials.(c)Relative improvements of stereo–motion over stereo alone.

[5]P.K.Ho and R.Chung,“Stereo-motion that complements

stereo and motion analyses,”in Proc.IEEE CVPR,pp.213–218,1997.

[6]M.Pollefeys,R.Koch,and L.V.Gool,“Self-calibration and

metric reconstruction in spite of varying and unknown inter-nal camera parameters,”in Proc.ICCV,pp.90–95,1998. [7] D.G.Lowe,“Fitting parameterized three–dimensional mod-

els to images,”IEEE Trans.PAMI,13(5),pp.441–450,1991.

[8]M.J.Black,Y.Yacoob,A.D.Jepson,and D.J.Fleet,“Learn-

ing parameterized models of image motion,”in Proc.IEEE CVPR,pp.561–567,1997.

[9]O.Faugeras and T.Papadopoulo,“A theory of the motion

?elds of curves,”https://www.360docs.net/doc/e0909399.html,p.Vis.,10(2),pp.125–156,1993.

[10]H.Ara′u jo,R.L.Carceroni,and C.M.Brown,“A fully

projective formulation to improve the accuracy of Lowe’s pose–estimation algorithm,”Comp.Vis.Image Und.,70(2), pp.227–238,1998.

[11]T.McInerney and D.Terzopoulos,“Deformable models in

medical image analysis:A survey,”Med.Image Anal.,1(2), 1996.

[12] D.Metaxas and I.A.Kakadiaris,“Elastically adaptive de-

formable models,”in Proc.ECCV,pp.550–559,1996. [13]P.J.Narayanan,P.W.Rander,and T.Kanade,“Constructing

virtual worlds using dense stereo,”in Proc.ICCV,pp.3–10, 1998.

[14]S.Moezzi,“Immersive telepresence,”IEEE Multimedia,

4(1),pp.17–26,1997.

[15]O.D.Faugeras and R.Keriven,“Variational principles,sur-

face evolution,PDE’s,level set methods,and the stereo prob-lem,”IEEE Trans.Image Proc.,7(3),pp.336–343,1998. [16] C.Bregler and J.Malik,“Tracking people with twists and

exponential maps,”in Proc.IEEE CVPR,pp.8–15,1998. [17] B.Guenter,C.Grimm,H.Malvar,and D.Wood,“Making

faces,”in Proc.SIGGRAPH,pp.55–66,1998.

[18] B.Bascle and R.Deriche,“Stereo matching,reconstruction

and re?nement of3D curves using deformable contours,”in Proc.ICCV,pp.421–430,1993.

[19]J.Porrill and S.Pollard,“Curve matching and stereo calibra-

tion,”Image https://www.360docs.net/doc/e0909399.html,p.,9(1),pp.45–50,1991.

[20]K.Astr¨o m,R.Cipolla,and P.J.Giblin,“Generalized epipolar

constraints,”in Proc.ECCV,pp.97–108,1996.

[21]R.Cipolla and A.Blake,“Surface shape from the deforma-

tion of apparent contours,”https://www.360docs.net/doc/e0909399.html,p.Vis.,9(2),pp.83–112,1992.

[22]R.Cipolla and A.Zisserman,“Qualitative surface shape from

deformation of image curves,”https://www.360docs.net/doc/e0909399.html,p.Vis.,8(1),pp.53–69,1992.

[23]M.P.D.Carmo,Differential Geometry of Curves and Sur-

faces.Prentice-Hall,1976.

[24]G.Baratoff and Y.Aloimonos,“Changes in surface convex-

ity and topology caused by distortions of stereoscopic visual space,”in Proc.ECCV,pp.226–240,1998.

[25]J.Weng,N.Ahuja,and T.S.Huang,“Optimal motion and

structure estimation,”IEEE Trans.PAMI,15(9),pp.864–884, 1993.[26]O.Faugeras and R.Keriven,“Complete dense stereovision

using level set methods,”in Proc.ECCV,pp.379–393,1998.

[27]K.N.Kutulakos and S.M.Seitz,“A theory of shape by space

carving,”TR692,https://www.360docs.net/doc/e0909399.html,p.Sci.,U.Rochester,1998. [28]P.Fua and Y.G.Leclerc,“Object-centered surface recon-

struction:Combining multi-image stereo and shading,”Int.

https://www.360docs.net/doc/e0909399.html,p.Vis.,16,pp.35–56,1995.

A Derivation of the Motion Error Functional Let be the unit vector de?ning the orientation of the nor-mal3D displacement,(Figure1b).is obtained by scal-ing the vector to unit norm,where is

the(unit)optical axis’orientation.The norm of is

(6)

A second expression for this norm can be obtained from the projected normal displacement,:

(7) where is de?ned in Figure3.Eq.(3)now follows by combining Eqs.(6)and(7):

(8)

where

.Hence,the ratio

,where is the camera’s focal length and is the depth of point.Since is equal to,we can de?ne as

.Note that can be computed from and the camera’s projection matrix.

相关主题
相关文档
最新文档