3D computational ghost imaging

Invited Paper3D computational ghost imagingMatthew P.Edgar a,Baoqing Sun a,Richard Bowman a,b,Stephen S.Welsh a andMiles J.Padgett aa SUPA,School of Physics and Astronomy,University of Glasgow,Glasgow,G128QQ.b Department of Physics,Cavendish Lab,University of Cambridge,Cambridge,CB30HE.ABSTRACTComputational ghost imaging is a technique that enables lensless single-pixel detectors to produce images.By illuminating a scene with a series of patterns from a digital light projector(DLP)and measuring the reflected or transmitted intensity,it is possible to retrieve a two-dimensional(2D)image when using a suitable computer algorithm.An important feature of this approach is that although the light travels from the DLP and is measured by the detector,the images produced reveal that the detector behaves like a source of light and the DLP behaves like a camera.By placing multiple single-pixel detectors in different locations it is possible to obtain multiple ghost images with different shading profiles,which together can be used to accurately calculate the three-dimensional(3D)surface geometry through a photometric stereo techniques.In this work we show that using four photodiodes and a850nm source of illumination,high quality3D images of a large toy soldier can be retrieved.The use of simplified lensless detectors in3D imaging allows different detector materials and architectures to be used whose sensitivity may extend beyond the visible spectrum,at wavelengths where existing camera based technology can become expensive or unsuitable.Keywords:3D imaging,structured illumination,infrared imaging,computational imaging,ghost imaging1.INTRODUCTIONComputational ghost imaging(GI)is an alternative technique to conventional imaging and removes the need for a spatially resolving detector.Instead,ghost imaging infers the scene by correlating the known spatial information of a changing incident lightfield with the reflected(or transmitted)intensity.The principles of GI were originally demonstrated using spatially entangled photon pairs produced by sponta-neous parametric down-conversion,known as quantum GI.1,2The two photons propagate along different paths: in thefirst path,the photon interacts with the object and if not absorbed is detected by a detector with no spatial resolution,in the second path the photon never interacts with the object,but its transverse position is measured by a scanning imaging system.It is by correlating the coincidence measurements over many photon pairs that enables an image of the object to be deduced.It was subsequently demonstrated that GI could be performed not only using an entangled light source but also with thermal light,a technique commonly termed classical GI.3–5In classical GI a copy of the lightfield is usually made with a beam splitter,one copy of the lightfield interacts with the object and a non spatially resolving detector and the other copy is recorded with a camera.Correlations between the two detectors again yield an image,albeit one with a higher background than in the quantum case.6The earlier controversy over the distinction between classical and quantum GI is now largely resolved.7Classical GI systems can be simplified by introducing a device capable of generating computer programmable lightfields,which negates the requirement for the beam splitter and the camera since knowledge of the light field is held in the computer memory.This type of system,termed computational GI,8,9has previously been performed using a programmable spatial light modulator(SLM)and a laser,but can also be achieved using a presentation-type programmable light projector.10,11We note that in this form computational GI is related to Further author information:(Send correspondence to M.P.E.or M.J.P.)M.P.E.:E-mail:matthew.edgar@,Telephone:+44(0)1413306432M.J.P.:E-mail:miles.padgett@,Telephone:+44(0)1413305389Emerging Technologies in Security and Defence; and Quantum Security II; and Unmanned Sensor Systems X,edited by K. L. Lewis, R. C. Hollins, T. J. Merlet, M. T. Gruneisen, M. Dusek, J. G. Rarity, E. M. Carapezza,Proc. of SPIE Vol. 8899, 889902 · © 2013 SPIE · CCC code: 0277-786X/13/$18 · doi: 10.1117/12.2032739single-pixelphotodetector computer light projectorobject analogue-digital converter structured illumination Figure 1.The DLP illuminates the object (toy soldier)with a series of computer generated binary Hadamard patterns.The light reflected from the object is collected on two spatially separated single-pixel photodetectors.The signals from the photodetectors are measured by the computer via the analogue-to-digital converter,and used to reconstruct a ghost image for each photodetector.the field of single-pixel cameras,12the difference being an interchange of the light source and detector,in other words the programmable component is either used to filter the detected light or to structure the illumination.In both single-pixel cameras and GI systems,inverting the known patterns and the measured intensities is a significant computational problem.A number of sophisticated algorithms have been developed over the years to improve the signal-to-noise ratio (SNR)for different GI systems 13,14but with appropriate normalization 15a simple iterative algorithm was adopted for this experiment.All previous GI experiments have been restricted to relatively small 2-dimensional (2D)images,mainly of 2D template objects or 2D outlines of 3-dimensional (3D)objects.In this work we overcome previous limitations of computational GI and capture the 3D spatial form of an object by using several single-pixel detectors in different locations.A spatially identical 2D image is derived from each detector but appears as if it is illuminated from a source located at the detector paring the shading information in the images allows the surface gradient and hence the 3D form of the surface to be reconstructed.2.EXPERIMENTAL SETUPThe experimental setup is shown in Fig.1.The source of structured illumination is a DLP (Texas Instruments Light Commander)that has been modified to use a high-power 850nm wavelength LED for providing near-infrared binary illumination.The light intensity reflected from the scene is measured by two silicon photodiodes (filtered for 850±20nm),located at different positions and directed towards a common point on the object.An analogue-to-digital converter is used to digitize the photodetector signals before they are processed by a computer which is also used to generate the structured illumination.The DLP contains a digital micro-mirror device (DMD),which is a 1024×768array of mirrors (10.8µm pitch)that can each be electronically controlled.The glass panel that protects the DMD is coated for optimum transmission at visible wavelengths,however the device itself has a much wider operational bandwidth (300nm −2µm),enabling the use of this technique at wavelengths potentially unsuitable for existing CCD technology.The illumination structure comprises of binary patterns having an equal black to white ratio,and which are projected onto the scene using a Nikon 50mm focal length lens.For every pattern that is projected a correspond-ing intensity is measured by each photodetector,which is used by the computer algorithm for reconstructing a ghost image.left right top bottomFigure2.The raw images produced from each photodetector located around the object(as indicated)are reconstructed using an iterative algorithm(described in the text).The spatial information in each image is identical;however,the apparent illumination source is determined by the location of the relevant photodetector.DMD-based projectors create colour images by displaying24binary images(bit planes)per frame in quick succession.By alternating between a binary pattern and its inverse in subsequent bit planes we can demodulate the measured signal at the frequency of the bit plane projection(1440Hz)to isolate the back-reflected signal from light sources at other frequencies such as room lighting.Importantly,the fact that the speckle pattern has equal numbers of black and white pixels enables normalization of the measured differential signals for each pattern,which has been shown to improve the SNR of thefinal reconstruction.153.TWO-DIMENSIONAL IMAGE RECONSTRUCTIONBoth iterative and inversion techniques can be applied when reconstructing computational ghost images from a series of M projected binary patterns P(x,y)and measured signals S i.In this experiment we employ an iterative algorithm to avoid storage of large pattern arrays and signals in computer memory.The2D reconstruction of the image I(x,y)is obtained by averaging the product of the(zero-mean)measured signals and patterns,given byI(x,y)=1MMi=1S i P i(x,y),(1)for M iterations.In our earlier work,random binary patterns have been used,however any overlap between patterns inherently introduces redundancy and thus the number of patterns,M,required to form a high-quality N-pixel image can be significantly greater than the Nyquist limit(M>>N).In this work we chose instead to project a complete set of Hadamard patterns,with a resolution set at96×96super-pixels(where1super-pixel is8×8pixels on the DMD).Each Hadamard pattern is orthogonal,and thus a new piece of spatial information is acquired with each iteration,reducing the number of patterns required to M=N.Using1we obtain a separate2D reconstruction of the object(toy soldier with approximate dimensions 25cm×10cm×4cm)for each photodetector,as shown in Fig.2.The ghost images produced with this technique are derived from the same set of patterns projected from a single source at afixed location,thus the spatial information in both images is identical.However,the intensity distribution in each image is different,because the apparent lighting of the object is dependent on the location of the detector used to record the backscat-tered light(a consequence of reciprocity in imaging systems).Therefore,computational ghost imaging has two key differences compared to a camera imaging system:the location of structured illumination determines the perspective and the detector position determines the lighting.4.THREE-DIMENSIONAL IMAGE RECONSTRUCTIONDepth information of a scene is normally lost in a2D image,however one well established technique for the retrieval of3D information,called‘shape from shading’16,17(SFS)uses the shading information in the image caused by geometrical features as an additional degree of freedom from which to infer surface normals.When using only one image the problem is underscontrained since the surface normals represent two degrees of freedom per pixel with respect to the line of sight,and thus many SFS methods make several assumptions,including uniform Lambertian surface reflectance.An extension of this technique called‘photometric stereo’18,19uses multiple images acquired from afixed perspective with different illumination positions,similar to the images acquired form a computational ghost imaging system with multiple single-pixel detectors.For our system we consider that the intensity of a pixel(x,y)in an image obtained from the i th detectorcan be expressed asI i(x,y)=I sαˆdi·ˆn,(2)where I s is the source intensity,αis the surface reflectivity,ˆd i is the unit detector vector pointing from the object to the detector,andˆn is the surface normal unit vector for the object.Thus,for P images,we can write Eq.2asI(x,y)=I sα(D·ˆn),(3) where D is an array containing the unit detector vectors and I(x,y)is an array containing the corresponding image intensities for that pixel.Thus for any pixel(x,y),the unit surface normal isˆn=D−1·I(x,y)I sα,(4)and the surface reflectivity(albedo)isα=|D−1·I(x,y)|.(5) The surface gradient at each pixel can therefore be calculated by evaluating the change between adjacent pixels in the surface normal map,from which3D surface information can be obtained by integration.The integration of surface gradients is performed by starting at the centre of the object and working outwards.The relative surface depth at any pixel is therefore estimated from the height of the neighbouring pixel and the surface gradient.Where more than one nearest neighbour exists,the height is estimated based on the mean of those pixels.A subsequent optimization step is performed on the3D surface,varying the depth of one pixel at a time for each iteration,where the cost function is the sum of the squared differences between the gradients of the reconstructed surface and the gradients recovered from the photometric stereo measurement.This step helps to improve the overall shape and remove spike features that occur at the edge of the3D surface.Our3D computational GI system has been shown to provide accurate3D reconstructions of large surfaces with reasonable geometric complexity,demonstrating a root mean square error of less than4mm.11From the final2D ghost images shown in Fig.2we applied the prescribed method for reconstructing a3D profile of a large toy soldier,shown in Fig.3.5.CONCLUSIONSUtilizing a high power850nm LED as the source of illumination within a commercially available digital light projector,we have demonstrated that large objects can be imaged by lensless single-pixel detectors using compu-tational ghost imaging methods.Our system was capable of projecting a series of orthogonal Hadamard patterns, and processing with an iterative algorithm at a rate of approximately650Hz,from which an image(96×96 pixel resolution)with reasonably high contrast was retrieved after approximately15seconds.This time may be reduced by employing a faster DMD or by reducing the image resolution.Spatially separating four single-pixel detectors is shown to produce ghost images with different shading profiles,determined by their respective po-sitions,thus enabling photometric stereo techniques to be applied for retrieval of3D surface information.ThisFigure3.Rendered views of the3D reconstructed toy soldier derived by integration of the surface normal data and overlaid with the surface reflectivity data.approach,demonstrated at near infrared wavelengths,can be extended for imaging applications at nearly any desired wavelength where light sources and single-pixel detectors exist,potentially in regions where camera based technology is expensive or limited.The deployment of simple,inexpensive,detectors for2D and3D imaging may have importance in various military applications,for example in chemical characterization and or use in hazardous environments.ACKNOWLEDGMENTSThe authors acknowledge the UK Engineering and Physical Sciences Research Council forfinancial support. M.J.P thanks the Royal Society and the Wolfson Foundation.REFERENCES1.Pittman,T.B.,Shih,Y.H.,Strekalov,D.V.,and Sergienko,A.V.,“Optical imaging by means of two-photon quantum entanglement,”Phys.Rev.A52,R3429–R3432(1995).2.Strekalov,D.V.,Sergienko,A.V.,Klyshko,D.N.,and Shih,Y.H.,“Observation of two-photon ghostinterference and diffraction,”Phys.Rev.Lett.74,3600–3603(1995).3.Bennink,R.S.,Bentley,S.J.,and Boyd,R.W.,““two-photon”coincidence imaging with a classical source,”Phys.Rev.Lett.89,113601(2002).4.Gatti,A.,Brambilla,E.,Bache,M.,and Lugiato,L.A.,“Correlated imaging,quantum and classical,”Phys.Rev.A70,013802(2004).5.Valencia,A.,Scarcelli,G.,DAngelo,M.,and Shih,Y.,“Two-photon imaging with thermal light,”Phys.Rev.Lett.94,063601(2005).6.Jack,B.,Leach,J.,Romero,J.,Franke-Arnold,S.,Ritsch-Marte,M.,Barnett,S.M.,,and Padgett,M.J.,“Holographic ghost imaging and the violation of a bell inequality,”Phys.Rev.Lett.103,083602(2009).7.Shapiro,J.H.and Boyd,R.W.,“The physics of ghost imaging,”Quantum Inf.Process11,949(2012).8.Shapiro,J.H.,“Computational ghost imaging,”Phys.Rev.A78,061802(R)(2008).9.Bromberg,Y.,Katz,O.,and Silberberg,Y.,“Ghost imaging with a single detector,”Phys.Rev.A79,053840(2009).10.Sen,P.,Chen,B.,Garg,G.,Marschner,S.R.,Horowitz,M.,Levoy,M.,and Lensch.,H.P.A.,“Dualphotography,”ACM Trans.Graph.24(3),745–755(2005).11.Sun,B.,Edgar,M.P.,Bowman,R.,Vittert,L.E.,Welsh,S.,Bowman,A.,and Padgett,M.J.,“3Dcomputational imaging with single-pixel detectors,”Science340,844–847(2013).12.Duarte,M.F.,Davenport,M.A.,Takhar,D.,Laska,J.N.,Sun,T.,Kelly,K.F.,and Baraniuk,R.G.,“Single-pixel imaging via compressive sampling,”IEEE Signal Process.Mag.25,83(2008).13.Katz,O.,Bromberg,Y.,and Silberberg,Y.,“Compressive ghost imaging,”Appl.Phys.Lett.95,131110(2009).14.Ferri,F.,Magatti,D.,Lugiato,L.A.,and Gatti,A.,“Differential ghost imaging,”Appl.Phys.Lett.104,253603(2010).15.Sun,B.,Welsh,S.S.,Edgar,M.P.,Shapiro,J.H.,and Padgett,M.J.,“Normalized ghost imaging,”Opt.Express20,16892(2012).16.Horn,B.K.P.,“Understanding image intensities,”Artif.Intell.8,201(1977).17.Brooks,M.J.and Horn,B.K.P.,[Shape from shading],MIT Press,Cambridge,MA(1989).18.Woodham,R.J.,“Photometric method for determining surface orientation from multiple images,”Opt.Eng.19,191139(1980).19.Horn,B.K.P.,[Robot Vision],MIT Press,Cambridge,MA(1986).。