Interaction with virtual game through hand gesture recognition

Interaction with Virtual Game through Hand GestureRecognitionSiddharth S. Rautaray1 and Anupam Agrawal21 Indian Institute of Information Technology, Allahabad, sr.rgpv@12 Indian Institute of Information Technology, Allahabad, anupam69@2Abstract—Hand gesture recognition systems for virtual reality applications provides the users an enhanced interaction experience as it integrates the virtual and the real world object. Growth in virtual environments based upon computer systems and development of user interfaces influence the changes in the Human-Computer Interaction (HCI). Gesture recognition based interaction interface, endow with more realistic and immersive interaction compared to the traditional devices. The system enables a physically realistic mode of interaction to the virtual environment. The Hand gesture recognition system based interface proposed and implemented in this paper consists of a detection, tracking and recognition module. For the implementation of these modules various image processing algorithms as Camshift, Lucas Kanade, Haar like features etc has been employed. Comprehensive user acceptability has been considered to exhibit the accuracy, usefulness and ease of use to the proposed and implemented hand gesture recognition system. Hand gesture communication based vocabulary offers many variations ranging from simple action of using our finger to point at to using hands for moving objects around to the rather complex one like expression of the feelings. The proposed hand gesture recognition system offers intensions to traditional input devices for interaction with the virtual environments. The gesture based interaction interface being proposed here can be substantially applied towards many applications like Virtual Reality, Sign Language and Games. Though the present paper considered games as the application domain.I.INTRODUCTIONThe evolution and research efforts in Human-Computer Interaction (HCI) have been enormously successful, and have vitally changed computing [1]. Even the magnificent growth of the World-Wide Web is unswerving result of HCI research: employing the hypertext technology to the browsers allows one to navigate a link worldwide by click of mouse. Interface enhancements more than anything else has actuated this quick-tempered growth. The basic interactions included Direct Manipulation of graphical objects, physical interacting devices like mouse, keyboard etc. Various upcoming interaction interfaces includes Gesture Recognition, Multi-Media, 3-D, Virtual Reality, "Augmented Reality”, Computer provides Cooperative Work, Natural language and speech. User interfaces are probable to value-addition to the competitive advantages to the future, as both hardware as well as the basic software have become merchandise. It seems through computer science done HCI research only disseminates out into products. The interfaces of the future need to use gesture based recognition systems, speech recognition and generation based systems, “intelligent agents," flexible as well as adaptive interfaces, video, and many other advanced technologies. The user interfaces in vogue are seldom designed with the dexterity performance in desired virtual reality. The key board, mouse etc lack the sensitivity desired in virtual reality application. Eventually the researchers working the area of Human Computer Interaction made a common emphasis to design and develop the user interfaces capable enough fulfill the intended performance criteria desired in the virtual environment. Such interfaces can be developed by exploiting the phenomena of instinctive communication and manipulation accomplishments of humans facilitating the human computer interaction even at the further higher degree. This section provides a brief overview and state of art related to hand gesture recognition in human computer interaction. Section 2 shows the architecture design of gesture recognition system. Section 3 discusses the implementation and performs the analysis of results obtained. The conclusion and future scope in mentioned in section 4.The advances in the computational and semiconductor technology made the intelligent devices affordable to extent by which a numerous applications have been developed. It involves multimedia technologies empowered virtual and augmented reality as there key component. Besides some interaction steps involving the swiftness in reflex such as double-click, and drop & drag dissuades users particularly elderly or affected by some physical or cognitive disorder [2]. Hence Human computer Interaction (HCI) is persistently in pursuance of being inexpensive, easier in use, versatile and fast so that it leads to the virtual environment to make a world where the interaction feels real. For accomplishing it, a successful a 3D user interface should be natural, intuitive,powerful enough and simple to acquire allowing users to accomplish the necessitated tasks. The present contact based devices like accelerometers, data glove [3], sensors/actuators and other input devices used to capture the user movement and control the selection, manipulation and movements of objects in virtual scenes [4]. The use of these devices is hindered by number factors such as awkwardness, unintuitive, rigidity, and prone to distortion from the physical environment. These devices have a cost which is prohibitive for their frequent use by the general user. Hence they are mostly used by highly trained professionals like the surgeons and ace pilots to train and accomplish their operations in the virtual environment. To overcome the challenges generated from these technologies more novel contemporaries of HCI has been focused. Gesture input can be categorized into different categories depending on various characteristic [5]. One of the categories is deictic gesture that refers to reaching for something or pointing an object. Accepting or refusing an action for an event is termed as mimetic gestures. It is useful for language representation of gestures. An iconic gesture is way to define an object or its features. Different researchers have proposed and implemented different pragmatic techniques for gesture as the input for human computer interfaces. Liu and Lovell [6], proposed an interesting technique for real time tracking of hand capturing gestures through a web camera and Intel Pentium based personal computer. The proposed technique is implemented without any use of sophisticated image processing algorithms and hardware. Controlling VLC media player using hand gesture recognition is done in real time environment using vision based techniques [7]. Xu et al. [8] used contact based devices like accelerometer and EMG sensors for controlling virtual games. Conci et al. [2] designs an interactive virtual blackboard by using video processing and gesture recognition engine for giving commands, writing and manipulating objects on a projected visual interface. There are several studies on the hand movements especially gestures, by modeling the human body and creating a amalgamate body of knowledge which generates challenges from a mathematical viewpoint [9].II.A RCHITECTURE D ESIGNThe application uses an integrated approach for hand gesture recognition. Figure 1 shows the architecture design a gesture recognition system. Images are captured from camera and passed through following phases/algorithms. The procedure starts by acquisition phase. As the standard input devices like keyboard, position & pointing devices are has been declared as non grata for this domain of applications. Our endeavor was oriented towards possible alternatives for user friendly and smart interfaces inspired by natural behavior of the users in real-world scenario. While making the choice of image capturing devices we have to look the point of installation as well. For example, we need robust and very sensitive camera in the hostile environment at the public places whereas the same performance can be accomplished by some relatively less sensitive camera in control environment like homes or laboratories. This is the reason that our system has to yield the better performance even in poor illumination and cheaper cameras. The system has to work within the real time constraints. Hence the procedure has to limit the parameters to the minimal possible level by removing the unnecessary information at the first instance.Figure 1. Virtual game architectureAs the initialization phase is over, the haar cascade [10], [11] classifier is responsible for locating hand position and classifying gestures (open, close, pointing, etc.). The Haar-like [11] characteristics are also comparatively robust to the noise and lighting changes as the features will figure the gray level deviation between the rectangles (black and the white). The noise and lighting variations also strike the pixel measures on the entire characteristic region, which could be counteracted. The hand is tracked using background subtraction and motion detection. The noise generated in the application is reduced using eroding and dilation. The application uses the Camshift and Lucas Kanade Optical Flow technique for tracking of hand. The following step was implemented to attain a robust hand motion tracking thin the video sequence. This is done by using CAMSHIFT tracking algorithm [13]. The tracking technique is made more robust by using Lucas-Kanade opticalflow [14]. It provides an accurate local tracking. As the hand is tracked a contour is mapped with the corresponding hand which further extracts corresponding convex hull (area). The recognition has been done through modeling of the hand by mapping to the number of defect formed in it. The system tracks the defects rendered by the hand and maps to a meaningful command that has been assigned in the application. The application tracks the gestures performed by the hand (defects formed) and maps to the appropriate action assigned in the gesture vocabulary. Thus focusing on areas matching to human skin color. Following sections provide the insight about the two main processing blocks: hand tracking and recognition.III.I MPLEMENTATION AND R ESULT A NALYSISThe image processing techniques used in the application for interactions with virtual game have been implemented in C++ with the use of OpenCV Library. Virtual games (front end) have been designed using OpenGL library. The web cam used in the experimental setup captures image sequences at the resolution of 320x240. Practical experiments show that our application is implemented well in environments with little noises (i.e., existence of objects whose color is similar to human skin) and with the balanced lightning condition.Once the hand has been detected the application further tracks different gestures of the user performed by his hand and generates contour around it. The application uses four hand gestures defined within the application for interaction with virtual game. Different gestures along with their assigned commands (functions) to control the application that is further used for the virtual game control. Following figures shows the results obtained from gestures which are used for interaction with the virtual game.•Depicting a virtual gaming environment used in the experimental application having a virtual man and aseries of virtual red birds trying to attack the virtualman. The game player can use various hand gesturesto implement different gaming commands asrepresented by figure 2.Figure 2. Virtual game•Depicting the gesture of ‘punch’ for virtual gaming environment used in the experimental application.Controlling the attack command attacking the virtualred birds in the virtual game scenario as depicted infigure 3.Figure 3. Gesture for punch •Depicting the gesture for ‘grab’ in virtual gaming environment used in the experimental application to implement the stick grabbing by virtual human as shown in figure 4.Figure 4. Gesture for grab•Depicting the gesture for ‘throw’ in virtual gaming environment used in the experimental application controlling the throw of stick from virtual man’s hand as depicted in figure 5.Figure 5. Gesture for throw•Depicting the gesture to ‘move forward’ in virtual gaming environment used in the experimental application to implement the forwarding of virtual birds as shown in figure 6.Figure 6. Gesture for move forwardThe performance and viability of the application is done in the experimental setup where we have interacted with the virtual game. Performance of the application is tested by manipulating with the virtual game through 15 different users. The performance have been tested for the application in different environmental condition based variations in parameters like lighting changes, number of different users etc. Also the application has been implemented for different gestures that could be associated with different possible commands in the virtual game. Following Graphical representation of results compares the hit and miss ratio of the hand gestures recognition in the virtual gaming environment,with fixed number of players being 15.Figure 7. Figure depicting the hits verses misses of hand gesture by thevirtual gameResults for the recognition rate (%) of different gestures used by different users for interaction with virtual game have been depicted in Figure 8. Results presented the maximum recognition rate of 93.33 %in the case of grab gesture and the lowest recognition rate of throw gesture having the percentage value of 80%.Though the move forward gesture also has the highest value of 93.33% of gesture recognition. The punchgesture has the recognition rate of 86.67%.Figure 8. Figure depicting the success percentage of the hand gesturerecognition in the virtual gaming environment.IV. C ONCLUSIONThe furtherance’s in the field of Human-Computer Interaction is majorly been in quality of interaction. Hence instead of designing regular interfaces, research branches have had focus on the concepts of intelligent and adaptive interfaces rather than command or action based ones. Many of input providing technologies require physical touch and also thereare other variations that provide input to the application without using physical touch as they may be based on other human characteristics like speech, hand gesture etc. The advantage of the usage of hand gesture based input mode is that this method provides the user ability of interacting with the application from a distance without using the traditional input devices like keyboard or mouse. The application of virtual game controlling through hand gestures proposed and implemented in the present paper provides a suitable efficient and user friendly interface between human and computer for interaction with virtual game using hand gestures. The user also can have an interaction with the gaming applicationthrough an interface that provide gesture as an input according to interested function defined within the application.R EFERENCES[1] X. Zabulis, H. Baltzakis and A. Argyros, Vision-based HandGesture Recognition for Human-Computer Interaction. In: The Universal Access Handbook. LEA, 2009.[2] N. Conci, P. Cerseato, F. G. B. De Natale, Natural Human-Machine Interface using an Interactive Virtual Blackboard, In Proceeding of ICIP 2007, 181-184, 2007[3] N. Karlsson, B. Karlsson, P. Wide, A glove equipped with fingerflexion sensors as a command generator used in a fuzzy control system, IEEE Trans. On Instrumentation and measurement, 1330-1334, October. 1998[4] S. Mitra, T. Acharya, Gesture Recognition: A survey, IEEETransactions on Systems, Man and Cybernetics (SMC) - Part C: Applications and Reviews 37(3): 211-324, 2007.[5] J. Eisenstein, R. Davis, Visual and linguistic information ingesture classification, In Proceedings of 6th International conference on Multimodal interfaces. ACM Press, 113-120, 2004. [6] N. Liu, B. Lovell, Mmx-accelerated realtime hand trackingsystem. In Proceedings of IVCNZ, 26–28, November 2001.[7] S.S. Rautaray, A. Agrawal, A Novel Human Computer InterfaceBased On Hand Gesture Recognition Using Computer Vision Techniques, In Proceedings of ACM IITM’10, 292-296, December 28-30, 2010.[8] Z. Xu, C. Xiang,W. Wen-hui, Y. Ji-hai, V. Lantz, W. Kong-qiao.Hand Gesture Recognition and Virtual Game Control Based on 3D Accelerometer and EMG Sensors, In Proceedings of IUI’09,401-406, February 2009[9] B. Yi, F. C. Harris Jr., L. Wang, Y. Yan, “Real-time natural handgestures”, In Proceedings of IEEE Computing in science and engineering, 92-96, , May-June 2005.[10] Q. Chen, N.D. Georganas, E.M. Petriu, Realtime Vision-basedHand Gesture Recognition Using Haar-like Features, In Proceedings of IEEE Instrument and Measurement Technology Conference, 1-6, 2007.[11] Viola and Jones, Rapid object detection using boosted cascade ofsimple features", In Proceedings of Computer Vision and Pattern Recognition, I-511 - I-518, 2001.[12] C.H, Messom, A.L.C., Barczak, Fast and Efficient Rotated Haar-like Features Using Rotated Integral Images", In Proceedings of Australian Conference on Robotics and Automation (ACRA2006), 1-6, 2006.[13] G. R. Bradski. Computer video faces tracking for use in aperceptual user interface. Intel Technology Journal, Q2, 1-15, 1998.[14] B. J. Yves, Pyramidal implementation of the lucas-kanade featuretracker. Microsoft Research Labs, Tech. Rep., 1999.。