Methods for Integrating Phonetic and Phonological Knowledge

Methods for Integrating Phonetic and Phonological Knowledge inSpeech InversionSORIN DUSANCenter for Advanced Information ProcessingRutgers University96 Frelinghuysen Road, Piscataway, NJ 08854U.S.A.sdusan@ /~sdusanAbstract: - Exploiting the information about the vocal tract shape that produced the speech has been appealing to speech researchers and scientists for a long period of time. Experimental studies that included the articulatory information from physiological measurements supported the idea that this information could be useful in a number of areas of speech science and technology. However, the estimation of articulatory trajectories from the speech acoustic signal is known as a difficult and ill-posed problem. Among the relatively recent methods proposed to solve this problem are those trying to integrate linguistic knowledge in the form of phonetic or phonological constraints. This paper reviews and discusses upon methods for applying phonetic and phonological constraints to provide unique solutions to the acoustic-to-articulatory inversion.Key-Words: -speech inversion, acoustic-to-articulatory mapping, vocal-tract estimation, phonetics, phonology1IntroductionThe recovery of articulatory state from the speech acoustic signal could have direct applications in different areas of speech technology and science, such as automatic speech recognition, speech synthesis, speech coding, in building aids for teaching the hearing impaired people and as a means of study in phonetics and phonology. But the acoustic-to-articulatory transformation is a complex non-linear and non-unique problem, in the sense that there is no unique vocal tract shape or articulatory vector mapping to a given speech spectrum or acoustic vector. Numerous studies over the years have approached this speech research area, usually called speech inversion, and tried to provide either analytical or data-driven solutions to solve this problem. Different methods have been proposed to constrain the search and obtain a single optimal solution to the acoustic-to-articulatory mapping.Some of the most accurate approaches of the speech inversion were those methods applying continuity or dynamical constraints to the state of articulators. Such methods based on dynamical constraints were successfully applied to obtain optimal solutions to the acoustic-to-articulatory mapping for some classes of speech sounds, especially for vowels. However, these methods did not provide the same accuracy in estimating the vocal tract shapes or the articulatory states for all classes of speech sounds. New constraints were necessary to extend these methods to all classes of speech sounds. It has been observed that different classes of speech sounds, or even different sounds from the same class, need to be modeled differently or with different parameters in order to obtain a relatively uniform accuracy over all speech sounds. The need of integrating some sort of linguistic knowledge emerged as a solution to further constrain the acoustic-to-articulatory mapping for all classes of speech sounds. Various methods have been proposed to this problem, from simply building two different models to treat differently the vowels and consonants, to applying more complex phonological and higher-level linguistic constraints. This paper has the following structure. Section 2 presents a short introduction of the speech inversion problem. Section 3 reviews and discusses methods for using phonetic and phonological knowledge in speech inversion. Section 4 presents a summary and some conclusions representing the opinion of the author of this paper.2The Speech InversionThe speech inversion is a distinctive speech research area, approached by researchers and scientists to provide better solutions to different applications in speech science and technology. This section presents a brief introduction to the speech inversion research area. Those interested in a more elaborate discussionon this topic are recommended to read a review paper, such as Schroeter and Sondhi [15]. The acoustic-to-articulatory transformation f*, represented in the equation x(t) = f* (y), is both non-linear and non-unique. Here x represents the articulatory vector, y is the acoustic vector and f is the articulatory-to-acoustic speech production function. The solution of this equation is usually obtained by applying a non-linear optimization technique, according to a certain criterion. Based upon the modeling technique of the acoustic-to-articulatory transformation f*, the speech inversion methods can be divided into two classes. The first class of methods is represented by those trying to provide analytical solutions using acoustic models of the vocal tract. Among the first researchers who approached the acoustic-to-articulatory inversion were Mermelstein and Schroeder, 1965 [8], who tried to estimate a smooth area function from formant frequencies using an analytical solution. Another early study based on analytical solution was published in 1973 by Wakita [18], who proposed an inverse filtering method of the acoustic speech waveform using the reflection coefficients of the acoustic tube model. But serious constraints prevented the application of these methods to provide unique and practical solutions to the speech inversion problem for all classes of speech sounds. The multiple-to-one nature of the articulatory-to-acoustic transformation has been proven by modeling the speech production using multiple acoustic tubes, Flanagan [5], and by bite-block experiments of articulatory compensation, Lindblom et al., [7]. The second class of methods is represented by those providing numerical, heuristic solutions based on data collections of simultaneously acquired acoustic and articulatory measurements. These methods are employing different mapping techniques between the acoustic and articulatory parameters, e.g., model matching, artificial neural networks, extended Kalman filtering, codebook search, etc., for providing optimal, unique solutions according to some optimality criteria. A special case of this second class of methods is that in which the acoustic-articulatory database is acquired from a vocal-tract model such as in Atal et al., [1]. That study and many other studies that followed it dealt with the non-linearity by employing local linearization techniques in a small neighborhood. To deal with the non-uniqueness of the speech inverse transformation, different continuity constraints have been applied upon the articulatory state. These constraints are also called dynamical constraints because of their influence upon the dynamics of the articulators or vocal tract shapes. However, these methods did not provide accurate, general solutions to the speech inversion problem for all classes of speech sounds. Higher-level constraints have been proposed to extend the speech inversion methods to all classes of speech sounds and those representing a special class are presented in the next section.3Using Phonetic and Phonological Knowledge in Speech InversionOne of the first studies that integrated phonetic information into a method of estimating articulatory parameters from the speech signal was proposed by Shirai and Kobayashi in 1983 [16]. The authors employed dynamical models with non-stationary dynamical parameters, corresponding to different phonemes, in an application of continuous speech recognition based on estimated articulatory targets. They successfully applied this method to estimate the motor commands from continuously uttered sequences of five Japanese vowels and to recognize these vowels. This technique was later applied by its authors in a study of estimating the articulatory motion from the speech signal using the model matching method, Shirai and Kobayashi [17]. However, it has been applied only to vowel sequences, although the final goal was to improve the accuracy in continuous speech recognition.Another method which used higher-level constraints upon the articulatory state in acoustic-to-articulatory mapping was proposed by Rahim et al., in 1991 [11]. Although this method did not impose purely phonetic or phonological constraints to estimate the articulatory parameters from speech acoustics, this study is worth to be mentioned here because of its similarity to those using this kind of constraints. In this study, the authors used an assembly of multi-layer perceptrons (MLP), each MLP being designed to cover a specific region of the articulatory space. This method of acoustic-to-articulatory mapping was used in a speech synthesis application. The assembly of MLPs covered 128 regions in the articulatory space, which did not have specific affiliation with the speech sounds. The acoustic input vector was provided to all MLPs simultaneously and the articulatory output was selected for each frame from a different MLP, according to a criterion that ensured the smoothness of the estimated vocal tract shapes. This method was applied to all classes of speech sounds. However, using the estimated articulatory parameters the synthesizer was unable to generate some sounds like /l/ and /r/.A new method for inferring articulation and recognizing gestures that integrated phonetic knowledge was proposed by Papcun et al., in 1992 [10]. The authors used a separate neural network for each articulator to map from acoustic parameters to articulatory parameters. They used real acoustic-articulatory data recorded from an X-ray system. This represented a new way of using phonetic knowledge in acoustic-to-articulatory mapping because the phonetic information integrated was about the place of articulation and not about the classification of the speech sounds. Three networks were used for mapping from acoustic vectors for the following articulators: lower lip, tongue tip and tongue dorsum. The method was tested on continuous utterances consisting of consonant-vowel sequences. The inferred articulatory trajectories and the recognition of gestures were accurate and the method proved that integrating phonetic knowledge in acoustic-to-articulatory mapping could be useful. Another approach that proposed the integration of phonetic knowledge into the process of recovering the hidden articulatory state from acoustics was published by Ramsay and Deng in 1995 [12] and 1996 [13]. This study approached the inversion of the articulatory-to-acoustic transformation as an internal Markov process of a stochastic target model for an automatic speech recognition system. The authors employed state-space equations with different dynamical parameters, corresponding to each phonetic unit, such as a phoneme or phone, and associated to these units some hidden articulatory targets. The phonetic constraints applied in this study were implemented in the form of different dynamical parameters in the state-space models and different targets for each phonetic unit. Although the idea of treating the articulatory state as a hidden stochastic Markov process is remarkable, the huge computational cost of that stochastic method prevented so far its application to estimating the articulatory states from real speech.A different approach for integrating phonetic knowledge into a method of deriving articulatory representations from speech was proposed by Richards et al., in 1996 [14]. In this study the authors used a dynamic programming method to search for an optimal articulatory vocal tract shape in an articulatory codebook derived from a vocal tract model. The search criterion was to minimize a cost function based on the acoustic similarity and the continuity of the estimated vocal tract shapes. The authors integrated into this method some simple phonetic knowledge by constructing two different articulatory codebooks, one for voiced and one for unvoiced speech sounds. The vocal tract shape estimation performed well and the method was successfully applied in estimating the place position for stop and fricative consonants.A new method of applying higher-level phonological constraints to speech inversion was proposed by Dusan in 1997 [2]. The new idea was to build different dynamical models and articulatory-acoustic codebooks for each possible pair of successive speech sounds in a language. This method integrated higher-level phonological knowledge into the acoustic-to-articulatory mapping by modeling phonological sequences of speech sounds and not elementary sounds like phonemes or phones. The general framework of this method was based on the extended Kalman filtering. The method was successfully implemented and published in Dusan [3] and Dusan and Deng [4]. In this study, about 80 different models were built, covering different combinations of vowels, fricatives, plosives and nasals. The most important contributions of this study were the application of phonological constraints to the acoustic-to-articulatory mapping in the form of diphone models, the modeling of the articulatory-acoustic function by sub-functions, each corresponding to a diphone and the integrated method of recognizing diphones based on estimated articulatory trajectories. This method was evaluated on different articulatory-acoustic codebooks, either generated from a vocal tract model or created from physiological measurements. Figure 1 presents a sample of estimated articulatory trajectories based on this method for a continuous utterance /eh s eh/, not included in the training data. The solid lines in this figure represent the real articulatory trajectories recorded simultaneously with the speech acoustic signal using an electromagnetic midsaggital articulograph (EMA) made by Carstens GmbH. The plots represent the x and y coordinates of three articulators, lower lip (LL), tongue body (T2) and tongue dorsum (T3), respectively. The horizontal axes represent the frame number. The average root mean square error is displayed in the seventh plot. At the bottom, the midsaggital profiles are displayed at the middle frames of each phoneme, including the profile of the hard palate. The main difficulty in applying this method is represented by the increased number of models necessary to account for all possible diphones in a language. However, the method showed a potential benefit in estimating the articulatory trajectories from the speech signal and the mentioned difficulty can be treated in a way similar to that in which the triphones are used in automatic speech recognition.An interesting approach trying to apply phonological constraints corresponding to sequences of more than two successive phonemes was proposed by King and Wrench in 1999 [6]. These authors applied the estimation of hidden articulatory state in a syllable classification task using the stochastic segment modeling and the dynamical modeling framework. They built 16 different dynamical models, each corresponding to a different CVC syllable, and applied this method for estimating articulatory trajectories from LPC acoustic parameters. The codebook was created using physiological measurements from a Carstens EMA system. In this approach, however, the authors used a linear dynamic transformation matrix and a linear acoustic observation function to model all the sounds in a syllable and this represents a very crude approximation. The dynamic parameters of different sounds are different and the articulatory-to-acoustic mapping is non-linear and different for different sounds in a syllable. Neither quantitative nor qualitative results regarding the estimated articulatory states were reported in that study. However, the authors compared the classification results of this method with those from other methods not based on a hidden articulatory state. Unfortunately, the classification results using the0653130619592612msec.y<h i n i-+a bu r a o s o s o guy O n a mo n o d a>10mm LVT4T3T2T1LLULLJspFig. 2: Observed (thin lines) and estimated (thick lines) articulatory trajectories for a continuous utterance, from Okadome et al., [9].estimated articulatory state were the worst and their approach of extending the dynamical modeling to syllable units was not successful in that experiment.A new method of using phonemic information to recover the articulatory movements was proposed by Okadome et al., in [9]. These authors employed a search technique of an acoustic-articulatory codebook generated from physiological measurements from a Carstens EMA system. The search was using the minimization of a cost function based on segmental spectral distance, articulatory distance between successive states and a distance between the position designated by the code vector and that predicted by the minimum-acceleration model of the phonemic sequence. The third distance was computed using a priori phonemic information of the speech sequence, which was obtained by manual segmentation of the speech acoustic-articulatory data. Figure 2 shows an example of estimated and measured articulatory trajectories for a continuous utterance in Japanese. Nine trajectories are displayed, representing the positions of different points on articulators. The method showed that the phonemic information from an utterance is useful in obtaining a more accurate estimation of articulatory trajectories from the speech acoustic signal.4Summary and ConclusionDuring the last 35 years, a specific speech research area, called speech inversion, has attracted many researchers and scientist. The number of studies published on this topic, during this period of time, has been estimated by the author to be over 100. All of these studies were concerned with the non-uniqueness nature of the acoustic-to-articulatory transformation. The point-to-point mapping between the acoustic space and the articulatory space has been proven to be ambiguous. An important way of constraining this mapping is by applying physiological constraints in the form of continuity of the articulatory states. Although these constraints have been applied in various ways, a complete practical solution has not been provided for all classes of speech sounds. New methods have been proposed to integrate phonetic and phonological knowledge in the form of higher-level constraints. In this paper, a number of 8 studies of speech inversion using this kind of constraints have been briefly reviewed. These studies have used the phonetic and phonological knowledge in different forms. Some of them used the phonetic classification of the speech sounds to build separate codebooks, others used the phonetic information relating to the place of articulation and others used the phonological constraints imposed on combinations of elementary speech sounds.Most of these approaches showed that including higher-level phonetic and phonological knowledge into the speech inversion techniques represents a potential benefit in improving the accuracy of estimated articulatory trajectories from the speech acoustic signal. Very few speech inversion approaches have evaluated the estimated articulatory trajectories by comparing them with the real ones obtained from physiological measurements. Among these, the studies presented in [9] and [3] reported average RMS errors of 1.65 mm and 2 mm, respectively.References:1Atal, B. S., Chang, J. J., Mathews, M. V. and Tukey, J. W., “Inversion of Articulatory-to-Acoustic Transformation in the Vocal Tract by a Computer-Sorting Technique,” J.A.S.A., 63(5):1535-1555, 1978.2Dusan, S., Estimation of Articulatory Dynamics from Speech Acoustics, Ph.D. research proposal, Dept. of Electrical and Computer Engineering, Waterloo University, Canada, March 27, 1997.3Dusan, S., Statistical Estimation of Articulatory Trajectories from the Speech Signal Using Dynamical and Phonological Constraints, Ph.D.thesis, Dept. of Electrical and Computer Engineering, Waterloo University, Canada, April 20, 2000.4Dusan, S. and Deng, L., “Acoustic-to-Articulatory Inversion Using Dynamical and Phonological Constraints,” 5th Seminar on Speech Production: Models and Data, Kloster Seeon, Germany, May 1-4, 2000.5Flanagan, J. L., Speech Analysis, Synthesis and Perception, Springer-Verlag, New York, 1972. 6King, S. and Wrench, A., “Dynamical System Modeling of Articulator Movement,” in Proceedings of the International Congress of Phonetic Sciences, 1999.7Lindblom, B., Lubker, J. and Gay, R., “Formant Frequencies of Some Fixed-Mandible Vowels and a Model of Speech Motor Programming by Predictive Simulation,” Journal of Phonetics, 7:146-161, 1979.8Mermelstein, P. and Schroeder, M. R.,“Determination of Smoothed Cross-Sectional Area Functions of the Vocal Tract from Formant Frequencies,” in Proceedings of the Fifth International Congress on Acoustics, Liege, 1965.9Okadome, T., Suzuki, S. and Honda, M.,“Recovery of Articulatory Movements from Acoustics with Phonemic Information,” 5th Seminar on Speech Production: Models and Data, Kloster Seeon, Germany, May 1-4, 2000. 10Papcun, G., Hochberg, J., Thomas, T. R., Laroche, F., Zacks, J. and Levy, S., “Inferring Articulation and Recognizing Gestures from Acoustics with a Neural Network Trained on X-ray Microbeam Data,” J.A.S.A., 92(2):688-700, 1992.11Rahim, M. G., Kleijn, W. B., Schroeter, J. and Goodyear, C. C., “Acoustic to Articulatory Parameter Mapping Using an Assembly of Neural Networks,” IEEE ICASSP p. 485-488, 1991.12Ramsay, G. and Deng, L., “Maximum-Likelihood Estimation for Articulatory Speech Recognition Using a Stochastic Target Model,”In Proc. EUROSPEECH’95, pages 1401-1404, 1995.13Ramsay, G. and Deng, L., “Optimal Filtering and Smoothing for Speech Recognition Using a Stochastic Target Model,” In Proc. ICSLP, pages 1113-1116, 1996.14Richards, H. B., Mason, J. S., Hunt, M. J. and Bridle, J. S., “Deriving ArticulatoryRepresentations from Speech with Various Excitation Modes,” In Proc. ICSLP, pages 1233-1236, 1996.15Schroeter, J. and Sondhi, M. M., “Techniques for Estimating Vocal-Tract Shapes from the Speech Signal,” IEEE Trans. Speech & Audio Proc., 2(1):133-150, 199416Shirai, K. and Kobayashi, T., “Considerations on Articulatory Dynamics for Continuous Speech Recognition,” In Proc. International Conference on Acoustics, Speech and Signal Processing, pages 324-327, 1983.17Shirai, K. and Kobayashi, T., “Estimating Articulatory Motion from Speech Wave,” In Speech Communication, 5:159-170, 1986.18Wakita, H., “Direct Estimation of the Vocal Tract Shape by Inverse Filtering of Acoustic Speech Waveforms,” IEEE Trans. Audio and Electroacoustics, 21(5):417-427, 1994.。