Introduction Radio Ad Filtering with Machine Learning 1

Radio Ad Filtering with Machine LearningMichael HolmesJosh JonesAdam FeldmanApril 17, 2003 IntroductionOur group explored an application of machine learning techniques to the binary classification of audio samples. The learned binary classification is intended to indicate whether a given sample is taken from music or from another type of audio signal (such as commercials or DJ talking).The datasets used for learning and evaluating this classifier consist of pairs of audio samples and their correct classification. The audio samples were obtained by digitally recording two different Internet radio stations – WREK and 99x. This method enabled the convenient acquisition of a large quantity of suitable audio data in digital format. These recordings were then broken into fixed-length chunks (samples). Each sample was transformed using Mel Frequency Cepstral Coefficients (MFCCs) in order to generate feature vectors.Once the feature vectors were generated, a random subset of data was used as a training set. The system to be trained was an SVM classifier. Several different kernels were experimented with (see results). The classification output from the SVM was used to train an HMM. This allowed accuracy and confidence information, as well as other data (such as time series data), to be considered in classifying a sample.This problem is interesting because it would allow for several applications. For example, many radio stations turn up the volume of their advertisements, which can be an inconvenience, to say the least. Therefore, a successful classifier could be used to monitor the start of commercials, and adjust the volume accordingly. Another application would be to serve as a“station changer”, allowing a listener to automatically change stations to ensure that music is always being played. Thus, a listener could supply a list of favorite stations that could be scrolled through whenever a commercial begins. Further, constant sampling across all the stations could help ensure that a station is chosen on which a song has recently started (as opposed to constantly tuning to stations for the tail ends of songs).Although existing research has focused on attempting to classify audio samples in various ways, little work has focused directly on music/non-music discrimination. Our learning task specifically requires determining what is integral to music, versus other kinds of sounds. This is a subtle, but significant difference between our project and the reference work we located. Most previous work has been on speech/music or genre discrimination. As an example of how this difference can influence system design decisions, notice that features that may be very useful for making certain kinds of distinctions between audio samples may not be useful for making other kinds of distinctions. For example, it might require a much more complicated decision to distinguish pure music from speech, speech with music in the background, effects noises, etc. than to simply distinguish music from speech. We also consider this project to be interesting and worthy of exploration because, to this point, no finished product with the capabilities we envision has been produced. Whether this is because no one has succeeded in enabling a machine to discriminate between music and non-music (by employing machine learning techniques or through other means), or simply because the technology has not been appropriately adapted to be useful in these contexts, it is clear that there is still progress to be made in this area.Prior WorkThe decision to use an SVM to produce initial classification confidences was based on a search of relevant literature detailing existing work on audio sample classification. Although little of this work focused directly on music/non-music classification, we considered it likely that techniques successful in some audio classification tasks would enjoy similar success when applied to our task. In particular, previous work shows SVMs to be superior to boosting for classification of audio samples into 16 distinct categories [1]. A number of additional studies confirm the applicability of SVM learning to this domain area [2, 5]. One other technique, “Nearest Feature Line” (NFL) was also described in several works, and appears in some cases within the audio classification domain to be superior to SVMs in both training time and accuracy of the resulting classifier [3,4]. The general principle behind NFL learning is to classify new instances by calculating the distance in feature space between the feature vector to be classified and a line between each possible pairing of training points within each class. The classification chosen is then whichever class contains the pair of training points generating the line in feature space from which the new example has minimum distance. In this way, it is similar to other memory-based methods. However, due to our greater familiarity with SVMs, the availability of high-quality software implementing SVM learning, and the existence of research leading to good results using SVMs for audio classification, we decided upon the use of SVM learning for this project. A potential avenue for future work could involve the implementation of a system similar to the one designed in this project, instead using NFL. Then, a direct comparison would be possible in the context of this task, with the possibility of producing interesting results that contrast the capabilities of the two techniques. But, such a comparison is outside the scope of this work.Most previous audio classification work has not incorporated time series information in any principled way. Scheirer and Slaney [7] implemented a music/speech discriminator that used 2.4-second window averages to boost performance on a frame-by-frame basis. However, this technique is rather ad hoc. We decided to incorporate time series information in Bayesian style through an HMM. Further, because the SVM output can be looked at as a measure of confidence in the classification, we developed a modification to normal HMM inference that uses the degree of confidence in the classification at each time step. This is in some sense similar to the hybrid HMM/neural network approach of [8], but their system used neural networks to change the probability table of observations given state at each time step, while our approach was to use the SVM to measure our belief in the observation, keeping the state/observation probability table constant throughout.The majority of previous work in audio classification has involved some form of Mel Frequency Cepstral Coefficients (MFCCs), which are a form of frequency spectrum adapted to the human perception of pitch. Two papers [4, 9] specifically contrast MFCCs with other audio features, and find that MFCCs perform favorably. This, combined with the availability of open source MFCC extractors, was the basis of our decision to use MFCC feature vectors in this work.ExperimentsOur first step was to gather a large amount of audio data. This was obtained by recording approximately two hours worth of Internet radio broadcasts from two different music stations. These audio files were then separated and regrouped into music and non-music segments, yielding approximately 600,000 samples.Since MFCCs are a well-established audio feature, we were able to use existing utilities to extract feature vectors from our audio samples. Specifically, we used the Edinburgh Speech Tools utility called sig2fv [10]. This gave us feature vectors containing twelve MFCCs for each 0.01-second interval. We wrote a utility to convert the output of sig2fv into a format useable by SVM lite [6], a well-regarded, open source, C-based SVM package. The data was then separated into a training set (5% or approximately 30,000 of the samples) and a test set (everything not in the training set). Because training times were prohibitively long, we decided against the normal 33%/67% splitting of the data. Several SVM kernels were tested, including linear, quadratic, cubic, quartic, etc. The cubic was found have best performance, and so was chosen for use in the time series system.Our initial approach to the incorporation of time series information was to create by hand a simple HMM and use the SVM classification as the HMM’s observed symbol at each time step. The HMM design is shown in Figure 1. The symbol M means music, and N means non-music.232.0)(,768.0)(==N M ππO=M O=N S=M0.9875 0.757 S=N 0.01250.243Figure 1: The HMM with transition probabilities, initial state probabilities, and observation probabilities.The numbers for the HMM were derived empirically from the raw data and the SVM results, i.e. 999947=MS=P because 99.9947% of the time a music sample is MS(=.0)'|πbecause 76.8% of the samples are music, followed by a music sample, 768M)(=.0S=MOP because the SVM classified 98.75% of the test music samples M=9875.0)|(=correctly.In order to appropriately test the HMM, it was necessary to obtain labeled data points in their original sequence. Previously all samples of a given type were grouped together, so only the set had to be labeled. We therefore cut an approximately 40-minute segment of the audio recordings and parsed it into music/non-music chunks, extracted ordered features and classifications from the chunks, and reassembled them into the full mixed sequence that included transitions between music and non-music. The fastest way to generate the sequential test set was to draw from only one radio station and not filter out 5% of data points that had been part of the training set. Time did not permit the creation of a more sophisticated sequential test set; however, 5% of the data points could not have had a large impact on overall performance.HMM inference was performed using the standard Viterbi algorithm [11], taking the final state of the most probable path for each time step as the state output for that step. The state output at each step was compared to the known label, and accuracies were computed for music and non-music classification. As we observed that music and non-music occur in large contiguous blocks, we added a domain-specific enhancement to the HMM: transitions out of a state were only allowed when the probability of being in the other state was higher than a threshold (0.9 to go from non-music to music, 0.55 to go from music to non-music; the reason for the asymmetry is that spurious SVM outputs are more common on non-music than on music samples).Finally, noting that the SVM output was not just a binary decision but the actual margin of each data point, we developed a modification to the HMM that used the SVM margin as a confidence measure in the correctness of the observation (M or N). This confidence measure was transformed into a probability that the observation is correct as follows. Since most margins had absolute value less than 1.5, the first confidence measure was either the absolute value of the margin if it was less than 1, or truncated to 1 if larger. This confidence measure alone was not a good estimate of the probability that the SVM gave the correct output; for example, a confidence of 0.2 that the sample is music does not mean that the probability of being non-music is 0.8. The key observation here is that a confidence of 0 should result in a 0.5 probability that the observation is correct, and a confidence of 1 should give a probability of 1. This leads to the formula)1(5.0)(confidence n Observatio Correct P +⋅=.So, for example, if the SVM classifies a sample as M with a certain confidence, then )1(5.0)(confidence M P +⋅= and )(1)(M P N P −=. Normally in Viterbi inference, the probabilities of the most probable path ending in state M would be updated as:)|(*),(*)(max )(,1M n Observatio P M j T j S P M S P t NM j t ====+. This assumes the observation is correct. We incorporate the possibility that the observation is incorrect by updating as follows:[][].)|(),()(max ))(1()|(),()(max )()(,1N n Observatio P M S T S P M Obs P M n Observatio P M j T j S P M Obs P M S P j jt j t N M j t ⋅⋅⋅=−+⋅⋅=⋅====+In other words, the new state probability is a linear combination of the normal Viterbi updates for each possible observation, weighted in proportion to the estimated probability that each observation was the correct one. We used the same state change thresholding in this modifiedHMM as we did in the normal HMM and again calculated accuracies for each class using the data labels.ResultsSeveral different SVM kernels were tested, with the results shown in Figure 2. The model dimensionality refers to the kernel used, ranging from a linear model (1), to a squared model (2), then a cubed model (3) and finally a quartic model (4). As the figure shows, the cubed model (3) has the best overall results. Specifically, this model correctly classified 98.8% of the tested music samples and 24.3% of all non-music samples, for an overall accuracy of 84.1%, which is higher than any other model in each measurement category. This was the basis of our decision to use the cubic model in the time series experiments.Figure 2: Test set accuracy for polynomial kernels of various dimensionalities.The SVM seemed adept at correctly classifying music samples, but performed very poorly on the non-music samples. Fortunately, this performance was increased greatly by the introduction of time series information. The accuracy results from the normal and modified HMMs are summarized with the original SVM accuracy in Table 1.Music Accuracy Non-Music Accuracy Overall Accuracy Static SVM 98.8% 24.3% 84.1% Normal HMM 88.6% 86.0% 88.0%Modified HMM 94.6% 73.5% 89.8%Table 1: Accuracy of normal and modified HMMs over the sequential test set. Note that the overall accuracy is not simply the average of the other two accuracies because there are more music than non-music data points.As can be seen, both types of HMM resulted in a relatively small decrease in the static music accuracy of 98.8%, while yielding very large increases over the 24.3% static non-music accuracy. Both HMMs outperformed the static SVM in overall accuracy, and the modified HMM performed 1.8% better than the normal HMM.ConclusionsAs the results of the experiment indicate, our method of classifying between music and non-music audio samples is successful. We have implemented a combination of SVM and HMM in order to provide the best possible chances of correctly identifying each sample. In this way, we have achieved an overall accuracy of 89.8%.Initially, MFCC is used to create a feature vectors for each audio sample. These feature vectors are then used to train a cubic SVM model. The SVM outputs a music or non-music margin, which we use for one-shot static sample classification, for normal HMM classification ina time series, and as a proto-confidence level in a modified HMM that uses a new version of Viterbi inference that takes into account uncertainty in the observation.We found that using the SVM alone produced very good results for classifying music samples (over 98%). However, the results of classifying non-music samples are significantly worse, at less than 25%. This averages to create an overall accuracy of approximately 84%. While this result is good on some level, the introduction of an HMM into the system yielded much better classifications. Specifically, the normal HMM had 88% accuracy, while our modified HMM technique yielded 90% accuracy. This is an especially interesting point, because the modified HMM technique is not at all domain specific, and the combination of a confidence-outputting classifier with the modified HMM technique could be a means of improving HMM performance in general.In the end, these results show promise, and indicate that these methods can be successful at classifying audio samples as music or non-music. Further, these results can possibly be improved upon through the addition of yet other techniques and/or making refinements to the techniques we employed. See below for possible future work and extensions to the project.Future WorkThere are several possible extensions to this project, some involving practical application of the classifier generated as the result of these experiments, and others focusing on improving the classifier itself by achieving a better understanding of applicable theories and methods within the context of binary audio classification. Ultimately, one goal would be to create a system for Internet radio listeners. This system would allow the user to define a series of favorite radio stations. Each radio station would be monitored for content – music vs. non-music. In this way,whenever a station is not playing music, it would be switched, in favor of a station that is playing music. Additionally, stations on which a song has recently started will be given priority over stations on which a song has been playing for longer (and therefore is nearer the end).Further, this concept could be extended, providing a similar service on television. In this domain, commercials are much more prone to increasing volume. Therefore, instead of changing the station, this system could be designed to automatically reduce the volume of the television. Taking this approach, as opposed to simply trying to constantly moderate the volume, would help prevent adjusting the volume from scene to scene in a given show or movie (which should not be interfered with).Another area of future interest involves different types of music. These techniques of discrimination should be examined in the context of classifying types of music (country vs. rap, for example). A motivation for this research would be to allow a listener to choose between channels based on type of music. Presumably in addition to removing commercials, this would ensure that a listener could listen to only the desired type of music, regardless of the fact that many stations play a variety of types.The general technique of combining a confidence-outputting classifier such as an SVM with the modified HMM inference is one that merits exploration in other domains. Testing in other areas and with other classifiers would lead to the discovery of whether this technique is a general improvement over regular HMMs or not.Any of these applications would require a broadening of the training set and improvement of the validation strategy. Alternative strategies for classification and feature extraction/selection could also be explored, possibly further improving the quality and practicalapplicability of the current classifier beyond the positive results that have been achieved as part of this study.References[1] Guo, Zhang and Li, "Boosting For Content-Based Audio Classification And Retrieval: An Evaluation," /503362.html[2] Lu, Li and Zhang, "Content-Based Audio Segmentation Using Support Vector Machines," /502778.html[3] Chen K., Wu T.Y., and Zhang H.J., “On the Use of Nearest Feature Line for Speaker Identification,” Pattern Recognition Letters 23: 1735-1746, (2002),.hk/~apnna/proceedings/iconip2001/papers/041a.pdf[4] Li, S. Z., "Content-Based Audio Classification and Retrieval Using the Nearest Feature Line Method," IEEE Transactions on Speech and Audio Processing, Vol. 8, No. 5, pp. 619--625, Sept. 2000, /~mjr59/reviews/nearest_line.pdf[5] S. Z. Li and G. Guo, “Content-based Audio Classification and Retrieval Using SVM Learning,” (invited talk), PCM, 2000,/china/papers/Content_Audio_Classification.pdf[6] T. Joachims, “Making large-Scale SVM Learning Practical. Advances in Kernel Methods - Support Vector Learning,” B. Schölkopf and C. Burges and A. Smola (ed.), MIT-Press, 1999, [7] E. Scheirer and M. Slaney, “Construction and Evaluation of a Robust Multi-featureSpeech/Music Discriminator,” Proceedings of the 1997 International Conference on Acoustics, Speech, and Signal Processing, 1997.[8] H. Franco, M. Cohen, N. Morgan, D. Rumelhart and V. Abrash, "Context-dependent connectionist probability estimation in a hybrid Hidden Markov Model – Neural Net speech recognition system," Computer Speech and Language, Vol. 8, No. 3, 1994.[9] B. Logan, “Mel Frequency Cepstral Coefficients for Music Modeling,” International Symposium on Music Information Retrieval, 2000.[10] P. Taylor, R. Caley, A. W. Black, S. King, “Edinburg Speech Tools Documentation,” 1999, /docs/speech_tools-1.2.0/.[11] “University of Leeds HMM Tutorial – Viterbi Algorithm,” /scs-only/teaching-materials/HiddenMarkovModels/html_dev/viterbi_algorithm/s1_pg1.html.。