After a lovely 20 hour flight from Brisbane, with stop-overs at every corner of the globe (it seemed), I arrived in Antwerp for the eight annual Interspeech conference. The Interspeech conferences replace the Eurospeech and ICSLP conferences, which used to alternate year-by-year. It is now considered taboo to mention these earlier names, as it is just Interspeech – at least this is what we were told at the welcome lecture.

Although speech is a fairly focussed area of signal processing, there is still a lot of topics that can be covered under Interspeech’s umbrella, and some of them weren’t of much interest to me. However, I did manage to attend a number of sessions on most of the 5 days of the conference. I was a little dissapointed that my area of research, multi-modal speech processing, had it’s only oral and poster session on at the same time!. That was quite annoying, but I did manage to see most of both sessions, even though I was presenting an oral in one of them. In particular I found some of the research into recognising speech with infrared sensors by Bo Zhu at MIT interesting.

The social program of Interspeech was quite nice, with lots of free food and Belgium beer available at various social events on most of the nights of the conference. Entrance to the Antwerp Zoo, next door to the conference venue, was also included in the conference registration, although all those animals in such a small area seemed a little sad to me.

On the final day of the Interspeech conference, I had to pack my bags and catch a hour or so train to Tilberg, Netherlands where I could catch an expensive taxi to Kasteel Groenendael in Hilvarenbeek for AVSP 2007. It probably would have been nice if AVSP had arranged a shuttle bus, as the taxi to Hilvarenbeek cost more than the train trip from Belgium, although I did get to share the cost with some other attendees on the way back.

The AVSP 2007 conference was a small workshop style conference specifically devoted to my area of research, although it does focus on human perception as well as automatic, which is more my style. I got to see a lot of interesting research at the AVSP workshop, although did seem to be a little perception heavy. However, I found the studies of how humans do what I am trying to perform with computers provided a good perspective on my research that I don’t normally encounter. Although even further away from my area of research, I found invited speaker Asif Ghazanfar’s talk on speech perception in monkeys (that is monkey-speech perception) to be very well presented and quite interesting.

Kasteel Groenendael is Philip Electronic’s executive training centre just outside the small village on Hilvarenbeek. Seeing what their executive training centre is like, I don’t think I’d mind working for Philips. Everything was provided for us at the workshop, and I’d probably even say it was worth loosing my weekend. I also got to meet and discuss research with a lot of interesting people over breakfast, lunch and dinner over the two days, and I hope to keep in touch with many of them.

Finally, I’d like to thank QUT and ASSTA for supplying the funding to travel to Europe and attend these conferences.

However, while perusing the wikipedia article on impact factors, I stumbled upon eigenfactor.org who take a much more page-ranky approach to journal impact. This apparently does a much better job than the more linear approach of the JCR. And what is even better is that it is easy to use. Here, go look at the journals listed in Computer Science – AI. Very easy to understand. And what is even more better is that it is completely free and any Joe or Jane on the internet can access it.

Audio-visual Speech Recognition

Automatic speech recognition is a very mature area of research, and one that is increasingly becoming involved in our day-to-day lives. While many systems that can recognise speech from an audio signal have shown promise when performing well defined tasks like dictation or call-centre navigation in reasonably controlled environments, automatic speech recognition has certainly not yet reached the stage where a user can seamlessly interact with a automatic speech interface [1]. One of the major stumbling blocks to speech becoming an alternative human-computer interface is the lack of robustness of present systems to channel or environmental noise, which can degrade performance by many orders of magnitude [2].

However, speech does not consist of the audio modality alone, and studies of human production and perception of speech have shown that the visual movement of the speaker’s face and lips are an important factor in human communication. Hiding or modifying one of the modalities independent of the other has been shown to cause errors in human speech perception [3, 4].

Fortunately many of the sources of audio degradation can be considered to have little effect on the visual signal, for example, a group of people talking out of view of the camera. A similar assumption can also be drawn about many sources of video degradation, such as face movement or minor occlusions. By taking advantage of visual speech in combination with traditional audio speech, automatic speech recognition systems can increase the robustness to degradation in both modalities.

The chosen method of combining these two orthogonal sources of information is still a major area of ongoing research in audio-visual speech recognition (AVSR). Early AVSR systems could be generally be divided into two main groups, early or late integration, based on whether the two modalities were combined before or after classification/scoring. Late integration had the advantage that the reliability of each modality’s classifier could be weighted easily before combination, but was difficult to use on anything but isolated word recognition due to the problem of aligning and fusing two possibly significantly different speech transcriptions. This was not a problem with early integration, where features are combined before using a single classifier, but, on the other hand, it would be very difficult to model the reliability of each modality.

To allow a compromise between these two extremes, middle integration schemes were developed that allow classifier scores to be combined in a weighted manner within the structure of the classifier itself. The simplest of the middle integration methods, and the subject of this paper, is the synchronous multi-stream HMM [1] (MSHMM). There are more complicated middle integration designs, primarily intended to allow modelling of the asynchronous nature of audio visual speech, such as asynchronous [5], product [1] or coupled HMMs [6]. These designs can be significantly more complicated to train and test, however, and the small performance increase may not be worth it, especially in embedded environments where processing power or memory might be limited.

References

[1] G. Potamianos, C. Neti, G. Gravier, A. Garg, and A. Senior, “Recent advances in the automatic recognition of audiovisual speech,” Proceedings of the IEEE, vol. 91, no. 9, pp. 1306–1326, 2003.

[2] Y. Gong, “Speech recognition in noisy environments: A survey,” Speech Communication, vol. 16, no. 3, pp. 261–291, 1995.

[3] H. McGurk and J. MacDonald, “Hearing lips and seeing voices,” Nature, vol. 264, no. 5588, pp. 746–748, Dec. 1976.

[4] S. M. Thomas and T. R. Jordan, “Contributions of oral and extraoral facial movement to visual and audiovisual speech perception,” Journal of Experimental Psychology: Human Perception and Performance, vol. 30, no. 5, pp. 873–888, 2004.

[5] S. Bengio, “Multimodal speech processing using asynchronous hidden markov models,” Information Fusion, vol. 5, no. 2, pp. 81–9, June 2004.

[6] A. Nefian, L. Liang, X. Pi, L. Xiaoxiang, C. Mao, and K. Murphy, “A coupled hmm for audio-visual speech recognition,” in Acoustics, Speech, and Signal Processing, 2002. Proceedings. (ICASSP ’02). IEEE International Conference on, vol. 2, 2002, pp. 2013–2016.

While we can communicate very well with audio alone, such as during a telephone call, our brains make use of many visual cues when we talk face-to-face. As well as more broad visual cues such as gestures and facial expressions, it may come as a surprise to learn that the actual motion of the lips play a very important part in the comprehension of human speech.

A useful demonstration of the impact of the visual modality on speech is the McGurk effect, first published by McGurk and McDonald in 1976. Rather than explain it in too much detail right now, go watch the video below from an episode of the Hackszine video podcast.

The basic and original McGurk effect was demonstrated by dubbing a video of a person saying ‘gah’ with audio of them saying ‘bah’. If you watch the dubbed video, they appear to be saying ‘dah’, but the audio along clearly says ‘bah’. This shows that even though you may not realise it, the visual lip movements are having an effect on your perception of speech. The hackszine video extends the McGurk effect to cover bad dubbing in general, but I would only consider the McGurk effect to cover when said bad dubbing appears to make the person say something that is neither in the video or dubbed audio.

Finally, this (I think) Japanese talk show appears to be very interested in the McGurk effect. It makes for fairly amusing watching.

More information:

• McGurk Effect at Wikipedia
• Hearing with your eyes: The McGurk Effect
• McGurk, Harry; and MacDonald, John (1976); “Hearing lips and seeing voices,” Nature, Vol 264(5588), pp. 746–748

This paper examines audio-visual speaker verification using a novel adaptation of fused hidden Markov models, in comparison to output fusion of individual classifiers in the audio and video modalities. A comparison of both hidden Markov model (HMM) and Gaussian mixture model (GMM) classifiers in both modalities under output fusion shows that the choice of audio classier is more important than video. Although temporal information allows a HMM to out-perform a GMM individually in video, this temporal information does not carry through to output fusion with an audio classier, where the difference between the two video classifiers is minor. An adaptation of fused hidden Markov models, designed to be more robust to within-speaker variation, is used to show that the temporal relationship between video observations and audio states can be harnessed to reduce errors in audio-visual speaker verification when compared to output fusion.

[ link | paper (pdf) | slides (ppt) ]

Fused hidden Markov models (FHMMs) have been shown to work well for the task of audio-visual speaker recognition, but only in an output decision-fusion configuration of both the audio- and video-biased versions of the FHMM structure. This paper looks at the performance of the audio and video-biased versions independently, and shows that the audio-biased version is considerably more capable for speaker recognition. Additionally, this paper shows that by taking advantage of the temporal relationship between the acoustic and visual data, the audio-biased FHMM provides better performance at less processing cost than best-performing output decision-fusion of regular HMMs.

[ link | paper (pdf) | slides (ppt) ]

This paper examines the utility of audio-visual speech for the two related tasks of speech and speaker recognition. A study of the confusion that exists between speaker and speech elements was performed to show that principal component analysis (PCA) based visual speech is considerably better for the task of speaker recognition than for speech. Decision fusion speech and speaker recognition engines were also tested under various levels of acoustic degradation to find that the optimal fusion configuration for speaker recognition was substantially different than that for speech. These results highlight the problem of employing similar visual features for both speech and speaker recognition.

[ link | paper (pdf) | slides (ppt) ]

The freely available nature of the CUAVE database allows it to provide a valuable platform to form benchmarks and compare research. This paper shows that the CUAVE database can successfully be used to test speaker identifications systems, with performance comparable to existing systems implemented on other databases. Additionally, this research shows that the optimal configuration for decision-fusion of an audio-visual speaker identification system relies heavily on the video modality in all but clean speech conditions.

[ link | paper (pdf) | poster (ppt) ]

Hi,

I have been updating my QUT eprints lately, and I would like to give a few suggestions as to how the system could be improved.

1) In lists of publications, make author names link to their list of publications. That is, if you see a list of publications and I am an author in one of them, my name should link to http://eprints.qut.edu.au/view/person/Dean,_David.html where the rest of my publications can be viewed.

2) A bibtex and Endnote-format file should be available for download on both individual papers and lists of papers

3) Author pages should have a header detailing contact information about the author, if available (similar to the user pages)

4) RSS or Atom syndication feeds should be available for authors so people can keep an eye on an author if needed. This would also be useful for other lists – topics, faculties, all papers, etc.

5) I’m not completely sure how (or if) you can attach other files to a paper, such as slides or datasets, etc.

Thankyou,

David Dean

UPDATE: Paula Callan, QUT eResearch Access Coordinator, contacted me and told me that all some of these things and more, presumably, will be available when the a system goes live in a month or so. I hope the old links still work (whoops, I read the email incorrectly – just the first two points should be in the new system, the rest are on the drawing board).