Speech = Words + ?
As a newcomer to this area, I'm struck by the considerable difference between words, sentences, as they are spoken; and those same words and sentences read from a page. Speech manages to convey essential semantic content that is not present in the words themselves. In Italian, for example, a spoken question is simply a statement that ends with a rising tone. With a rising tone, the sentence "feels" unfinished and in need of a response.
That unfinished feeling that is invoked in the listener is not something that the listener has to be told how to interpret. And nor is the rising tone something that the speaker has to be told to use when asking a question. Speech has its "body language".
If a part of the body is unable to make its contribution to the speech then perhaps that contribution can be provided by some other part. A gesture may replace the rising tone in someone who has lost their voice and is asking a question using a speech synthesis device.
But which gesture to use? This question assumes that there is a choice. Perhaps, like the use of the rising tone, we may be able to produce the gesture, without being shown or told. What would happen, in a small community of people, if everyone was given the exact same mechanical expressionless voice?
Leonardo Bottaci
Posted 14th September 2011 at 12:31 PM
Comments
Hi Leonardo, What you're referring to is 'prosody' - intonation, rhythm and timing - all of which are richly present in the spoken form, but which only appear superficially in the written form (e.g. as punctuation). There is much research on prosody and the link that it provides between body movement (especially the eyebrows!) and meaning. Interestingly, the majority of automatic speech recognition devices throw away this information (mainly because no-one knows what to do with it), and text-to-speech synthesis atempts to guess the prosody from the punctuation and lexical forms. It would indeed be interesting to give prosodic control over to a user via some gestural sensor (how about a Kinect?). Cheers, Roger
Roger Moore
Commented 20th September 2011 at 4:55 PM