speech word recognition definition

Speech recognition, also known as automatic speech recognition (ASR), computer speech recognition or speech-to-text, is a capability that enables a program to process human speech into a written format.

While speech recognition is commonly confused with voice recognition, speech recognition focuses on the translation of speech from a verbal format to a text one whereas voice recognition just seeks to identify an individual user’s voice.

IBM has had a prominent role within speech recognition since its inception, releasing of “Shoebox” in 1962. This machine had the ability to recognize 16 different words, advancing the initial work from Bell Labs from the 1950s. However, IBM didn’t stop there, but continued to innovate over the years, launching VoiceType Simply Speaking application in 1996. This speech recognition software had a 42,000-word vocabulary, supported English and Spanish, and included a spelling dictionary of 100,000 words.

While speech technology had a limited vocabulary in the early days, it is utilized in a wide number of industries today, such as automotive, technology, and healthcare. Its adoption has only continued to accelerate in recent years due to advancements in deep learning and big data.  Research  (link resides outside ibm.com) shows that this market is expected to be worth USD 24.9 billion by 2025.

Explore the free O'Reilly ebook to learn how to get started with Presto, the open source SQL engine for data analytics.

Register for the guide on foundation models

Many speech recognition applications and devices are available, but the more advanced solutions use AI and machine learning . They integrate grammar, syntax, structure, and composition of audio and voice signals to understand and process human speech. Ideally, they learn as they go — evolving responses with each interaction.

The best kind of systems also allow organizations to customize and adapt the technology to their specific requirements — everything from language and nuances of speech to brand recognition. For example:

• Language weighting: Improve precision by weighting specific words that are spoken frequently (such as product names or industry jargon), beyond terms already in the base vocabulary.
• Speaker labeling: Output a transcription that cites or tags each speaker’s contributions to a multi-participant conversation.
• Acoustics training: Attend to the acoustical side of the business. Train the system to adapt to an acoustic environment (like the ambient noise in a call center) and speaker styles (like voice pitch, volume and pace).
• Profanity filtering: Use filters to identify certain words or phrases and sanitize speech output.

Meanwhile, speech recognition continues to advance. Companies, like IBM, are making inroads in several areas, the better to improve human and machine interaction.

The vagaries of human speech have made development challenging. It’s considered to be one of the most complex areas of computer science – involving linguistics, mathematics and statistics. Speech recognizers are made up of a few components, such as the speech input, feature extraction, feature vectors, a decoder, and a word output. The decoder leverages acoustic models, a pronunciation dictionary, and language models to determine the appropriate output.

Speech recognition technology is evaluated on its accuracy rate, i.e. word error rate (WER), and speed. A number of factors can impact word error rate, such as pronunciation, accent, pitch, volume, and background noise. Reaching human parity – meaning an error rate on par with that of two humans speaking – has long been the goal of speech recognition systems. Research from Lippmann (link resides outside ibm.com) estimates the word error rate to be around 4 percent, but it’s been difficult to replicate the results from this paper.

Various algorithms and computation techniques are used to recognize speech into text and improve the accuracy of transcription. Below are brief explanations of some of the most commonly used methods:

• Natural language processing (NLP): While NLP isn’t necessarily a specific algorithm used in speech recognition, it is the area of artificial intelligence which focuses on the interaction between humans and machines through language through speech and text. Many mobile devices incorporate speech recognition into their systems to conduct voice search—e.g. Siri—or provide more accessibility around texting. 
• Hidden markov models (HMM): Hidden Markov Models build on the Markov chain model, which stipulates that the probability of a given state hinges on the current state, not its prior states. While a Markov chain model is useful for observable events, such as text inputs, hidden markov models allow us to incorporate hidden events, such as part-of-speech tags, into a probabilistic model. They are utilized as sequence models within speech recognition, assigning labels to each unit—i.e. words, syllables, sentences, etc.—in the sequence. These labels create a mapping with the provided input, allowing it to determine the most appropriate label sequence.
• N-grams: This is the simplest type of language model (LM), which assigns probabilities to sentences or phrases. An N-gram is sequence of N-words. For example, “order the pizza” is a trigram or 3-gram and “please order the pizza” is a 4-gram. Grammar and the probability of certain word sequences are used to improve recognition and accuracy.
• Neural networks: Primarily leveraged for deep learning algorithms, neural networks process training data by mimicking the interconnectivity of the human brain through layers of nodes. Each node is made up of inputs, weights, a bias (or threshold) and an output. If that output value exceeds a given threshold, it “fires” or activates the node, passing data to the next layer in the network. Neural networks learn this mapping function through supervised learning, adjusting based on the loss function through the process of gradient descent.  While neural networks tend to be more accurate and can accept more data, this comes at a performance efficiency cost as they tend to be slower to train compared to traditional language models.
• Speaker Diarization (SD): Speaker diarization algorithms identify and segment speech by speaker identity. This helps programs better distinguish individuals in a conversation and is frequently applied at call centers distinguishing customers and sales agents.

A wide number of industries are utilizing different applications of speech technology today, helping businesses and consumers save time and even lives. Some examples include:

Automotive: Speech recognizers improves driver safety by enabling voice-activated navigation systems and search capabilities in car radios.

Technology: Virtual agents are increasingly becoming integrated within our daily lives, particularly on our mobile devices. We use voice commands to access them through our smartphones, such as through Google Assistant or Apple’s Siri, for tasks, such as voice search, or through our speakers, via Amazon’s Alexa or Microsoft’s Cortana, to play music. They’ll only continue to integrate into the everyday products that we use, fueling the “Internet of Things” movement.

Healthcare: Doctors and nurses leverage dictation applications to capture and log patient diagnoses and treatment notes.

Sales: Speech recognition technology has a couple of applications in sales. It can help a call center transcribe thousands of phone calls between customers and agents to identify common call patterns and issues. AI chatbots can also talk to people via a webpage, answering common queries and solving basic requests without needing to wait for a contact center agent to be available. It both instances speech recognition systems help reduce time to resolution for consumer issues.

Security: As technology integrates into our daily lives, security protocols are an increasing priority. Voice-based authentication adds a viable level of security.

Convert speech into text using AI-powered speech recognition and transcription.

Convert text into natural-sounding speech in a variety of languages and voices.

AI-powered hybrid cloud software.

Enable speech transcription in multiple languages for a variety of use cases, including but not limited to customer self-service, agent assistance and speech analytics.

Learn how to keep up, rethink how to use technologies like the cloud, AI and automation to accelerate innovation, and meet the evolving customer expectations.

IBM watsonx Assistant helps organizations provide better customer experiences with an AI chatbot that understands the language of the business, connects to existing customer care systems, and deploys anywhere with enterprise security and scalability. watsonx Assistant automates repetitive tasks and uses machine learning to resolve customer support issues quickly and efficiently.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• HHS Author Manuscripts

Speech Perception, Word Recognition and the Structure of the Lexicon *

This paper reports the results of three projects concerned with auditory word recognition and the structure of the lexicon. The first project was designed to experimentally test several specific predictions derived from MACS, a simulation model of the Cohort Theory of word recognition. Using a priming paradigm, evidence was obtained for acoustic-phonetic activation in word recognition in three experiments. The second project describes the results of analyses of the structure and distribution of words in the lexicon using a large lexical database. Statistics about similarity spaces for high and low frequency words were applied to previously published data on the intelligibility of words presented in noise. Differences in identification were shown to be related to structural factors about the specific words and the distribution of similar words in their neighborhoods. Finally, the third project describes efforts at developing a new theory of word recognition known as Phonetic Refinement Theory. The theory is based on findings from human listeners and was designed to incorporate some of the detailed acoustic-phonetic and phonotactic knowledge that human listeners have about the internal structure of words and the organization of words in the lexicon, and how, they use this knowledge in word recognition. Taken together, the results of these projects demonstrate a number of new and important findings about the relation between speech perception and auditory word recognition, two areas of research that have traditionally been approached from quite different perspectives in the past.

Introduction

Much of the research conducted in our laboratory over the last few years has been concerned, in one way or another, with the relation between early sensory input and the perception of meaningful linguistic stimuli such as words and sentences. Our interest has been with the interface between the acoustic-phonetic input -- the physical correlates of speech -- on the one hand, and more abstract levels of linguistic analysis that are used to comprehend the message. Research on speech perception over the last thirty years has been concerned principally, if not exclusively, with feature and phoneme perception in isolated CV or CVC nonsense syllables. This research strategy has undoubtedly been pursued because of the difficulties encountered when one deals with the complex issues surrounding the role of early sensory input in word recognition and spoken language understanding and its interface with higher levels of linguistic analysis. Researchers in any field of scientific investigation typically work on tractable problems and issues that can be studied with existing methodologies and paradigms. However, relative to the bulk of speech perception research on isolated phoneme perception, very little is currently known about how the early sensory-based acoustic-phonetic information is used by the human speech processing system in word recognition, sentence perception or comprehension of fluent connected speech.

Several general operating principles have guided the choice of problems we have decided to study. We believe that continued experimental and theoretical work is needed in speech perception in order to develop new models and theories that can capture significant aspects of the process of speech sound perception and spoken language understanding. To say, as some investigators have, that speech perception is a “special” process requiring specialized mechanisms for perceptual analysis is, in our view, only to define one of several general problems in the field of speech perception and not to provide a principled explanatory account of any observed phenomena. In our view, it is important to direct research efforts in speech perception toward somewhat broader issues that use meaningful stimuli in tasks requiring the use of several sources of linguistic knowledge by the listener.

Word Recognition and Lexical Representation in Speech

Although the problems of word recognition and the nature of lexical representations have been long-standing concerns of cognitive psychologists, these problems have not generally been studied by investigators working in the mainstream of speech perception research (see [ 1 , 2 ]). For many years these two lines of research, speech perception and word recognition, have remained more-or-less distinct from each other. This was true for several reasons. First, the bulk of work on word recognition was concerned with investigating visual word recognition processes with little, if any, attention directed to questions of spoken word recognition. Second, most of the interest and research effort in speech perception was directed toward feature and phoneme perception. Such an approach is appropriate for studying the “low level” auditory analysis of speech but it is not useful in dealing with questions surrounding how words are recognized in isolation or in connected speech or how various sources of knowledge are used by the listener to recover the talker’s intended message.

Many interesting and potentially important problems in speech perception involve the processes of word recognition and lexical access and bear directly on the nature of the various types of representations in the mental lexicon. For example, at the present time, it is of considerable interest to determine precisely what kinds of representations exist in the mental lexicon. Do words, morphemes, phonemes, or sequences of spectral templates characterize the representation of lexical entries? Is a word accessed on the basis of an acoustic, phonetic or phonological code? Why are high frequency words recognised so rapidly? We are interested in how human listeners hypothesize words for a given stretch of speech. Furthermore, we are interested in characterizing the sensory information in the speech signal that listeners use to perceive words and how this information interacts with other sources of higher-level linguistic knowledge. These are a few of the problems we have begun to study in our laboratory over the past few years.

Past theoretical work in speech perception has not been very well developed, nor has the link between theory and empirical data been very sophisticated. Moreover, work in the field of speech perception has tended to be defined by specific experimental paradigms or particular phenomena (see [ 3 , 4 ]). The major theoretical issues in speech perception often seem to be ignored, or alternatively, they take on only a secondary role and therefore receive little serious attention by investigators who are content with working on the details of specific experimental paradigms.

Over the last few years, some work has been carried out on questions surrounding the interaction of knowledge sources in speech perception, particularly research on word recognition in fluent speech. A number of interesting and important findings have been reported recently in the literature and several models of spoken word recognition have been proposed to account for a variety of phenomena in the area. In the first section of this paper we will briefly summarize several recent accounts of spoken word recognition and outline the general assumptions that follow from this work that are relevant to our own recent research. Then we will identify what we see as the major issues in word recognition. Finally, we will summarize the results of three ongoing projects that use a number of different research strategies and experimental paradigms to study word recognition and the structure of the lexicon. These sections are designed to give the reader an overview of the kinds of problems we are currently studying as we attempt to link research in speech perception with auditory word recognition.

Word Recognition and Lexical Access

Before proceeding, it will be useful to distinguish between word recognition and lexical access, two terms that are often used interchangeably in the literature. We will use the term word recognition to refer to those computational processes by which a listener identifies the acoustic-phonetic and/or phonological form of spoken words (see [ 5 ]). According to this view, word recognition may be simply thought of as a form of pattern recognition. The sensory and perceptual processes used in word recognition are assumed to be the same whether the input consists of words or pronounceable nonwords. We view the “primary recognition process” as the problem of characterizing how the form of a spoken utterance is recognized from an analysis of the acoustic waveform. This description of word recognition should be contrasted with the term lexical access which we use to refer to those higher-level computational processes that are involved in the activation of the meaning or meanings of words that are currently present in the listener’s mental lexicon (see [ 5 ]). By this view, the meaning of a word is accessed from the lexicon after its phonetic and/or phonological form makes contact with some appropriate representation previously stored in memory.

Models of Word Recognition

A number of contemporary models of word recognition have been concerned with questions of processing words in fluent speech and have examined several types of interactions between bottom-up and top-down sources of knowledge. However, little, if any, attention has been directed at specifying the precise nature of the early sensory-based input or how it is actually used in word recognition processes. Klatt’s recent work on the LAFS model (Lexical Access From Spectra) is one exception [ 6 ]. His proposed model of word recognition is based on sequences of spectral templates in networks that characterize the properties of the sensory input. One important aspect of Klatt’s model is that it explicitly avoids any need to compute a distinct level of representation corresponding to discrete phonemes. Instead, LAFS uses a precompiled acoustically-based lexicon of all possible words in a network of diphone power spectra. These spectral templates are assumed to be context-sensitive like “Wickelphones” [ 7 ] because they characterize the acoustic correlates of phones in different phonetic environments. They accomplish this by encoding the spectral characteristics of the segments themselves and the transitions from the middle of one segment to the middle of the next.

Klatt [ 6 ] argues that diphone concatenation is sufficient to capture much of the context-dependent variability observed for phonetic segments in spoken words. According to this model, word recognition involves computing a spectrum of the input speech every 10 ms and then comparing this input spectral sequence with spectral templates stored in the network. The basic idea, adopted from HARPY, is to find the path through the network that best represents the observed input spectra [ 8 ]. This single path is then assumed to represent the optimal phonetic transcription of the input signal.

Another central problem in word recognition and lexical access deals with the interaction of sensory input and higher-level contextual information. Some investigators, such as Forster [ 9 , 10 ] and Swinney [ 11 ] maintain that early sensory information is processed independently of higher-order context, and that the facilitation effects observed in word recognition are due to post-perceptual processes involving decision criteria (see also [ 12 ]). Other investigators such as Morton [ 13 , 14 , 15 ], Marslen-Wilson and Tyler [ 16 ], Tyler and Marslen-Wilson [ 17 , 18 ], Marslen-Wilson and Welsh [ 19 ], Cole and Jakimik [ 20 ] and Foss and Blank [ 21 ] argue that context can, in fact, influence the extent of early sensory analysis of the input signal.

Although Foss and Blank [ 21 ] explicitly assume that phonemes are computed during the perception of fluent speech and are subsequently used during the process of word recognition and lexical access, other investigators such as Marslen-Wilson and Welsh [ 19 ] and Cole and Jakimik [ 20 , 22 ] have argued that words, rather than phonemes, define the locus of interaction between the initial sensory input and contextual constraints made available from higher sources of knowledge. Morton’s [ 13 , 14 , 15 ] well-known Logogen Theory of word recognition is much too vague, not only about the precise role that phonemes play in word recognition, but also as to the specific nature of the low-level sensory information that is input to the system.

It is interesting to note in this connection that Klatt [ 6 ], Marslen-Wilson and Tyler [ 16 ] and Cole & Jakimik [ 22 ] all tacitly assume that words are constructed out of linear sequences of smaller elements such as phonemes. Klatt implicitly bases his spectral templates on differences that can be defined at a level corresponding to phonemes; likewise, Marslen-Wilson and Cole & Jakimik implicitly differentiate lexical items on the basis of information about the constituent segmental structure of words. This observation is, of course, not surprising since it is precisely the ordering and arrangement of different phonemes in spoken languages that specifies the differences between different words. The ordering and arrangement of phonemes in words not only indicates where words are different but also how they are different from each other (see [ 23 ] for a brief review of these arguments). These relations therefore provide the criterial information about the internal structure of words and their constituent morphemes required to access the meanings of words from the lexicon.

Although Klatt [ 6 ] argues that word recognition can take place without having to compute phonemes along the way, Marslen-Wilson has simply ignored the issue entirely by placing his major emphasis on the lexical level. According to his view, top-down and bottom-up sources of information about a word’s identity are integrated together to produce what he calls the primary recognition decision which is assumed to be the immediate lexical interpretation of the input signal. Since Marslen-Wilson’s “Cohort Theory” of word recognition has been worked out in some detail, and since it occupies a prominent position in contemporary work on auditory word recognition and spoken language processing, it will be useful to summarize several of the assumptions and some of the relevant details of this approach. Before proceeding to Cohort Theory, we examine several assumptions of its predecessor, Morton’s Logogen Theory.

Logogen and Cohort Theory of Word Recognition

In some sense, Logogen Theory and Cohort Theory are very similar. According to Logogen Theory, word recognition occurs when the activation of a single lexical entry (i.e., a logogen) crosses some critical threshold value [ 14 ]. Each word in the mental lexicon is assumed to have a logogen, a theoretical entity that contains a specification of the word’s defining characteristics (i.e., its syntactic, semantic, and sound properties). Logogens function as “counting devices” that accept input from both the bottom-up sensory analyzers and the top-down contextual mechanisms. An important aspect of Morton’s Logogen Model is that both sensory and contextual information interact in such a way that there is a trade-off relationship between them; the more contextual information input to a logogen from top-down sources, the less sensory information is needed to bring the Logogen above threshold for activation. This feature of the Logogen model enables it to account for the observed facilitation effects of syntactic and semantic constraints on speed of lexical access (see e.g., [ 24 , 25 , 26 ]) as well as the word frequency and word apprehension effects reported in the literature. In the presence of constraining prior contexts, the time needed to activate a logogen from the onset of the relevant sensory information will be less than when such constraints are not available because less sensory information will be necessary to bring the logogen above its threshold value.

In contrast to Logogen Theory which assumes activation of only a single lexical item after its threshold value is reached, Cohort Theory views Word recognition as a process of eliminating possible candidates by deactivation (see [ 16 , 27 , 28 , 29 , 30 ]). A set of potential word-candidates is activated during the earliest phases of the word recognition process solely on the, basis of bottom-up sensory information. According to Marslen-Wilson and Welsh [ 19 ], the set of word-initial cohorts consists of the entire set of words in the language that begins wish a particular initial sound sequence. The length of the initial sequence defining the initial cohort is not very large, corresponding roughly to the information in the first 200–250 ms of a word. According to the Cohort Theory, a word is recognized at the point that a particular word can be uniquely distinguished from any of the other words in the word-initial cohort set that was defined exclusively by the bottom-up information in the signal. This is known as the “critical recognition point” of a word. Upon first hearing a word, all words sharing the same initial sound characteristics become activated in the system. As the system detects mismatches between the initial bottom-up sensory information and the top-down information about the expected sound representation of words generated by context, inappropriate candidates within the initial cohort are deactivated.

In Cohort Theory, as in the earlier Logogen Theory, word recognition and subsequent lexical access are viewed as a result of a balance between the available sensory and contextual information about a word at any given time. In particular, when deactivation occurs on the basis of contextual mismatches, less sensory information is therefore needed for a single word candidate to emerge. According to the Cohort Theory, once word recognition has occurred the perceptual system carries out a much less detailed analysis of the sound structure of the remaining input. As Marslen-Wilson and Welsh [ 19 ] have put it, “No more and no less bottom-up information needs to be extracted than is necessary in a given context”, Pp. 58.

Acoustic-Phonetic Priming and Cohort Theory

As outlined above, Cohort theory proposes that in the initial stage of word recognition, a “cohort” of all lexical elements whose words begin with a particular acoustic-phonetic sequence will be activated. Several recent studies in our laboratory (see [ 31 ]) have been concerned with testing the extent to which initial acoustic-phonetic information is used to activate a cohort of possible word candidates in word recognition. Specifically, a series of auditory word recognition experiments were conducted using a priming paradigm.

Much of the past research that has used priming techniques was concerned with the influence of the meaning of a prime word on access to the meaning of a target word (e.g., [ 32 ]). However, it has been suggested by a number of researchers that the acoustic-phonetic representation of a prime stimulus may also facilitate or inhibit recognition of a subsequent test word (see [ 33 ]). A lexical activation model of Cohort Theory called MACS was developed in our lab to test the major assumptions of Cohort Theory [ 29 , 31 ]. Several predictions of the MACS model suggested that phonetic overlap between two items could influence auditory word recognition. Specifically, it was suggested that the residual activation of word candidates following recognition of a prime word could influence the activation of lexical candidates during recognition of a test word. Furthermore, the relationship between the amount of acoustic-phonetic overlap and the amount of residual activation suggested that identification should improve with increasing amounts of acoustic-phonetic overlap between the beginnings of the prime and test words.

In order to test these predictions, we performed an experiment in which subjects heard a prime word followed by a test word. On some trials, the prime and test words were either unrelated or identical. On other trials, although the prime and test words were different, they contained the same initial acoustic-phonetic information. For these trials, the prime and test words shared the same initial phoneme, the first two phonemes or the first three phonemes. Thus, we examined five levels of acoustic-phonetic overlap between the prime and target: 0, 1, 2, 3, or 4 phonemes in common.

By way of example, consider in this context the effects of presenting a single four phoneme word (e.g., the prime) on the recognition system. Following recognition of the prime, the different cohorts activated by the prime will retain a residual amount of activation corresponding to the point at which the candidates were eliminated. When the test word is presented, the effect of this residual activation will depend on the acoustic-phonetic overlap or similarity between the prime and the test word. A prime that shares only the first phoneme of a test word should have less of an effect on identification than a prime that is identical to the test word. The residual activation of the candidates therefore differentially contributes to the rate of reactivation of the cohorts for the test word.

In this experiment, we examined the effect of word primes on the identification of word targets presented in masking noise at various signal-to-noise ratios. Primes and targets were related as outlined above. The prime items were presented over headphones in the clear; targets were presented 50 msec after the prime items embedded in noise. Subjects were instructed to listen to the pair of items presented on each trial and to respond by identifying the second item (the target word embedded in noise). The results of the first experiment supported the predictions of the MACS model and provided support for Cohort Theory. The major findings are shown in Figure 1 .

Results displaying the probability of correct identification in a priming experiment using word primes and word targets presented at various signal-to-noise ratios. Crosses in each panel represent unprimed trials and squares represent primed trials when prime-target overlap equals: (a) 0 phonemes, (b) 1 phoneme, (c) 2 phonemes, (d) 3 phonemes and (4) identical (Data from [ 31 ]).

Specifically, the probability of correctly identifying targets increased as the acoustic-phonetic overlap between the prime and the target increased. Subjects showed the highest performance in identifying targets when they were preceded by an identical prime. Moreover, probability of correct identification was greater for primes and targets that shared three phonemes than those that shared two phonemes, which were, in turn, greater than pairs that shared one phoneme or pairs that were unrelated.

The results of this experiment demonstrate that acoustic-phonetic priming can be obtained for identification of words that have initial phonetic information in common. However, this experiment did not test the lexical status of the prime. The priming results may have been due to the fact that only word primes preceded the target items. In order to demonstrate that priming was, in fact, based on acoustic-phonetic similarity (as opposed to some lexical effect), we conducted a second identification experiment in which the prime items were phonologically admissable pseudowords. As in the first experiment, the primes shared 3, 2, or 1 initial phonemes with the target or they were unrelated to the target. Because of the difference in lexical status between primes and targets, there was no identical prime-target condition in this experiment. The subject’s task was the same as in the first experiment.

As in the first experiment, we found an increased probability of correctly identifying target items as acoustic-phonetic overlap between the prime and target increased. Thus, the lexical status of the prime item did not influence identification of the target. Taken together, the results of both studies demonstrate acoustic-phonetic priming in word recognition. The facilitation we observed in identification of target words embedded in noise suggests the presence of residual activation of the phonetic forms of words in the lexicon. Furthermore, the results provide additional support for the MACS lexical activation model based on Cohort Theory by demonstrating that priming is due to the segment-by-segment activation of lexical representations in word recognition.

One of the major assumptions of Cohort Theory that was incorporated in our lexical activation model is that a set of candidates is activated based on word initial acoustic-phonetic information. Although we obtained strong support for acoustic-phonetic activation of word candidates, the outcome of both experiments did not establish that the acoustic-phonetic information needs to be exclusively restricted to word initial position. In order to test this assumption, we conducted a third priming experiment using the same identification paradigm. In this experiment, word primes and word targets were selected so that the acoustic-phonetic overlap occurred between word primes and targets at the ends of the words. Primes and targets were identical or 0, 1, 2, or 3 phonemes were the same from the end of the words.

As in the first two experiments, we found evidence of acoustic-phonetic priming. The probability of correctly identifying a target increased as the acoustic-phonetic overlap between the prime and target increased from the ends of the-items. These results demonstrate that listeners are as sensitive to acoustic-phonetic overlap at the ends of words as they are to overlap at the beginnings of words. According to the MACS model and Cohort Theory, only words that share the initial sound sequences of a prime item should be activated by the prime. Thus, both MACS and Cohort Theory predict that no priming should have been observed. However, the results of the third experiment demonstrated priming from the ends of words, an outcome that is clearly inconsistent with the predictions of the MACS model and Cohort Theory.

The studies reported here were an initial step in specifying how words might be recognized in the lexicon. The results of these studies demonstrate the presence of some form of residual activation based on acoustic-phonetic properties of words. Using a priming task, we observed changes in word identification performance as a function of the acoustic-phonetic similarity of prime and target items. However, at least one of the major assumptions made about word recognition in Cohort Theory appears to be incorrect. In addition to finding acoustic-phonetic priming from the beginning of words, we also observed priming from the ends of words as well. This latter result suggests that activation of potential word candidates may not be restricted to only a cohort of words sharing word initial acoustic-phonetic information. Indeed, other parts of words may also be used by listeners in word recognition. Obviously, these findings will need to be incorporated into any theory of auditory word recognition. Phonetic Refinement Theory, as outlined in the last section of this paper, was designed to deal with this finding as well as several other problems with Cohort Theory.

Measures of Lexical Density and the Structure of the Lexicon

A seriously neglected topic in word recognition and lexical access has been the precise structural organization of entries in the mental lexicon. Although search theories of word recognition such as Forster’s [ 9 , 10 ] have assumed that lexical items are arranged according to word frequency, little work has been devoted to determining what other factors might figure into the organization of the lexicon (see however [ 34 ]). Landauer and Streeter [ 35 ] have shown that one must take the phonemic, graphemic, and syllabic structure of lexical items into account when considering the word frequency effect in visual recognition experiments. They have shown that a number of important structural differences between common and rare words may affect word recognition. Their results suggest that the frequency and organization of constituent phonemes and graphemes in a word maybe an important determinant of its ease of recognition. Moreover, Landauer and Streeter, as well as Eukel [ 36 ], have argued that “similarity neighborhoods” or “phonotaatic density” may affect word recognition and lexical access in ways that a simple “experienced” word frequency account necessarily ignores. For example, it would be of great theoretical and practical interest to determine if word recognition is controlled by the relative density of the neighborhood from which a given word is drawn, the frequency of the neighboring items, and the interaction of these variables with the frequency of the word in question. In short, one may ask how lexical distance in this space (as measured, for example, by the Greenberg and Jenkins [ 37 ] method) interacts with word frequency in word recognition.

As a first step toward approaching these important issues, we have acquired several large databases. One of these, based on Kenyon and Knott’s A Pronouncing Dictionary of American English [ 38 ] and Webster’s Seventh Collegiate Dictionary [ 39 ], contains approximately 300,000 entries. Another smaller database of 20,000 words is based on Webster’s Pocket Dictionary Each entry contains the standard orthography of a word, a phonetic transcription, and special codes indicating the syntactic functions of the word. We have developed a number of algorithms for determining, in various ways, the similarity neighborhood’s, or “lexical density,” for any given entry in the dictionary. This information has provided some useful information about the structural properties of words in the lexicon and how this information might be used by human listeners in word recognition.

Lexical Density, Similarity Spaces and the Structure of the Lexicon

Word frequency effects obtained in perceptual and memory research have typically been explained in terms of frequency of usage (e.g., [ 13 , 9 ]), the time between the current and last encounter with the word in question [ 40 ], and similar such ideas. In each of these explanations of word frequency effects, however, it has been at least implicitly assumed that high and low frequency words are “perceptually equivalent” [ 41 , 42 , 43 , 13 , 44 , 45 ]. That is, it has often been assumed that common and rare words are structurally equivalent in terms of phonemic and orthographic composition. Landauer and Streeter [ 35 ] have shown, however, that the assumption of perceptual equivalence of high and low frequency words is not necessarily warranted. In their study, Landauer and Streeter demonstrated that common and rare words differ on two structural dimensions. For printed words, they found that the “similarity neighborhoods” of common and rare words differ in both size and composition: High frequency words have more words in common (in terms of one letter substitutions) than low frequency words, and high frequency words tend to have high frequency neighbors, whereas low frequency words tend to have low frequency neighbors. Thus, for printed words, the similarity neighborhoods for high and low frequency words show marked differences. Landauer and Streeter also demonstrated that for spoken words, certain phonemes are more prevalent in high frequency words than in low frequency words and vice versa (see also [ 46 ]).

One of us [ 47 ] has undertaken a project that is aimed at extending and elaborating the original Landauer and Streeter study (see also [ 48 ]). In this research, both the similarity neighborhoods and phonemic constituencies of high and low frequency words have been examined in order to determine the extent to which spoken common and rare words differ in the nature and number of “neighbors” as well as phonemic configuration. To address these issues, an on-line version of Webster’s Pocket Dictionary (WPD) was employed to compute statistics about the structural organization of words. Specifically, the phonetic representations of approximately 20,000 words were used to compute similarity neighborhoods and examine phoneme distributions. (See Luce [ 47 ] for a more detailed description). Some initial results of this project are reported below.

Similarity Neighborhoods of Spoken Common and Rare Words

In an intial attempt to characterize the similarity neighborhoods of common and rare words, a subset of high and low frequency target words were selected from the WPD for evaluation. High frequency words were defined as those equal to or exceeding 1000 words per million in the Kucera and Francis [ 49 ] word count. Low frequency words were defined as those between 10 and 30 words per million inclusively. For each target word meeting these a priori frequency criteria, similarity neighborhoods were computed based on one-phoneme substitutions at each position within the target word. There were 92 high frequency words and 2063 low frequency words. The mean number of words within the similarity neighborhoods for the high and low frequency words were computed, as well as the mean frequencies of the neighbors. In addition, a decision rule was computed as a measure of the distinctiveness of a given target word relative to its neighborhood according to the following formul:

where T equals the frequency of the target word and N equals the frequency of the i-th neighbor of that target word (see [ 35 ]). Larger values for the decision rule indicate a target word that “stands out” in its neighborhood; smaller values indicate a target word that is relatively less distinctive in its neighborhood.

The results of this analysis, broken down by the length of the target word, are shown in Table I . (Mean frequencies of less, than one were obtained because some words included in the WPD were not listed in Kucera and Francis; these words were assigned a value of zero in the present analysis.) Of primary interest are the data for words of lengths two through four (in which more than two words were found for each length at each frequency). For these word lengths, it was found that although the mean number of neighbors for high and low frequency target words were approximately equal, the mean frequencies of the similarity neighborhoods for high frequency target words of lengths two and three were higher than the mean frequencies of the similarity neighborhoods of the low frequency target words.

Similarity neighborhood statistics for high and low frequency words as a function of word length. (Data from [ 47 ]).

No such difference was obtained, however, for target words consisting of four phonemes. Thus, these results only partially replicate Landauer and Streeter’s earlier results obtained from printed high and low frequency words, with the exception that the number of neighbors was not substantially different for high and low frequency words nor were the mean frequencies of the neighborhoods different for words consisting of four phonemes.

The finding that high frequency words tend to have neighbors of higher frequency than low frequency words suggests, somewhat paradoxically, that high frequency words are more, rather than less, likely to be confused with other words than low frequency words. At first glance, this finding would appear to contradict the results of many studies demonstrating that high frequency words are recognized more easily than low frequency words. However, as shown in Table I , the decision rule applied to high and low frequency target words predicts that high frequency words should be perceptually distinctive relative to the words in their neighborhoods whereas low frequency targets will not. This is shown by the substantially larger values of this index for high frequency words than low frequency words of the same length. Work is currently underway in our laboratory to determine if this decision rule predicts identification responses when frequencies of the target words are fixed and the values of the decision rule vary. If the relationship of a target word to its neighborhood, and not the frequency of the target word itself, is the primary predictor of identification performance, this would provide strong evidence that structural factors, rather than experienced frequency per se, underlie the word frequency effect (see also [ 36 , 35 ] for similar arguments).

Also of interest in Table I are the values of the decision rule and the percentage of unique target words (i.e., words with no neighbors) as a function of word length. For target words of both frequencies, the decision rule predicts increasingly better performance for words of greater length (except for the unique situation of one-phoneme high frequency words). In addition, it can be seen that for words consisting of more than three phonemes, the percentage of unique words increases substantially as word length increases. This finding demonstrates that simply increasing the length of a word increases the probability that the phonotactic configuration of that word will be unique and eventually diverge from all other words in the lexicon. Such a result suggests the potentially powerful contribution of word length in combination with various structural factors to the isolation of a given target word in the lexicon.

Phoneme Distributions in Common and Rare Words

The finding that high frequency spoken words tend to be more similar to other high frequency words than to low frequency words also suggests that certain phonemes or phonotactic configurations may be more common in high frequency words than in low frequency words [ 50 , 46 ]. As a first attempt to evaluate this claim, Luce [ 47 ] has examined the distribution of phonemes in words having frequencies of 100 or greater and words having a frequency of one. For each of the 45 phonemes used in the transcriptions contained in the WPD, percentages of the total number of possible phonemes for four and five phoneme words were computed for the high and low frequency subsets. (For the purposes of this analysis, function words were excluded. Luce [ 47 ] has demonstrated that function words are structurally quite different from content words of equivalent frequencies. In particular, function words tend to have many fewer neighbors than content words. Thus, in order to eliminate any contribution of word class effects, only content words were examined.)

Of the trends uncovered by these analyses, two were the most compelling. First, the percentages of bilabials, interdentals, palatals, and labiodentals tended to remain constant or decrease slightly from the low to high frequency words. However, the pattern of results for the alveolars and velars was quite different. For the alveolars, increases from low to high frequency words of 9.07% for the four phoneme words and 3.63% for the five phoneme words were observed. For the velars, however, the percentage of phonemes dropped from the low to high frequency words by 2.33% and 1.14% for the four and five phoneme words, respectively. In the second trend of interest, there was an increase of 4.84% for the nasals from low to high frequency words accompanied by a corresponding drop of 4.38% in the overall percentage of stops for the five phoneme words.

The finding that high frequency words tend to favor consonants having an alveolar place of articulation and disfavor those having a velar place of articulation suggests that frequently used words may have succumbed to pressures over the history of the language to exploit consonants that are in some senue easier to articulate [ 50 , 51 ]. This result, in conjunction with the finding for five phoneme words regarding the differential use of nasals and stops in common and rare words, strongly suggests that, at leas; in terms of phonemic constituency, common words differ structurally from rare words in terms of their choice or selection of constituent elements. Further analyses of the phonotactic configuration of high and low frequency words should reveal even more striking structural differences between high and low frequency words in light of the results obtained from the crude measure of structural differences based on the overall distributions of phonemes in common and rare words (see [ 47 ]).

Similarity Neighborhoods and Word Identification

In addition to the work summarized above demonstrating differences in structural characteristics of common and rare words, Luce [ 47 ] has demonstrated that the notion of similarity neighborhoods or lexical density may be used to derive predictions regarding word intelligibility that surpasses a simple frequency of usage explanation. A subset of 300 words published by Hood and Poole [ 52 ] which were ranked according to their intelligibility in white noise has been examined. As Hood and Poole pointed out, frequency of usage was not consistently correlated with word intelligibility scores for their data. It is there re likely that some metric based on the similarity neighborhoods of these words would be better at capturing the observed differences in intelligibility than simple frequency of occurrence.

To test this possibility, Luce [ 47 ] examined 50 of the words provided by Hood and Foole, 25 of which constituted the easiest words and 25 of which constituted the most difficult in their data. In keeping with Hood and Poole’s observation regarding word frequency, Luce found that the 25 easiest and 25 most difficult words were not, in fact, significantly different in frequency. However, it was found that the relationship of the easy words to their neighbors differed substantially from the relationship of the difficult words to their neighbors. More specifically, on the average, 56.41% of the words in the neighborhoods of the difficult words were equal to or higher in frequency than the difficult words themselves, whereas only 23.62% of the neighbors of the easy words were of equal or higher frequency. Thus, it appears that the observed differences in intelligibility may have been due, at least in part, to the frequency composition of the neighborhoods of the easy and difficult words, and were not primarily due to the frequencies of the words themselves (see also [ 53 , 54 ]). In particular, it appears that the difficult words in Hoode and Poole’s study were more difficult to perceive because they had relatively more “competition” from their neighbors than the easy words.

In summary, the results obtained thus far by Luce suggest that the processes involved in word recognition may be highly contingent on structural factors related to the organization of words in the lexicon and the relation of words to other phonetically similar words in surrounding neighborhoods in the lexicon. In particular, the present findings suggest that the classic word frequency effect may be due, in whole or in part, to structural differences between high and low frequency words, and not to experienced frequency per se. The outcome of this work should prove quite useful in discovering not only the underlying structure of the mental lexicon, but also in detailing the implications these structural constraints may have for the real-time processing of spoken language by human listeners as well as machines. In the case of machine recognition, these findings may provide a principled way to develop new distance metrics based on acoustic-phonetic similarity of words in large vocabularies.

Phonetic Refinement Theory

Within the last few years three major findings have emerged from a variety of experiments on spoken word recognition (see [ 22 , 21 , 27 , 19 ]). First, spoken words appear to be recognized from left-to-right; that is, words are recognized in the same temporal sequence by which they are produced. Second, the beginnings of words appear to be far more important for directing the recognition process than either the middles or the ends of words. Finally, word recognition involves an interaction between bottom-up pattern processing and top-down expectations derived from context and linguistic knowledge.

Although Cohort Theory was proposed to account for word recognition as an interactive process that depends on the beginnings of words for word candidate selection, it is still very similar to other theories of word recognition. Almost all of the current models of human auditory word recognition are based on pattern matching techniques. In these models, the correct recognition of a word depends on the exact match of an acoustic property or linguistic unit (e.g., a phoneme) derived from a stimulus word with a mental representation of that property or unit in the lexicon of the listener. For example, in Cohort Theory, words are recognized by a sequential match between input and lexical representations. However, despite the linear, serial nature of the matching process, most theories of word recognition generally have had little to say about the specific nature of the units that are being matched or the internal structure of words (see [ 5 ]). In addition, these theories make few, if any, claims about the structure or organization of words in the lexicon. This is unfortunate because models dealing with the process of word recognition may not be independent from the representations of words or the organization of words in the lexicon.

Recently, two of us [ 55 , 56 ] have proposed a different approach to word recognition that can account for the same findings as Cohort Theory. Moreover, the approach explicitly incorporates information about the Internal structure of words and the organization of words in the lexicon. This theoretical perspective, which we have called Phonetic Refinement Theory, proposes that word recognition should be viewed not as pattern matching but instead as constraint satisfaction. In other words, rather than assume that word recognition is a linear process of comparing elements of a stimulus pattern to patterns in the mental lexicon, word recognition is viewed from this perspective as a process more akin to relaxation labeling (e.g., [ 57 ]) in which a global interpretation of a visual pattern results from the simultaneous interaction of a number of local constraints. Translating this approach into terms more appropriate for auditory word recognition, the process of identifying a spoken word therefore depends on finding a word in the lexicon that simultaneously satisfies a number of constraints imposed by the stimulus, the structure of words in the lexicon, and the context in which the word was spoken.

Constraint Satisfaction

Phonetic Refinement Theory is based on the general finding that human listeners can and do use fine phonetic information in the speech waveform and use this information to recognize words, even when the acoustic-phonetic input is incomplete or only partially specified, or when it contains errors or is noisy. At present, the two constraints we consider most important for the bottom-up recognition of words (i.e., excluding the role of lingustic context) are the phonetic refinement of each segment in a word and its word length in terms of the number of segments in the word. Phonetic refinement refers to the process of identifying the phonetic information that is encoded in the acoustic pattern of a word. We assume that this process occurs over time such that each segment is first characterized by an acoustic event description. As more and more acoustic information is processed, acoustic events are characterized using increasingly finer and finer phonetic descriptions. The most salient phonetic properties of a segment are first described (e.g., manner); less salient properties are identified later as more acoustic information accumulates. Thus, we assume that decoding the phonetic structure of a word from the speech waveform requires time during which new acoustic segments are acquired and contribute to the phonetic refinement of earlier segments.

The constraints on word recognition can therefore be summarized as an increasingly better characterization of each of the phonetic segments of a word over time, as well as the development of an overall phonotactic pattern that emerges from the sequence of phonetic segments. These two constraints increase simultaneously over time and can be thought of as narrowing down the set of possible words. At some point, the left-to-right phonotactic constraint converges on the constraint provided by the increasing phonetic refinement of phonetic segments to specify a single word from among a number of phonetically-similar potential candidates.

Organization of the Lexicon

According to this view, words are recognized using a one-pass, left-to-right strategy with no backtracking as in Cohort Theory, LAFS, and Logogen Theory. However, unlike these theories, Phonetic Refinement Theory assumes that words in the lexicon are organized as sequences of phonetic segments in a multi-dimensional acoustic-phonetic space [ 45 ]. In this space, words that are more similar in their acoustic-phonetic structures are closer to each other, in the lexicon. Furthermore, it is possible to envision the lexicon as structured so that those portions of words that are similar in location and structure are closer together in this space. For example, words that rhyme with each other are topologically deformed to bring together those parts of the words that are phonetically similar and separate those portions of words that are phonetically distinct.

We assume that the recognition process takes place in this acoustic-phonetic space by activating pathways corresponding to words in the lexicon. Partial or incomplete phonetic descriptions of the input activate regions of the lexicon that consist of phonetically similar pathways. As more information is obtained about an utterance by continued phonetic refinement and acquisition of new segments, a progressive narrowing occurs in both the phonetic specification of the stimulus and the set of activated word candidates that are phonetically similar to the input signal. As more segments are acquired from the input and earlier segments are progressively refined, the constraints on the region of activation are increased until the word is recognized. According to Phonetic Refinement Theory, a word is recognized when the activation path for one word through the phonetic space is higher than any competing paths or regions through the lexicon.

Comparison with Cohort Theory

By focusing on the structural properties of words and the process of constraint satisfaction, Phonetic Refinement Theory is able to account for much of the same data that Cohort Theory was developed to deal with. Moreover, it is able to deal with some of the problems that Cohort Theory has been unable to resolve. First, by allowing linguistic context to serve as another source of constraint on word recognition, Phonetic Refinement Theory provides an interactive account of context effects that is similar to the account suggested by Cohort Theory (cf. [ 16 , 19 ]).

Second, Phonetic Refinement Theory can account for the apparent importance of word beginnings in recognition. In Cohort Theory, the acoustic-phonetic information at the beginning of a word entirely determines the set of cohorts (potential word candidates) that are considered for recognition. Word beginnings are important for recognition by fiat; that is, they are important because the theory has axiomatically assumed that they have a privileged status in determining which candidates are activated in recognition. In contrast, the importance of word beginnings in Phonetic Refinement Theory is simply a consequence of the temporal structure of spoken language and the process of phonetic refinement. Word beginnings do not exclude inconsistent word candidates; rather they activate candidates that are consistent with them to the degree that the candidates are consistent with word-initial information in the signal. Since the beginnings of words are, by necessity, processed first, they receive the most phonetic refinement earliest on in processing and therefore provide the strongest initial constraint on word candidates. As a consequence, Phonetic Refinement Theory can account for the ability of listeners to identify words from only partial information at the beginnings of words (e.g., [ 27 , 28 , 58 ]). In addition, Phonetic Refinement Theory predicts the finding that subjects can detect nonwords at the first phoneme that causes an utterance to become a nonword; that is, at the point where the nonword becomes different from all the words in the lexicon [ 59 ]. The theory makes this prediction directly because the segment that causes the input pattern to become a nonword directs the activation pathway to an “empty” region in the acoustic-phonetic lexical space.

More importantly, Phonetic Refinement Theory can account for a number of results that are inconsistent with Cohort Theory. For example, Cohort Theory cannot directly account for the ability of subjects to identify words in a gating study based on word endings alone [ 60 , 28 , 58 ]. However, in Phonetic Refinement Theory, word endings are a valid form of constraint on the recognition process and the theory predicts that listeners can and do use this information. The extent to which listeners use word endings in recognition depends, of course, on the relative efficiency of this constraint compared to the constraint provided by word beginnings. Therefore, Phonetic Refinement Theory predicts that listeners should be sensitive to phonetic overlap between prime and test words, whether that overlap occurs at the beginning or the ending of a word (see above and [ 31 ] for further details).

In addition, Cohort Theory cannot account for word frequency effects in perception (cf. [ 15 ]). The theory incorporates no mechanisms that would predict any effect of frequency whatsoever on word recognition. By comparison, Phonetic Refinement Theory incorporates two possible sources of word frequency effects. The first is based on findings suggesting the possibility that word frequency effects may be explained by the different structural properties of high and low frequency words [ 36 , 35 ]. According to this view, high and low frequency words occupy different acoustic-phonetic regions of the lexicon (see above and Luce [ 7 ] for further details). Thus, the density characteristics of these regions in the lexicon could account for the relative ease of perception of high and low frequency words, if high frequency words were in sparse neighborhoods while low frequency words resided in denser regions of the lexicon.

A second account of word frequency effects in perception appeals to the use of experienced frequency or familiarity as a selectional constraint to be used for generating a response once a region of the lexicon has been activated. In the case of isolated words, this selectional constraint represents the subject’s “best guess” in the case that no other stimulus properties could be employed to resolve a word path from an activated region. In understanding fluent speech, this selection constraint would be supplanted by the more reasonable constraint imposed by expectations derived from linguistic context [ 61 , 62 ]. Thus, word frequency effects should be substantially attenuated or even eliminated when words are placed in meaningful contexts. This is precisely the result observed by Luce [ 62 ] in a study on auditory word identification in isolation and in sentence’ context.

Finally, Phonetic Refinement Theory is able to deal with the effects of noise, segmental ambiguity, and mispronunciations in a much more elegant manner than Cohort Theory. In Cohort Theory, word-initial acoustic-phonetic information determines the set of possible word candidates from which the recognized word is chosen. If there is a mispronunciation of the initial segment of a word (see [ 12 ]), the wrong set of word candidates will be activated and there will be no way to recover gracefully from the error. However, in Phonetic Refinement Theory, if a phonetic segment is incorrectly recognized, two outcomes are possible. If the mispronunciation yields an utterance that is a nonword, correct recognition should be possible by increasing other constraints such as acquiring more segments from the input. At some point in the utterance, the pathway with the highest activation will lead into a “hole” in the lexicon where no word is found. However, the next highest pathway will specify the correct word. An incorrect phoneme that occurs early in a word will probably terminate a pathway in empty space quite early so that, by the end of the utterance, the correct word will actually have a higher aggregate level of pathway activation than the aborted path corresponding to the nonword. This is a simple consequence of the correct pathway having more similar segments to the utterance over the entire path than the nonword sequence ending in a hole in the lexicon. If the error occurs late in the word, it may actually occur after the constraints on the word were sufficient to permit recognition. Thus, Phonetic Refinement Theory has little difficulty recovering from errors that result in nonwords. However, for the second type of error -- those that result in a real word other than the intended word -- there is no way the Phonetic Refinement Theory could recover from this error without using linguistic context as a constraint. Of course, this sort of error could not be recovered from by any recognition system, including a human listener, unless context was allowed to play a direct role in the early recognition process; this assumption is still the topic of intense controversy and we will not attempt to deal with it here.

Structural Constraints on Word Recognition

Although it has been asserted by several researchers that words can be recognized from only a partial specification of the phonetic content of words, these claims are based primarily on data from gating experiments (e.g., [ 27 , 60 , 58 , 28 ]). Since we argue that it is the structure of words in the lexicon that determines the performance of human listeners and not simply some form of sophisticated guessing strategy, it is important to learn more about the relative power of different phonetic and phonotactic constraints in reducing the search space of word candidates during recognition.

The approach we have taken to this problem was motivated by several recent studies that were conducted to investigate the relative heuristic power of various classification schemes for large vocabulary word recognition by computers [ 63 , 64 , 34 , 65 ]. The goal of this research has been to find a classification scheme that reduces the lexical search space from a very large vocabulary (i.e., greater than 20,000 words) to a very few candidates. An optimal classification heuristic would be one that yields candidate sets that contain an average of one word each, without requiring complete identification of all the phonemes in each word. However, even if one heuristic is not optimal, it may still reduce the search space by a significant amount at a very low cost in computational complexity; other constraints can then be applied to finally “recognize” the word from among the members contained in the reduced search space. Thus, instead of a serial search through a very large number of words, heuristics that reduce the search space can quickly rule out very large subsets of words that are totally inconsistent with an utterance without requiring highly detailed pattern matching.

In a number of recent papers, Zue and his colleagues [ 64 , 34 ] have shown that a partial phonetic specification of every phoneme in a word results in an average candidate set size of about 2 words for a ocabulary of 20,000 words. The partial phonetic specification consisted of six gross manner classes of phonemes. Instead of using 40 to 50 phonemes to transcribe a spoken word, only six gross categories were used: stop consonant, strong fricative, weak fricative, nasal, liquid/glide, or vowel. These categories obviously represent a relatively coarse level of phonetic description and yet when combined with word length, they provide a powerful phonotactic constraint on the size of the lexical search space.

Using a slightly different approach, Crystal et al. [ 63 ] demonstrated that increasing the phonetic refinement of every phoneme in a word from four gross categories to ten slightly more refined categories produced large improvements in the number of unique words that could be isolated in a large corpus of text. However, both of these computational studies examined the consequences of partially classifying every segment in a word. Thus, they actually employed two constraints: (1) the partial classification of each segment and (2) the broad phonotactic shape of each word resulting from the combination of word length with gross phonetic category information.

We have carried out several analyses recently using a large lexical database containing phonetic transcriptions of 126,000 words to study the effects of different constraints on search space reduction in auditory word recognition [ 55 , 56 ]. Figure 2 shows the results of one analysis based on word length constraints. Knowing only the length of a word in phonemes reduces the search space from about 126,000 words to 6,342 words. Clearly, word length is a powerful constraint in reducing the lexicon on the average of about two orders of magnitude, even without any detailed segmental phonetic information. Of course, as Figure 2 shows, the length constraint is strongest for relatively long words.

The effects of word length constraints on the number of candidates in a lexicon of 126,000 words. The open squares show the number of words in the database at each word length (in number of phonemes). The plus symbols (+) indicate the increased constraint over and above word length that occurs when each segment is classified as either a consonant or a vowel.

Figure 2 also shows the results of another analysis, the effect of adding the constraint of minimal phonetic information about every phoneme in The words -- that is, simply classifying each segment as either a consonant or vowel. The reduction in the search space over and above the length constraint by this minimal phonotactic constraint is enormous. The number of words. considered as potential candidates for recognition is reduced from the original 126,000 word lexicon to about 34 words per candidate set on average.

Figure 3 shows a comparison of the log weighted-mean candidate set sizes for the minimal phonotactic constraint of classification of phonemes into two categories (consonants and vowels), with the six gross manner class scheme used by Zue and his colleagues. We have found that their results obtained on a 20,000 word lexicon generalize to our 126,000 word lexicon -- the unweighted-mean candidate set size computed in the same way as Shipman and Zue [ 34 ] is 2.4 words. Figure 3 also shows the constraint afforded by complete identification of only some of the phonemes in words. While a partial specification of all the phonemes in words provides overall “word shape” information (in some sense), it is not at all clear that complete information about some of the phonemes in a word would be as effective in reducing the search space. We classified just over half of the phonemes in each of the words in the original 126,000 word lexicon from the beginning of the words and from the ends of the words. The results of this analysis are shown in Figure 3 The weighted-mean candidate set size for words classified from the beginning was about 1.7 words; for words classified from the end the weighted mean was about 1.8 words. These results demonstrate that detailed phonetic information, even only partial information about the phonetic content of a word is a very effective heuristic for reducing the number of possible words to be recognized.

The relative effectiveness of different types of phonetic and phonotactic constraints in a lexicon of 126,000 words. The constraints shown are; (a) every segment in each word classified as a consonant or vowel (CV), (b) every segment labeled as a member or one of six gross manner classes (6 CLASSES), (c) the phonemes in the first half of each word identified exactly (INITIAL), and (d) the phonemes in the last half of each word identified exactly (FINAL). Constraint effectiveness is indexed by the log weighted-mean of the number of word candidates that result from the application of each constraint to the total vocabulary.

Thus, from these initial findings, it is quite apparent that the basic approach of Phonetic Refinement Theory is valid and has much to offer. Having refined roughly the first half of a word, a listener need only compute a fairly coarse characterization of the remainder of a word to uniquely identify it. This finding suggests an account of the observed failure of listeners to detect word-final mispronunciations with the same accuracy as word-initial mispronunciations (see [ 22 ]). Given reliable Information in the early portions of words, listeners do not need to precisely identify all the phonemes in the latter half of words. Furthermore, the word candidate set sizes resulting from phonetic refinement of the endings of words indicates that, in spite of the large number of common English word-final inflections and affixes, word-final phonetic information also provides strong constraints on word recognition. For words between three and ten phonemes long, the mean cohort size resulting from classification of the beginning was 2.4 word whereas for classification of word endings, the mean cohort size was 2.8 words. This small difference in effectiveness between word-initial and word-final constraints suggests that listeners might be slightly better in identifying words from their beginnings than from their endings -- a result that was observed recently with human listeners by Salasoo and Pisoni [ 28 ] using the gating paradigm.

Taken together, Phonetic Refinement Theory is able to account for many of the findings in auditory word recognition by reference to structural constraints in the lexicon. Moreover, it is also clear that there are advantages to the phonetic refinement approach with respect to recovery from phonetic classification errors due to noise, ambiguity, or mispronunciations. Phonetic Refinement Theory can account for the ability of listeners to identify words from word endings and their sensitivity to acoustic-phonetic overlap between prime and test words in word recognition. Both of these findings would be difficult, if not impossible, to account for with the current version of Cohort Theory [ 59 ] which emphasizes the primacy of word-initial acoustic-phonetic information in controlling activation of potential word candidates in the early stages of word recognition (cf. [ 29 ]).

Summary and Conclusions

In this report we have briefly summarized research findings from three on-going projects that are concerned with the general problem of auditory word recognition. Data on the perceptual sensitivity of listeners to the distribution of acoustic-phonetic information in the structure of words, taken together with new research on the organization of words in the lexicon has identified a number of serious problems with the Cohort Theory of auditory word recognition. To deal with these problems, we have proposed a new approach to word recognition known as Phonetic Refinement Theory. The theory was designed to account for the way listeners use detailed knowledge about the internal structure of words and the organization of words in the lexicon in word recognition. Whether the details of our approach will continue to be supported by subsequent research remains to be seen. Regardless of this outcome, however, we feel that the most important contribution of the theory will probably lie in directing research efforts towards the study of how the perceptual processing of the acoustic-phonetic structure of speech interacts with the listener’s knowledge of the structure of words and the organization of words in his/her lexicon. This is an important and seemingly neglected area of research on language processing that encompasses both speech perception and word recognition.

* Preparation of this paper was supported, in part, by NIH research grant NS-12179-08 to Indiana University in Bloomington. We thank Beth Greene for her help in editing the manuscript, Arthur House and Tom Crystal for providing us with a machine readable version of the lexical database used in our analyses and Chris Davis for his outstanding contributions to the software development efforts on the SRL Project Lexicon. This paper was written in honor of Ludmilla Chistovich, one of the great pioneers of speech research, on her 60th birthday. We hope that the research described in this paper will influence other researchers in the future in the way Dr. Chistovich’s now classic work has influenced our own thinking about the many important problems in speech perception and production. It is an honor for us to submit this report as a small token of our appreciation of her contributions to the field of speech communications. We extend our warmest regards and congratulations to her on this occasion and wish her many more years of good health and productivity. This paper will appear in Speech Communication, 1985.

Speech Recognition: Everything You Need to Know in 2024

Speech recognition, also known as automatic speech recognition (ASR) , enables seamless communication between humans and machines. This technology empowers organizations to transform human speech into written text. Speech recognition technology can revolutionize many business applications , including customer service, healthcare, finance and sales.

In this comprehensive guide, we will explain speech recognition, exploring how it works, the algorithms involved, and the use cases of various industries.

If you require training data for your speech recognition system, here is a guide to finding the right speech data collection services.

What is speech recognition?

Speech recognition, also known as automatic speech recognition (ASR), speech-to-text (STT), and computer speech recognition, is a technology that enables a computer to recognize and convert spoken language into text.

Speech recognition technology uses AI and machine learning models to accurately identify and transcribe different accents, dialects, and speech patterns.

What are the features of speech recognition systems?

Speech recognition systems have several components that work together to understand and process human speech. Key features of effective speech recognition are:

• Audio preprocessing: After you have obtained the raw audio signal from an input device, you need to preprocess it to improve the quality of the speech input The main goal of audio preprocessing is to capture relevant speech data by removing any unwanted artifacts and reducing noise.
• Feature extraction: This stage converts the preprocessed audio signal into a more informative representation. This makes raw audio data more manageable for machine learning models in speech recognition systems.
• Language model weighting: Language weighting gives more weight to certain words and phrases, such as product references, in audio and voice signals. This makes those keywords more likely to be recognized in a subsequent speech by speech recognition systems.
• Acoustic modeling : It enables speech recognizers to capture and distinguish phonetic units within a speech signal. Acoustic models are trained on large datasets containing speech samples from a diverse set of speakers with different accents, speaking styles, and backgrounds.
• Speaker labeling: It enables speech recognition applications to determine the identities of multiple speakers in an audio recording. It assigns unique labels to each speaker in an audio recording, allowing the identification of which speaker was speaking at any given time.
• Profanity filtering: The process of removing offensive, inappropriate, or explicit words or phrases from audio data.

What are the different speech recognition algorithms?

Speech recognition uses various algorithms and computation techniques to convert spoken language into written language. The following are some of the most commonly used speech recognition methods:

• Hidden Markov Models (HMMs): Hidden Markov model is a statistical Markov model commonly used in traditional speech recognition systems. HMMs capture the relationship between the acoustic features and model the temporal dynamics of speech signals.
• Estimate the probability of word sequences in the recognized text
• Convert colloquial expressions and abbreviations in a spoken language into a standard written form
• Map phonetic units obtained from acoustic models to their corresponding words in the target language.
• Speaker Diarization (SD): Speaker diarization, or speaker labeling, is the process of identifying and attributing speech segments to their respective speakers (Figure 1). It allows for speaker-specific voice recognition and the identification of individuals in a conversation.

Figure 1: A flowchart illustrating the speaker diarization process

• Dynamic Time Warping (DTW): Speech recognition algorithms use Dynamic Time Warping (DTW) algorithm to find an optimal alignment between two sequences (Figure 2).

Figure 2: A speech recognizer using dynamic time warping to determine the optimal distance between elements

5. Deep neural networks: Neural networks process and transform input data by simulating the non-linear frequency perception of the human auditory system.

6. Connectionist Temporal Classification (CTC): It is a training objective introduced by Alex Graves in 2006. CTC is especially useful for sequence labeling tasks and end-to-end speech recognition systems. It allows the neural network to discover the relationship between input frames and align input frames with output labels.

Speech recognition vs voice recognition

Speech recognition is commonly confused with voice recognition, yet, they refer to distinct concepts. Speech recognition converts  spoken words into written text, focusing on identifying the words and sentences spoken by a user, regardless of the speaker’s identity. 

On the other hand, voice recognition is concerned with recognizing or verifying a speaker’s voice, aiming to determine the identity of an unknown speaker rather than focusing on understanding the content of the speech.

What are the challenges of speech recognition with solutions?

While speech recognition technology offers many benefits, it still faces a number of challenges that need to be addressed. Some of the main limitations of speech recognition include:

Acoustic Challenges:

• Assume a speech recognition model has been primarily trained on American English accents. If a speaker with a strong Scottish accent uses the system, they may encounter difficulties due to pronunciation differences. For example, the word “water” is pronounced differently in both accents. If the system is not familiar with this pronunciation, it may struggle to recognize the word “water.”

Solution: Addressing these challenges is crucial to enhancing  speech recognition applications’ accuracy. To overcome pronunciation variations, it is essential to expand the training data to include samples from speakers with diverse accents. This approach helps the system recognize and understand a broader range of speech patterns.

• For instance, you can use data augmentation techniques to reduce the impact of noise on audio data. Data augmentation helps train speech recognition models with noisy data to improve model accuracy in real-world environments.

Figure 3: Examples of a target sentence (“The clown had a funny face”) in the background noise of babble, car and rain.

Linguistic Challenges:

• Out-of-vocabulary words: Since the speech recognizers model has not been trained on OOV words, they may incorrectly recognize them as different or fail to transcribe them when encountering them.

Figure 4: An example of detecting OOV word

Solution: Word Error Rate (WER) is a common metric that is used to measure the accuracy of a speech recognition or machine translation system. The word error rate can be computed as:

Figure 5: Demonstrating how to calculate word error rate (WER)

• Homophones: Homophones are words that are pronounced identically but have different meanings, such as “to,” “too,” and “two”. Solution: Semantic analysis allows speech recognition programs to select the appropriate homophone based on its intended meaning in a given context. Addressing homophones improves the ability of the speech recognition process to understand and transcribe spoken words accurately.

Technical/System Challenges:

• Data privacy and security: Speech recognition systems involve processing and storing sensitive and personal information, such as financial information. An unauthorized party could use the captured information, leading to privacy breaches.

Solution: You can encrypt sensitive and personal audio information transmitted between the user’s device and the speech recognition software. Another technique for addressing data privacy and security in speech recognition systems is data masking. Data masking algorithms mask and replace sensitive speech data with structurally identical but acoustically different data.

Figure 6: An example of how data masking works

• Limited training data: Limited training data directly impacts  the performance of speech recognition software. With insufficient training data, the speech recognition model may struggle to generalize different accents or recognize less common words.

Solution: To improve the quality and quantity of training data, you can expand the existing dataset using data augmentation and synthetic data generation technologies.

13 speech recognition use cases and applications

In this section, we will explain how speech recognition revolutionizes the communication landscape across industries and changes the way businesses interact with machines.

Customer Service and Support

• Interactive Voice Response (IVR) systems: Interactive voice response (IVR) is a technology that automates the process of routing callers to the appropriate department. It understands customer queries and routes calls to the relevant departments. This reduces the call volume for contact centers and minimizes wait times. IVR systems address simple customer questions without human intervention by employing pre-recorded messages or text-to-speech technology . Automatic Speech Recognition (ASR) allows IVR systems to comprehend and respond to customer inquiries and complaints in real time.
• Customer support automation and chatbots: According to a survey, 78% of consumers interacted with a chatbot in 2022, but 80% of respondents said using chatbots increased their frustration level.
• Sentiment analysis and call monitoring: Speech recognition technology converts spoken content from a call into text. After  speech-to-text processing, natural language processing (NLP) techniques analyze the text and assign a sentiment score to the conversation, such as positive, negative, or neutral. By integrating speech recognition with sentiment analysis, organizations can address issues early on and gain valuable insights into customer preferences.
• Multilingual support: Speech recognition software can be trained in various languages to recognize and transcribe the language spoken by a user accurately. By integrating speech recognition technology into chatbots and Interactive Voice Response (IVR) systems, organizations can overcome language barriers and reach a global audience (Figure 7). Multilingual chatbots and IVR automatically detect the language spoken by a user and switch to the appropriate language model.

Figure 7: Showing how a multilingual chatbot recognizes words in another language

• Customer authentication with voice biometrics: Voice biometrics use speech recognition technologies to analyze a speaker’s voice and extract features such as accent and speed to verify their identity.

Sales and Marketing:

• Virtual sales assistants: Virtual sales assistants are AI-powered chatbots that assist customers with purchasing and communicate with them through voice interactions. Speech recognition allows virtual sales assistants to understand the intent behind spoken language and tailor their responses based on customer preferences.
• Transcription services : Speech recognition software records audio from sales calls and meetings and then converts the spoken words into written text using speech-to-text algorithms.

Automotive:

• Voice-activated controls: Voice-activated controls allow users to interact with devices and applications using voice commands. Drivers can operate features like climate control, phone calls, or navigation systems.
• Voice-assisted navigation: Voice-assisted navigation provides real-time voice-guided directions by utilizing the driver’s voice input for the destination. Drivers can request real-time traffic updates or search for nearby points of interest using voice commands without physical controls.

Healthcare:

• Recording the physician’s dictation
• Transcribing the audio recording into written text using speech recognition technology
• Editing the transcribed text for better accuracy and correcting errors as needed
• Formatting the document in accordance with legal and medical requirements.
• Virtual medical assistants: Virtual medical assistants (VMAs) use speech recognition, natural language processing, and machine learning algorithms to communicate with patients through voice or text. Speech recognition software allows VMAs to respond to voice commands, retrieve information from electronic health records (EHRs) and automate the medical transcription process.
• Electronic Health Records (EHR) integration: Healthcare professionals can use voice commands to navigate the EHR system , access patient data, and enter data into specific fields.

Technology:

• Virtual agents: Virtual agents utilize natural language processing (NLP) and speech recognition technologies to understand spoken language and convert it into text. Speech recognition enables virtual agents to process spoken language in real-time and respond promptly and accurately to user voice commands.

Further reading

• Top 5 Speech Recognition Data Collection Methods in 2023
• Top 11 Speech Recognition Applications in 2023

External Links

• 1. Databricks
• 2. PubMed Central
• 3. Qin, L. (2013). Learning Out-of-vocabulary Words in Automatic Speech Recognition . Carnegie Mellon University.
• 4. Wikipedia

Next to Read

10+ speech data collection services in 2024, top 5 speech recognition data collection methods in 2024, top 4 speech recognition challenges & solutions in 2024.

Your email address will not be published. All fields are required.

Related research

Top 11 Voice Recognition Applications in 2024

Speech Recognition: Definition, Importance and Uses

Transkriptor 2024-01-17

Speech recognition, known as voice recognition or speech-to-text, is a technological development that converts spoken language into written text. It has two main benefits, these include enhancing task efficiency and increasing accessibility for everyone including individuals with physical impairments.

The alternative of speech recognition is manual transcription. Manual transcription is the process of converting spoken language into written text by listening to an audio or video recording and typing out the content.

There are many speech recognition software, but a few names stand out in the market when it comes to speech recognition software; Dragon NaturallySpeaking, Google's Speech-to-Text and Transkriptor.

The concept behind "what is speech recognition?" pertains to the capacity of a system or software to understand and transform oral communication into written textual form. It functions as the fundamental basis for a wide range of modern applications, ranging from voice-activated virtual assistants such as Siri or Alexa to dictation tools and hands-free gadget manipulation.

The development is going to contribute to a greater integration of voice-based interactions into an individual's everyday life.

What is Speech Recognition?

Speech recognition, known as ASR, voice recognition or speech-to-text, is a technological process. It allows computers to analyze and transcribe human speech into text.

How does Speech Recognition work?

Speech recognition technology works similar to how a person has a conversation with a friend. Ears detect the voice, and the brain processes and understands.The technology does, but it involves advanced software as well as intricate algorithms. There are four steps to how it works.

The microphone records the sounds of the voice and converts them into little digital signals when users speak into a device. The software processes the signals to exclude other voices and enhance the primary speech. The system breaks down the speech into small units called phonemes.

Different phonemes give their own unique mathematical representations by the system. It is able to differentiate between individual words and make educated predictions about what the speaker is trying to convey.

The system uses a language model to predict the right words. The model predicts and corrects word sequences based on the context of the speech.

The textual representation of the speech is produced by the system. The process requires a short amount of time. However, the correctness of the transcription is contingent on a variety of circumstances including the quality of the audio.

What is the importance of Speech Recognition?

The importance of speech recognition is listed below.

• Efficiency: It allows for hands-free operation. It makes multitasking easier and more efficient.
• Accessibility: It provides essential support for people with disabilities.
• Safety: It reduces distractions by allowing hands-free phone calls.
• Real-time translation: It facilitates real-time language translation. It breaks down communication barriers.
• Automation: It powers virtual assistants like Siri, Alexa, and Google Assistant, streamlining many daily tasks.
• Personalization: It allows devices and apps to understand user preferences and commands.

What are the Uses of Speech Recognition?

The 7 uses of speech recognition are listed below.

• Virtual Assistants. It includes powering voice-activated assistants like Siri, Alexa, and Google Assistant.
• Transcription services. It involves converting spoken content into written text for documentation, subtitles, or other purposes.
• Healthcare. It allows doctors and nurses to dictate patient notes and records hands-free.
• Automotive. It covers enabling voice-activated controls in vehicles, from playing music to navigation.
• Customer service. It embraces powering voice-activated IVRs in call centers.
• Educatio.: It is for easing in language learning apps, aiding in pronunciation, and comprehension exercises.
• Gaming. It includes providing voice command capabilities in video games for a more immersive experience.

Who Uses Speech Recognition?

General consumers, professionals, students, developers, and content creators use voice recognition software. Voice recognition sends text messages, makes phone calls, and manages their devices with voice commands. Lawyers, doctors, and journalists are among the professionals who employ speech recognition. Using speech recognition software, they dictate domain-specific information.

What is the Advantage of Using Speech Recognition?

The advantage of using speech recognition is mainly its accessibility and efficiency. It makes human-machine interaction more accessible and efficient. It reduces the human need which is also time-consuming and open to mistakes.

It is beneficial for accessibility. People with hearing difficulties use voice commands to communicate easily. Healthcare has seen considerable efficiency increases, with professionals using speech recognition for quick recording. Voice commands in driving settings help maintain safety and allow hands and eyes to focus on essential duties.

What is the Disadvantage of Using Speech Recognition?

The disadvantage of using speech recognition is its potential for inaccuracies and its reliance on specific conditions. Ambient noise or  accents confuse the algorithm. It results in misinterpretations or transcribing errors.

These inaccuracies are problematic. They are crucial in sensitive situations such as medical transcribing or legal documentation. Some systems need time to learn how a person speaks in order to work correctly. Voice recognition systems probably have difficulty interpreting multiple speakers at the same time. Another disadvantage is privacy. Voice-activated devices may inadvertently record private conversations.

What are the Different Types of Speech Recognition?

The 3 different types of speech recognition are listed below.

• Automatic Speech Recognition (ASR)
• Speaker-Dependent Recognition (SDR)
• Speaker-Independent Recognition (SIR)

Automatic Speech Recognition (ASR) is one of the most common types of speech recognition . ASR systems convert spoken language into text format. Many applications use them like Siri and Alexa. ASR focuses on understanding and transcribing speech regardless of the speaker, making it widely applicable.

Speaker-Dependent recognition recognizes a single user's voice. It needs time to learn and adapt to their particular voice patterns and accents. Speaker-dependent systems are very accurate because of the training. However, they struggle to recognize new voices.

Speaker-independent recognition interprets and transcribes speech from any speaker. It does not care about the accent, speaking pace, or voice pitch. These systems are useful in applications with many users.

What Accents and Languages Can Speech Recognition Systems Recognize?

The accents and languages that speech recognition systems can recognize are English, Spanish, and Mandarin to less common ones. These systems frequently incorporate customized models for distinguishing dialects and accents. It recognizes the diversity within languages. Transkriptor, for example, as a dictation software, supports over 100 languages.

Is Speech Recognition Software Accurate?

Yes, speech recognition software is accurate above 95%. However, its accuracy varies depending on a number of things. Background noise and audio quality are two examples of these.

How Accurate Can the Results of Speech Recognition Be?

Speech recognition results can achieve accuracy levels of up to 99% under optimal conditions. The highest level of speech recognition accuracy requires controlled conditions such as audio quality and background noises. Leading speech recognition systems have reported accuracy rates that exceed 99%.

How Does Text Transcription Work with Speech Recognition?

Text transcription works with speech recognition by analyzing and processing audio signals. Text transcription process starts with a microphone that records the speech and converts it to digital data. The algorithm then divides the digital sound into small pieces and analyzes each one to identify its distinct tones.

Advanced computer algorithms aid the system for matching these sounds to recognized speech patterns. The software compares these patterns to a massive language database to find the words users articulated. It then brings the words together to create a logical text.

How are Audio Data Processed with Speech Recognition?

Speech recognition processes audio data by splitting sound waves, extracting features, and mapping them to linguistic parts. The system collects and processes continuous sound waves when users speak into a device. The software advances to the feature extraction stage.

The software isolates specific features of the sound. It focuses on phonemes that are crucial for identifying one phoneme from another. The process entails evaluating the frequency components.

The system then starts using its trained models. The software combines the extracted features to known phonemes by using vast databases and machine learning models.

The system takes the phonemes, and puts them together to form words and phrases. The system combines technology skills and language understanding to convert noises into intelligible text or commands.

What is the best speech recognition software?

The 3 best speech recognition software are listed below.

Transkriptor

• Dragon NaturallySpeaking
• Google's Speech-to-Text

However, choosing the best speech recognition software depends on personal preferences.

Transkriptor is an online transcription software that uses artificial intelligence for quick and accurate transcription. Users are able to translate their transcripts with a single click right from the Transkriptor dashboard. Transkriptor technology is available in the form of a smartphone app, a Google Chrome extension, and a virtual meeting bot. It is compatible with popular platforms like Zoom, Microsoft Teams, and Google Meet which makes it one of the Best Speech Recognition Software.

Dragon NaturallySpeaking allows users to transform spoken speech into written text. It offers accessibility as well as adaptations for specific linguistic languages. Users like software’s adaptability for different vocabularies.

Google's Speech-to-Text is widely used for its scalability, integration options, and ability to support multiple languages. Individuals use it in a variety of applications ranging from transcription services to voice-command systems.

Is Speech Recognition and Dictation the Same?

No, speech recognition and dictation are not the same. Their principal goals are different, even though both voice recognition and dictation make conversion of spoken language into text. Speech recognition is a broader term covering the technology's ability to recognize and analyze spoken words. It converts them into a format that computers understand.

Dictation refers to the process of speaking aloud for recording. Dictation software uses speech recognition to convert spoken words into written text.

What is the Difference between Speech Recognition and Dictation?

The difference between speech recognition and dictation are related to their primary purpose, interactions, and scope. Itss primary purpose is to recognize and understand spoken words. Dictation has a more definite purpose. It focuses on directly transcribing spoken speech into written form.

Speech Recognition covers a wide range of applications in terms of scope. It helps voice assistants respond to user questions. Dictation has a narrower scope.

It provides a more dynamic interactive experience, often allowing for two-way dialogues. For example, virtual assistants such as Siri or Alexa not only understand user requests but also provide feedback or answers. Dictation works in a more basic fashion. It's typically a one-way procedure in which the user speaks and the system transcribes without the program engaging in a response discussion.

Frequently Asked Questions

Transkriptor stands out for its ability to support over 100 languages and its ease of use across various platforms. Its AI-driven technology focuses on quick and accurate transcription.

Yes, modern speech recognition software is increasingly adept at handling various accents. Advanced systems use extensive language models that include different dialects and accents, allowing them to accurately recognize and transcribe speech from diverse speakers.

Speech recognition technology greatly enhances accessibility by enabling voice-based control and communication, which is particularly beneficial for individuals with physical impairments or motor skill limitations. It allows them to operate devices, access information, and communicate effectively.

Speech recognition technology's efficiency in noisy environments has improved, but it can still be challenging. Advanced systems employ noise cancellation and voice isolation techniques to filter out background noise and focus on the speaker's voice.

Speech to Text

Convert your audio and video files to text

Audio to Text

Video Transcription

Transcription Service

Privacy Policy

Terms of Service

Contact Information

[email protected]

© 2024 Transkriptor

• Search Menu
• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Urban Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Literature
• Classical Reception
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Epigraphy
• Greek and Roman Law
• Greek and Roman Papyrology
• Greek and Roman Archaeology
• Late Antiquity
• Religion in the Ancient World
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Emotions
• History of Agriculture
• History of Education
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Acquisition
• Language Evolution
• Language Reference
• Language Variation
• Language Families
• Lexicography
• Linguistic Anthropology
• Linguistic Theories
• Linguistic Typology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature
• Bibliography
• Children's Literature Studies
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies (Modernism)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Religion
• Music and Media
• Music and Culture
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Science
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Lifestyle, Home, and Garden
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Politics
• Law and Society
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Neuroanaesthesia
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Toxicology
• Medical Oncology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Clinical Neuroscience
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Medical Ethics
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Security
• Computer Games
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Psychology
• Cognitive Neuroscience
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business Strategy
• Business Ethics
• Business History
• Business and Government
• Business and Technology
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic Systems
• Economic History
• Economic Methodology
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Natural Disasters (Environment)
• Social Impact of Environmental Issues (Social Science)
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• International Political Economy
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Political Behaviour
• Political Economy
• Political Institutions
• Political Theory
• Politics and Law
• Public Administration
• Public Policy
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Developmental and Physical Disabilities Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

• < Previous chapter
• Next chapter >

33 Speech Recognition

Lori Lamel is a Senior Research Scientist at LIMSI CNRS, which she joined as a permanent researcher in October 1991. She holds a PhD degree in EECS from MIT, and an HDR in CS from the University of Paris XI. She also has over 330 peer-reviewed publications. Her research covers a range of topics in the field of spoken language processing and corpus-based linguistics. She is a Fellow of ISCA and IEEE and serves on the ISCA board.

Jean-Luc Gauvain is a Senior Research Scientist at the CNRS and Spoken Language Processing Group Head at LISN. He received a doctorate in electronics and a computer science HDR degree from Paris-Sud University. His research centres on speech technologies, including speech recognition, audio indexing, and language and speaker recognition. He has contributed over 300 publications to this field and was awarded a CNRS silver medal in 2007. He served as co-editor-in-chief for Speech Communication Journal in 2006–2008, and as scientific coordinator for the Quaero research programme in 2008–2013. He is an ISCA Fellow.

• Published: 10 December 2015
• Cite Icon Cite
• Permissions Icon Permissions

Speech recognition is concerned with converting the speech waveform, an acoustic signal, into a sequence of words. Today’s best-performing approaches are based on a statistical modelization of the speech signal. This chapter provides an overview of the main topics addressed in speech recognition: that is, acoustic-phonetic modelling, lexical representation, language modelling, decoding, and model adaptation. The focus is on methods used in state-of-the-art, speaker-independent, large-vocabulary continuous speech recognition (LVCSR). Some of the technology advances over the last decade are highlighted. Primary application areas for such technology initially addressed dictation tasks and interactive systems for limited domain information access (usually referred to as spoken language dialogue systems). The last decade has witnessed a wider coverage of languages, as well as growing interest in transcription systems for information archival and retrieval, media monitoring, automatic subtitling and speech analytics. Some outstanding issues and directions of future research are discussed.

33.1 Introduction

Speech recognition is principally concerned with the problem of transcribing the speech signal as a sequence of words. Today’s best-performing systems use statistical models (Chapter 12 ) of speech. From this point of view, speech generation is described by a language model which provides estimates of Pr( w ) for all word strings w independently of the observed signal, and an acoustic model that represents, by means of a probability density function f(x|w ), the likelihood of the signal x given the message w . The goal of speech recognition is to find the most likely word sequence given the observed acoustic signal. The speech-decoding problem thus consists of maximizing the probability of w given the speech signal x , or equivalently, maximizing the product Pr( w)f(x|w ).

The principles on which these systems are based have been known for many years now, and include the application of information theory to speech recognition ( Bahl et al. 1976 ; Jelinek 1976 ), the use of a spectral representation of the speech signal ( Dreyfus-Graf 1949 ; Dudley and Balashek 1958 ), the use of dynamic programming for decoding ( Vintsyuk 1968 ), and the use of context-dependent acoustic models ( Schwartz et al. 1984 ). Despite the fact that some of these techniques were proposed well over two decades agos, considerable progress has been made in recent years in part due to the availability of large speech and text corpora (Chapters 19 and 20 ) and improved processing power, which have allowed more complex models and algorithms to be implemented. Compared with the state-of-the-art technology a decade ago, advances in acoustic modelling have enabled reasonable transcription performance for various data types and acoustic conditions.

The main components of a generic speech recognition system are shown in Figure 33.1 . The elements shown are the main knowledge sources (speech and textual training materials and the pronunciation lexicon), the feature analysis (or parameterization), the acoustic and language models which are estimated in a training phase, and the decoder. The next four sections are devoted to discussing these main components. The last two sections provide some indicative measures of state-of-the-art performance on some common tasks as well as some perspectives for future research.

System diagram of a generic speech recognizer based using statistical models, including training and decoding processes

33.2 Acoustic Parameterization and Modelling

Acoustic parameterization is concerned with the choice and optimization of acoustic features in order to reduce model complexity while trying to maintain the linguistic information relevant for speech recognition. Acoustic modelling must take into account different sources of variability present in the speech signal: those arising from the linguistic context and those associated with the non-linguistic context, such as the speaker (e.g. gender, age, emotional state, human non-speech sounds, etc.) and the acoustic environment (e.g. background noise, music) and recording channel (e.g. direct microphone, telephone). Most state-of-the-art systems make use of hidden Markov models (HMMs) for acoustic modelling, which consists of modelling the probability density function of a sequence of acoustic feature vectors. In this section, common parameterizations are described, followed by a discussion of acoustic model estimation and adaptation.

33.2.1 Acoustic Feature Analysis

The first step of the acoustic feature analysis is digitization, where the continuous speech signal is converted into discrete samples. The most commonly used sampling rates are 16 kHz and 10 kHz for direct microphone input and 8 kHz for telephone signals. The next step is feature extraction (also called parameterization or front-end analysis), which has the goal of representing the audio signal in a more compact manner by trying to remove redundancy and reduce variability, while keeping the important linguistic information ( Hunt 1996 ).

A widely accepted assumption is that although the speech signal is continually changing, due to physical constraints on the rate at which the articulators can move, the signal can be considered quasi-stationary for short periods (on the order of 10 ms to 20 ms). Therefore most recognition systems use short-time spectrum-related features based either on a Fourier transform or a linear prediction model. Among these features, cepstral parameters are popular because they are a compact representation, and are less correlated than direct spectral components. This simplifies estimation of the HMM parameters by reducing the need for modelling the feature dependency.

The two most popular sets of features are cepstrum coefficients obtained with a Mel Frequency Cepstral (MFC) analysis ( Davis and Mermelstein 1980 ) or with a Perceptual Linear Prediction (PLP) analysis ( Hermansky 1990 ). In both cases, a Mel scale short-term power spectrum is estimated on a fixed window (usually in the range of 20 to 30 ms). In order to avoid spurious high-frequency components in the spectrum due to discontinuities caused by windowing the signal, it is common to use a tapered window such as a Hamming window. The window is then shifted (usually a third or a half the window size) and the next feature vector computed. The most commonly used offset is 10 ms. The Mel scale approximates the frequency resolution of the human auditory system, being linear in the low-frequency range (below 1,000 Hz) and logarithmic above 1,000 Hz. The cepstral parameters are obtained by taking an inverse transform of the log of the filterbank parameters. In the case of the MFC coefficients, a cosine transform is applied to the log power spectrum, whereas a root Linear Predictive Coding (LPC) analysis is used to obtain the PLP cepstrum coefficients. Both sets of features have been used with success for large-vocabulary continuous speech recognition (LVCSR), but PLP analysis has been found for some systems to be more robust in the presence of background noise. The set of cepstral coefficients associated with a windowed portion of the signal is referred to as a frame or a parameter vector . Cepstral mean removal (subtraction of the mean from all input frames) is commonly used to reduce the dependency on the acoustic recording conditions. Computing the cepstral mean requires that all of the signal is available prior to processing, which is not the case for certain applications where processing needs to be synchronous with recording. In this case, a modified form of cepstral subtraction can be carried out where a running mean is computed from the N last frames (N is often on the order of 100, corresponding to 1 s of speech). In order to capture the dynamic nature of the speech signal, it is common to augment the feature vector with ‘delta’ parameters. The delta parameters are computed by taking the first and second differences of the parameters in successive frames. Over the last decade there has been a growing interest in capturing longer-term dynamics of speech than of the standard cepstral features. A variety of techniques have been proposed from simple concatenation of sequential frames to the use of TempoRAl Patterns (TRAPs) ( Hermansky and Sharma 1998 ). In all cases the wider context results in a larger number of parameters that consequently need to be reduced. Discriminative classifiers such as Multi-Layer Perceptrons (MLPs), a type of neural network, are efficient methods for discriminative feature estimation. Over the years, several groups have developed mature techniques for extracting probabilistic MLP features and incorporating them in speech-to-text systems ( Zhu et al. 2005 ; Stolcke et al. 2006 ). While probabilistic features have not been shown to consistently outperform cepstral features in LVCSR, being complementary they have been shown to significantly improve performance when used together ( Fousek et al. 2008 ).

33.2.2 Acoustic Models

Hidden Markov models are widely used to model the sequences of acoustic feature vectors ( Rabiner and Juang 1986 ). These models are popular as they are per-formant and their parameters can be efficiently estimated using well-established techniques. They are used to model the production of speech feature vectors in two steps. First, a Markov chain is used to generate a sequence of states, and then speech vectors are drawn using a probability density function (PDF) associated with each state. The Markov chain is described by the number of states and the transitions probabilities between states.

The most widely used elementary acoustic units in LVCSR systems are phone-based where each phone is represented by a Markov chain with a small number of states, where phones usually correspond to phonemes. Phone-based models offer the advantage that recognition lexicons can be described using the elementary units of the given language, and thus benefit from many linguistic studies. It is of course possible to perform speech recognition without using a phonemic lexicon, either by use of ‘word models’ (as was the more commonly used approach 20 years ago) or a different mapping such as the fenones ( Bahl et al. 1988 ). Compared with larger units (such as words, syllables, demisyllables), small subword units reduce the number of parameters, enable cross-word modelling, facilitate porting to new vocabularies, and most importantly, can be associated with back-off mechanisms to model rare contexts. Fenones offer the additional advantage of automatic training, but lack the ability to include a priori linguistic models. For some languages, most notably tonal languages such as Chinese, longer units corresponding to syllables or demisyllables (also called onsets and offsets or initials and finals) have been explored. While the use of larger units remains relatively limited to phone units, they may better capture tone information and may be well suited to casual speaking styles.

While different topologies have been proposed, all make use of left-to-right state sequences in order to capture the spectral change across time. The most commonly used configurations have between three and five emitting states per model, where the number of states imposes a minimal time duration for the unit. Some configurations allow certain states to be skipped, so as to reduce the required minimal duration. The probability of an observation (i.e. a speech vector) is assumed to be dependent only on the state, which is known as a first-order Markov assumption.

Strictly speaking, given an n-state HMM with parameter vector λ, the HMM stochastic process is described by the following joint probability density function f ( x ,   s | λ ) of the observed signal x   =   ( x 1 ,     … , x T ) and the unobserved state sequence s   =   ( s 0 ,     … ,   s T ) ⁠ ,

where π i is the initial probability of state i , a ij is the transition probability from state i to state j , and f(|s) is the emitting PDF associated with each state s. Figure 33.2 shows a three-state HMM with the associated transition probabilities and observation PDFs.

A typical three-state phone HMM with no skip state (top) which generates feature vectors ( x 1 … x n ) representing speech segments

A given HMM can represent a phone without consideration of its neighbours (context-independent or monophone model) or a phone in a particular context (context-dependent model). The context may or may not include the position of the phone within the word (word-position dependent), and word-internal and cross-word contexts may be merged or considered separated models. The use of cross-word contexts complicates decoding (see section 33.5 ). Different approaches are used to select the contextual units based on frequency or using clustering techniques, or decision trees, and different context types have been investigated: single-phone contexts, triphones, generalized triphones, quadphones and quinphones, with and without position dependency (within-word or cross-word). The model states are often clustered so as to reduce the model size, resulting in what are referred to as ‘tied-state’ models.

Acoustic model training consists of estimating the parameters of each HMM. For continuous density Gaussian mixture HMMs, this requires estimating the means and covariance matrices, the mixture weights and the transition probabilities. The most popular approaches make use of the Maximum Likelihood (ML) criterion, ensuring the best match between the model and the training data (assuming that the size of the training data is sufficient to provide robust estimates).

Estimation of the model parameters is usually done with the Expectation Maximization (EM) algorithm ( Dempster et al. 1977 ) which is an iterative procedure starting with an initial set of model parameters. The model states are then aligned to the training data sequences and the parameters are re-estimated based on this new alignment using the Baum–Welch re-estimation formulas ( Baum et al. 1970 ; Liporace 1982 ; Juang 1985 ). This algorithm guarantees that the likelihood of the training data given the model’s increases at each iteration. In the alignment step a given speech frame can be assigned to multiple states (with probabilities summing to 1) using the forward-backward algorithm or to a single state (with probability 1) using the Viterbi algorithm. This second approach yields a slightly lower likelihood but in practice there is very little difference in accuracy especially when large amounts of data are available. It is important to note that the EM algorithm does not guarantee finding the true ML parameter values, and even when the true ML estimates are obtained they may not be the best ones for speech recognition. Therefore, some implementation details such as a proper initialization procedure and the use of constraints on the parameter values can be quite important.

Since the goal of training is to find the best model to account for the observed data, the performance of the recognizer is critically dependent upon the representativity of the training data. Some methods to reduce this dependency are discussed in the next subsection. Speaker independence is obtained by estimating the parameters of the acoustic models on large speech corpora containing data from a large speaker population. There are substantial differences in speech from male and female talkers arising from anatomical differences (on average females have a shorter vocal tract length resulting in higher formant frequencies, as well as a higher fundamental frequency) and social ones (female voice is often ‘breathier’ caused by incomplete closure of the vocal folds). It is thus common practice to use separate models for male and female speech in order to improve recognition performance, which requires automatic identification of the gender.

Previously only used for small-vocabulary tasks ( Bahl et al. 1986 ), discriminative training of acoustic models for large-vocabulary speech recognition using Gaussian mixture hidden Markov models was introduced in Povey and Woodland (2000) . Different criteria have been proposed, such as maximum mutual information estimation (MMIE), criterion minimum classification error (MCE), minimum word error (MWE), and minimum phone error (MPE). Such methods can be combined with the model adaptation techniques described in the next section.

33.2.3 Adaptation

The performances of speech recognizers drop substantially when there is a mismatch between training and testing conditions. Several approaches can be used to minimize the effects of such a mismatch, so as to obtain a recognition accuracy as close as possible to that obtained under matched conditions. Acoustic model adaptation can be used to compensate for mismatches between the training and testing conditions, such as differences in acoustic environment, microphones and transmission channels, or particular speaker characteristics. The techniques are commonly referred to as noise compensation, channel adaptation, and speaker adaptation, respectively. Since in general no prior knowledge of the channel type, the background noise characteristics, or the speaker is available, adaptation is performed using only the test data in an unsupervised mode.

The same tools can be used in acoustic model training in order to compensate for sparse data, as in many cases only limited representative data are available. The basic idea is to use a small amount of representative data to adapt models trained on other large sources of data. Some typical uses are to build gender-specific, speaker-specific, or task-specific models, and to use speaker adaptive training (SAT) to improve performance. When used for model adaption during training, it is common to use the true transcription of the data, known as supervised adaptation.

Three commonly used schemes to adapt the parameters of an HMM can be distinguished: Bayesian adaptation ( Gauvain and Lee 1994 ); adaptation based on linear transformations ( Leggetter and Woodland 1995 ); and model composition techniques ( Gales and Young 1995 ). Bayesian estimation can be seen as a way to incorporate prior knowledge into the training procedure by adding probabilistic constraints on the model parameters. The HMM parameters are still estimated with the EM algorithm but using maximum a posteriori (MAP) re-estimation formulas ( Gauvain and Lee 1994 ). This leads to the so-called MAP adaptation technique where constraints on the HMM parameters are estimated based on parameters of an existing model. Speaker-independent acoustic models can serve as seed models for gender adaptation using the gender-specific data. MAP adaptation can be used to adapt to any desired condition for which sufficient labelled training data are available. Linear transforms are powerful tools to perform unsupervised speaker and environmental adaptation. Usually these transformations are ML-trained and are applied to the HMM Gaussian means, but can also be applied to the Gaussian variance parameters. This ML linear regression (MLLR) technique is very appropriate to unsupervised adaptation because the number of adaptation parameters can be very small. MLLR adaptation can be applied to both the test data and training data. Model composition is mostly used to compensate for additive noise by explicitly modelling the background noise (usually with a single Gaussian) and combining this model with the clean speech model. This approach has the advantage of directly modelling the noisy channel as opposed to the blind adaptation performed by the MLLR technique when applied to the same problem.

The chosen adaptation method depends on the type of mismatch and on the amount of available adaptation data. The adaptation data may be part of the training data, as in adaptation of acoustic seed models to a new corpus or a subset of the training material (specific to gender, dialect, speaker, or acoustic condition) or can be the test data (i.e. the data to be transcribed). In the former case, supervised adaptation techniques can be applied, as the reference transcription of the adaptation data can be readily available. In the latter case, only unsupervised adaptation techniques can be applied.

33.2.4 Deep Neural Networks

In addition to using MLPs for feature extraction, neural networks (NNs) can also be used to estimate the HMM state likelihoods in place of using Gaussian mixtures. This approach relying on very large MLPs (the so-called deep neural networks or DNNs) has been very successful in recent years, leading to some significant reduction of the error rates ( Hinton et al. 2012 ). In this case, the neural network outputs correspond to the states of the acoustic model and they are used to predict the state posterior probabilities. The NN output probabilities are divided by the state prior probabilities to get likelihoods that can be used to replace the GMM likelihoods. Given the large number of context-dependent HMM states used in state-of-the-art systems, the number of targets can be over 10,000, which leads to an MLP with more than 10 million weights.

33.3 Lexical and Pronunciation Modelling

The lexicon is the link between the acoustic-level representation and the word sequence output by the speech recognizer. Lexical design entails two main parts: definition and selection of the vocabulary items and representation of each pronunciation entry using the basic acoustic units of the recognizer. Recognition performance is obviously related to lexical coverage, and the accuracy of the acoustic models is linked to the consistency of the pronunciations associated with each lexical entry.

The recognition vocabulary is usually selected to maximize lexical coverage for a given size lexicon. Since on average, each out-of-vocabulary (OOV) word causes more than a single error (usually between 1.5 and two errors), it is important to judiciously select the recognition vocabulary. Word list selection is discussed in section 33.4 . Associated with each lexical entry are one or more pronunciations, described using the chosen elementary units (usually phonemes or phone-like units). This set of units is evidently language-dependent. For example, some commonly used phone set sizes are about 45 for English, 49 for German, 35 for French, and 26 for Spanish. In generating pronunciation baseforms, most lexicons include standard pronunciations and do not explicitly represent allophones. This representation is chosen as most allophonic variants can be predicted by rules, and their use is optional. More importantly, there is often a continuum between different allophones of a given phoneme and the decision as to which occurred in any given utterance is subjective. By using a phonemic representation, no hard decision is imposed, and it is left to the acoustic models to represent the observed variants in the training data. While pronunciation lexicons are usually (at least partially) created manually, several approaches to automatically learn and generate word pronunciations have been investigated ( Cohen 1989 ; Riley and Ljojle 1996 ).

There are a variety of words for which frequent alternative pronunciation variants are observed that are not allophonic differences. An example is the suffixization which can be pronounced with a diphthong (/ a i /) or a schwa (/ə/). Alternate pronunciations are also needed for homographs (words spelled the same, but pronounced differently) which reflect different parts of speech (verb or noun) such as excuse, record, produce . Some common three-syllable words such as interest and company are often pronounced with only two syllables. Figure 33.3 shows two examples of the word interest by different speakers reading the same text prompt: ‘ In reaction to the news, interest rates plunged … ’. The pronunciations are those chosen by the recognizer during segmentation using forced alignment. In the example on the left, the /t/ is deleted, and the /n/ is produced as a nasal flap. In the example on the right, the speaker said the word with two syllables, the second starting with a /tr/ cluster. Segmenting the training data without pronunciation variants is illustrated in the middle. Whereas no /t/ is observed in the first example, two /t/ segments were aligned. An optimal alignment with a pronunciation dictionary including all required variants is shown on the bottom. Better alignment results in more accurate acoustic phone models. Careful lexical design improves speech recognition performance.

In speech from fast speakers or speakers with relaxed speaking styles it is common to observe poorly articulated (or skipped) unstressed syllables, particularly in long words with sequences of unstressed syllables. Although such long words are typically well recognized, often a nearby function word is deleted. To reduce these kinds of errors, alternate pronunciations for long words such as positioning (/pǝzIʃǝnɨŋ/ or /pǝzIʃnɨŋ/), can be included in the lexicon allowing schwa deletion or syllabic consonants in unstressed syllables. Compound words have also been used as a way to represent reduced forms for common word sequences such as ‘did you’ pronounced as ‘dija’ or ‘going to’ pronounced as ‘gonna’. Alternatively, such fluent speech effects can be modelled using phonological rules ( Oshika et al. 1975 ). The principle behind the phonological rules is to modify the allowable phone sequences to take into account such variations. These rules are optionally applied during training and recognition. Using phonological rules during training results in better acoustic models, as they are less ‘polluted’ by wrong transcriptions. Their use during recognition reduces the number of mismatches. The same mechanism has been used to handle liaisons, mute-e, and final consonant cluster reduction for French. Most of today’s state-of-the-art systems include pronunciation variants in the dictionary, associating pronunciation probabilities with the variants ( Bourlard et al. 1999 ; Fosler-Lussier et al. 2005 ).

Spectrograms of the word interest with pronunciation variants: /InɝIs/ (left) and /IntrIs/ (right) taken from the WSJ corpus (sentences 20tc0106, 40lc0206). The grid is 100 ms by 1 kHz. Segmentation of these utterances with a single pronunciation of interest /IntrIst/ (middle) and with multiple variants /IntrIst/ /IntrIs/ /InɝIs/ (bottom).

As speech recognition research has moved from read speech to spontaneous and conversational speech styles, the phone set has been expanded to include non-speech events. These can correspond to noises produced by the speaker (breath noise, coughing, sneezing, laughter, etc.) or can correspond to external sources (music, motor, tapping, etc.). There has also been growing interest in exploring multilingual modelling at the acoustic level, with IPA or Unicode representations of the underlying units (see Gales et al. 2015 ; Dalmia et al. 2018 ).

33.4 Language Modelling

Language models (LMs) are used in speech recognition to estimate the probability of word sequences. Grammatical constraints can be described using a context-free grammar (for small to medium-size vocabulary tasks these are usually manually elaborated) or can be modelled stochastically, as is common for LVCSR. The most popular statistical methods are n -gram models, which attempt to capture the syntactic and semantic constraints by estimating the frequencies of sequences of n words. The assumption is made that the probability of a given word string ( w 1 , w 2 ,   … , w k ) can be approximated by Π i = 1 k Pr ( w i | w i − n + 1 ,     ​ ​ … , w i − 2 , w i − 1 ) ⁠ , therefore reducing the word history to the preceding n − 1 words.

A back-off mechanism is generally used to smooth the estimates of the probabilities of rare n -grams by relying on a lower-order n -gram when there is insufficient training data, and to provide a means of modelling unobserved word sequences ( Katz 1987 ). For example, if there are not enough observations for a reliable ML estimate of a 3-gram probability, it is approximated as follows: P r ( w i | w i − 1 , w i − 2 ) ≃ P r ( w i | w i − 1 ) B ( w i − 1 , w i − 2 ) ⁠ , where B ( w i , w i − 1 ) is a back-off coefficient needed to ensure that the total probability mass is still 1 for a given context. Based on this equation, many methods have been proposed to implement this smoothing.

While trigram LMs are the most widely used, higher-order (n>3) and word-class-based (counts are based on sets of words rather than individual lexical items) n -grams and adapted LMs are recent research areas aiming to improve LM accuracy. Neural network language models have been used to address the data sparseness problem by performing the estimation in a continuous space ( Bengio et al. 2001 ).

Given a large text corpus it may seem relatively straightforward to construct n -gram language models. Most of the steps are pretty standard and make use of tools that count word and word sequence occurrences. The main differences arise in the choice of the vocabulary and in the definition of words, such as the treatment of compound words or acronyms, and the choice of the back-off strategy. There is, however, a significant amount of effort needed to process the texts before they can be used.

A common motivation for normalization in all languages is to reduce lexical variability so as to increase the coverage for a fixed-size-task vocabulary. Normalization decisions are generally language-specific. Much of the speech recognition research for American English has been supported by ARPA and has been based on text materials which were processed to remove upper/lower-case distinction and compounds. Thus, for instance, no lexical distinction is made between Gates, gates or Green, green . In the French Le Monde corpus, capitalization of proper names is distinctive with different lexical entries for Pierre, pierre or Roman, roman .

The main conditioning steps are text mark-up and conversion. Text mark-up consists of tagging the texts (article, paragraph, and sentence markers) and garbage bracketing (which includes not only corrupted text materials, but all text material unsuitable for sentence-based language modelling, such as tables and lists). Numerical expressions are typically expanded to approximate the spoken form ($150 → one hundred and fifty dollars). Further semi-automatic processing is necessary to correct frequent errors inherent in the texts (such as obvious mispellings milllion, officals ) or arising from processing with the distributed text processing tools. Some normalizations can be considered as ‘decompounding’ rules in that they modify the word boundaries and the total number of words. These concern the processing of ambiguous punctuation markers (such as hyphen and apostrophe), the processing of digit strings, and treatment of abbreviations and acronyms (ABCD → A. B. C. D.). Other normalizations (such as sentence-initial capitalization and case distinction) keep the total number of words unchanged, but reduce graphemic variability. In general, the choice is a compromise between producing an output close to correct standard written form of the language and lexical coverage, with the final choice of normalization being largely application-driven.

Better language models can be obtained using texts transformed to be closer to the observed reading style, where the transformation rules and corresponding probabilities are automatically derived by aligning prompt texts with the transcriptions of the acoustic data. For example, the word hundred followed by a number can be replaced by hundred and 50% of the time; 50% of the occurences of one eighth are replaced by an eighth , and 15% of million dollars are replaced with simply million .

In practice, the selection of words is done so as to minimize the system’s OOV rate by including the most useful words. By useful we mean that the words are expected as an input to the recognizer, but also that the LM can be trained given the available text corpora. In order to meet the latter condition, it is common to choose the N most frequent words in the training data. This criterion does not, however, guarantee the usefulness of the lexicon, since no consideration of the expected input is made. Therefore, it is common practice to use a set of additional development data to select a word list adapted to the expected test conditions.

There is sometimes the conflicting need for sufficient amounts of text data to estimate LM parameters and assuring that the data is representative of the task. It is also common that different types of LM training material are available in differing quantities. One easy way to combine training material from different sources is to train a language model for each source and to interpolate them. The interpolation weights can be directly estimated on some development data with the EM algorithm. An alternative is to simply merge the n -gram counts and train a single language model on these counts. If some data sources are more representative than others for the task, the n -gram counts can be empirically weighted to minimize the perplexity on a set of development data. While this can be effective, it has to be done by trial and error and cannot easily be optimized. In addition, weighting the n -gram counts can pose problems in properly estimating the back-off coefficients. For these reasons, the language models in most of today’s state-of-the-art systems are obtained via the interpolation methods, which can also allow for task adaptation by simply modifying the interpolation coefficients ( Chen et al. 2004 ; Liu et al. 2008 ).

The relevance of a language model is usually measured in terms of test set perplexity defined as Px = Pr ( text | LM ) − 1 n ⁠ , where n is the number of words in the text. The perplexity is a measure of the average branching factor, i.e. the vocabulary size of a memoryless uniform language model with the same entropy as the language model under consideration.

33.5 Decoding

In this section we discuss the LVCSR decoding problem, which is the design of an efficient search algorithm to deal with the huge search space obtained by combining the acoustic and language models. Strictly speaking, the aim of the decoder is to determine the word sequence with the highest likelihood, given the lexicon and the acoustic and language models. In practice, however, it is common to search for the most likely HMM state sequence, i.e. the best path through a trellis (the search space) where each node associates an HMM state with given time. Since it is often prohibitive to exhaustively search for the best path, techniques have been developed to reduce the computational load by limiting the search to a small part of the search space. Even for research purposes, where real-time recognition is not needed, there is a limit on computing resources (memory and CPU time) above which the development process becomes too costly. The most commonly used approach for small and medium vocabulary sizes is the one-pass frame-synchronous Viterbi beam search ( Ney 1984 ) which uses a dynamic programming algorithm. This basic strategy has been extended to deal with large vocabularies by adding features such as dynamic decoding, multi-pass search, and N-best rescoring.

Dynamic decoding can be combined with efficient pruning techniques in order to obtain a single-pass decoder that can provide the answer using all the available information (i.e. that in the models) in a single forward decoding pass over the speech signal. This kind of decoder is very attractive for real-time applications. Multi-pass decoding is used to progressively add knowledge sources in the decoding process and allows the complexity of the individual decoding passes to be reduced. For example, a first decoding pass can use a 2-gram language model and simple acoustic models, and later passes will make use of 3-gram and 4-gram language models with more complex acoustic models. This multiple-pass paradigm requires a proper interface between passes in order to avoid losing information and engendering search errors. Information is usually transmitted via word graphs, although some systems use N-best hypotheses (a list of the most likely word sequences with their respective scores). This approach is not well suited to real-time applications since no hypothesis can be returned until the entire utterance has been processed.

It can sometimes be difficult to add certain knowledge sources into the decoding process especially when they do not fit in the Markovian framework (i.e. short-distance dependency modelling). For example, this is the case when trying to use segmental information or to use grammatical information for long-term agreement. Such information can be more easily integrated in multi-pass systems by rescoring the recognizer hypotheses after applying the additional knowledge sources.

Mangu, Brill, and Stolcke (2000) proposed the technigue of confusion network decoding (also called consensus decoding) which minimizes an approximate WER, as opposed to MAP decoding which minimizes the sentence error rate (SER). This technique has since been adopted in most state-of-the-art systems, resulting in lower WERs and better confidence scores. Confidence scores are a measure of the reliability of the recognition hypotheses, and give an estimate of the word error rate (WER). For example, an average confidence of 0.9 will correspond to a word error rate of 10% if deletions are ignored. Jiang (2004) provides an overview of confidence measures for speech recognition, commenting on the capacity and limitations of the techniques.

33.6 State-of-the-Art Performance

The last decade has seen large performance improvements in speech recognition, particularly for large-vocabulary, speaker-independent, continuous speech. This progress has been substantially aided by the availability of large speech and text corpora and by significant increases in computer processing capabilities which have facilitated the implementation of more complex models and algorithms. 1 In this section we provide some illustrative results for different LVCSR tasks, but make no attempt to be exhaustive.

The commonly used metric for speech recognition performance is the ‘word error’ rate, which is a measure of the average number of errors taking into account three error types with respect to a reference transcription: substitutions (one word is replaced by another word), insertions (a word is hypothesized that was not in the reference), and deletions (a word is missed). The word error rate is defined as #subs+#ins+#del #reference   words ⁠ , and is typically computed after a dynamic programming alignment of the reference and hypothesized transcriptions. Note that given this definition the word error can be more than 100%.

Three types of tasks can be considered: small-vocabulary tasks, such as isolated command words, digits or digit strings; medium-size (1,000–3,000-word) vocabulary tasks such as are typically found in spoken dialogue systems (Chapter 44 ); and large-vocabulary tasks (typically over 100,000 words). Another dimension is the speaking style which can be read, prepared, spontaneous, or conversational. Very low error rates have been reported for small-vocabulary tasks, below 1% for digit strings, which has led to some commercial products, most notably in the telecommunications domain. Early benchmark evaluations focused on read speech tasks: the state of the art in speaker-independent, continuous speech recognition in 1992 is exemplified by the Resource Management task (1,000-word vocabulary, word-pair grammar, four hours of acoustic training data) with a word error rate of 3%. In 1995, on read newspaper texts (the Wall Street Journal task, 160 hours of acoustic training data and 400 million words of language model texts) word error rates around 8% were obtained using a 65,000-word vocabulary. The word errors roughly doubled for speech in the presence of noise, or on texts dictated by journalists. The maturity of the technology led to the commercialization of speaker-dependent continuous speech dictation systems for which comparable benchmarks are not publicly available.

Over the last decade the research has focused on ‘found speech’, originating with the transcription of radio and television broadcasts and moving to any audio found on the Internet (podcasts). This was a major step for the community in that the test data is taken from a real task, as opposed to consisting of data recorded for evaluation purposes. The transcription of such varied data presents new challenges as the signal is one continuous audio stream that contains segments of different acoustic and linguistic natures. Today well-trained transcription systems for broadcast data have been developed for at least 25 languages, achieving word error rates on the order of under 20% on unrestricted broadcast news data. The performance on studio-quality speech from announcers is often comparable to that obtained on WSJ read speech data.

Word error rates of under 20% have been reported for the transcription of conversational telephone speech (CTS) in English using the Switchboard corpus, with substantially higher WERs (30–40%) on the multiple language Callhome (Spanish, Arabic, Mandarin, Japanese, German) data and on data from the IARPA Babel Program (< http://www.iarpa.gov/index.php/research-programs/babel >; Sainath et al. 2013 ). A wide range of word error rates have been reported for the speech recognition components of spoken dialogue systems (Chapters 8 , 44 , and 45 ), ranging from under 5% for simple travel information tasks using close-talking microphones to over 25% for telephone-based information retrieval systems. It is quite difficult to compare results across systems and tasks as different transcription conventions and text normalizations are often used.

Speech-to-text (STT) systems historically produce a case-insensitive, unpunctuated output. Recently there have been a number of efforts to produce STT outputs with correct case and punctuation, as well as conversion of numbers, dates, and acronymns to a standard written form. This is essentially the reverse process of the text normalization steps described in section 33.4 . Both linguistic and acoustic information (essentially pause and breath noise cues) are used to add punctuation marks in the speech recognizer output. An efficient method is to rescore word lattices that have been expanded to permit punctuation marks after each word, sentences boundaries at each pause, with a specialized case-sensitive, punctuated language model.

33.7 Discussion and Perspectives

Despite the numerous advances made over the last decade, speech recognition is far from a solved problem. Current research topics aim to develop generic recognition models with increased use of data perturbation and augmentation techniques for both acoustic and language modelling ( Ko et al. 2015 ; Huang et al. 2017 ; Park et al. 2019 ) and to use unannotated data for training purposes, in an effort to reduce the reliance on manually annotated training corpora. There has also been growing interest in End-to-End neural network models for speech recognition (< http://iscslp2018.org/Tutorials.html >, as well as tutorials at Interspeech 2019–2021, some of which also describe freely available toolkits) which aim to simultaneously train all automatic speech recognition (ASR) components optimizing the targeted evaluation metric (usually the WER), as opposed to the more traditional training described in this chapter.

Much of the progress in LVCSR has been fostered by supporting infrastructure for data collection, annotation, and evaluation. The Speech Group at the National Institute of Standards and Technology (NIST) has been organizing benchmark evaluations for a range of human language technologies (speech recognition, speaker and language recognition, spoken document retrieval, topic detection and tracking, automatic content extraction, spoken term detection) for over 20 years, recently extended to also include related multimodal technologies. 2 In recent years there has been a growing number of challenges and evaluations, often held in conjunction with major conferences, to promote research on a variety of topics. These challenges typically provide common training and testing data sets allowing different methods to be compared on a common basis.

While the performance of speech recognition technology has dramatically improved for a number of ‘dominant’ languages (English, Mandarin, Arabic, French, Spanish … ), generally speaking technologies for language and speech processing are available only for a small proportion of the world’s languages. By several estimations there are over 7,000 spoken languages in the world, but only about 15% of them are also written. Text corpora, which can be useful for training the language models used by speech recognizers, are becoming more and more readily available on the Internet. The site < http://www.omniglot.com > lists about 800 languages that have a written form.

It has often been observed that there is a large difference in recognition performance for the same system between the best and worst speakers. Unsupervised adaption techniques do not necessarily reduce this difference—in fact, they often improve performance on good speakers more than on bad ones. Interspeaker differences are not only at the acoustic level, but also the phonological and word levels. Today’s modelling techniques are not able to take into account speaker-specific lexical and phonological choices.

Today’s systems often also provide additional information which is useful for structuring audio data. In addition to the linguistic message, the speech signal encodes information about the characteristics of the speaker, the acoustic environment, the recording conditions, and the transmission channel. Acoustic meta-data can be extracted from the audio to provide a description, including the language(s) spoken, the speaker’s (or speakers’) accent(s), acoustic background conditions, the speaker’s emotional state, etc. Such information can be used to improve speech recognition performance, and to provide an enriched text output for downstream processing. The automatic transcription can also be used to provide information about the linguistic content of the data (topic, named entities, speech style … ). By associating each word and sentence with a specific audio segment, an automatic transcription can allow access to any arbitrary portion of an audio document. If combined with other meta-data (language, speaker, entities, topics), access via other attributes can be facilitated.

A wide range of potential applications can be envisioned based on automatic annotation of broadcast data, particularly in light of the recent explosion of such media, which required automated processing for indexation and retrieval (Chapters 37 , 38 , and 40 ), machine translation (Chapters 35 and 36 ), and question answering (Chapter 39 ). Important future research will address keeping vocabulary up-to-date, language model adaptation, automatic topic detection and labelling, and enriched transcriptions providing annotations for speaker turns, language, acoustic conditions, etc. Another challenging problem is recognizing spontaneous speech data collected with far-field microphones (such as meetings and interviews), which have difficult acoustic conditions (reverberation, background noise) and often have overlapping speech from different speakers.

Further Reading and Relevant Resources

An excellent reference is Corpus Based Methods in Language and Speech Processing , edited by Young and Bloothooft (1997) . This book provides an overview of currently used statistically based techniques, their basic principles and problems. A theoretical presentation of the fundamentals of the subject is given in the book Statistical Methods for Speech Recognition by Jelinek (1997) . A general introductory tutorial on HMMs can be found in Rabiner (1989) . Pattern Recognition in Speech and Language Processing by Chou and Juang (2003) , Spoken Language Processing: A Guide to Theory, Algorithm, and System Development by Huang, Acero, and Hon (2001) , and Multilingual Speech Processing by Schultz and Kirchhoff (2006) provide more advanced reading. Two recent books, The Voice in the Machine: Building Computers That Understand Speech by Roberto Pieraccini (2012) which targets general audiences and Automatic Speech Recognition: A Deep Learning Approach by Dong Yu and Li Deng (2015) provide an overview of the recent advance in the field. For general speech processing reference, the classical book Digital Processing of Speech Signals ( Rabiner and Shafer 1978 ) remains relevant. The most recent work in speech recognition can be found in the proceedings of major conferences (IEEE ICASSP, ISCA Interspeech) and workshops (most notably DARPA/IARPA, ISCA ITRWs, IEEE ASRU, SLT), as well as the journals Speech Communication and Computer Speech and Language .

Several websites of interest are:

European Language Resources Association (ELRA), < http://www.elda.fr/en/ >.

International Speech Communication Association (ISCA) < http://www.isca-speech.org >.

Linguistic Data Consortium (LDC), < http://www.ldc.upenn.edu/ >.

NIST Spoken Natural-Language Processing, < http://www.itl.nist.gov/iad/mig/tests >.

Survey of the State of the Art in Human Language Technology, < http://www.cslu.ogi.edu/HLTsurvey >.

Languages of the world, < http://www.omniglot.com >.

OLAC: Open Language Archives Community, < http://www.language-archives.org >.

Speech recognition software, < http://en.wikipedia.org/wiki/List_of_speech_recognition_software >.

Bahl, Lalit , James Baker , Paul Cohen , N. Rex Dixon , Frederick Jelinek , Robert Mercer , and Harvey Silverman (1976). ‘Preliminary Results on the Performance of a System for the Automatic Recognition of Continuous Speech’. In Proceedings of the IEEE Conference on Acoustics Speech and Signal Processing (ICASSP-76) , Philadelphia, 425–429. IEEE.

Bahl, Lalit , Peter Brown , Peter de Souza , and Robert Mercer (1986). ‘Maximum Mutual Information Estimation of Hidden Markov Model Parameters for Speech Recognition’. In Proceedings of the IEEE Conference on Acoustics Speech and Signal Processing (ICASSP-86) , Tokyo, 49–52. IEEE Press.

Bahl, Lalit , Peter Brown , Peter de Souza , Robert Mercer , and Michael Picheny (1988). ‘Acoustic Markov Models Used in the Tangora Speech Recognition System’. In Proceedings of the IEEE Conference on Acoustics Speech and Signal Processing (ICASSP-88) , New York, 497–500. IEEE Press.

Baum, Leonard , Ted Petrie , Georges Soules , and Norman Weiss ( 1970 ). ‘ A Maximization Technique Occurring in the Statistical Analysis of Probabilistic Functions of Markov Chains ’, Annals of Mathematical Statistics 41: 164–171.

Google Scholar

Bengio, Yoshua , Réjean Ducharme , and Pascal Vincent ( 2001 ). ‘A Neural Probabilistic Language Model’. In T. Leen , T. Dietterich , and V. Tresp (eds), Advances in Neural Information Processing Systems 13 (NIPS ‘00) , 932–938. Cambridge, MA: MIT Press.

Google Preview

Bourlard, Hervé , Sadaoki Furui , Nelson Morgan , and Helmer Strik (eds) ( 1999 ). Special issue on ‘Modeling Pronunciation Variation for Automatic Speech Recognition ’, Speech Communication 29(2–4), November.

Chen, Langzhou , Jean-Luc Gauvain , Lori Lamel , and Gilles Adda (2004). ‘Dynamic Language Modeling for Broadcast News’. In 8th International Conference on Spoken Language Processing, ICSLP-04 , Jeju Island, Korea, 1281–1284. ISCA (International Speech Communication Association).

Chou, Wu and Juang, Biing-Hwang ( 2003 ). Pattern Recognition in Speech and Language Processing . CRC Press.

Cohen, Michael (1989). ‘Phonological Structures for Speech Recognition’. PhD thesis, University of California, Berkeley.

Dalmia, Siddharth , Ramon Sanabria , Florian Metze , and Alan W. Black (2018). ‘Sequence-based Multi-lingual Low Resource Speech Recognition’. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 4909–4913. IEEE.

Davis, Steven and Paul Mermelstein ( 1980 ). ‘ Comparison of Parametric Representations of Monosyllabic Word Recognition in Continuously Spoken Sentences ’, IEEE Transactions on Acoustics, Speech, and Signal Processing 28(4): 357–366.

Dempster, Arthur , Nan Laird , and Donald Rubin ( 1977 ). ‘ Maximum Likelihood from Incomplete Data via the EM Algorithm ’, Journal of the Royal Statistical Society Series B (Methodological) 39(1): 1–38.

Dreyfus-Graf, Jean ( 1949 ). ‘ Sonograph and Sound Mechanics ’, Journal of the Acoustic Society of America 22: 731–739.

Dudley, Homer and S. Balashek ( 1958 ). ‘ Automatic Recognition of Phonetic Patterns in Speech ’, Journal of the Acoustical Society of America 30: 721–732.

Fosler-Lussier, Eric , William Byrne , and Dan Jurafsky (eds) ( 2005 ). Special issue on ‘Pronunciation Modeling and Lexicon Adaptation ’, Speech Communication 46(2), June.

Fousek Petr , Lori Lamel , and Jean-Luc Gauvain ( 2008 ). ‘On the Use of MLP Features for Broadcast News Transcription’. In P. Sojka et al. (eds), TSD ’08 , Lecture Notes in Computer Science 5246, 303–310. Berlin and Heidelberg: Springer-Verlag.

Frederick Jelinek ( 1997 ). Statistical Methods for Speech Recognition. Language, Speech and Communication series . Cambridge, MA: MIT Press (Bradford Book).

Gales, Mark J. F. , Kate M. Knill , and Anton Ragni (2015). ‘Unicode-based Graphemic Systems for Limited Resource Languages’. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 5186–5190. IEEE, 2015.

Gales, Mark and Steven Young ( 1995 ). ‘ Robust Speech Recognition in Additive and Convolutional Noise Using Parallel Model Combination ’, Computer Speech and Language 9(4): 289–307.

Gauvain, Jean-Luc and Chin-Hui Lee ( 1994 ). ‘Maximum a posteriori Estimation for Multivariate Gaussian Mixture Observations of Markov Chains’, IEEE Transactions on Speech and Audio Processing 2(2): 291–298.

Hermansky, Hynek ( 1990 ) ‘ Perceptual Linear Predictive (PLP) Analysis of Speech ’, Journal of the Acoustic Society America 87(4): 1738–1752.

Hermansky, Hynek and Sangita Sharma ( 1998 ). ‘TRAPs—Classifiers of Temporal Patterns’. In Robert H. Mannell and Jordi Robert-Ribes (eds), 5th International Conference on Spoken Language Processing, ICSLP ’98 , Sydney, 1003–1006. Australian Speech Science and Technology Association (ASSTA).

Hinton, Geoffrey , Li Deng , Dong Yu , George Dahl , Abdelrahman Mohamed , Navdeep Jaitly , Andrew Senior , Vincent Vanhoucke , Patrick Nguyen , Tara Sainath , and Brian Kingsbury ( 2012 ). ‘ Deep Neural Networks for Acoustic Modeling in Speech Recognition ’, IEEE Signal Processing Magazine 29(6): 82–97.

Huang, Xuedong , Acero, Alex and Hon, Hsiao-Wuen ( 2001 ). Spoken Language Processing: A Guide to Theory, Algorithm, and System Development . London: Prentice Hall.

Huang, Guangpu , Thiago Fraga Da Silva , Lori Lamel , Jean-Luc Gauvain , Arseniy Gorin , Antoine Laurent , Rasa Lileikyté , and Abdelkhalek Messaoudi ( 2017 ). ‘ An Investigation into Language Model Data Augmentation for Low-resourced STT and KWS ’. In Proceedings of the IEEE-ICASSP , 5790–5794. New Orleans.

Hunt, Melvin ( 1996 ). ‘Signal Representation’. In Ron Cole et al. (eds), Survey of the State of the Art in Human Language Technology , 10–15 (ch. 1.3). Cambridge: Cambridge University Press and Giardini, < http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.50.7794&rep=rep1&type=pdf >.

Jelinek, Frederik (1976). ‘ Continuous Speech Recognition by Statistical Methods ’, Proceed­ings of the IEEE: Special Issue on Man–Machine Communication by Voice 64(4), April: 532–556.

Jiang, Hui ( 2004 ). ‘ Confidence Measures for Speech Recognition: A Survey ’, Speech Communi­cation 45(4): 455–470.

Juang, Biing-Hwang ( 1985 ). ‘ Maximum-Likelihood Estimation for Mixture Multivariate Stochastic Observations of Markov Chains ’, AT&T Technical Journal 64(6), July–August: 1235–1249.

Katz, Slava ( 1987 ). ‘ Estimation of Probabilities from Sparse Data for the Language Model Component of a Speech Recognizer ’, IEEE Transactions on Acoustics, Speech, and Signal Processing 35(3): 400–401.

Ko, Tom , Vijayaditya Peddinti , Daniel Povey , and Sanjeev Khudanpur (2015). ‘Audio Augmentation for Speech Recognition’. In Sixteenth Annual Conference of the International Speech Communication Association .

Leggetter, Chris and Philip Woodland ( 1995 ). ‘ Maximum Likelihood Linear Regression for Speaker Adaptation of Continuous Density Hidden Markov Models ’, Computer Speech and Language 9: 171–185.

Liporace, Louis ( 1982 ). ‘ Maximum Likelihood Estimation for Multivariate Observations of Markov Sources ’, IEEE Transactions on Information Theory 28(5): 729–734.

Liu, Xunying , Mark Gales , and Philip Woodland (2008). ‘Context-Dependent Language Model Adaptation’. In Interspeech ’08: 9th Annual Conference of the International Speech Communication Association , Brisbane, 837–840. International Speech Communication Association.

Mangu, Lidia , Eric Brill , and Andreas Stolcke ( 2000 ). ‘ Finding Consensus in Speech Recognition: Word Error Minimization and Other Applications of Confusion Networks ’, Computer Speech and Language 144: 373–400.

Ney, Hermann (1984). ‘ The Use of a One-Stage Dynamic Programming Algorithm for Connected Word Recognition ’, IEEE Transactions on Acoustics, Speech, and Signal Processing 32(2), April: 263–271.

Oshika, Beatrice , Victor Zue , Rollin Weeks , Helene Neu , and Joseph Aurbach ( 1975 ). ‘ The Role of Phonological Rules in Speech Understanding Research ’, IEEE Transactions on Acoustics, Speech, and Signal Processing 23: 104–112.

Park, Daniel S. , William Chan , Yu Zhang , Chung-Cheng Chiu , Barret Zoph , Ekin D. Cubuk , and Quoc V. Le (2019). ‘Specaugment: A Simple Data Augmentation Method for Automatic Speech Recognition’. arXiv preprint arXiv:1904.08779.

Povey, Daniel and Philip Woodland (2000). ‘Large-Scale MMIE Training for Conversational Telephone Speech Recognition’, in Proceedings of the NIST Speech Transcription Workshop , College Park, MD.

Rabiner, Lawrence ( 1989 ). ‘ A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition ’, Proceedings of the IEEE 77(2), February: 257–286.

Rabiner, Lawrence and Biing-Hwang Juang ( 1986 ). ‘ An Introduction to Hidden Markov Models ’, IEEE Acoustics, Speech, and Signal Processing Magazine (ASSP) 3(1), January: 4–16.

Rabiner, Lawrence and Ronald Schafer ( 1978 ). Digital Processing of Speech Signals . Englewood Cliffs, NJ and London: Prentice-Hall.

Riley, Michael and Andrej Ljojle ( 1996 ). ‘Automatic Generation of Detailed Pronunciation Lexicons’. In Chin-Hui Lee , Frank K. Soong , and Kuldip K. Paliwal (eds), Automatic Speech and Speaker Recognition , 285–301. Dordrecht: Kluwer Academic Publishers.

Roberto Pieraccini ( 2012 ). The Voice in the Machine: Building Computers that Understand Speech . Cambridge, MA: MIT Press.

Sainath, Tara , Brian Kingsbury , Florian Metze , Nelson Morgan , and Stavros Tsakalidis ( 2013 ). ‘ An Overview of the Base Period of the Babel Program ’, SLTC Newsletter, November, < http://www.signalprocessingsociety.org/technical-committees/list/sl-tc/spl-nl/2013-11/BabelBaseOverview >.

Schultz, Tanja and Katrin Kirchhoff ( 2006 ). Multilingual Speech Processing . London: Academic Press.

Schwartz, Richard , Yen-Lu Chow , Salim Roucos , Michael Krasner , and John Makhoul (1984). ‘Improved Hidden Markov Modeling of Phonemes for Continuous Speech Recognition’. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP ’84) , San Diego, 35.6.1–35.6.4. IEEE Press.

Stolcke, Andreas , Barry Chen , Horacio Franco , Venkata Gadde , M. Graciarena , Mei-Yuh Hwang , Katrin Kirchhoff , Arindam Mandal , Nelson Morgan , Xin Lei , Tim Ng , Mari Ostendorf , Kemal Sonmez , Anand Venkataraman , Dimitra Vergyri , Wen Wang , Jing Zheng , and Qifeng Zhu ( 2006 ). ‘ Recent Innovations in Speech-to-Text Transcription at SRI-ICSI-UW ’, IEEE Transactions on Audio, Speech, and Language Processing 14(5): 1729–1744.

Vintsyuk, Taras ( 1968 ). ‘ Speech Discrimination by Dynamic Programming ’, Kibnernetika 4: 81–88.

Young, Steve and Gerrit Bloothooft (eds) ( 1997 ). Corpus-Based Methods in Language and Speech Processing . Text, Speech and Language Technology Series. Amsterdam: Springer Netherlands.

Yu, Dong and Li Deng ( 2015 ). Automatic Speech Recognition: A Deep Learning Approach . Amsterdam: Springer.

Zhu, Qifeng , Andreas Stolcke , Barry Chen , and Nelson Morgan (2005). ‘Using MLP Features in SRI’s Conversational Speech Recognition System’. In Interspeech ’05: Proceedings of the 9th European Conference on Speech Communication and Technology , Lisbon, 2141–2144.

These advances can be clearly seen in the context of DARPA-supported benchmark evaluations. This framework, known in the community as the DARPA evaluation paradigm, has provided the training materials (transcribed audio and textual corpora for training acoustic and language models), test data, and a common evaluation framework. The data have been generally provided by the Linguistics Data Consortium (LDC) and the evaluations organized by the National Institute of Standards and Technology (NIST) in collaboration with representatives from the participating sites and other government agencies.

See < http://www.nist.gov/speech/tests >.

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

What Is Speech Recognition?

Time to read: 4 minutes

• Facebook logo
• Twitter Logo Follow us on Twitter
• LinkedIn logo

The human voice allows people to express their thoughts, emotions, and ideas through sound. Speech separates us from computing technology, but both similarly rely on words to transform ideas into shared understanding. In the past, we interfaced with computers and applications only through keyboards, controllers, and consoles—all hardware. But today, speech recognition software bridges the gap that separates speech and text.

First, let’s start with the meaning of automatic speech recognition: it’s the process of converting what speakers say into written or electronic text. Potential business applications include everything from customer support to translation services.

Now that you understand what speech recognition is, read on to learn how speech recognition works, different speech recognition types, and how your business can benefit from speech recognition applications.

How does speech recognition work?

Speech recognition technologies capture the human voice with physical devices like receivers or microphones. The hardware digitizes recorded sound vibrations into electrical signals. Then, the software attempts to identify sounds and phonemes—the smallest unit of speech—from the signals and match these sounds to corresponding text. Depending on the application, this text displays on the screen or triggers a directive—like when you ask your smart speaker to play a specific song and it does.

Background noise, accents, slang, and cross talk can interfere with speech recognition, but advancements in artificial intelligence (AI) and machine learning technologies filter through these anomalies to increase precision and performance.

Thanks to new and emerging machine learning algorithms, speech recognition offers advanced capabilities:

• Natural language processing is a branch of computer science that uses AI to emulate how humans engage in and understand speech and text-based interactions.
• Hidden Markov Models (HMM) are statistical models that assign text labels to units of speech—like words, syllables, and sentences—in a sequence. Labels map to the provided input to determine the correct label or text sequence.
• N-grams are language models that assign probabilities to sentences or phrases to improve speech recognition accuracy. These contain sequences of words and use prior sequences of the same words to understand or predict new words and phrases. These calculations improve the predictions of sentence automatic completion systems, spell-check results, and even grammar checks.
• Neural networks consist of node layers that together emulate the learning and decision-making capabilities of the human brain. Nodes contain inputs, weights, a threshold, and an output value. Outputs that exceed the threshold activate the corresponding node and pass data to the next layer. This means remembering earlier words to continually improve recognition accuracy.
• Connectionist temporal classification is a neural network algorithm that uses probability to map text transcript labels to incoming audio. It helps train neural networks to understand speech and build out node networks.

Features of speech recognition

Not all speech recognition works the same. Implementations vary by application, but each uses AI to quickly process speech at a high—but not flawless—quality level. Many speech recognition technologies include the same features:

• Filtering identifies and censors—or removes—specified words or phrases to sanitize text outputs.
• Language weighting assigns more value to frequently spoken words—like proper nouns or industry jargon—to improve speech recognition precision.
• Speaker labeling distinguishes between multiple conversing speakers by identifying contributions based on vocal characteristics.
• Acoustics training analyzes conditions—like ambient noise and particular speaker styles—then tailors the speech recognition software to that environment. It’s useful when recording speech in busy locations, like call centers and offices.
• Voice recognition helps speech recognition software pivot the listening approach to each user’s accent, dialect, and grammatical library.

5 benefits of speech recognition technology

The popularity and convenience of speech recognition technology have made speech recognition a big part of everyday life. Adoption of this technology will only continue to spread, so learn more about how speech recognition transforms how we live and work:

• Speed: Speaking with your voice is faster than typing with your fingers—in most cases.
• Assistance: Listening to directions from users and taking action accordingly is possible thanks to speech recognition technology. For instance, if your vehicle’s sound system has speech recognition capabilities, you can tell it to tune the radio to a particular channel or map directions to a specified address.
• Productivity: Dictating your thoughts and ideas instead of typing them out, saves time and effort to redirect toward other tasks. To illustrate, picture yourself dictating a report into your smartphone while walking or driving to your next meeting.
• Intelligence: Learning from and adapting to your unique speech habits and environment to identify and understand you better over time is possible thanks to speech recognition applications.
• Accessibility: Entering text with speech recognition is possible for people with visual impairments who can’t see a keyboard thanks to this technology. Software and websites like Google Meet and YouTube can accommodate hearing-impaired viewers with text captions of live speech translated to the user’s specific language.

Business speech recognition use cases

Speech recognition directly connects products and services to customers. It powers interactive voice recognition software that delivers customers to the right support agents—each more productive with faster, hands-free communication. Along the way, speech recognition captures actionable insights from customer conversations you can use to bolster your organization’s operational and marketing processes.

Here are some real-world speech recognition contexts and applications:

• SMS/MMS messages: Write and send SMS or MMS messages conveniently in some environments.
• Chatbot discussions: Get answers to product or service-related questions any time of day or night with chatbots.
• Web browsing : Browse the internet without a mouse, keyboard, or touch screen through voice commands.
• Active learning: Enable students to enjoy interactive learning applications—such as those that teach a new language—while teachers create lesson plans.
• Document writing: Draft a Google or Word document when you can't access a physical or digital keyboard with speech-to-text. You can later return to the document and refine it once you have an opportunity to use a keyboard. Doctors and nurses often use these applications to log patient diagnoses and treatment notes efficiently.
• Phone transcriptions: Help callers and receivers transcribe a conversation between 2 or more speakers with phone APIs .
• Interviews: Turn spoken words into a comprehensive speech log the interviewer can reference later with this software. When a journalist interviews someone, they may want to record it to be more active and attentive without risking misquotes.

Try Twilio’s Speech Recognition API

Speech-to-text applications help you connect to larger and more diverse audiences. But to deploy these capabilities at scale, you need flexible and affordable speech recognition technology—and that’s where we can help.

Twilio’s Speech Recognition API performs real-time translation and converts speech to text in 119 languages and dialects. Make your customer service more accessible on a pay-as-you-go plan, with no upfront fees and free support. Get started for free !

Related Posts

Related Resources

Twilio docs, from apis to sdks to sample apps.

API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.

Resource Center

The latest ebooks, industry reports, and webinars.

Learn from customer engagement experts to improve your own communication.

Twilio's developer community hub

Best practices, code samples, and inspiration to build communications and digital engagement experiences.

Pardon our stardust! You've reached our interactive prototype, where we're polishing and adding new content daily!

• View Our Glossary
• Top Q&As
• Our Community

Where am I in the Reading Universe Taxonomy?

What Is Word Recognition?

word recognition skills. 

When we teach children to read, we spend a lot of time in the early years teaching them word recognition skills, including phonological awareness and phonics . We need to teach them how to break the code of our alphabet — decoding — and then to become fluent with that code, like we are. This will allow your students to focus on the meaning of what they read, which is the main thing! Word recognition is a tool or a means to the end goal of reading comprehension. 

What Does Word Recognition Look Like?

Here's a child in the early stages of decoding . She's working hard and having success sounding out individual words — and quickly recognizing some that she's read a few times before.

Produced by Reading Universe, a partnership of WETA, Barksdale Reading Institute, and First Book

View Video Transcript

This next child is a little bit further along with her decoding skills.

The second child's brain can focus on the story. We can tell because she's able to use expression. She only has to slow down when she comes across a new word. 

How Does Word Recognition Connect to Reading for Meaning?

Of course, it helps immensely that the young girl in the second video knows the meanings of the words she's decoding . And that's understated. Word recognition without language comprehension won't work. In order to be able to read for meaning … to comprehend what they read … children need to be able to recognize words and apply meaning to those words. The faster and easier they can do both, the more they'll be able to gain from reading in their life.

The Integration of Word Recognition and Language Comprehension

When we read, these two sets of skills — word recognition and language comprehension — intertwine and overlap, and your children will need your help to integrate them. As they read a new story word by word, they'll need to be able to sound out each word (or recognize it instantly), call up its meaning, connect it with their knowledge about the meaning, and apply it to the context of what they're reading — as quickly as possible.

In your classroom, whether you're a pre-K teacher or a second grade teacher, you'll spend time every day on both word recognition skills and language comprehension skills, often at the same time!

Picture singing a rhyming song and talking about the characters in the rhyme … that's an integrated lesson!

The word recognition section of the Reading Universe Taxonomy and is where we break each word recognition skill out, because, in the beginning, we need to spend significant time teaching skills in isolation, so that students can master them.

If you'd like a meatier introduction to the two sets of skills children need to read, we've got a one-hour presentation by reading specialist Margaret Goldberg for you to watch. Orthographic mapping, anyone?

How Children Learn to Read, with Margaret Goldberg

How Can Reading Universe Help You Teach Word Recognition?

syllables and suffixes to r-controlled vowels and the schwa . Each skill explainer has a detailed description of how to teach each skill, along with lesson plans, decodable text s, practice activities, and assessments. This continuum displays the many phonological awareness and phonics skills that all students need to master in order to become confident and fluent readers. We can help you teach all of them. Get started now .

Related Content

• skill Explainer

Onset-Rime Skill Explainer

• Word Recognition
• Phonological Awareness

Sign in to save your favorites. Don't have an account? It's easy and free to sign up !

How to Use Decodable Texts

Copyright © 2024 AudiologyOnline - All Rights Reserved

• Back to Basics: Speech Audiometry

Janet R. Schoepflin, PhD

• Hearing Evaluation - Adults

Editor's Note: This is a transcript of an AudiologyOnline live seminar. Please download supplemental course materials . Speech is the auditory stimulus through which we communicate. The recognition of speech is therefore of great interest to all of us in the fields of speech and hearing. Speech audiometry developed originally out of the work conducted at Bell Labs in the 1920s and 1930s where they were looking into the efficiency of communication systems, and really gained momentum post World War II as returning veterans presented with hearing loss. The methods and materials for testing speech intelligibility were of interest then, and are still of interest today. It is due to this ongoing interest as seen in the questions that students ask during classes, by questions new audiologists raise as they begin their practice, and by the comments and questions we see on various audiology listservs about the most efficient and effective ways to test speech in the clinical setting, that AudiologyOnline proposed this webinar as part of their Back to Basics series. I am delighted to participate. I am presenting a review of the array of speech tests that we use in clinical evaluation with a summary of some of the old and new research that has come about to support the recommended practices. The topics that I will address today are an overview of speech threshold testing, suprathreshold speech recognition testing, the most comfortable listening level testing, uncomfortable listening level, and a brief mention of some new directions that speech testing is taking. In the context of testing speech, I will assume that the environment in which you are testing meets the ANSI permissible noise criteria and that the audiometer transducers that are being used to perform speech testing are all calibrated to the ANSI standards for speech. I will not be talking about those standards, but it's of course important to keep those in mind.

Speech Threshold testing involves several considerations. They include the purposes of the test or the reasons for performing the test, the materials that should be used in testing, and the method or procedure for testing. Purposes of Speech Threshold Testing A number of purposes have been given for speech threshold testing. In the past, speech thresholds were used as a means to cross-check the validity of pure tone thresholds. This purpose lacks some validity because we have other physiologic and electrophysiologic procedures like OAEs and imittance test results to help us in that cross-check. However, the speech threshold measure is a test of hearing. It is not entirely invalid to be performed as a cross-check for pure tone hearing. I think sometimes we are anxious to get rid of things because we feel we have a better handle from other tests, but in this case, it may not be the wisest thing to toss out. Also in past years, speech thresholds were used to determine the level for suprathreshold speech recognition testing. That also lacks validity, because the level at which suprathreshold testing is conducted depends on the reason you are doing the test itself. It is necessary to test speech thresholds if you are going to bill 92557. Aside from that, the current purpose for speech threshold testing is in the evaluation of pediatric and difficult to test patients. Clinical practice surveys tell us that the majority of clinicians do test speech thresholds for all their patients whether it is for billing purposes or not. It is always important that testing is done in the recommended, standardized manner. The accepted measures for speech thresholds are the Speech Recognition Threshold (SRT) and the Speech Detection Threshold (SDT). Those terms are used because they specify the material or stimulus, i.e. speech, as well as the task that the listener is required to do, which is recognition or identification in the case of the SRT, and detection or noticing of presence versus absence of the stimulus in the case of SDT. The terms also specify the criterion for performance which is threshold or generally 50%. The SDT is most commonly performed on those individuals who have been unable to complete an SRT, such as very young children. Because recognition is not required in the speech detection task, it is expected that the SDT will be about 5 to 10 dB better than the SRT, which requires recognition of the material. Materials for Speech Threshold Testing The materials that are used in speech threshold testing are spondees, which are familiar two-syllable words that have a fairly steep psychometric function. Cold running speech or connected discourse is an alternative for speech detection testing since recognition is not required in that task. Whatever material is used, it should be noted on the audiogram. It is important to make notations on the audiogram about the protocols and the materials we are using, although in common practice many of us are lax in doing so. Methods for Speech Threshold Testing The methods consideration in speech threshold testing is how we are going to do the test. This would include whether we use monitored live voice or recorded materials, and whether we familiarize the patient with the materials and the technique that we use to elicit threshold. Monitored live voice and recorded speech can both be used in SRT testing. However, recorded presentation is recommended because recorded materials standardize the test procedure. With live voice presentation, the monitoring of each syllable of each spondee, so that it peaks at 0 on the VU meter can be fairly difficult. The consistency of the presentation is lost then. Using recorded materials is recommended, but it is less important in speech threshold testing than it is in suprathreshold speech testing. As I mentioned with the materials that are used, it is important to note on the audiogram what method of presentation has been used. As far as familiarization goes, we have known for about 50 years, since Tillman and Jerger (1959) identified familiarity as a factor in speech thresholds, that familiarization of the patient with the test words should be included as part of every test. Several clinical practice surveys suggest that familiarization is not often done with the patients. This is not a good practice because familiarization does influence thresholds and should be part of the procedure. The last consideration under methods is regarding the technique that is going to be used. Several different techniques have been proposed for the determination of SRT. Clinical practice surveys suggest the most commonly used method is a bracketing procedure. The typical down 10 dB, up 5 dB is often used with two to four words presented at each level, and the threshold then is defined as the lowest level at which 50% or at least 50% of the words are correctly repeated. This is not the procedure that is recommended by ASHA (1988). The ASHA-recommended procedure is a descending technique where two spondees are presented at each decrement from the starting level. There are other modifications that have been proposed, but they are not widely used.  

Suprathreshold speech testing involves considerations as well. They are similar to those that we mentioned for threshold tests, but they are more complicated than the threshold considerations. They include the purposes of the testing, the materials that should be used in testing, whether the test material should be delivered via monitored live voice or recorded materials, the level or levels at which the testing should be conducted, whether a full list, half list, or an abbreviated word list should be used, and whether or not the test should be given in quiet or noise. Purposes of Suprathreshold Testing There are several reasons to conduct suprathreshold tests. They include estimating the communicative ability of the individual at a normal conversational level; determining whether or not a more thorough diagnostic assessment is going to be conducted; hearing aid considerations, and analysis of the error patterns in speech recognition. When the purpose of testing is to estimate communicative ability at a normal conversational level, then the test should be given at a level around 50 to 60 dBHL since that is representative of a normal conversational level at a communicating distance of about 1 meter. While monosyllabic words in quiet do not give a complete picture of communicative ability in daily situations, it is a procedure that people like to use to give some broad sense of overall communicative ability. If the purpose of the testing is for diagnostic assessment, then a psychometric or performance-intensity function should be obtained. If the reason for the testing is for hearing aid considerations, then the test is often given using words or sentences and either in quiet or in a background of noise. Another purpose is the analysis of error patterns in speech recognition and in that situation, a test other than some open set monosyllabic word test would be appropriate. Materials for Suprathreshold Testing The choice of materials for testing depends on the purpose of the test and on the age and abilities of the patients. The issues in materials include the set and the test items themselves.  

Closed set vs. Open set. The first consideration is whether a closed set or an open set is appropriate. Closed set tests limit the number of response alternatives to a fairly small set, usually between 4 and 10 depending on the procedure. The number of alternatives influences the guess rate. This is a consideration as well. The Word Intelligibility by Picture Identification or the WIPI test is a commonly used closed set test for children as it requires only the picture pointing response and it has a receptive language vocabulary that is as low as about 5 years. It is very useful in pediatric evaluations as is another closed set test, the Northwestern University Children's Perception of Speech test (NU-CHIPS).

In contrast, the open set protocol provides an unlimited number of stimulus alternatives. Therefore, open set tests are more difficult. The clinical practice surveys available suggest for routine audiometric testing that monosyllabic word lists are the most widely used materials in suprathreshold speech recognition testing for routine evaluations, but sentences in noise are gaining popularity for hearing aid purposes.  

CID W-22 vs. NU-6. The most common materials for speech recognition testing are the monosyllabic words, the Central Institute of the Deaf W-22 and the Northwestern University-6 word list. These are the most common open set materials and there has been some discussion among audiologists concerning the differences between those. From a historical perspective, the CID W-22 list came from the original Harvard PAL-PB50 words and the W-22s are a group of the more familiar of those. They were developed into four 50-word lists. They are still commonly used by audiologists today. The NU-6 lists were developed later and instead of looking for phonetic balance, they considered a more phonemic balance. The articulation function for both of those using recorded materials is about the same, 4% per dB. The NU-6 tests are considered somewhat more difficult than the W-22s. Clinical surveys show that both materials are used by practicing audiologists, with usage of the NU-6 lists beginning to surpass usage of W-22s.

Nonsense materials. There are other materials that are available for suprathreshold speech testing. There are other monosyllabic word lists like the Gardner high frequency word list (Gardner, 1971) that could be useful for special applications or special populations. There are also nonsense syllabic tasks which were used in early research in communication. An advantage of the nonsense syllables is that the effects of word familiarity and lexical constraints are reduced as compared to using actual words as test materials. A few that are available are the City University of New York Nonsense Syllable test, the Nonsense Syllable test, and others.

Sentence materials. Sentence materials are gaining popularity, particularly in hearing aid applications. This is because speech that contains contextual cues and is presented in a noise background is expected to have better predictive validity than words in quiet. The two sentence procedures that are popular are the Hearing In Noise Test (HINT) (Nilsson, Soli,& Sullivan, 1994) and the QuickSIN (Killion, Niquette, Gudmundsen, Revit & Banerjee, 2004). Other sentence tests that are available that have particular applications are the Synthetic Sentence Identification test (SSI), the Speech Perception and Noise test (SPIN), and the Connected Speech test.

Monitored Live Voice vs. Recorded. As with speech threshold testing, the use of recorded materials for suprathreshold speech testing standardizes the test administration. The recorded version of the test is actually the test in my opinion. This goes back to a study in 1969 where the findings said the test is not just the written word list, but rather it is a recorded version of those words.

Inter-speaker and intra-speaker variability makes using recorded materials the method of choice in almost all cases for suprathreshold testing. Monitored live voice (MLV) is not recommended. In years gone by, recorded materials were difficult to manipulate, but the ease and flexibility that is afforded us by CDs and digital recordings makes recorded materials the only way to go for testing suprathreshold speech recognition. Another issue to consider is the use of the carrier phrase. Since the carrier phrase is included on recordings and recorded materials are the recommended procedure, that issue is settled. However, I do know that monitored live voice is necessary in certain situations and if monitored live voice is used in testing, then the carrier phrase should precede the test word. In monitored live voice, the carrier phrase is intended to allow the test word to have its own natural inflection and its own natural power. The VU meter should peak at 0 for the carrier phrase and the test word then is delivered at its own natural or normal level for that word in the phrase.  

Levels. The level at which testing is done is another consideration. The psychometric or performance-intensity function plots speech performance in percent correct on the Y-axis, as a function of the level of the speech signal on the X-axis. This is important because testing at only one level, which is fairly common, gives us insufficient information about the patient's optimal performance or what we commonly call the PB-max. It also does not allow us to know anything about any possible deterioration in performance if the level is increased. As a reminder, normal hearers show a function that reaches its maximum around 25 to 40 dB SL (re: SRT) and that is the reason why suprathreshold testing is often conducted at that level. For normals, the performance remains at that level, 100% or so, as the level increases. People with conductive hearing loss also show a similar function. Individuals with sensorineural hearing loss, however, show a performance function that reaches its maximum at generally less than 100%. They can either show performance that stays at that level as intensity increases, or they can show a curve that reaches its maximum and then decreases in performance as intensity increases. This is known as roll-over. A single level is not the best way to go as we cannot anticipate which patients may have rollover during testing, unless we test at a level higher than where the maximum score was obtained. I recognize that there are often time constraints in everyday practice, but two levels are recommended so that the performance-intensity function can be observed for an individual patient at least in an abbreviated way.

Recently, Guthrie and Mackersie (2009) published a paper that compared several different presentation levels to ascertain which level would result in maximum word recognition in individuals who had different hearing loss configurations. They looked at a number of presentation levels ranging from 10 dB above the SRT to a level at the UCL (uncomfortable listening level) -5 dB. Their results indicated that individuals with mild to moderate losses and those with more steeply sloping losses reached their best scores at a UCL -5 dB. That was also true for those patients who had moderately-severe to severe losses. The best phoneme recognition scores for their populations were achieved at a level of UCL -5 dB. As a reminder about speech recognition testing, masking is frequently needed because the test is being presented at a level above threshold, in many cases well above the threshold. Masking will always be needed for suprathreshold testing when the presentation level in the test ear is 40 dB or greater above the best bone conduction threshold in the non-test ear if supra-aural phones are used.  

Full lists vs. half-lists. Another consideration is whether a full list or a half-list should be administered. Original lists were composed of 50 words and those 50 words were created for phonetic balance and for simplicity in scoring. It made it easy for the test to be scored if 50 words were administered and each word was worth 2%. Because 50-word lists take a long time, people often use half-lists or even shorter lists for the purpose of suprathreshold speech recognition testing. Let's look into this practice a little further.

An early study was done by Thornton and Raffin (1978) using the Binomial Distribution Model. They investigated the critical differences between one score and a retest score that would be necessary for those scores to be considered statistically significant. Their findings showed that with an increasing set size, variability decreased. It would seem that more items are better. More recently Hurley and Sells (2003) conducted a study that looked at developing a test methodology that would identify those patients requiring a full 50 item suprathreshold test and allow abbreviated testing of patients who do not need a full 50 item list. They used Auditec recordings and developed 10-word and 25-word screening tests. They found that the four lists of NU-6 10-word and the 25-word screening tests were able to differentiate listeners who had impaired word recognition who needed a full 50-word list from those with unimpaired word recognition ability who only needed the 10-word or 25-word list. If abbreviated testing is important, then it would seem that this would be the protocol to follow. These screening lists are available in a recorded version and their findings were based on a recorded version. Once again, it is important to use recorded materials whether you are going to use a full list or use an abbreviated list.  

Quiet vs. Noise. Another consideration in suprathreshold speech recognition testing is whether to test in quiet or in noise. The effects of sensorineural hearing loss beyond the threshold loss, such as impaired frequency resolution or impaired temporal resolution, makes speech recognition performance in quiet a poor predictor for how those individuals will perform in noise. Speech recognition in noise is being promoted by a number of experts because adding noise improves the sensitivity of the test and the validity of the test. Giving the test at several levels will provide for a better separation between people who have hearing loss and those who have normal hearing. We know that individuals with hearing loss have a lot more difficulty with speech recognition in noise than those with normal hearing, and that those with sensorineural hearing loss often require a much greater signal-to-noise ratio (SNR), 10 to 15 better, than normal hearers.

Monosyllabic words in noise have not been widely used in clinical evaluation. However there are several word lists that are available. One of them is the Words in Noise test or WIN test which presents NU-6 words in a multi-talker babble. The words are presented at several different SNRs with the babble remaining at a constant level. One of the advantages of using these kinds of tests is that they are adaptive. They can be administered in a shorter period of time and they do not run into the same problems that we see with ceiling effects and floor effects. As I mentioned earlier, sentence tests in noise have become increasingly popular in hearing aid applications. Testing speech in noise is one way to look at amplification pre and post fitting. The Hearing in Noise Test and QuickSin, have gained popularity in those applications. The HINT was developed by Nilsson and colleagues in 1994 and later modified. It is scored as the dB to noise ratio that is necessary to get a 50% correct performance on the sentences. The sentences are the BKB (Bamford-Kowal-Bench) sentences. They are presented in sets of 10 and the listener listens and repeats the entire sentence correctly in order to get credit. In the HINT, the speech spectrum noise stays constant and the signal level is varied to obtain that 50% point. The QuickSin is a test that was developed by Killion and colleagues (2004) and uses the IEEE sentences. It has six sentences per list with five key words that are the scoring words in each sentence. All of them are presented in a multi-talker babble. The sentences get presented one at a time in 5 dB decrements from a high positive SNR down to 0 dB SNR. Again the test is scored as the 50% point in terms of dB signal-to-noise ratio. The guide proposed by Killion on the SNR is if an individual has somewhere around a 0 to 3 dB SNR it would be considered normal, 3 to 7 would be a mild SNR loss, 7 to15 dB would be a moderate SNR loss, and greater than 15 dB would be a severe SNR loss.  

Scoring. Scoring is another issue in suprathreshold speech recognition testing. It is generally done on a whole word basis. However phoneme scoring is another option. If phoneme scoring is used, it is a way of increasing the set size and you have more items to score without adding to the time of the test. If whole word scoring is used, the words have to be exactly correct. In this situation, being close does not count. The word must be absolutely correct in order to be judged as being correct. Over time, different scoring categorizations have been proposed, although the percentages that are attributed to those categories vary among the different proposals.

The traditional categorizations include excellent, good, fair, poor, and very poor. These categories are defined as:  

• Excellent or within normal limits = 90 - 100% on whole word scoring
• Good or slight difficulty = 78 - 88%
• Fair to moderate difficulty = 66 - 76%
• Poor or great difficulty = 54 - 64 %
• Very poor is < 52%

A very useful test routinely administered to those who are being considered for hearing aids is the level at which a listener finds listening most comfortable. The materials that are used for this are usually cold running speech or connected discourse. The listener is asked to rate the level at which listening is found to be most comfortable. Several trials are usually completed because most comfortable listening is typically a range, not a specific level or a single value. People sometimes want sounds a little louder or a little softer, so the range is a more appropriate term for this than most comfortable level. However whatever is obtained, whether it is a most comfortable level or a most comfortable range, should be recorded on the audiogram. Again, the material used should also be noted on the audiogram. As I mentioned earlier the most comfortable level (MCL) is often not the level at which a listener achieves maximum intelligibility. Using MCL in order to determine where the suprathreshold speech recognition measure will be done is not a good reason to use this test. MCL is useful, but not for determining where maximum intelligibility will be. The study I mentioned earlier showed that maximum intelligibility was reached for most people with hearing loss at a UCL -5. MCL is useful however in determining ANL or acceptable noise level.  

The uncomfortable listening level (UCL) is also conducted with cold running speech. The instructions for this test can certainly influence the outcome since uncomfortable or uncomfortably loud for some individuals may not really be their UCL, but rather a preference for listening at a softer level. It is important to define for the patient what you mean by uncomfortably loud. The utility of the UCL is in providing an estimate for the dynamic range for speech which is the difference between the UCL and the SRT. In normals, this range is usually 100 dB or more, but it is reduced in ears with sensorineural hearing loss often dramatically. By doing the UCL, you can get an estimate of the individual's dynamic range for speech.  

Acceptable Noise Level (ANL) is the amount of background noise that a listener is willing to accept while listening to speech (Nabelek, Tucker, & Letowski, 1991). It is a test of noise tolerance and it has been shown to be related to the successful use of hearing aids and to potential benefit with hearing aids (Nabelek, Freyaldenhoven, Tampas, & Muenchen, 2006). It uses the MCL and a measure known as BNL or background noise level. To conduct the test, a recorded speech passage is presented to the listener in the sound field for the MCL. Again note the use of recorded materials. The noise is then introduced to the listener to a level that will be the highest level that that person is able to accept or "put up with" while they are listening to and following the story in the speech passage. The ANL then becomes the difference between the MCL and the BNL. Individuals that have very low scores on the ANL are considered successful hearing aid users or good candidates for hearing aids. Those that have very high scores are considered unsuccessful users or poor hearing aid candidates. Obviously there are number of other applications for speech in audiologic practice, not the least of which is in the assessment of auditory processing. Many seminars could be conducted on this topic alone. Another application or future direction for speech audiometry is to more realistically assess hearing aid performance in "real world" environments. This is an area where research is currently underway.  

Question: Are there any more specific instructions for the UCL measurement? Answer: Instructions are very important. We need to make it clear to a patient exactly what we expect them to do. I personally do not like things loud. If I am asked to indicate what is uncomfortably loud, I am much below what is really my UCL. I think you have to be very direct in instructing your patients in that you are not looking for a little uncomfortable, but where they just do not want to hear it or cannot take it. Question: Can you sum up what the best methods are to test hearing aid performance? I assume this means with speech signals. Answer: I think the use of the HINT or the QuickSin would be the most useful on a behavioral test. We have other ways of looking at performance that are not behavioral. Question: What about dialects? In my area, some of the local dialects have clipped words during speech testing. I am not sure if I should count those as correct or incorrect. Answer: It all depends on your situation. If a patient's production is really reflective of the dialect of that region and they are saying the word as everyone else in that area would say it, then I would say they do have the word correct. If necessary, if you are really unclear, you can always ask the patient to spell the word or write it down. This extra time can be inconvenient, but that is the best way to be sure that they have correctly identified the word. Question: Is there a reference for the bracketing method? Answer: The bracketing method is based on the old modified Hughson-Westlake that many people use for pure tone threshold testing. It is very similar to that traditional down 10 dB, up 5 dB. I am sure there are more references, but the Hughson-Westlake is what bracketing is based on. Question: Once you get an SRT result, if you want to compare it to the thresholds to validate your pure tones, how do you compare it to the audiogram? Answer: If it is a flat hearing loss, then you can compare to the 3-frequency pure tone average (PTA). If there is a high frequency loss, where audibility at perhaps 2000 Hz is greatly reduced, then it is better to use just the average of 500Hz and 1000Hz as your comparison. If it is a steeply sloping loss, then you look for agreement with the best threshold, which would probably be the 500 Hz threshold. The reverse is also true for patients who have rising configurations. Compare the SRT to the best two frequencies of the PTA, if the loss has either a steep slope or a steep rise, or the best frequency in the PTA if it is a really precipitous change in configuration. Question: Where can I find speech lists in Russian or other languages? Answer: Auditec has some material available in languages other than English - it would be best to contact them directly. You can also view their catalog at www.auditec.com Carolyn Smaka: This raises a question I have. If an audiologist is not fluent in a particular language, such as Spanish, is it ok to obtain a word list or recording in that language and conduct speech testing? Janet Schoepflin: I do not think that is a good practice. If you are not fluent in a language, you do not know all the subtleties of that language and the allophonic variations. People want to get an estimation of suprathreshold speech recognition and this would be an attempt to do that. This goes along with dialect. Whether you are using a recording, or doing your best to say these words exactly as there are supposed to be said, and your patient is fluent in a language and they say the word back to you, since you are not familiar with all the variations in the language it is possible that you will score the word incorrectly. You may think it is correct when it is actually incorrect, or you may think it is incorrect when it is correct based on the dialect or variation of that language. Question: In school we were instructed to use the full 50-word list for any word discrimination testing at suprathreshold, but if we are pressed for time, a half word list would be okay. However, my professor warned us that we absolutely must go in order on the word list. Can you clarify this? Answer: I'm not sure why that might have been said. I was trained in the model to use the 50-word list. This was because the phonetic balance that was proposed for those words was based on the 50 words. If you only used 25 words, you were not getting the phonetic balance. I think the more current findings from Hurley and Sells show us that it is possible to use a shorter list developed specifically for this purpose. It should be the recorded version of those words. These lists are available through Auditec. Question: On the NU-6 list, the words 'tough' and 'puff' are next to each other. 'Tough' is often mistaken for 'puff' so then when we reads 'puff', the person looks confused. Is it okay to mix up the order on the word list? Answer: I think in that case it is perfectly fine to move that one word down. Question: When do you recommend conducting speech testing, before or after pure tone testing? Answer: I have always been a person who likes to interact with my patients. My own procedure is to do an SRT first. Frequently for an SRT I do use live voice. I do not use monitored live voice for suprathreshold testing. It gives me a time to interact with the patient. People feel comfortable with speech. It is a communicative act. Then I do pure tone testing. Personally I would not do suprathreshold until I finished pure tone testing. My sequence is often SRT, pure tone, and suprathreshold. If this is not a good protocol for you based on time, then I would conduct pure tone testing, SRT, and then suprathreshold. Question: Some of the spondee words are outdated such as inkwell and whitewash. Is it okay to substitute other words that we know are spondee words, but may not be on the list? Or if we familiarize people, does it matter? Answer: The words that are on the list were put there for their so-called familiarity, but also because they were somewhat homogeneous and equal in intelligibility. I think inkwell, drawbridge and whitewash are outdated. If you follow a protocol where you are using a representative sample of the words and you are familiarizing, I think it is perfectly fine to eliminate those words you do not want to use. You just do not want to end up only using five or six words as it will limit the test set. Question: At what age is it appropriate to expect a child to perform suprathreshold speech recognition testing? Answer: If the child has a receptive language age of around 4 or 5 years, even 3 years maybe, it is possible to use the NU-CHIPS as a measure. It really does depend on language more than anything else, and the fact that the child can sit still for a period of time to do the test. Question: Regarding masking, when you are going 40 dB above the bone conduction threshold in the non-test ear, what frequency are you looking at? Are you comparing speech presented at 40 above a pure tone average of the bone conduction threshold? Answer: The best bone conduction threshold in the non-test ear is what really should be used. Question: When seeing a patient in follow-up after an ENT prescribes a steroid therapy for hydrops, do you recommend using the same word list to compare their suprathreshold speech recognition? Answer: I think it is better to use a different list, personally. Word familiarity as we said can influence even threshold and it certainly can affect suprathreshold performance. I think it is best to use a different word list. Carolyn Smaka: Thanks to everyone for their questions. Dr. Schoepflin has provided her email address with the handout. If your question was not answered or if you have further thoughts after the presentation, please feel free to follow up directly with her via email. Janet Schoepflin: Thank you so much. It was my pleasure and I hope everyone found the presentation worthwhile.

American Speech, Language and Hearing Association. (1988). Determining Threshold Level for Speech [Guidelines]. Available from www.asha.org/policy Gardner, H.(1971). Application of a high-frequency consonant discrimination word list in hearing-aid evaluation. Journal of Speech and Hearing Disorders, 36 , 354-355. Guthrie, L. & Mackersie, C. (2009). A comparison of presentation levels to maximize word recognition scores. Journal of the American Academy of Audiology, 20 (6), 381-90. Hurley, R. & Sells, J. (2003). An abbreviated word recognition protocol based on item difficulty. Ear & Hearing, 24 (2), 111-118. Killion, M., Niquette, P., Gudmundsen, G., Revit, L., & Banerjee, S. (2004). Development of a quick speech-in-noise test for measuring signal-to-noise ratio loss in normal-hearing and hearing-impaired listeners. Journal of the Acoustical Society of America, 116 (4 Pt 1), 2395-405. Nabelek, A., Freyaldenhoven, M., Tampas, J., Burchfield, S., & Muenchen, R. (2006). Acceptable noise level as a predictor of hearing aid use. Journal of the American Academy of Audiology, 17 , 626-639. Nabelek, A., Tucker, F., & Letowski, T. (1991). Toleration of background noises: Relationship with patterns of hearing aid use by elderly persons. Journal of Speech and Hearing Research, 34 , 679-685. Nilsson, M., Soli. S,, & Sullivan, J. (1994). Development of the hearing in noise test for the measurement of speech reception thresholds in quiet and in noise. Journal of the Acoustical Society of America, 95 (2), 1085-99. Thornton, A.. & Raffin, M, (1978). Speech-discrimination scores modeled as a binomial variable. Journal of Speech and Hearing Research, 21 , 507-518. Tillman, T., & Jerger, J. (1959). Some factors affecting the spondee threshold in normal-hearing subjects. Journal of Speech and Hearing Research, 2 , 141-146.

Chair, Communication Sciences and Disorders, Adelphi University

Janet Schoepflin is an Associate Professor and Chair of the Department of Communication Sciences and Disorders at Adelphi University and a member of the faculty of the Long Island AuD Consortium.  Her areas of research interest include speech perception in children and adults, particularly those with hearing loss, and the effects of noise on audition and speech recognition performance.

Related Courses

Empowerment and behavioral insights in client decision making, presented in partnership with nal, course: #37124 level: intermediate 1 hour, cognition and audition: supporting evidence, screening options, and clinical research, course: #37381 level: introductory 1 hour, innovative audiologic care delivery, course: #38661 level: intermediate 4 hours, aurical hit applications part 1 - applications for hearing instrument fittings and beyond, course: #28678 level: intermediate 1 hour, rethinking your diagnostic audiology battery: using value added tests, course: #29447 level: introductory 1 hour.

Our site uses cookies to improve your experience. By using our site, you agree to our Privacy Policy .

• Engineering Mathematics
• Discrete Mathematics
• Operating System
• Computer Networks
• Digital Logic and Design
• C Programming
• Data Structures
• Theory of Computation
• Compiler Design
• Computer Org and Architecture
• Top 10 Highest Goalscorers of All Time in Football [Updated 2024]
• Specific Heat Calculator - Free Online Calculator
• Millennials Vs Gen Z: Key Differences and How They Impact Customer Experience
• mg to mL Converter - Free Online Tool
• Care Bear Names 2024 (Names and Characters)
• Top 50+ Funniest Fantasy Football Team Names
• MLB Teams - List of Baseball Teams
• Michael Jackson's Kids: Prince Michael Jackson, Paris Jackson, Bigi Jackson
• What Is a Swap File and How Does It Work?
• 50 Most Common Passwords List in 2024
• Top 50 Most Common and Powerful Last Names in the US
• Top 10 Business Intelligence Tools (Best Business Intelligence BI)
• How to Say Sorry in Spanish - Top 10 Ways in 2024
• Blood Group Compatibility Chart: A-, A+, B-, B+, AB-, AB+, O- and O+
• What is a Boot Sector Virus? (Definition, Risks and Prevention)
• What is a Resident Virus? Examples and Protection
• List of ICC Women's Cricket World Cup Winners (From 1973 to 2023)
• Ink Master Winners: All Season Winners and Where are They Now?
• Spanish Numbers - How to Count From 1 to 100

What is Speech Recognition?

Speech recognition or speech-to-text recognition, is the capacity of a machine or program to recognize spoken words and transform them into text. Speech Recognition is an important feature in several applications used such as home automation, artificial intelligence, etc. In this article, we are going to discuss every point about speech recognition.

Speech Recognition, also known as automatic speech recognition (ASR), computer speech recognition, or speech-to-text, focuses on enabling computers to understand and interpret human speech. Speech recognition involves converting spoken language into text or executing commands based on the recognized words. This technology relies on sophisticated algorithms and machine learning models to process and understand human speech in real-time, despite the variations in accents, pitch, speed, and slang.

Features of Speech Recognition

• Accuracy and Speed: They can process speech in real-time or near real-time, providing quick responses to user inputs.
• Natural Language Understanding (NLU): NLU enables systems to handle complex commands and queries, making technology more intuitive and user-friendly .
• Multi-Language Support: Support for multiple languages and dialects, allowing users from different linguistic backgrounds to interact with technology in their native language.
• Background Noise Handling: This feature is crucial for voice-activated systems used in public or outdoor settings.

Speech Recognition Algorithms

Speech recognition technology relies on complex algorithms to translate spoken language into text or commands that computers can understand and act upon. Here are the algorithms and approaches used in speech recognition:

1. Hidden Markov Models (HMM)

Hidden Markov Models have been the backbone of speech recognition for many years. They model speech as a sequence of states, with each state representing a phoneme (basic unit of sound) or group of phonemes. HMMs are used to estimate the probability of a given sequence of sounds, making it possible to determine the most likely words spoken. Usage : Although newer methods have surpassed HMM in performance, it remains a fundamental concept in speech recognition, often used in combination with other techniques.

2. Natural language processing (NLP)

NLP is the area of  artificial intelligence  which focuses on the interaction between humans and machines through language through speech and text. Many mobile devices incorporate speech recognition into their systems to conduct voice search. Example such as: Siri or provide more accessibility around texting. 

3. Deep Neural Networks (DNN)

DNNs have improved speech recognition’s accuracy a lot. These networks can learn hierarchical representations of data, making them particularly effective at modeling complex patterns like those found in human speech. DNNs are used both for acoustic modeling, to better understand the sound of speech, and for language modeling, to predict the likelihood of certain word sequences.

4. End-to-End Deep Learning

Now, the trend has shifted towards end-to-end deep learning models, which can directly map speech inputs to text outputs without the need for intermediate phonetic representations. These models, often based on advanced RNNs, Transformers, or Attention Mechanisms, can learn more complex patterns and dependencies in the speech signal.

How does Speech Recognition Work?

Speech recognition systems works on computer algorithms to process and interpret spoken words before converting them into text. A software program converts the sound into written text that computers and humans can understand by analyzing the audio, broke down into segments, digitize into readable format and apply most suitable algorithm. Human speech is very diverse and context-specific, thus speech recognition software has to adapt accordingly. The software algorithms that interpret and organise audio into text are trained on a variety of speech patterns, speaking styles, languages, dialects, accents, and phrasing. The software also distinguishes spoken audio from noise from the background. Speech recognition uses two types model:

• Acoustic Model: An acoustic model is responsible for converting an audio signal into a sequence of phonemes or sub-word units. It represents the relationship between acoustic signals and phonemes or sub-word units.
• Language Model: A language model is responsible for assigning probabilities to sequences of words or phrases. It captures the likelihood of certain word sequences occurring in a given language. Language models can be based on n-gram models, recurrent neural networks (RNNs) , or transformer-based architectures like GPT (Generative Pre-trained Transformer).

Speech Recognition Use Cases

• Virtual Assistants : These assistants use speech recognition to understand user commands and questions, enabling hands-free interaction for tasks like setting reminders, searching the internet, controlling smart home devices, and more. For Ex – Siri, Alexa, Google Assistant
• Accessibility Tools : Speech recognition improves accessibility, allowing individuals with physical disabilities to interact with technology and communicate more easily. For Ex – Voice control features in smartphones and computers, specialized applications for individuals with disabilities.
• Automotive Systems : Drivers can use voice commands to control navigation systems, music, and phone calls, reducing distractions and enhancing safety on the road. For Ex – Voice-activated navigation and infotainment systems in cars.
• Healthcare : Doctors and medical staff use speech recognition for faster documentation, allowing them to spend more time with patients. Additionally, voice-enabled bots can assist in patient care and inquiries. For Ex – Dictation solutions for medical documentation, patient interaction bots.
• Customer Service : Speech recognition is used to route customer calls to the appropriate department or to provide automated assistance, improving efficiency and customer satisfaction. For Ex – Voice-operated call centers, customer service bots.
• Education and E-Learning : Speech recognition aids in language learning by providing immediate feedback on pronunciation. It also helps in transcribing lectures and seminars for better accessibility. For Ex – Language learning apps, lecture transcription services.
• Security and Authentication : Speech recognition combined with voice biometrics offers a secure and convenient way to authenticate users for banking services, secure facilities, and personal devices. For Ex – Voice biometrics in banking and secure access.
• Entertainment and Media : Users can find content using voice search, making navigation easier and more intuitive. Voice-controlled games offer a unique, hands-free gaming experience. For Ex – Voice biometrics in banking and secure access.

Speech Recognition Vs Voice Recognition

Speech Recognition is better for applications where the goal is to understand and convert spoken language into text or commands . This makes it ideal for creating hands-free user interfaces, transcribing meetings or lectures, enabling voice commands for devices, and assisting users with disabilities. Whereas Voice Recognition is better for applications focused on identifying or verifying the identity of a speaker. This technology is crucial for security and personalized interaction, such as biometric authentication, personalized user experiences based on the identified speaker, and access control systems. Its value comes from its ability to recognize the unique characteristics of a person’s voice, offering a layer of security or customization.

Advantages of Speech Recognition

• Accessibility: Speech recognition technology improves accessibility for individuals with disabilities, including those with mobility impairments or vision loss.
• Increased Productivity: Speech recognition can significantly enhance productivity by enabling faster data entry and document creation.
• Hands-Free Operation:  Enables hands-free interaction with devices and systems, improving safety and convenience, especially in tasks like driving or cooking.
• Efficiency:  Speeds up data entry and interaction with devices, as speaking is often faster than typing or using a keyboard.
• Multimodal Interaction:  Supports multimodal interfaces, allowing users to combine speech with other input methods like touch and gestures for more natural interactions.

Disadvantages of Speech Recognition

• Inconsistent performance: The systems may be unable to record words accurately due to variations in pronunciation, a lack of capability for particular languages, and the inability to sift through background noise.
• Speed: Some voice recognition programs require time to implement and learn. Speech processing is relatively slow.
• Source file issues: Speech recognition is dependent on the recording equipment utilised, not simply the programme.
• Dependence on Infrastructure: Effective speech recognition frequently relies on strong infrastructure, such as consistent internet connectivity and computing resources.

Speech recognition is a powerful technology that lets computers understand and process human speech. It’s used everywhere, from asking your smartphone for directions to controlling your smart home devices with just your voice. This tech makes life easier by helping with tasks without needing to type or press buttons, making gadgets like virtual assistants more helpful. It’s also super important for making tech accessible to everyone, including those who might have a hard time using keyboards or screens. As we keep finding new ways to use speech recognition, it’s becoming a big part of our daily tech life, showing just how much we can do when we talk to our devices.

Frequently Asked Question on Speech Recognition – FAQs

What are examples of speech recognition.

Note Taking/Writing: An example of speech recognition technology in use is speech-to-text platforms such as Speechmatics or Google’s speech-to-text engine. In addition, many voice assistants offer speech-to-text translation.

Is speech recognition secure?

Security concerns related to speech recognition primarily involve the privacy and protection of audio data collected and processed by speech recognition systems. Ensuring secure data transmission, storage, and processing is essential to address these concerns.

What is speech recognition in AI?

Speech recognition is the process of converting sound signals to text transcriptions. Steps involved in conversion of a sound wave to text transcription in a speech recognition system are: Recording: Audio is recorded using a voice recorder. Sampling: Continuous audio wave is converted to discrete values.

How accurate is speech recognition technology?

The accuracy of speech recognition technology can vary depending on factors such as the quality of audio input, language complexity, and the specific application or system being used. Advances in machine learning and deep learning have improved accuracy significantly in recent years.

Please Login to comment...

Similar reads.

• Computer Subject

Improve your Coding Skills with Practice

What kind of Experience do you want to share?

How Does Speech Recognition Work? (9 Simple Questions Answered)

• by Team Experts
• July 2, 2023 July 3, 2023

Discover the Surprising Science Behind Speech Recognition – Learn How It Works in 9 Simple Questions!

Speech recognition is the process of converting spoken words into written or machine-readable text. It is achieved through a combination of natural language processing , audio inputs, machine learning , and voice recognition . Speech recognition systems analyze speech patterns to identify phonemes , the basic units of sound in a language. Acoustic modeling is used to match the phonemes to words , and word prediction algorithms are used to determine the most likely words based on context analysis . Finally, the words are converted into text.

What is Natural Language Processing and How Does it Relate to Speech Recognition?

How do audio inputs enable speech recognition, what role does machine learning play in speech recognition, how does voice recognition work, what are the different types of speech patterns used for speech recognition, how is acoustic modeling used for accurate phoneme detection in speech recognition systems, what is word prediction and why is it important for effective speech recognition technology, how can context analysis improve accuracy of automatic speech recognition systems, common mistakes and misconceptions.

Natural language processing (NLP) is a branch of artificial intelligence that deals with the analysis and understanding of human language. It is used to enable machines to interpret and process natural language, such as speech, text, and other forms of communication . NLP is used in a variety of applications , including automated speech recognition , voice recognition technology , language models, text analysis , text-to-speech synthesis , natural language understanding , natural language generation, semantic analysis , syntactic analysis, pragmatic analysis, sentiment analysis, and speech-to-text conversion. NLP is closely related to speech recognition , as it is used to interpret and understand spoken language in order to convert it into text.

Audio inputs enable speech recognition by providing digital audio recordings of spoken words . These recordings are then analyzed to extract acoustic features of speech, such as pitch, frequency, and amplitude. Feature extraction techniques , such as spectral analysis of sound waves, are used to identify and classify phonemes . Natural language processing (NLP) and machine learning models are then used to interpret the audio recordings and recognize speech. Neural networks and deep learning architectures are used to further improve the accuracy of voice recognition . Finally, Automatic Speech Recognition (ASR) systems are used to convert the speech into text, and noise reduction techniques and voice biometrics are used to improve accuracy .

Machine learning plays a key role in speech recognition , as it is used to develop algorithms that can interpret and understand spoken language. Natural language processing , pattern recognition techniques , artificial intelligence , neural networks, acoustic modeling , language models, statistical methods , feature extraction , hidden Markov models (HMMs), deep learning architectures , voice recognition systems, speech synthesis , and automatic speech recognition (ASR) are all used to create machine learning models that can accurately interpret and understand spoken language. Natural language understanding is also used to further refine the accuracy of the machine learning models .

Voice recognition works by using machine learning algorithms to analyze the acoustic properties of a person’s voice. This includes using voice recognition software to identify phonemes , speaker identification, text normalization , language models, noise cancellation techniques , prosody analysis , contextual understanding , artificial neural networks, voice biometrics , speech synthesis , and deep learning . The data collected is then used to create a voice profile that can be used to identify the speaker .

The different types of speech patterns used for speech recognition include prosody , contextual speech recognition , speaker adaptation , language models, hidden Markov models (HMMs), neural networks, Gaussian mixture models (GMMs) , discrete wavelet transform (DWT), Mel-frequency cepstral coefficients (MFCCs), vector quantization (VQ), dynamic time warping (DTW), continuous density hidden Markov model (CDHMM), support vector machines (SVM), and deep learning .

Acoustic modeling is used for accurate phoneme detection in speech recognition systems by utilizing statistical models such as Hidden Markov Models (HMMs) and Gaussian Mixture Models (GMMs). Feature extraction techniques such as Mel-frequency cepstral coefficients (MFCCs) are used to extract relevant features from the audio signal . Context-dependent models are also used to improve accuracy . Discriminative training techniques such as maximum likelihood estimation and the Viterbi algorithm are used to train the models. In recent years, neural networks and deep learning algorithms have been used to improve accuracy , as well as natural language processing techniques .

Word prediction is a feature of natural language processing and artificial intelligence that uses machine learning algorithms to predict the next word or phrase a user is likely to type or say. It is used in automated speech recognition systems to improve the accuracy of the system by reducing the amount of user effort and time spent typing or speaking words. Word prediction also enhances the user experience by providing faster response times and increased efficiency in data entry tasks. Additionally, it reduces errors due to incorrect spelling or grammar, and improves the understanding of natural language by machines. By using word prediction, speech recognition technology can be more effective , providing improved accuracy and enhanced ability for machines to interpret human speech.

Context analysis can improve the accuracy of automatic speech recognition systems by utilizing language models, acoustic models, statistical methods , and machine learning algorithms to analyze the semantic , syntactic, and pragmatic aspects of speech. This analysis can include word – level , sentence- level , and discourse-level context, as well as utterance understanding and ambiguity resolution. By taking into account the context of the speech, the accuracy of the automatic speech recognition system can be improved.

• Misconception : Speech recognition requires a person to speak in a robotic , monotone voice. Correct Viewpoint: Speech recognition technology is designed to recognize natural speech patterns and does not require users to speak in any particular way.
• Misconception : Speech recognition can understand all languages equally well. Correct Viewpoint: Different speech recognition systems are designed for different languages and dialects, so the accuracy of the system will vary depending on which language it is programmed for.
• Misconception: Speech recognition only works with pre-programmed commands or phrases . Correct Viewpoint: Modern speech recognition systems are capable of understanding conversational language as well as specific commands or phrases that have been programmed into them by developers.

From Talk to Tech: Exploring the World of Speech Recognition

What is Speech Recognition Technology?

Imagine being able to control electronic devices, order groceries, or dictate messages with just voice. Speech recognition technology has ushered in a new era of interaction with devices, transforming the way we communicate with them. It allows machines to understand and interpret human speech, enabling a range of applications that were once thought impossible.

Speech recognition leverages machine learning algorithms to recognize speech patterns, convert audio files into text, and examine word meaning. Siri, Alexa, Google's Assistant, and Microsoft's Cortana are some of the most popular speech to text voice assistants used today that can interpret human speech and respond in a synthesized voice.

From personal assistants that can understand every command directed towards them to self-driving cars that can comprehend voice instructions and take the necessary actions, the potential applications of speech recognition are manifold. As technology continues to advance, the possibilities are endless.

How do Speech Recognition Systems Work?

speech to text processing is traditionally carried out in the following way:

Recording the audio:  The first step of speech to text conversion involves recording the audio and voice signals using a microphone or other audio input devices.

Breaking the audio into parts: The recorded voice or audio signals are then broken down into small segments, and features are extracted from each piece, such as the sound's frequency, pitch, and duration.

Digitizing speech into computer-readable format:  In the third step, the speech data is digitized into a computer-readable format that identifies the sequence of characters to remember the words or phrases that were most likely spoken.

Decoding speech using the algorithm:  Finally, language models decode the speech using speech recognition algorithms to produce a transcript or other output.

To adapt to the nature of human speech and language, speech recognition is designed to identify patterns, speaking styles, frequency of words spoken, and speech dialects on various levels. Advanced speech recognition software are also capable of eliminating background noises that often accompany speech signals.

When it comes to processing human speech, the following two types of models are used:

Acoustic Models

Acoustic models are a type of machine learning model used in speech recognition systems. These models are designed to help a computer understand and interpret spoken language by analyzing the sound waves produced by a person's voice.

Language Models

Based on the speech context, language models employ statistical algorithms to forecast the likelihood of words and phrases. They compare the acoustic model's output to a pre-built vocabulary of words and phrases to identify the most likely word order that makes sense in a given context of the speech. 

Applications of Speech Recognition Technology

Automatic speech recognition is becoming increasingly integrated into our daily lives, and its potential applications are continually expanding. With the help of speech to text applications, it's now becoming convenient to convert a speech or spoken word into a text format, in minutes.

Speech recognition is also used across industries, including healthcare , customer service, education, automotive, finance, and more, to save time and work efficiently. Here are some common speech recognition applications:

Voice Command for Smart Devices

Today, there are many home devices designed with voice recognition. Mobile devices and home assistants like Amazon Echo or Google Home are among the most widely used speech recognition system. One can easily use such devices to set reminders, place calls, play music, or turn on lights with simple voice commands.

Online Voice Search

Finding information online is now more straightforward and practical, thanks to speech to text technology. With online voice search, users can search using their voice rather than typing. This is an excellent advantage for people with disabilities and physical impairments and those that are multitasking and don't have the time to type a prompt.

Help People with Disabilities

People with disabilities can also benefit from speech to text applications because it allows them to use voice recognition to operate equipment, communicate, and carry out daily duties. In other words, it improves their accessibility. For example, in case of emergencies, people with visual impairment can use voice commands to call their friends and family on their mobile devices.

Business Applications of Speech Recognition

Speech recognition has various uses in business, including banking, healthcare, and customer support. In these industries, voice recognition mainly aims at enhancing productivity, communication, and accessibility. Some common applications of speech technology in business sectors include:

Speech recognition is used in the banking industry to enhance customer service and expedite internal procedures. Banks can also utilize speech to text programs to enable clients to access their accounts and conduct transactions using only their voice.

Customers in the bank who have difficulties entering or navigating through complicated data will find speech to text particularly useful. They can simply voice search the necessary data. In fact, today, banks are automating procedures like fraud detection and customer identification using this impressive technology, which can save costs and boost security.

Voice recognition is used in the healthcare industry to enhance patient care and expedite administrative procedures. For instance, physicians can dictate notes about patient visits using speech recognition programs, which can then be converted into electronic medical records. This also helps to save a lot of time, and correct data is recorded in the best way possible with this technology.

Customer Support

Speech recognition is employed in customer care to enhance the customer experience and cut expenses. For instance, businesses can automate time-consuming processes using speech to text so that customers can access information and solve problems without speaking to a live representative. This could shorten wait times and increase customer satisfaction.

Challenges with Speech Recognition Technology

Although speech recognition has become popular in recent years and made our lives easier, there are still several challenges concerning speech recognition that needs to be addressed.

Accuracy may not always be perfect

A speech recognition software can still have difficulty accurately recognizing speech in noisy or crowded environments or when the speaker has an accent or speech impediment. This can lead to incorrect transcriptions and miscommunications.

The software can not always understand complexity and jargon

Any speech recognition software has a limited vocabulary, so it may struggle to identify uncommon or specialized vocabulary like complex sentences or technical jargon, making it less useful in specific industries or contexts. Errors in interpretation or translation may happen if the speech recognition fails to recognize the context of words or phrases.

Concern about data privacy, data can be recorded.

Speech recognition technology relies on recording and storing audio data, which can raise concerns about data privacy. Users may be uncomfortable with their voice recordings being stored and used for other purposes. Also, voice notes, phone calls, and recordings may be recorded without the user's knowledge, and hacking or impersonation can be vulnerable to these security breaches. These things raise privacy and security concerns.

Software that Use Speech Recognition Technology

Many software programs use speech recognition technology to transcribe spoken words into text. Here are some of the most popular ones:

Nuance Dragon.

Amazon Transcribe.

Google Text to Speech

Watson Speech to Text

To sum up, speech recognition technology has come a long way in recent years. Given its benefits, including increased efficiency, productivity, and accessibility, its finding applications across a wide range of industries. As we continue to explore the potential of this evolving technology, we can expect to see even more exciting applications emerge in the future.

With the power of AI and machine learning at our fingertips, we're poised to transform the way we interact with technology in ways we never thought possible. So, let's embrace this exciting future and see where speech recognition takes us next!

What are the three steps of speech recognition?

The three steps of speech recognition are as follows:

Step 1: Capture the acoustic signal

The first step is to capture the acoustic signal using an audio input device and later pre-process the motion to remove noise and other unwanted sounds. The movement is then broken down into small segments, and features such as frequency, pitch, and duration are extracted from each piece.

Step 2: Combining the acoustic and language models

The second step involves combining the acoustic and language models to produce a transcription of the spoken words and word sequences.

Step 3: Converting the text into a synthesized voice

The final step is converting the text into a synthesized voice or using the transcription to perform other actions, such as controlling a computer or navigating a system.

What are examples of speech recognition?

Speech recognition is used in a wide range of applications. The most famous examples of speech recognition are voice assistants like Apple's Siri, Amazon's Alexa, and Google Assistant. These assistants use effective speech recognition to understand and respond to voice commands, allowing users to ask questions, set reminders, and control their smart home devices using only voice.

What is the importance of speech recognition?

Speech recognition is essential for improving accessibility for people with disabilities, including those with visual or motor impairments. It can also improve productivity in various settings and promote language learning and communication in multicultural environments. Speech recognition can break down language barriers, save time, and reduce errors.

You should also read:

Understanding Speech to Text in Depth

Top 10 Speech to Text Software in 2024

How Speech Recognition is Changing Language Learning

Speech Recognition

What Is Speech Recognition?

Speech recognition is the technology that allows a computer to recognize human speech and process it into text. It’s also known as automatic speech recognition ( ASR ), speech-to-text, or computer speech recognition. 

Speech recognition systems rely on technologies like artificial intelligence (AI) and machine learning (ML) to gain larger samples of speech, including different languages, accents, and dialects. AI is used to identify patterns of speech, words, and language to transcribe them into a written format.

In this blog post, we’ll take a deeper dive into speech recognition and look at how it works, its real-world applications, and how platforms like aiOla are using it to change the way we work.

Basic Speech Recognition Concepts

To start understanding speech recognition and all its applications, we need to first look at what it is and isn’t. While speech recognition is more than just the sum of its parts, it’s important to look at each of the parts that contribute to this technology to better grasp how it can make a real impact. Let’s take a look at some common concepts.

Speech Recognition vs. Speech Synthesis

Unlike speech recognition, which converts spoken language into a written format through a computer, speech synthesis does the same in reverse. In other words, speech synthesis is the creation of artificial speech derived from a written text, where a computer uses an AI-generated voice to simulate spoken language. For example, think of the language voice assistants like Siri or Alexa use to communicate information.

Phonetics and Phonology

Phonetics studies the physical sound of human speech, such as its acoustics and articulation. Alternatively, phonology looks at the abstract representation of sounds in a language including their patterns and how they’re organized. These two concepts need to be carefully weighed for speech AI algorithms to understand sound and language as a human might.

Acoustic Modeling

In acoustic modeling , the acoustic characteristics of audio and speech are looked at. When it comes to speech recognition systems, this process is essential since it helps analyze the audio features of each word, such as the frequency in which it’s used, the duration of a word, or the sounds it encompasses.

Language Modeling

Language modeling algorithms look at details like the likelihood of word sequences in a language. This type of modeling helps make speech recognition systems more accurate as it mimics real spoken language by looking at the probability of word combinations in phrases.

Speaker-Dependent vs. Speaker-Independent Systems

A system that’s dependent on a speaker is trained on the unique voice and speech patterns of a specific user, meaning the system might be highly accurate for that individual but not as much for other people. By contrast, a system that’s independent of a speaker can recognize speech for any number of speakers, and while more versatile, may be slightly less accurate.

How Does Speech Recognition Work?

There are a few different stages to speech recognition, each one providing another layer to how language is processed by a computer. Here are the different steps that make up the process.

• First, raw audio input undergoes a process called preprocessing , where background noise is removed to enhance sound quality and make recognition more manageable.
• Next, the audio goes through feature extraction , where algorithms identify distinct characteristics of sounds and words. 
• Then, these extracted features go through acoustic modeling , which as we described earlier, is the stage where acoustic and language models decide the most accurate visual representation of the word. These acoustic modeling systems are based on extensive datasets, allowing them to learn the acoustic patterns of different spoken words.
• At the same time, language modeling looks at the structure and probability of words in a sequence, which helps provide context. 
• After this, the output goes into a decoding sequence, where the speech recognition system matches data from the extracted features with the acoustic models. This helps determine the most likely word sequence.
• Finally, the audio and corresponding textual output go through post-processing , which refines the output by correcting errors and improving coherence to create a more accurate transcription.

When it comes to advanced systems, all of these stages are done nearly instantaneously, making this process almost invisible to the average user. All of these stages together have made speech recognition a highly versatile tool that can be used in many different ways, from virtual assistants to transcription services and beyond.

Types of Speech Recognition Systems

Speech recognition technology is used in many different ways today, transforming the way humans and machines interact and work together. From professional settings to helping us make our lives a little easier, this technology can take on many forms. Here are some of them.

Virtual Assistants

In 2022, 62% of US adults used a voice assistant on various mobile devices. Siri, Google Assistant, and Alexa are all examples of speech recognition in our daily lives. These applications respond to vocal commands and can interact with humans through natural language in order to complete tasks like sending messages, answering questions, or setting reminders.

Voice Search

Search engines like Google can be searched using voice instead of typing in a query, often with voice assistants. This allows users to conveniently search for a quick answer without sorting through content when they need to be hands-free, like when driving or multitasking. This technology has become so popular over the last few years that now 50% of US-based consumers use voice search every single day.

Transcription Services

Speech recognition has completely changed the transcription industry. It has enabled transcription services to automate the process of turning speech into text, increasing efficiency in many fields like education, legal services, healthcare, and even journalism.

Accessibility

With speech recognition, technologies that may have seemed out of reach are now accessible to people with disabilities. For example, for people with motor impairments or who are visually impaired, AI voice-to-text technology can help with the hands-free operation of things like keyboards, writing assistance for dictation, and voice commands to control devices.

Automotive Systems

Speech recognition is keeping drivers safer by giving them hands-free control over in-car features. Drivers can make calls, adjust the temperature, navigate, or even control the music without ever removing their hands from the wheel and instead just issuing voice commands to a speech-activated system.

How Does aiOla Use Speech Recognition?

aiOla’s AI-powered speech platform is revolutionizing the way certain industries work by bringing advanced speech recognition technology to companies in fields like aviation, fleet management, food safety, and manufacturing.

Traditionally, many processes in these industries were manual, forcing organizations to use a lot of time, budget, and resources to complete mission-critical tasks like inspections and maintenance. However, with aiOla’s advanced speech system, these otherwise labor and resource-intensive tasks can be reduced to a matter of minutes using natural language.

Rather than manually writing to record data during inspections, inspectors can speak about what they’re verifying and the data gets stored instantly. Similarly, through dissecting speech, aiOla can help with predictive maintenance of essential machinery, allowing food manufacturers to produce safer items and decrease downtime.

Since aiOla’s speech recognition platform understands over 100 languages and countless accents, dialects, and industry-specific jargon, the system is highly accurate and can help turn speech into action to go a step further and automate otherwise manual tasks.

Embracing Speech Recognition Technology

Looking ahead, we can only expect the technology that relies on speech recognition to improve and become more embedded into our day-to-day. Indeed, the market for this technology is expected to grow to $19.57 billion by 2030 . Whether it’s refining virtual assistants, improving voice search, or applying speech recognition to new industries, this technology is here to stay and enhance our personal and professional lives.

aiOla, while also a relatively new technology, is already making waves in industries like manufacturing, fleet management, and food safety. Through technological advancements in speech recognition, we only expect aiOla’s capabilities to continue to grow and support a larger variety of businesses and organizations.

Schedule a demo with one of our experts to see how aiOla’s AI speech recognition platform works in action.

What is speech recognition software? Speech recognition software is a technology that enables computers to convert speech into written words. This is done through algorithms that analyze audio signals along with AI, ML, and other technologies. What is a speech recognition example? A relatable example of speech recognition is asking a virtual assistant like Siri on a mobile device to check the day’s weather or set an alarm. While speech recognition can complete a lot more advanced tasks, this exemplifies how this technology is commonly used in everyday life. What is speech recognition in AI? Speech recognition in AI refers to how artificial intelligence processes are used to aid in recognizing voice and language using advanced models and algorithms trained on vast amounts of data. What are some different types of speech recognition? A few different types of speech recognition include speaker-dependent and speaker-independent systems, command and control systems, and continuous speech recognition. What is the difference between voice recognition and speech recognition? Speech recognition converts spoken language into text, while voice recognition works to identify a speaker’s unique vocal characteristics for authentication purposes. In essence, voice recognition is more tied to identity rather than transcription.

Ready to put your speech in motion? We’re listening.

Share your details to schedule a call

We will contact you soon!

Dictate text using Speech Recognition

On Windows 11 22H2 and later, Windows Speech Recognition (WSR) will be replaced by voice access starting in September 2024. Older versions of Windows will continue to have WSR available. To learn more about voice access, go to Use voice access to control your PC & author text with your voice .

You can use your voice to dictate text to your Windows PC. For example, you can dictate text to fill out online forms; or you can dictate text to a word-processing program, such as WordPad, to type a letter.

Dictating text

When you speak into the microphone, Windows Speech Recognition converts your spoken words into text that appears on your screen.

 To dictate text

Say "start listening" or click the Microphone button to start the listening mode.

Open the program you want to use or select the text box you want to dictate text into.

Say the text that you want dictate.

Correcting dictation mistakes

There are several ways to correct mistakes made during dictation. You can say "correct that" to correct the last thing you said. To correct a single word, say "correct" followed by the word that you want to correct. If the word appears more than once, all instances will be highlighted and you can choose the one that you want to correct. You can also add words that are frequently misheard or not recognized by using the Speech Dictionary.

To use the Alternates panel dialog box

Do one of the following:

To correct the last thing you said, say "correct that."

To correct a single word, say "correct" followed by the word that you want to correct.

In the Alternates panel dialog box, say the number next to the item you want, and then "OK."  

Note:  To change a selection, in the Alternates panel dialog box, say "spell" followed by the number of the item you want to change, and then "OK."

To use the Speech Dictionary

Say "open Speech Dictionary."

Do any of the following:

To add a word to the dictionary, click or say Add a new word , and then follow the instructions in the wizard.

To prevent a specific word from being dictated, click or say Prevent a word from being dictated , and then follow the instructions in the wizard.

To correct or delete a word that is already in the dictionary, click or say Change existing words , and then follow the instructions in the wizard.

Note:  Speech Recognition is available only in English, French, Spanish, German, Japanese, Simplified Chinese, and Traditional Chinese.

Need more help?

Want more options.

Explore subscription benefits, browse training courses, learn how to secure your device, and more.

Microsoft 365 subscription benefits

Microsoft 365 training

Microsoft security

Accessibility center

Communities help you ask and answer questions, give feedback, and hear from experts with rich knowledge.

Ask the Microsoft Community

Microsoft Tech Community

Windows Insiders

Microsoft 365 Insiders

Find solutions to common problems or get help from a support agent.

Online support

Was this information helpful?

Thank you for your feedback.