eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Natural language processing (NLP) is a branch of artificial intelligence (AI) that focuses on computers incorporating speech and text in a manner similar to humans understanding. This area of computer science relies on computational linguistics—typically based on statistical and mathematical methods—that model human language use.
NLP plays an increasingly prominent role in computing—and in the everyday lives of humans. Smart assistants such as Apple’s Siri, Amazon’s Alexa and Microsoft’s Cortana are examples of systems that use NLP.
In addition, various other tools rely on natural language processing. Among them: navigation systems in automobiles; speech-to-text transcription systems such as Otter and Rev; chatbots; and voice recognition systems used for customer support. In fact, NLP appears in a rapidly expanding universe of applications, tools, systems and technologies.
In every instance, the goal is to simplify the interface between humans and machines. In many cases, the ability to speak to a system or have it recognize written input is the simplest and most straightforward way to accomplish a task.
While computers cannot “understand” language the same way humans do, natural language technologies are increasingly adept at recognizing the context and meaning of phrases and words and transforming them into appropriate responses—and actions.
Also see: Top Natural Language Processing Companies
Natural Language Processing: A Brief History
The idea of machines understanding human speech extends back to early science fiction novels. However, the field of natural language processing began to take shape in the 1950s, after computing pioneer Alan Turing published an article titled “Computing Machinery and Intelligence.” It introduced the Turing Test, which provided a basic way to gauge a computer’s natural language abilities.
During the ensuing decade, researchers experimented with computers translating novels and other documents across spoken languages, though the process was extremely slow and prone to errors. In the 1960s, MIT professor Joseph Weizenbaum developed ELIZA, which mimicked human speech patterns remarkably well. Over the next quarter century, the field continued to evolve. As computing systems became more powerful in the 1990s, researchers began to achieve notable advances using statistical modeling methods.
Dictation and language translation software began to mature in the 1990s. However, early systems required training, they were slow, cumbersome to use and prone to errors. It wasn’t until the introduction of supervised and unsupervised machine learning in the early 2000s, and then the introduction of neural nets around 2010, that the field began to advance in a significant way.
With these developments, deep learning systems were able to digest massive volumes of text and other data and process it using far more advanced language modeling methods. The resulting algorithms had become far more accurate and utilitarian.
Also see: Top AI Software
How Does Natural Language Processing Work?
Early NLP systems relied on hard coded rules, dictionary lookups and statistical methods to do their work. They frequently supported basic decision-tree models. Eventually, machine learning automated tasks while improving results.
Today’s natural language processing frameworks use far more advanced—and precise—language modeling techniques. Most of these methods rely on convolutional neural networks (CNNs) to study language patterns and develop probability-based outcomes.
For example, a method called word vectors applies complex mathematical models to weight and relate words, phrases and constructs. Another method called Recognizing Textual Entailment (RTE), classifies relationships of words and sentences through the lens of entailment, contradiction, or neutrality. For instance, the premise “a dog has paws” entails that “dogs have legs” but contradicts “dogs have wings” while remaining neutral to “all dogs are happy.”
A key part of NLP is word embedding. It refers to establishing numerical weightings for words in specific context. The process is necessary because many words and phrases can mean different things in different meanings or contexts (go to a club, belong to a club or swing a club). Words can also be pronounced the same way but mean different things (through, threw or witch, which). There’s also a need to understand idiomatic phrases that do not make sense literally, such as “You are the apple of my eye” or “it doesn’t cut the mustard.”
Today’s models are trained on enormous volumes of language data—in some cases several hundred gigabytes of books, magazines articles, websites, technical manuals, emails, song lyrics, stage plays, scripts and publicly available sources such as Wikipedia. As the deep learning system parse through millions or even billions of combinations—relying on hundreds of thousands of CPU or GPU cores—they analyze patterns, connect the dots and learn semantic properties of words and phrases.
It’s also often necessary to refine natural language processing systems for specific tasks, such as a chatbot or a smart speaker. But even after this takes place, a natural language processing system may not always work as billed. Even the best NLPs make errors. They can encounter problems when people misspell or mispronounce words and they sometimes misunderstand intent and translate phrases incorrectly. In some cases, these errors can be glaring—or even catastrophic.
Today, prominent natural language models are available under licensing models. These include the OpenAI codex, LaMDA by Google, IBM Watson and software development tools such as CodeWhisperer and CoPilot. In addition, some organizations build their own proprietary models.
How is Natural Language Processing Used?
There are a growing array of uses for natural language processing. These include:
Conversational AI. The ability of computers to recognize words introduces a variety of applications and tools. Personal assistants like Siri, Alexa and Microsoft Cortana are prominent examples of conversational AI. They allow humans to make a call from a mobile phone while driving or switch lights on or off in a smart home. Increasingly, these systems understand intent and act accordingly. For example, chatbots can respond to human voice or text input with responses that seem as if they came from another person. What’s more, these systems use machine learning to constantly improve.
Machine translation. There’s a growing use of NLP for machine translation tasks. These include language translations that replace words in one language for another (English to Spanish or French to Japanese, for example). Google Translate and DeepL are examples of this technology. But machine translation can also take other forms. For example, NLP can convert spoken words—either in the form of a recording or live dictation—into subtitles on a TV show or a transcript from a Zoom or Microsoft Teams meeting. Yet while these systems are increasingly accurate and valuable, they continue to generate some errors.
Sentiment analysis. NLP has the ability to parse through unstructured data—social media analysis is a prime example—extract common word and phrasing patterns and transform this data into a guidepost for how social media and online conversations are trending. This capability is also valuable for understanding product reviews, the effectiveness of advertising campaigns, how people are reacting to news and other events, and various other purposes. Sentiment analysis finds things that might otherwise evade human detection.
Content analysis. Another use case for NLP is making sense of complex systems. For example, the technology can digest huge volumes of text data and research databases and create summaries or abstracts that relate to the most pertinent and salient content. Similarly, content analysis can be used for cybersecurity, including spam detection. These systems can reduce or eliminate the need for manual human involvement.
Text and image generation. A rapidly emerging part of natural language processing focuses on text, image and even music generation. Already, some news organizations produce short articles using natural language processing. Meanwhile, OpenAI has developed a tool that generates text and computer code through a natural language interface. Another OpenAI tool, dubbed Dall-E-2, creates high quality images through an NLP interface. Type the words “black cat under a stairway” and an image appears. GitHub Copilot and Amazon CodeWhisperer can auto-complete and auto-generate computer code through natural language.
Also see: Top Data Visualization Tools
NLP Business Use Cases
The use of NLP is increasingly common in the business world. Among the top use cases:
Chatbots and voice interaction systems. Retailers, health care providers and others increasingly rely on chatbots to interact with customers, answer basic questions and route customers to other online resources. These systems can also connect a customer to a live agent, when necessary. Voice systems allow customers to verbally say what they need rather than push buttons on the phone.
Transcription. As organizations shift to virtual meetings on Zoom and Microsoft Teams, there’s often a need for a transcript of the conversation. Services such as Otter and Rev deliver highly accurate transcripts—and they’re often able to understand foreign accents better than humans. In addition, journalists, attorneys, medical professionals and others require transcripts of audio recordings. NLP can deliver results from dictation and recordings within seconds or minutes.
International translation. NLP has revolutionized interactions between businesses in different countries. While the need for translators hasn’t disappeared, it’s now easy to convert documents from one language to another. This has simplified interactions and business processes for global companies while simplifying global trade.
Scoring systems. Natural language is used by financial institutions, insurance companies and others to extract elements and analyze documents, data, claims and other text-based resources. The same technology can also aid in fraud detection, financial auditing, resume evaluations and spam detection. In fact, the latter represents a type of supervised machine learning that connects to NLP.
Market intelligence and sentiment analysis. Marketers and others increasingly rely on NLP to deliver market intelligence and sentiment trends. Semantic engines scrape content from blogs, news sites, social media sources and other sites in order to detect trends, attitudes and actual behaviors. Similarly, NLP can help organizations understand website behavior, such as search terms that identify common problems and how people use an e-commerce site. This data can lead to design and usability changes.
Software development. A growing trend is the use of natural language for software coding. Low-code and no-code environments can transform spoken and written requests into actual lines of software code. Systems such as Amazon’s CodeWhisperer and GitHub’s CoPilot include predictive capabilities that autofill code in much the same way that Google Mail predicts what a person will type next. They also can pull information from an integrated development environment (IDE) and produce several lines of code at a time.
Text and image generation. The OpenAI codex can generate entire documents, based a basic request. This makes it possible to generate poems, articles and other text. Open AI’s DALL-E 2 generates photorealistic images and art through natural language input. This can aid designers, artists and others.
Also see: Best Data Analytics Tools
What Ethical Concerns Exist for NLP?
Concerns about natural language processing are heavily centered on the accuracy of models and ensuring that bias doesn’t occur. Many of these deep learning algorithms are so-called “black boxes,” meaning that there’s no way to understand how the underlying model works and whether it is free of biases that could affect critical decisions about lending, healthcare and more.
There is also debate about whether these systems are “sentient.” The question of whether AI can actually think and feel like a human has been expressed in films such as 2001: A Space Odyssey and Star Wars. It also reappeared in 2022, when former Google data scientist Blake Lemoine published human-to-machine discussions with LaMDA. Lemoine claimed that the system had gained sentience. However, numerous linguistics experts and computer scientists countered that a silicon-based system cannot think and feel the way humans do. It merely parrots language in a highly convincing way.
In fact, researchers who have experimented with NLP systems have been able to generate egregious and obvious errors by inputting certain words and phrases. Getting to 100% accuracy in NLP is nearly impossible because of the nearly infinite number of word and conceptual combinations in any given language.
Another issue is ownership of content—especially when copyrighted material is fed into the deep learning model. Because many of these systems are built from publicly available sources scraped from the Internet, questions can arise about who actually owns the model or material, or whether contributors should be compensated. This has so far resulted in a handful of lawsuits along with broader ethical questions about how models should be developed and trained.
Also see: AI vs. ML: Artificial Intelligence and Machine Learning
What Role Will NLP Play in the Future?
There’s no question that natural language processing will play a prominent role in future business and personal interactions. Personal assistants, chatbots and other tools will continue to advance. This will likely translate into systems that understand more complex language patterns and deliver automated but accurate technical support or instructions for assembling or repairing a product.
NLP will also lead to more advanced analysis of medical data. For example, a doctor might input patient symptoms and a database using NLP would cross-check them with the latest medical literature. Or a consumer might visit a travel site and say where she wants to go on vacation and what she wants to do. The site would then deliver highly customized suggestions and recommendations, based on data from past trips and saved preferences.
For now, business leaders should follow the natural language processing space—and continue to explore how the technology can improve products, tools, systems and services. The ability for humans to interact with machines on their own terms simplifies many tasks. It also adds value to business relationships.
Also see: The Future of Artificial Intelligence