Natural language processing, usually shortened to NLP, is the branch of computing that lets software read, interpret and produce human language. It sits behind spell-checkers, search engines, translation tools, voice assistants and the chatbots described in the guide to rule-based vs AI chatbots.
Language is hard for computers because it is ambiguous, full of context and constantly changing. NLP is the set of methods used to turn that mess into something a program can work with.
Why language is difficult
Take the sentence “I saw her duck.” It could mean you saw her lower her head, or saw a bird belonging to her. Words have several meanings, sentences can be parsed in different ways, and meaning often depends on what was said before, on tone or on shared knowledge. Add spelling variants, slang, typos and many languages, and the scale of the problem is clear.
Computers store text as numbers, so every NLP system must first decide how to represent words as numbers in a way that keeps their meaning.
Core steps in a classic pipeline
Older systems broke the task into stages. Many modern systems still use some of them.
- Tokenisation: splitting text into units called tokens, such as words, parts of words or punctuation marks. “Don’t stop” might become “Do”, “n’t”, “stop”, though conventions vary between tools. Modern models often use sub-word tokens, so a rare word is split into familiar pieces.
- Normalisation: lowercasing, removing stray symbols or handling accents.
- Stemming and lemmatisation: reducing words to a base form. Stemming chops endings crudely, so “studies” may become “studi”; lemmatisation uses a dictionary to return “study”.
- Part-of-speech tagging: labelling words as nouns, verbs and so on.
- Parsing: working out grammatical structure.
- Named entity recognition: finding names of people, organisations, places or dates.
From counting words to learning meaning
Early statistical methods represented a document as a count of its words, ignoring order. That is simple and works for tasks like sorting spam, but it cannot tell that “good” and “excellent” are close in meaning.
A major step forward came with word embeddings, which map each word to a list of numbers so that words used in similar contexts end up close together. Models such as word2vec, published by researchers at Google in 2013, popularised this.
The next leap was the transformer architecture, introduced in the 2017 paper “Attention Is All You Need” by Vaswani and colleagues. Instead of reading text strictly word by word, transformers use a mechanism called attention to weigh how much each token relates to every other token in the passage. Large language models (LLMs) are built on this design and trained on very large text collections to predict the next token. This simple training aim leads to systems that can summarise, translate, answer questions and write code, though how well they do each varies.
Common NLP tasks
| Task | What it does | Example |
|---|---|---|
| Classification | Assigns a label to text | Marking an email as spam |
| Sentiment analysis | Estimates opinion | Rating a review as positive or negative |
| Machine translation | Converts between languages | English to Spanish |
| Summarisation | Shortens long text | Condensing a report |
| Question answering | Finds or writes an answer | A help assistant |
| Speech recognition | Turns audio into text | Voice dictation |
How NLP fits into chatbots
A chatbot needs to interpret the user’s message and decide what to do. In older designs, NLP components classified the intent and extracted details. In newer designs, one language model can handle interpretation and reply writing together, often combined with retrieval from trusted documents so that answers are grounded. For a wider view of how this evolved, see the short history of chatbots.
Where NLP goes wrong
- Bias: models learn patterns from their training text, including stereotypes.
- Ambiguity and sarcasm: tone is often invisible in plain text.
- Low-resource languages: tools perform better in languages with a lot of digital text.
- Fabrication: generative models can state false things fluently.
- Domain shift: a model trained on news may stumble on medical or legal writing.
Careful evaluation on realistic examples is the standard remedy.
Frequently asked questions
Is NLP the same as AI?
NLP is a subfield of artificial intelligence and linguistics focused on language. AI is the broader area, covering vision, planning and robotics as well.
What is tokenisation in simple terms?
Tokenisation is cutting text into pieces that a model can process. Depending on the system, the pieces can be whole words, fragments of words or single characters.
Do I need to code to learn NLP?
Not to understand the concepts. To build systems you will usually learn a language such as Python, along with basic statistics, but many tools now allow experiments with little code.
What is the difference between NLP and an LLM?
NLP is the field. A large language model is one type of tool within it, trained on large amounts of text to predict and generate language.
A starting point
NLP is the toolkit that lets software deal with words. Start with tokens and embeddings, then look at how transformers use attention, and test every idea against real examples. The rule-based vs AI chatbots comparison shows how these methods change what a chatbot can do.