A large language model, or LLM, is a software system trained to predict what text should come next given text that came before. That is its core function: next-word prediction, applied at enormous scale.
To build one, a company collects a vast corpus of text, hundreds of billions of words drawn from websites, books, articles, and other written sources, and feeds it through a mathematical architecture that learns statistical patterns: which words tend to follow which other words, in which contexts, with what frequency. IBM's technical documentation describes the result as a system that 'works as a giant statistical prediction machine that repeatedly predicts the next word in a sequence.'1
The word 'large' refers to scale. Modern LLMs contain billions of numerical weights, the parameters that encode all the patterns learned during training. GPT-4 is estimated to have hundreds of billions of parameters. Those parameters are fixed after training; the model does not continue learning from the conversations it has with users unless its developers explicitly retrain it.
When a student types a question into a chatbot, the LLM does not look anything up. It does not search a database of verified facts. It generates a response by predicting, word by word, what a plausible continuation of that text would look like, based on the patterns it absorbed during training.
Stanford University's IT department, in its AI Demystified series for non-technical users, describes this plainly: LLMs 'rely on patterns in the data rather than genuine comprehension, which can lead to plausible but incorrect or nonsensical outputs.'2 The model has no mechanism for distinguishing between something it knows to be true and something that merely sounds right given the surrounding words.
This is the origin of what researchers call hallucination: the tendency of LLMs to produce fluent, confident-sounding text that is factually wrong. The model is not lying. It has no concept of lying. It is completing a pattern, and sometimes the pattern it completes does not correspond to anything real.
The hallucination problem is not abstract for schools. A 2024 peer-reviewed study by researchers at the J.D. Williams Library at the University of Mississippi examined suspect citations submitted with freshman-level student papers and to the library help desk. Most of the flagged citations were plausible-sounding but fabricated references that combined real-sounding author names, journal titles, and dates into sources that did not exist.3
Students using AI to help with research are frequently working with a tool that cannot tell them when it is making something up. The output looks authoritative. It uses the correct formatting. It sounds like something a source might say. A student who does not already know enough to evaluate the claim has no way to tell the difference without checking independently.
This is a specific, teachable limitation, not a reason to ban the tools. But it is a reason to ensure that students understand what these tools are before they use them. An LLM is not a search engine. It is not a database. It is a very sophisticated pattern-completion system, and its outputs require the same critical evaluation that students are taught to apply to any other source.
Most AI products marketed to schools are built on top of a large language model, either one developed in-house or accessed through an API from a provider like OpenAI, Google, or Anthropic. The vendor's interface, safety filters, and content policies sit on top of that underlying model, but they cannot change how the model fundamentally generates text.
When evaluating any AI tool for classroom use, it is worth asking two questions that follow directly from the architecture. First: what happens when this tool is wrong? Does it flag uncertainty, or does it present all outputs with equal confidence? Second: are students expected to verify the tool's outputs independently, and does the curriculum account for that step?
The districts handling AI well are the ones where those questions have answers. Understanding how the technology works is what makes it possible to ask them.
