How AI Tools Actually Work (And Why They Still Think 2+2=5 Sometimes)

By Katherine McKean, Junior and President of my high school AI Exploration Club

Living in the Bay Area, you hear a lot of wild things about AI. Like, “It’s going to revolutionize education,” or “My kid used it to write a sonnet in the voice of Steve Jobs.” Cool. But the first time I asked ChatGPT to check my math homework, it told me the square root of 81 was 8. Then doubled down on it. So let’s talk about what these tools actually do—and why they sometimes act like confident liars in a group project.

what even is a large language model?

Large language models (LLMs) are basically next-level autocomplete. Like the Notes app version of someone who has read every conversation ever posted on Reddit, every Wikipedia article, and way too many startup blog posts. They don’t understand language—they just predict what should come next.

So when you type, “The mitochondria is the…”, it knows “powerhouse of the cell” probably comes next. But if you type, “The mitochondria is located in… South San Francisco?” it’ll nod and say, “Sounds right.” That’s when you know you’re on your own.

why 2+2 equals 5 (sometimes)

AI is not a calculator. It doesn’t actually “do” math—it recognizes math-looking patterns. So if it’s seen a lot of text that mistakenly says 2+2=5, that might sound statistically reasonable. And LLMs are all about what sounds right, not what is right.

Case in point: I once asked it to help solve a physics problem about velocity. It confidently told me the final answer was “6 giraffes per hour.” Then moved on like nothing had happened. I’m all for creative units, but still.

how it’s trained (and why it sometimes gets weird)

These tools are trained on massive amounts of internet text—everything from academic papers to fan fiction. During training, they learn patterns in how words and phrases appear next to each other. That’s it. They’re not Googling answers. They’re not consulting databases. They’re completing sentences.

When I asked it to explain the causes of the Great Depression, it gave a great summary. But then it blamed it on the lack of avocado toast. Someone out there clearly joked about that online, and now it lives forever in training data. Great.

homework help… kind of

These tools are weirdly helpful. They’ll write an intro paragraph, outline your essay, and suggest ways to make your science fair project sound more dramatic. But they’ll also throw in a few hallucinated facts. Like the time Claude said Ada Lovelace invented the Chromebook. Good effort.

Here’s my strategy: I use them for ideas, not answers. I double-check math. I rewrite anything that sounds like it was written by a very polite robot. And I always keep a calculator nearby.

remembering stuff (sort of)

Memory with AI is hit or miss. ChatGPT remembers previous conversations if you let it, but sometimes forgets halfway through explaining something. Claude forgets everything once you hit refresh. Gemini forgets mid-sentence and redirects to Wikipedia like it’s trying to avoid a breakup.

I tried to teach it that I live near Berkeley. It responded by suggesting vegan cafes in Utah. Close enough?

when ai gets emotional (not really)

I once typed, “I just bombed my math final,” and asked for advice. ChatGPT told me I was a resilient learner on a growth journey. Claude suggested mindfulness breathing. Gemini said, “That’s unfortunate,” and linked me to a Harvard Business Review article about failure. Touching.

These bots don’t feel emotions—they just simulate empathy based on how people talk about it. So they’ll give you comfort words, but also might respond to “I failed my test” with “Have you considered baking?” It’s hit or miss.

weirdest ai fail we’ve seen

During an AI Club challenge, we asked ChatGPT to write a rap battle between Isaac Newton and a Roomba. The Newton lines were historically solid. The Roomba only beeped and said “I clean, therefore I am.” 10/10 entertainment. Zero actual logic.

how we explain it in club

We tell new members this: think of AI as your overconfident cousin who once read an encyclopedia and now wants to explain everything at dinner. It’s not always right, but it sounds right. That’s the magic—and the risk.

In our meetings, we test prompts like: “Write a monologue from a frustrated electric car stuck in Bay Area traffic.” Or: “Debate me on why cats are the true founders of Silicon Valley.” It’s a mix of chaos, learning, and fact-checking. Lots of fact-checking.

why knowing how it works actually helps

When you understand how LLMs actually work, you stop treating them like answer machines. You start using them as assistants, collaborators, idea generators. And when they fail, you know why—and you know how to steer them back.

Also, you develop a pretty solid radar for AI-written content. Like that time my classmate turned in a “personal reflection” that started with, “As a conscious being navigating the complexities of existence…” Yeah, we knew.

when in doubt, cross-check

If AI says something that sounds fishy, don’t panic. Cross-check it with a trusted source. Ask your teacher. Google it. Or just ask the club—we’ve seen every mistake these tools make, and we keep a running tally.

Also, don’t ask AI to plan your college essay. It tends to use phrases like “pursuing excellence in all endeavors,” which sounds like it came from a motivational poster in a dentist’s office.

Bottom line? AI is a tool. A cool, weird, occasionally wrong tool. Learn how to use it—and when to put it down.

Want to bring the power of AI to your school? Check out this step-by-step guide on How to Launch a High School AI Club in 10 Easy Steps.