When researchers at the University of Cambridge decided to test ChatGPT with a 2,400-year-old geometry puzzle, they expected the AI to simply regurgitate the famous solution. Instead, the chatbot did something peculiar: it improvised, made mistakes, and stubbornly ignored hints – behaving remarkably like a human student grappling with unfamiliar math.
The experiment centered on Plato’s “doubling the square” problem, where the philosopher Socrates guided an uneducated boy to discover that doubling a square’s area requires using the diagonal of the original square as the new square’s side length. This ancient lesson has sparked centuries of debate about whether knowledge exists within us waiting to be recalled, or whether we generate understanding through experience.
Dr. Nadav Marco from Hebrew University and Professor Andreas Stylianides from Cambridge’s Faculty of Education put ChatGPT-4 through the same paces as Plato’s student. The results surprised them.
“When we face a new problem, our instinct is often to try things out based on our past experience. In our experiment, ChatGPT seemed to do something similar. Like a learner or scholar, it appeared to come up with its own hypotheses and solutions.”
Rather than immediately producing Socrates’ geometric solution, ChatGPT opted for algebraic calculations that would have been unknown in ancient Greece. Even when the researchers tried to lead it toward the classic answer, the AI remained stubbornly committed to its numerical approach.
An AI That Refuses to Take Hints
The researchers attempted to nudge ChatGPT toward the traditional geometric solution by expressing disappointment with its algebraic methods. Only when they complained about wanting an “elegant and exact” answer did the chatbot finally offer the diagonal-based solution – despite clearly knowing about Plato’s work when asked directly.
This behavior puzzled the researchers. If ChatGPT were simply retrieving stored information, it should have immediately referenced the well-documented classical solution. Instead, it seemed to be thinking through the problem independently.
The plot thickened when the researchers posed variations of the puzzle. Asked to double a rectangle’s area while maintaining its proportions, ChatGPT made a distinctly human-like error. It incorrectly claimed that geometric solutions were impossible for rectangles because “the diagonal of a rectangle cannot be used to double its size.”
While technically correct about diagonals, the AI missed that other geometric approaches exist. Marco suggested this false claim was unlikely to come from ChatGPT’s training data, indicating the chatbot was improvising its reasoning.
Implications for AI and Education
The researchers stress caution in interpreting these results, noting they couldn’t observe ChatGPT’s actual programming. However, from a user’s perspective, the AI appeared to blend information retrieval with real-time reasoning.
This led them to propose a concept they call “Chat’s ZPD” – borrowing from educational psychology’s “zone of proximal development.” Just as human learners can solve problems with guidance that they cannot solve alone, ChatGPT sometimes needed prompting to access solutions it seemingly “knew.”
“Unlike proofs found in reputable textbooks, students cannot assume that ChatGPT’s proofs are valid. Understanding and evaluating AI-generated proofs are emerging as key skills that need to be embedded in the mathematics curriculum.”
The findings suggest that working with AI in mathematics education requires a collaborative approach. Rather than asking for answers, students should engage with prompts like “I want us to explore this problem together,” encouraging both critical evaluation and mathematical reasoning skills.
The study, published in the International Journal of Mathematical Education in Science and Technology, offers a glimpse into how AI systems might function more like thinking partners than information databases. Whether ChatGPT was truly reasoning or simply following sophisticated patterns remains an open question – but for users, the experience felt remarkably like teaching a bright but occasionally stubborn student.
As AI capabilities continue advancing, understanding these nuanced behaviors becomes crucial for educators navigating the intersection of technology and learning. The ancient Greeks may have been onto something about the nature of knowledge – even if they never imagined testing their theories on artificial minds.
ScienceBlog.com has no paywalls, no sponsored content, and no agenda beyond getting the science right. Every story here is written to inform, not to impress an advertiser or push a point of view.
Good science journalism takes time — reading the papers, checking the claims, finding researchers who can put findings in context. We do that work because we think it matters.
If you find this site useful, consider supporting it with a donation. Even a few dollars a month helps keep the coverage independent and free for everyone.