When AI Writes Like Your Boss, Trust Becomes the Vulnerability

An email lands in your inbox. The tone is familiar, the phrasing matches your manager’s style down to the occasional comma splice. It asks you to review a document on a shared drive. You click. By the time you realize the sender wasn’t your boss, your password is already logged on a server halfway across the world. This is high-fidelity phishing, and it runs on the same technology that helps you draft presentations.

Researchers from Shanghai Jiao Tong University and East China Normal University have mapped the security landscape around large language models in a review published in Frontiers of Engineering Management (DOI: 10.1007/s42524-025-4082-6). After screening more than 10,000 documents and analyzing 73 key works, the team outlines how fluent text generation has become a dual-use tool: helpful for drafting emails, dangerous when weaponized for impersonation, phishing, and misinformation at scale.

The paper frames the problem across two fronts. One is misuse, where bad actors exploit the model’s fluency to automate fraud. The other is direct attacks on the model itself, including techniques that extract private training data, poison datasets, or manipulate outputs through what the researchers call prompt injection. That last method works like slipping hidden instructions into a conversation, causing the model to ignore its safety rules and follow the attacker’s script instead.

Jailbreaks and Watermarks

On the defense side, the review surveys three main strategies. Parameter processing aims to strip redundant model elements that could serve as attack surfaces. Input preprocessing detects adversarial triggers or paraphrases prompts before they reach the model, no retraining required. Adversarial training, including red-teaming frameworks, simulates attacks in order to harden the system before deployment.

The paper also describes detection tools designed to flag AI-generated text. Semantic watermarking embeds a hidden signature that special software can trace. One tool mentioned, CheckGPT, reportedly identifies model-generated content with 98% to 99% accuracy. These methods act as a silent filter, catching malicious prompts before they turn into harmful outputs.

But technical defenses have a shelf life. The authors argue that attack strategies evolve faster than engineering solutions, especially when those solutions only work in narrow, controlled settings. They call for scalable, low-cost approaches that can adapt across languages and deployment contexts, not just lab environments.

“They argue that hallucination, bias, privacy leakage, and misinformation are social-level risks, not merely engineering problems,” the study authors write.

Trust as Infrastructure

The review’s broader message is that some risks spill beyond code. When LLM outputs are treated as authoritative in healthcare, education, or public services, hallucinations and bias can produce institutional harms. A confidently stated lie can carry weight if it sounds plausible enough, and fluency alone doesn’t guarantee accuracy.

To address that layer, the authors recommend ethical governance alongside technical safeguards. They highlight transparency, verifiable traceability, cross-disciplinary oversight, dataset audits, and public awareness education as necessary components. The goal is to reduce misuse and protect vulnerable populations from systems that generate convincing but unreliable information.

In practical terms, this could shape how societies adopt LLM-based tools. The review points to potential impacts ranging from protecting financial systems against phishing to reducing medical misinformation and maintaining scientific integrity. Red-teaming and watermark-based traceability may become standard deployment practices, not optional add-ons.

The warning embedded in this work is straightforward: the same fluency that makes LLMs useful also makes their failures and their abuse feel convincingly real. That gap between perception and reality is the hidden threat the authors want decision-makers to recognize before trust itself becomes the next exploitable surface.

Frontiers of Engineering Management: 10.1007/s42524-025-4082-6