Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods

Original Paper: https://arxiv.org/abs/2210.07321

By: Evan Crothers, Nathalie Japkowicz, Herna Viktor

Abstract:

Machine generated text is increasingly difficult to distinguish from human authored text. Powerful open-source models are freely available, and user-friendly tools that democratize access to generative models are proliferating. ChatGPT, which was released shortly after the first edition of this survey, epitomizes these trends. The great potential of state-of-the-art natural language generation (NLG) systems is tempered by the multitude of avenues for abuse. Detection of machine generated text is a key countermeasure for reducing abuse of NLG models, with significant technical challenges and numerous open problems. We provide a survey that includes both 1) an extensive analysis of threat models posed by contemporary NLG systems, and 2) the most complete review of machine generated text detection methods to date. This survey places machine generated text within its cybersecurity and social context, and provides strong guidance for future work addressing the most critical threat models, and ensuring detection systems themselves demonstrate trustworthiness through fairness, robustness, and accountability.

Summary Notes

In the world of artificial intelligence (AI), distinguishing between text written by humans and machines is getting harder. Machine-generated (MG) text, created by advanced technologies like natural language generation (NLG), presents new challenges in cybersecurity and information integrity.

This post covers the dangers of MG text, looks at the threats it poses, examines how we can detect it, and discusses ways to deal with these emerging issues.

The Dangers of Machine-Generated Text

MG text is causing concern due to its ability to:

• Phishing and Fraud: Create convincing scam emails that slip past usual detection methods.
• Disinformation: Spread false information widely to sway public opinion.
• Fraudulent Reviews: Generate fake reviews en masse, misleading consumers.
• Academic Dishonesty: Produce essays or reports that compromise academic standards.
• Toxic Spam: Overwhelm platforms with harmful content.

The careful sharing of models like GPT-3 shows the industry's awareness of these potential misuses.

Understanding the Survey

Previous surveys have looked at NLG models or MG text aspects separately, but a comprehensive review including threat models and detection techniques is missing.

This survey addresses that gap, considering sociotechnical and human-centric factors in line with the EU's Trustworthy AI principles.

Defining Machine-Generated Text

MG text refers to any text produced or altered by machines, mainly focusing on natural language outputs like automated news or fake reviews, but not including code generation.

Evolution of Natural Language Generation

The development of NLG from the early Turing test to today's advanced Transformer models has made it possible to create text that closely mimics human writing.

These technologies power a variety of applications, from summarization to dialogue systems.

Threat Models

MG text can be used maliciously in several ways:

• Malicious Chatbots: Trick users into revealing sensitive information or spread false information.
• Spamming Attacks: Generate vast amounts of spam that dodge usual filters.
• Mass Fake Reviews: Damage the credibility of platforms with bogus reviews.
• Political Disinformation Campaigns: Affect politics or public opinion with fabricated stories.

This part reviews the potential attackers, their tactics, and how to counteract them.

Detecting Machine-Generated Text

Detecting MG text ranges from basic feature checks to sophisticated AI models. This section looks at:

• Effectiveness and Limitations: The success rate of different techniques and their limits.
• Application Areas: Where these detection methods work best.
• Challenges: The hurdles in identifying MG text across various languages and contexts.

Future Challenges and Directions

The ongoing development of MG text and its detection requires continuous research to tackle new threats and enhance detection capabilities. Creating detection systems that are effective, fair, and respect privacy is crucial.

Conclusion

NLG technologies have great potential for good, from streamlining tasks to creating entertainment. Yet, their risks must be managed carefully.

A balanced strategy that promotes responsible use and oversight of these technologies is vital for maximizing their positive impact while minimizing dangers.

This survey underlines the need for collaborative efforts from multiple disciplines to ensure NLG technologies are used ethically and securely, emphasizing the importance of joint action in navigating the complexities of machine-generated text.