Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration

Original Paper: https://arxiv.org/abs/2311.06062

By: Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, Tao Jiang

Abstract:

Membership Inference Attacks (MIA) aim to infer whether a target data record has been utilized for model training or not. Prior attempts have quantified the privacy risks of language models (LMs) via MIAs, but there is still no consensus on whether existing MIA algorithms can cause remarkable privacy leakage on practical Large Language Models (LLMs). Existing MIAs designed for LMs can be classified into two categories: reference-free and reference-based attacks. They are both based on the hypothesis that training records consistently strike a higher probability of being sampled. Nevertheless, this hypothesis heavily relies on the overfitting of target models, which will be mitigated by multiple regularization methods and the generalization of LLMs. The reference-based attack seems to achieve promising effectiveness in LLMs, which measures a more reliable membership signal by comparing the probability discrepancy between the target model and the reference model. However, the performance of reference-based attack is highly dependent on a reference dataset that closely resembles the training dataset, which is usually inaccessible in the practical scenario. Overall, existing MIAs are unable to effectively unveil privacy leakage over practical fine-tuned LLMs that are overfitting-free and private. We propose a Membership Inference Attack based on Self-calibrated Probabilistic Variation (SPV-MIA). Specifically, since memorization in LLMs is inevitable during the training process and occurs before overfitting, we introduce a more reliable membership signal, probabilistic variation, which is based on memorization rather than overfitting. Furthermore, we introduce a self-prompt approach, which constructs the dataset to fine-tune the reference model by prompting the target LLM itself. In this manner, the adversary can collect a dataset with a similar distribution from public APIs.

Summary Notes

A New Strategy for Improving Attacks on Language Model Privacy

Large Language Models (LLMs) like GPT and LLaMA have transformed how machines create human-like text, leading to innovations in chatbots and content creation.

However, their ability to potentially memorize and expose training data raises privacy concerns.

This post introduces a cutting-edge method, Self-calibrated Probabilistic Variation (SPV-MIA), which aims to increase the effectiveness of Membership Inference Attacks (MIAs) by exploiting LLMs' tendency to memorize data.

Exploring LLM Privacy Risks

LLMs are a boon and a bane. While they've enabled significant advancements across various sectors, they also pose privacy risks, especially when they inadvertently memorize and leak sensitive data from their training sets.

Traditional MIAs, which focus on detecting model overfitting, have found it challenging to effectively target LLMs, leading to the need for more advanced techniques.

Introducing SPV-MIA: A Novel Approach

SPV-MIA offers a breakthrough in conducting MIAs against LLMs through two main innovations:

• Probabilistic Variation: This new metric detects when a model has memorized specific data points by looking for peaks in the probability distributions it generates, moving beyond the reliance on overfitting.
• Self-Prompt Technique: By using the LLM to create a dataset that resembles its training data, SPV-MIA can compare the model's behavior on known versus unknown data, helping to identify what it has learned by heart.

Testing and Results

The effectiveness of SPV-MIA was tested on various LLMs and datasets, showing an average improvement of 23.6% in identifying membership over existing methods.

These tests highlight the crucial role of the reference model's quality, with better results achieved when the reference data closely matches the training data.

Why SPV-MIA Excels

SPV-MIA outperforms traditional MIAs for several reasons:

• Focus on Memorization: Directly targeting memorization allows SPV-MIA to bypass the challenges of overfitting mitigation that limit conventional MIAs.
• Innovative Reference Data Creation: Generating reference data without needing the original training set makes SPV-MIA more practical and effective.
• Reliable Metric: Probabilistic variation is a robust metric for detecting membership, indicating whether a data point was part of the training set.

Future Directions and Implications

SPV-MIA's development emphasizes the need for awareness and mitigation strategies against privacy risks in LLMs. Future research could extend SPV-MIA's application to various models and refine the technique for different scenarios.

In summary, SPV-MIA represents a significant step forward in protecting privacy in the age of advanced language models by offering a more precise way to determine if specific data was used in training.

This advancement underscores the ongoing need for improved privacy measures in AI development.