Probabilistic Discoverable Extraction is a method designed to measure the memorization of training data in large language models (LLMs). Traditional discoverable extraction methods split a training example into a prefix and suffix, prompting the LLM with the prefix to see if it can generate the matching suffix using greedy sampling. However, this approach is unreliable due to the non-determinism in more realistic sampling schemes. Probabilistic Discoverable Extraction addresses this by considering multiple queries to quantify the probability of extracting a target sequence, providing more nuanced information about extraction risk. This method evaluates across different models, sampling schemes, and training-data repetitions, offering a more comprehensive understanding of memorization in LLMs.
Probabilistic extraction
Large Language Models
Custom datasets for memorization evaluation
Extraction probability, memorization risk
Research environments
No
Yes
Measures memorization risk, probabilistic approach
No
Standard computing resources
Linux, Windows, macOS
Compatible with various LLMs
Focus on data privacy
N/A
N/A
Yes
Research community
N/A
Varies based on evaluation
Depends on model size
Standard for LLMs
Provides insights into memorization
Data privacy concerns
Relies on probabilistic measures
AI research
Evaluating LLM memorization
Researchers
Integrates with LLMs
Scalable with model size
Community support
N/A
Command-line
No
N/A
Open-source
Yes
Research institutions
N/A
N/A
N/A
Research tool
No
N/A
Open-source
0.00
N/A
Open-source
01/01/1970
01/01/1970
N/A
N/A
Yes