SqueakyCleanText

A comprehensive text cleaning and preprocessing pipeline for machine learning and NLP tasks.

In the world of machine learning and natural language processing, clean and well-structured text data is crucial for building effective downstream models and managing token limits in language models.

SqueakyCleanText simplifies the process by automatically addressing common text issues, ensuring your data is clean and well-structured with minimal effort on your part.

Key Features

• Encoding Issues: Corrects text encoding problems and handles bad Unicode characters.
• HTML and URLs: Removes or replaces HTML tags and URLs with configurable tokens.
• Contact Information: Handles emails, phone numbers, and other contact details with customizable replacement tokens.
• Named Entity Recognition (NER):
  • Multi-language support (English, Dutch, German, Spanish)
  • Ensemble voting technique for improved accuracy
  • Configurable confidence thresholds
  • Efficient batch processing
  • Automatic text chunking for long documents
  • GPU acceleration support
• Text Normalization:
  • Removes isolated letters and symbols
  • Normalizes whitespace
  • Handles currency symbols
  • Year detection and replacement
  • Number standardization
• Language Support:
  • Automatic language detection
  • Language-specific NER models
  • Language-aware stopword removal
• Dual Output Formats:
  • Language Model format (preserves structure with tokens)
  • Statistical Model format (optimized for classical ML)
• Performance Optimization:
  • Batch processing support
  • Configurable batch sizes
  • Memory-efficient processing of large texts
  • GPU memory management

Benefits

For Language Models

• Maintains text structure while anonymizing sensitive information
• Configurable token replacements
• Preserves context while removing noise
• Handles long documents through intelligent chunking

For Statistical Models

• Removes stopwords and punctuation
• Case normalization
• Special symbol removal
• Optimized for classification tasks

Advanced NER Processing

• Ensemble approach reduces missed entities
• Language-specific models improve accuracy
• Confidence thresholds for precision control
• Efficient batch processing for large datasets
• Automatic handling of long documents

Installation

pip install SqueakyCleanText

Usage

Basic Usage

from sct import sct, config

# Initialize the TextCleaner
cleaner = sct.TextCleaner()

# Input text
text = "Contact John Doe at [email protected]. Meeting on 2023-10-01."

# Process the text
lm_text, stat_text, lang = cleaner.process(text)

print(f"Language Model format:    {lm_text}")
# Output: "Contact <PERSON> at <EMAIL>. Meeting on <DATE>."

print(f"Statistical Model format: {stat_text}")
# Output: "contact meeting"

print(f"Detected Language: {lang}")
# Output: "ENGLISH"

Advanced Configuration

from sct import sct, config

# Customize NER settings
config.CHECK_NER_PROCESS = True
config.NER_CONFIDENCE_THRESHOLD = 0.85
config.POSITIONAL_TAGS = ['PER', 'LOC', 'ORG']

# Customize replacement tokens
config.REPLACE_WITH_URL = "<URL>"
config.REPLACE_WITH_EMAIL = "<EMAIL>"
config.REPLACE_WITH_PHONE_NUMBERS = "<PHONE>"

# Set known language (skips detection)
config.LANGUAGE = "ENGLISH"  # Options: ENGLISH, DUTCH, GERMAN, SPANISH

# Initialize with custom settings
sx = sct.TextCleaner()

Batch Processing with Custom Configuration**

from sct import sct, config

# Customize configuration
config.CHECK_REMOVE_STOPWORDS = True
config.CHECK_REMOVE_PUNCTUATION = True
config.CHECK_NER_PROCESS = True
config.POSITIONAL_TAGS = ['PERSON', 'ORG', 'LOC']
config.NER_CONFIDENCE_THRESHOLD = 0.90

# Initialize the TextCleaner with custom config
cleaner = sct.TextCleaner()

# Sample texts
texts = [
    "Email [email protected] for more info.",  # Spanish
    "Besuchen Sie uns im Büro in Berlin.",           # German
    "Voor vragen, bel +31 20 123 4567.",             # Dutch
]

# Process texts in batch
results = cleaner.process_batch(texts, batch_size=2)

for lm_text, stat_text, lang in results:
    print(f"Language: {lang}")
    print(f"LM Format:    {lm_text}")
    print(f"Stat Format:  {stat_text}")
    print("-" * 40)

API

sct.TextCleaner

process(text: str) -> Tuple[str, str, str]

Processes the input text and returns a tuple containing: - Cleaned text formatted for language models. - Cleaned text formatted for statistical models (stopwords removed). - Detected language of the text.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request or open an issue.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgements

The package took inspirations from the following repo:

• clean-text