恶意样本分类与聚类识别

---
name: malware-classification
description: Classify malware samples by extracting static and behavioral features, computing similarity scores, clustering related samples, and identifying malware families using machine learning techniques.
---

# Malware Classification

Classify malware samples into families and types by extracting features from binaries, computing similarity scores, clustering related samples, and maintaining a local classification database.

## Prerequisites

- **Python 3.10+** with `json` and standard library modules
- **pefile**: PE file parsing for static feature extraction (`pip install pefile`)
- **ssdeep**: Fuzzy hashing library (`pip install ssdeep`)
- **scikit-learn**: Clustering algorithms (DBSCAN, hierarchical) and ML classifiers (`pip install scikit-learn`)
- **capa**: Capability detection for enriching feature vectors (optional)
- **YARA**: Rule matching as classification features (optional)
- **EMBER model** or **XGBoost**: Pre-trained ML models for family identification (optional)

## Steps

### 1. Extract Static Features from Binaries

Extract feature vectors from PE headers, imports, sections, and other static properties:

```bash
# Extract features from a single sample
python3 scripts/malware_classifier.py --extract-features \
    --input sample.exe \
    --output features.json

# Extract features from a directory of samples
python3 scripts/malware_classifier.py --extract-features \
    --input ./samples/ \
    --output batch_features.json

# Extract specific feature categories
python3 scripts/malware_classifier.py --extract-features \
    --input sample.exe \
    --feature-types imports,sections,header,strings \
    --output features.json
```

Static features extracted:
- **PE header fields**: machine type, timestamp, subsystem, DLL characteristics, entry point
- **Section properties**: names, sizes, entropy, virtual sizes, characteristics flags
- **Import hash (imphash)**: MD5 of ordered import table (used for family clustering)
- **Rich header hash**: Hash of the Visual Studio build metadata
- **Import table**: DLL names, function names, import counts per library
- **Export table**: Exported function names and ordinals
- **Resource section**: Resource types, sizes, languages, entropy
- **String features**: URL patterns, file paths, registry keys, suspicious strings

### 2. Extract Behavioral Features

Extract features from dynamic analysis results (sandbox reports, API traces):

```bash
# Extract behavioral features from a CAPE/Cuckoo JSON report
python3 scripts/malware_classifier.py --extract-behavioral \
    --input sandbox_report.json \
    --output behavioral_features.json

# Extract from an API call trace log
python3 scripts/malware_classifier.py --extract-behavioral \
    --input api_trace.log \
    --format strace \
    --output behavioral_features.json
```

Behavioral features:
- **API call sequences**: Ordered list of API calls with arguments
- **System call traces**: Linux strace/ltrace output parsing
- **Network behavior**: Protocols used, connection patterns, DNS queries
- **File operations**: Files created, modified, deleted, and their paths
- **Registry operations**: Keys created, modified, values set
- **Process behavior**: Child processes spawned, injection targets

### 3. Compute Similarity Between Samples

Compare samples using multiple similarity metrics:

```bash
# Compare two samples
python3 scripts/malware_classifier.py --compare \
    --input sample_a.exe \
    --reference sample_b.exe \
    --output comparison.json

# Compare a sample against a feature database
python3 scripts/malware_classifier.py --compare \
    --input sample.exe \
    --database classification_db.json \
    --top-k 10 \
    --output matches.json
```

Similarity metrics:
- **Imphash match**: Exact import hash match (strong family indicator)
- **SSDeep fuzzy hash**: Context-triggered piecewise hashing for binary similarity
- **Section hash comparison**: Per-section hash matching
- **Import set Jaccard similarity**: Overlap of imported functions
- **String set similarity**: Overlap of extracted strings
- **Feature vector cosine similarity**: Numeric feature vector distance

### 4. Integrate with Existing Analysis Tools

Use capa results and YARA matches as classification features:

```bash
# Use capa results as feature input
capa sample.exe -j > capa_results.json
python3 scripts/malware_classifier.py --extract-features \
    --input sample.exe \
    --capa-results capa_results.json \
    --output features.json

# Use YARA match results as features
yara -s rules.yar sample.exe > yara_matches.txt
python3 scripts/malware_classifier.py --extract-features \
    --input sample.exe \
    --yara-results yara_matches.txt \
    --output features.json
```

This enriches the feature vector with:
- Matched capa capabilities as binary features
- Matched YARA rules as binary features
- ATT&CK technique coverage as a feature vector

### 5. Cluster Similar Samples

Group related samples into clusters for family identification:

```bash
# Cluster samples using DBSCAN on feature vectors
python3 scripts/malware_classifier.py --cluster \
    --input batch_features.json \
    --algorithm dbscan \
    --eps 0.3 \
    --min-samples 2 \
    --output clusters.json

# Hierarchical clustering with dendrogram output
python3 scripts/malware_classifier.py --cluster \
    --input batch_features.json \
    --algorithm hierarchical \
    --distance-threshold 0.5 \
    --output clusters.json
```

Clustering parameters:
- **DBSCAN**: `--eps` (neighborhood radius), `--min-samples` (minimum cluster size)
- **Hierarchical**: `--distance-threshold`, `--linkage` (ward, complete, average, single)
- Feature selection: `--features imphash,imports,sections,strings` to control which features drive clustering

### 6. Use ML Models for Family Identification

Leverage pre-trained models and embeddings for classification:

```bash
# Classify using a local model (EMBER-style feature extraction)
python3 scripts/malware_classifier.py --classify \
    --input sample.exe \
    --model ember_model.pkl \
    --output classification.json

# Generate feature embedding for similarity search
python3 scripts/malware_classifier.py --embed \
    --input sample.exe \
    --output embedding.json
```

Model integration options:
- **EMBER dataset**: Train gradient-boosted models on 2,381 features extracted from PE files
- **MalConv**: Deep learning on raw bytes (requires GPU, good for novel samples)
- **capa + XGBoost**: Use capa capability vectors as input features for a trained classifier
- **Custom models**: Train scikit-learn classifiers on your own labeled sample set

### 7. Build and Maintain a Classification Database

Maintain a local database for ongoing classification:

```bash
# Initialize a new classification database
python3 scripts/malware_classifier.py --init-db \
    --output classification_db.json

# Add a classified sample to the database
python3 scripts/malware_classifier.py --add-to-db \
    --input sample.exe \
    --family "emotet" \
    --type "trojan" \
    --campaign "2026-Q1" \
    --confidence 0.92 \
    --database classification_db.json

# Query the database for similar samples
python3 scripts/malware_classifier.py --query-db \
    --family "emotet" \
    --database classification_db.json

# Export database statistics
python3 scripts/malware_classifier.py --db-stats \
    --database classification_db.json
```

### 8. Handle Unknown and Novel Samples

Score confidence and flag unknown samples:

```bash
# Classify with confidence scoring
python3 scripts/malware_classifier.py --classify \
    --input unknown_sample.exe \
    --database classification_db.json \
    --threshold 0.7 \
    --output classification.json
```

Confidence interpretation:
- **>0.9**: High confidence — strong match to a known family
- **0.7-0.9**: Medium confidence — likely match, manual review recommended
- **0.5-0.7**: Low confidence — partial match, could be variant or new family
- **<0.5**: Unknown — novel sample, no strong match to known families

For unknown samples:
- Flag for manual analyst review
- Compare against multiple databases (VirusTotal, MalwareBazaar)
- Run additional dynamic analysis to extract behavioral features
- Consider if it represents a new family or a significantly modified variant

## Output Format

```json
{
  "sample": {
    "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    "file_name": "sample.exe",
    "file_size": 245760
  },
  "classification": {
    "family": "emotet",
    "type": "trojan",
    "confidence": 0.92,
    "method": "feature_similarity",
    "matched_features": ["imphash", "import_set", "string_patterns"]
  },
  "features": {
    "static": {
      "imphash": "a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6",
      "rich_header_hash": "f0e1d2c3b4a5968778695a4b3c2d1e0f",
      "section_count": 5,
      "section_names": [".text", ".rdata", ".data", ".rsrc", ".reloc"],
      "section_entropies": [6.8, 5.2, 4.1, 3.9, 6.2],
      "import_count": 142,
      "import_dlls": ["kernel32.dll", "advapi32.dll", "ws2_32.dll", "wininet.dll"],
      "suspicious_imports": ["CreateRemoteThread", "VirtualAllocEx", "WriteProcessMemory"],
      "entry_point": 4096,
      "compile_timestamp": "2026-01-15T08:30:00Z"
    },
    "behavioral": {
      "api_call_count": 1247,
      "unique_apis": 89,
      "network_connections": 3,
      "files_created": 5,
      "registry_keys_modified": 2,
      "processes_spawned": 1
    }
  },
  "similar_samples": [
    {
      "sha256": "aabb...",
      "family": "emotet",
      "similarity": 0.95,
      "matching_features": ["imphash", "import_set", "section_hashes"]
    },
    {
      "sha256": "ccdd...",
      "family": "emotet",
      "similarity": 0.88,
      "matching_features": ["import_set", "string_patterns"]
    }
  ],
  "cluster_id": 3,
  "is_novel": false
}
```

## Tips

- Imphash is the single most effective feature for PE family clustering — samples compiled from the same source with the same imports will share an imphash even across recompilations
- Entropy analysis of PE sections quickly identifies packed or encrypted content: sections above 7.0 entropy are likely packed, above 7.9 are nearly random (encrypted or compressed)
- Rich header hashes can link samples built with the same Visual Studio toolchain, even when the code itself differs
- SSDeep fuzzy hashing works best for detecting minor variants (recompiled with small changes) but fails against significant code changes or packing
- When clustering, normalize features to prevent high-cardinality features (like string counts) from dominating the distance metric
- Maintain separate classification databases for different contexts (e.g., one per campaign, one per malware type) to reduce noise
- Always verify ML model classifications against a known-good sample set to measure accuracy before trusting automated results
- For novel malware with no database matches, fall back to behavioral classification: what the sample does matters more than what it looks like
- Combine static and behavioral features for the highest classification accuracy — static features alone miss packed/encrypted samples, behavioral features alone miss environment-aware malware
- Update your classification database regularly with newly analyzed samples to improve future matching accuracy