# Common Encryption and Encoding in Malware

## Overview

Malware uses encryption and encoding to hide strings, configuration data,
communication payloads, and secondary payloads. This reference covers
identification and decryption of commonly encountered schemes.

## Encoding Schemes

### Base64

**Identification:**
- Character set: `A-Z`, `a-z`, `0-9`, `+`, `/`, `=` (padding)
- Length is always a multiple of 4 (with padding)
- Encoded data is ~33% larger than original

**Variants:**
- **Standard**: RFC 4648 alphabet
- **URL-safe**: `+` replaced with `-`, `/` replaced with `_`
- **Custom alphabet**: Any 64-character substitution (common in malware)
- **Modified padding**: Using characters other than `=` or no padding

**Decryption:**
```bash
echo "SGVsbG8gV29ybGQ=" | base64 -d
python3 -c "import base64; print(base64.b64decode('SGVsbG8gV29ybGQ='))"
```

### Hex Encoding

**Identification:**
- Only characters `0-9`, `A-F` (or `a-f`)
- Length is exactly 2x the original data
- May have separators: spaces, colons, dashes

**Decryption:**
```bash
echo "48656c6c6f" | xxd -r -p
python3 -c "print(bytes.fromhex('48656c6c6f'))"
```

### URL Encoding (Percent Encoding)

**Identification:**
- `%XX` patterns where XX are hex digits
- Common in web-based malware, phishing URLs

### Decimal/Char Code Encoding

**Identification:**
- Lists of decimal numbers: `72,101,108,108,111`
- Often seen in JavaScript/VBScript malware
- May use `String.fromCharCode()` or `Chr()` functions

## XOR-Based Encryption

### Single-Byte XOR

**Identification:**
- Encrypted data has unusual byte frequency distribution
- Key can be recovered if plaintext is partially known
- Null bytes in plaintext become the key byte in ciphertext
- Common keys: `0xFF`, `0x41` (`A`), `0x55`, `0xAA`

**Known-plaintext attack:**
If you know the plaintext starts with "MZ" (PE file) or "http":
```python
key = encrypted[0] ^ ord('M')  # If decrypted should start with 'M'
```

**Frequency analysis:**
The most common byte in the ciphertext is likely the key XORed with
null (0x00) or space (0x20).

### Multi-Byte XOR

**Identification:**
- Repeating patterns in ciphertext when key length aligns with data patterns
- Key length can be determined using Kasiski examination or index of coincidence
- Longer keys provide stronger encryption but same fundamental weakness

**Key length detection:**
```python
# Try key lengths 1-32, look for lowest index of coincidence
for key_len in range(1, 33):
    # Split ciphertext into key_len streams
    # Calculate IC for each stream
    # Low IC across all streams = likely key length
```

### Rolling XOR

**Identification:**
- No repeating pattern (unlike multi-byte XOR)
- Key changes with each byte based on previous plaintext or ciphertext
- Common in custom malware encryptors

**Variants:**
1. `key[i+1] = (key[i] + plaintext[i]) & 0xFF`
2. `key[i+1] = (key[i] + ciphertext[i]) & 0xFF`
3. `key[i+1] = ROL(key[i], 1)`
4. `key[i+1] = key[i] ^ plaintext[i]`

### XOR with ADD/SUB

**Identification:**
- XOR combined with addition or subtraction
- Pattern: `plaintext = (ciphertext XOR key1) - key2`

## Stream Ciphers

### RC4 (ARC4)

**Identification in disassembly:**
- 256-byte array initialization (KSA): loop from 0 to 255
- Two index variables (i, j) with modular arithmetic
- Swap operations within a 256-element array
- No block size requirements (encrypts byte by byte)

**Identifying constants:**
```
; KSA initialization
mov ecx, 256
xor eax, eax
init_loop:
    mov [esi+eax], al    ; S[i] = i
    inc eax
    loop init_loop
```

**Common RC4 usage in malware:**
- Configuration decryption (Emotet, TrickBot)
- C2 communication encryption
- Payload decryption

**Key locations:**
- Hardcoded in the binary (look for byte arrays near KSA)
- Derived from system information (hostname, volume serial)
- Received from C2 server
- Embedded in resource sections

### ChaCha20 / Salsa20

**Identification:**
- Constants: `"expand 32-byte k"` or `"expand 16-byte k"`
- Quarter-round operations: add, XOR, rotate
- 64-byte block output
- Used by modern ransomware (e.g., some LockBit variants)

## Block Ciphers

### AES (Advanced Encryption Standard)

**Identification in disassembly:**
- S-box constant: first bytes are `0x63, 0x7C, 0x77, 0x7B, 0xF2, 0x6B, 0x6F, 0xC5`
- Round constant (Rcon): `0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80`
- Fixed block size of 16 bytes
- 10/12/14 rounds for 128/192/256-bit keys
- Windows CryptoAPI calls: `BCryptEncrypt`, `CryptEncrypt`
- Look for `AES` or `Rijndael` strings

**Common modes in malware:**
- **AES-CBC**: Most common; requires IV (often prepended to ciphertext)
- **AES-ECB**: No IV; identical plaintext blocks produce identical ciphertext
- **AES-CTR**: Counter mode; operates as stream cipher
- **AES-GCM**: Authenticated encryption (newer malware)

**Key derivation patterns:**
- Hardcoded key bytes
- SHA256 of a password string
- PBKDF2 derivation
- First 16/32 bytes of a larger buffer

### DES / 3DES

**Identification:**
- 8-byte block size
- Specific permutation tables (IP, FP)
- Legacy, still seen in older malware families
- Windows API: `CryptEncrypt` with `CALG_DES` or `CALG_3DES`

### Blowfish

**Identification:**
- Variable key length (32-448 bits)
- 8-byte block size
- P-array (18 entries) and S-boxes (4 x 256 entries)
- Initial P-array values derived from pi

## Asymmetric Encryption

### RSA

**Identification:**
- Large numbers (1024/2048/4096 bits)
- Public exponent typically 65537 (0x10001)
- Windows API: `CryptEncrypt` with `CALG_RSA_KEYX`
- Often used to encrypt symmetric keys (hybrid encryption)

**In ransomware:**
- Master RSA public key embedded in binary
- Per-victim RSA key pair generated
- File encryption key (AES) encrypted with victim RSA public key
- Victim RSA private key encrypted with master RSA public key

### Elliptic Curve (ECC)

**Identification:**
- Curve parameters (P-256, Curve25519, secp256k1)
- Point multiplication operations
- Smaller key sizes than RSA for equivalent security
- Modern ransomware families (BlackCat/ALPHV)

## Malware-Specific Patterns

### Configuration Encryption

Many malware families encrypt their configuration blobs:

| Family | Encryption | Key Location |
|--------|-----------|-------------|
| Emotet | RC4 / XOR | Hardcoded, per-build |
| TrickBot | RC4 | Embedded in PE resources |
| QakBot | RC4 + XOR | Hardcoded |
| Cobalt Strike | XOR / rolling XOR | In beacon config |
| AgentTesla | AES + Base64 | Hardcoded |
| Formbook | Custom XOR + hashing | Derived from binary |
| Remcos | RC4 | In PE resources |

### Shellcode Encryption

- **XOR stub**: Small XOR decoder prepended to encrypted shellcode
- **Shikata Ga Nai**: Polymorphic XOR encoder (Metasploit)
  - Identified by: FPU instructions for GetEIP, followed by XOR loop
- **Custom stagers**: Download and decrypt second-stage payload

### String Stacking

Not encryption, but an obfuscation technique where strings are built
character-by-character on the stack or in a buffer:

```c
char s[8];
s[0] = 'c'; s[1] = 'm'; s[2] = 'd';
s[3] = '.'; s[4] = 'e'; s[5] = 'x'; s[6] = 'e'; s[7] = 0;
```

**Detection:** Look for sequential byte/word MOV instructions to adjacent
memory addresses. FLOSS tool can detect these automatically.

## Decryption Workflow

1. **Identify the algorithm**: Check for constants, API calls, code patterns
2. **Locate the key**: Hardcoded, derived, or received from C2
3. **Extract encrypted data**: From sections, resources, or overlay
4. **Determine mode/parameters**: IV, block mode, padding scheme
5. **Decrypt offline**: Use `string_decryptor.py` or custom script
6. **Validate result**: Check for readable strings, valid file headers
7. **Document**: Record algorithm, key, and decrypted content for the report