Troubleshooting#

This guide helps you diagnose and resolve common issues with DBSurveyor.

Quick Diagnostics#

Check Installation#

# Verify binaries are installed and working
dbsurveyor-collect --version
dbsurveyor --version

# Check compiled features
dbsurveyor-collect list

# Test with minimal example
echo "CREATE TABLE test (id INTEGER);" | sqlite3 test.db
dbsurveyor-collect sqlite://test.db
rm test.db schema.dbsurveyor.json

Enable Debug Logging#

# Enable debug logging for all modules
export RUST_LOG=debug
dbsurveyor-collect postgres://localhost/db

# Enable trace logging for specific modules
export RUST_LOG=dbsurveyor_collect=trace,dbsurveyor_core=debug
dbsurveyor-collect postgres://localhost/db

# Log to file
export RUST_LOG=debug
dbsurveyor-collect postgres://localhost/db 2> debug.log

Connection Issues#

Database Connection Failures#

Symptoms: Connection timeouts, authentication failures, network errors

PostgreSQL Connection Issues#

# Test basic connectivity
ping localhost
telnet localhost 5432

# Test with psql
psql -h localhost -U user -d db -c "SELECT 1;"

# Check PostgreSQL logs
sudo tail -f /var/log/postgresql/postgresql-*.log

# Common connection string issues
# ❌ Wrong: postgres://user:pass@localhost/db:5432
# ✅ Correct: postgres://user:pass@localhost:5432/db

# SSL issues
dbsurveyor-collect "postgres://user:pass@localhost/db?sslmode=disable"
dbsurveyor-collect "postgres://user:pass@localhost/db?sslmode=require"

MySQL Connection Issues#

# Test basic connectivity
telnet localhost 3306

# Test with mysql client
mysql -h localhost -u user -p -e "SELECT 1;" db

# Check MySQL logs
sudo tail -f /var/log/mysql/error.log

# Common issues
# Character set problems
dbsurveyor-collect "mysql://user:pass@localhost/db?charset=utf8mb4"

# SSL issues
dbsurveyor-collect "mysql://user:pass@localhost/db?ssl-mode=DISABLED"

SQLite Connection Issues#

# Check file exists and is readable
ls -la /path/to/database.db
file /path/to/database.db

# Test with sqlite3 command
sqlite3 /path/to/database.db ".tables"

# Permission issues
chmod 644 /path/to/database.db

# Use absolute paths
dbsurveyor-collect "sqlite:///$(pwd)/database.db"

MongoDB Connection Issues#

# Test basic connectivity
telnet localhost 27017

# Test with mongo client
mongosh "mongodb://user:pass@localhost:27017/db"

# Authentication issues
dbsurveyor-collect "mongodb://user:pass@localhost/db?authSource=admin"

# Replica set issues
dbsurveyor-collect "mongodb://user:pass@host1,host2/db?replicaSet=rs0"

Permission and Authentication Issues#

Symptoms: Access denied, insufficient privileges, authentication failed

PostgreSQL Permissions#

-- Check current user permissions
SELECT current_user, session_user;

-- Check database access
SELECT datname FROM pg_database WHERE datname = 'your_db';

-- Check table permissions
SELECT schemaname, tablename 
FROM pg_tables 
WHERE schemaname NOT IN ('information_schema', 'pg_catalog');

-- Grant necessary permissions
GRANT CONNECT ON DATABASE mydb TO dbsurveyor_user;
GRANT USAGE ON SCHEMA public TO dbsurveyor_user;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO dbsurveyor_user;

MySQL Permissions#

-- Check current user
SELECT USER(), CURRENT_USER();

-- Check permissions
SHOW GRANTS FOR 'dbsurveyor_user'@'%';

-- Grant necessary permissions
GRANT SELECT ON mydb.* TO 'dbsurveyor_user'@'%';
GRANT SELECT ON information_schema.* TO 'dbsurveyor_user'@'%';
FLUSH PRIVILEGES;

MongoDB Permissions#

// Check current user
db.runCommand({
    connectionStatus: 1
})

// Check permissions
use mydb
db.runCommand({
    usersInfo: "dbsurveyor_user"
})

// Grant read permissions
use admin
db.grantRolesToUser("dbsurveyor_user", [{
    role: "read",
    db: "mydb"
}])

Collection Issues#

Schema Collection Failures#

Symptoms: Partial collection, missing objects, collection errors

Missing Tables or Objects#

# Enable verbose logging to see what's being collected
dbsurveyor-collect -vvv postgres://localhost/db

# Check if objects exist in database
psql -h localhost -U user -d db -c "\dt" # PostgreSQL tables
mysql -u user -p -e "SHOW TABLES;" db # MySQL tables
sqlite3 db.sqlite ".tables" # SQLite tables

Large Database Timeouts#

# Increase timeouts
dbsurveyor-collect --connect-timeout 60 --query-timeout 120 postgres://localhost/db

# Use throttling to reduce load
dbsurveyor-collect --throttle 1000 postgres://localhost/db

# Disable sample collection for speed
dbsurveyor-collect --sample 0 postgres://localhost/db

Memory Issues#

# Monitor memory usage
top -p $(pgrep dbsurveyor-collect)

# Use compression to reduce memory usage
dbsurveyor-collect --compress postgres://localhost/db

# Process smaller chunks (multi-database collection)
dbsurveyor-collect --exclude-databases large_db1,large_db2 postgres://localhost

Output File Issues#

Symptoms: Empty files, corrupted output, permission errors

File Permission Issues#

# Check output directory permissions
ls -la $(dirname schema.dbsurveyor.json)

# Ensure write permissions
chmod 755 $(dirname schema.dbsurveyor.json)

# Use explicit output path
dbsurveyor-collect --output /tmp/schema.json postgres://localhost/db

Corrupted Output Files#

# Validate schema file
dbsurveyor validate schema.dbsurveyor.json

# Check file size and format
ls -la schema.dbsurveyor.json
file schema.dbsurveyor.json

# Test JSON parsing
jq . schema.dbsurveyor.json > /dev/null

Compression/Encryption Issues#

# Test compression
dbsurveyor-collect --compress postgres://localhost/db
zstd -t schema.dbsurveyor.json.zst # Test compressed file

# Test encryption (will prompt for password)
dbsurveyor-collect --encrypt postgres://localhost/db
dbsurveyor validate schema.enc # Test encrypted file

Documentation Generation Issues#

Processing Failures#

Symptoms: Generation errors, empty output, format issues

Input File Issues#

# Validate input file first
dbsurveyor validate schema.dbsurveyor.json

# Check file format
file schema.dbsurveyor.json

# Test with minimal processing
dbsurveyor --format json schema.dbsurveyor.json

Encrypted File Issues#

# Verify encryption format
dbsurveyor validate schema.enc

# Test decryption
dbsurveyor --format json schema.enc

# Check password
# Note: Passwords are case-sensitive and don't show characters

Large Schema Processing#

# Use JSON format for large schemas (most efficient)
dbsurveyor --format json large-schema.json

# Monitor memory usage
top -p $(pgrep dbsurveyor)

# Process in parts (planned feature)
# Currently: split large schemas manually

Output Format Issues#

Symptoms: Malformed output, missing content, rendering problems

HTML Generation Issues#

# Test with simple format first
dbsurveyor --format markdown schema.json

# Check HTML output
dbsurveyor --format html schema.json
# Open in browser to check for issues

Mermaid Diagram Issues#

# Generate Mermaid format
dbsurveyor --format mermaid schema.json

# Validate Mermaid syntax
# Copy content to https://mermaid.live/ for testing

Output Formatting and Progress Indicators#

Symptoms: Progress spinners or colored output causing problems, garbled output with escape codes, missing progress indicators

Disabling Colors and Progress Spinners#

For CI/CD pipelines, automation scripts, or non-interactive environments where colored output or progress spinners may cause issues:

# Disable ANSI color codes and progress spinners
export NO_COLOR=1
dbsurveyor-collect postgres://localhost/db
dbsurveyor --format markdown schema.json

# Alternative: Use TERM=dumb
TERM=dumb dbsurveyor-collect postgres://localhost/db
TERM=dumb dbsurveyor --format json schema.json

# For CI environments (typically set automatically)
export NO_COLOR=1
export TERM=dumb
dbsurveyor-collect postgres://localhost/db > collection.log

Common scenarios:

CI/CD pipelines: Set NO_COLOR=1 to prevent ANSI escape codes in logs
Log files: Progress spinners are automatically hidden when output is redirected
Automated scripts: Use TERM=dumb for consistent non-interactive behavior
Windows systems: Some terminals may display escape codes incorrectly

Troubleshooting Progress Indicators#

# Check if output is going to a TTY
# Progress spinners only show when stdout is a terminal
test -t 1 && echo "TTY detected" || echo "Not a TTY"

# Force spinner to be hidden (for debugging)
export NO_COLOR=1
dbsurveyor --format html schema.json

# Check environment variables
echo "NO_COLOR=${NO_COLOR:-not set}"
echo "TERM=${TERM:-not set}"

Note: The tool automatically detects when output is piped or redirected and hides progress indicators. Setting NO_COLOR or TERM=dumb ensures consistent behavior across all environments.

Performance Issues#

Slow Collection#

Symptoms: Long collection times, high resource usage

Database Performance#

# Use connection pooling
dbsurveyor-collect --max-connections 5 postgres://localhost/db

# Add throttling
dbsurveyor-collect --throttle 500 postgres://localhost/db

# Disable sample collection
dbsurveyor-collect --sample 0 postgres://localhost/db

# Monitor database load
# PostgreSQL: SELECT * FROM pg_stat_activity;
# MySQL: SHOW PROCESSLIST;

Network Performance#

# Test network latency
ping database-host

# Use local connections when possible
dbsurveyor-collect postgres://localhost/db # Better than remote

# Consider compression for remote databases
dbsurveyor-collect --compress postgres://remote-host/db

Memory Usage#

Symptoms: High memory consumption, out of memory errors

Memory Optimization#

# Monitor memory usage
ps aux | grep dbsurveyor
top -p $(pgrep dbsurveyor)

# Use streaming for large datasets (automatic)
# Reduce sample size
dbsurveyor-collect --sample 10 postgres://localhost/db

# Use compression
dbsurveyor-collect --compress postgres://localhost/db

Security Issues#

Credential Exposure#

Symptoms: Passwords in logs, credential leakage

Verify Credential Sanitization#

# Check logs for credentials
export RUST_LOG=debug
dbsurveyor-collect postgres://user:secret@localhost/db 2>&1 | grep -i secret
# Should return no results

# Test credential sanitization
dbsurveyor-collect test postgres://user:secret@localhost/db
# Should show: postgres://user:****@localhost/db

Secure Credential Handling#

# Use environment variables
export DATABASE_URL="postgres://user:secret@localhost/db"
dbsurveyor-collect

# Avoid shell history
set +o history # Disable history
dbsurveyor-collect postgres://user:secret@localhost/db
set -o history # Re-enable history

Encryption Issues#

Symptoms: Encryption failures, decryption errors

Test Encryption#

# Test encryption roundtrip
echo "test data" > test.txt
dbsurveyor-collect --encrypt sqlite://test.db
dbsurveyor validate schema.enc
rm test.db test.txt schema.enc

Password Issues#

# Ensure password consistency
# Passwords are case-sensitive
# No visual feedback during password entry (security feature)

# Test with simple password first
# Use ASCII characters only initially

Build and Installation Issues#

Compilation Failures#

Symptoms: Build errors, missing dependencies, feature conflicts

Rust Toolchain Issues#

# Update Rust toolchain
rustup update

# Check Rust version (minimum 1.87)
rustc --version

# Clean and rebuild
cargo clean
cargo build --release

Feature Compilation Issues#

# Check available features
cargo build --help | grep -A 20 "FEATURES:"

# Build with specific features
cargo build --release --features postgresql,sqlite

# Debug feature compilation
cargo build --release --features postgresql --verbose

System Dependencies#

# Ubuntu/Debian: Install system dependencies
sudo apt-get update
sudo apt-get install build-essential libssl-dev pkg-config

# For PostgreSQL support
sudo apt-get install libpq-dev

# For MySQL support
sudo apt-get install libmysqlclient-dev

# macOS: Install dependencies
brew install openssl pkg-config
# For PostgreSQL: brew install postgresql
# For MySQL: brew install mysql

Runtime Dependencies#

Symptoms: Missing shared libraries, runtime errors

Check Dependencies#

# Check binary dependencies
ldd target/release/dbsurveyor-collect # Linux
otool -L target/release/dbsurveyor-collect # macOS

# Test minimal functionality
dbsurveyor-collect --version
dbsurveyor-collect list

Getting Help#

Collect Debug Information#

# System information
uname -a
rustc --version
cargo --version

# DBSurveyor information
dbsurveyor-collect --version
dbsurveyor-collect list

# Feature compilation
cargo build --release --verbose 2>&1 | grep -i feature

# Runtime debug
export RUST_LOG=debug
dbsurveyor-collect test postgres://localhost/db 2> debug.log

Create Minimal Reproduction#

# Create minimal test case
sqlite3 minimal.db "CREATE TABLE test (id INTEGER PRIMARY KEY, name TEXT);"
sqlite3 minimal.db "INSERT INTO test VALUES (1, 'test');"

# Test collection
dbsurveyor-collect sqlite://minimal.db

# Test documentation
dbsurveyor generate schema.dbsurveyor.json

# Clean up
rm minimal.db schema.dbsurveyor.json schema.md

Report Issues#

When reporting issues, include:

System Information: OS, Rust version, DBSurveyor version
Command Used: Exact command that failed (sanitize credentials)
Error Output: Complete error message and stack trace
Debug Logs: Output with RUST_LOG=debug
Minimal Reproduction: Smallest example that reproduces the issue

Security Note: Never include actual database credentials in issue reports. Use placeholder values like user:password@localhost/db.

Community Resources#

GitHub Issues: Report bugs and request features
Documentation: Complete user guide
Security Issues: Email security@evilbitlabs.io

Professional Support#

For enterprise users requiring professional support, contact support@evilbitlabs.io for:

Priority issue resolution
Custom feature development
Integration consulting
Security auditing and compliance