Overview
Self-hosted email security platform combining rule-based analysis, supervised ML (XGBoost, TF-IDF), anomaly detection (Isolation Forest, One-Class SVM), and SpamAssassin with weighted decision fusion. Features multi-layer evasion detection (homograph, zero-width chars, JS obfuscation, RTL override, brand spoofing), continuous automated ML retraining, async queue pipeline, role-based mail oversight (superadmin, admin, end user), and zero-trust mailbox isolation.
The Problem
Modern email security faces increasingly sophisticated threat vectors — traditional rule-based spam filters fail against zero-day phishing, obfuscated HTML/JS payloads, homograph domain spoofing, and subtle anomaly patterns. Organizations struggle with email security platforms that lack multi-layered detection, continuous machine learning adaptation, transparent security explanations, and strict mailbox-level isolation between administrative oversight and end-user access.
The Solution
CogniMail is a self-hosted ML-Powered Anti-Phishing & Spam Filtering platform designed to receive, analyze, classify, and present email safely. It combines rule-based analysis (SpamAssassin), supervised machine learning (XGBoost + TF-IDF), anomaly detection (Isolation Forest & One-Class SVM), and deep evasion detection into a weighted decision fusion engine with CLEAN, WARN, and QUARANTINE outcomes. The system features an async worker pipeline, continuous ML retraining with model versioning and automated accuracy validation, and 3-tier role-based access control (superadmin, admin, end user) with zero-trust mailbox data isolation enforced strictly at the backend level.
Key Features
Layered Threat Detection Engine
Combines SpamAssassin rules, supervised XGBoost classification (TF-IDF features), Isolation Forest & One-Class SVM anomaly detection, MIME attachment auditing, and URL/domain intelligence into a single unified analysis pipeline.
Weighted Decision Fusion
Evaluates cross-layer detection confidence scores to route messages into 3 distinct outcomes — CLEAN (delivered to end-user inbox), WARN (administrative security review), or QUARANTINE (phishing/spam quarantine).
Advanced Evasion Detection Module
Identifies sophisticated evasion techniques including homograph Cyrillic/Greek script domain spoofing, zero-width invisible character injection, JS/HTML obfuscation (eval, atob, entity overuse), RTL override filename spoofing, and brand impersonation.
Continuous ML Retraining System
Automated background retraining worker that incorporates user feedback, maintains timestamped model version backups, prevents catastrophic forgetting by blending new and baseline data, and verifies >85% accuracy prior to production deployment.
Zero-Trust Mailbox Isolation & RBAC
Strict 3-tier role-based access control (superadmin, admin, end user). Mailbox ownership and security-review data boundaries are strictly validated on the backend API, ensuring end users cannot view administrative quarantine or other mailboxes.
Full Webmail & Security Dashboard
Complete webmail experience (inbox, compose, reply, forward, drafts, sent, starred, search) for end users, alongside detailed administrative dashboards with security analytics, threat reports, audit logs, and XAI feature contribution graphs.
Architecture
Decoupled microservices architecture. Incoming mail is received by SMTP receiver service and queued into Redis. Asynchronous Analysis Worker pulls jobs from Redis, orchestrating analysis across SpamAssassin, ML Classifier service (FastAPI/PyTorch/XGBoost), and Evasion Detection modules. Results are stored in PostgreSQL 16. The FastAPI application API serves the React 19 (Vite + TanStack Query) frontend webmail and administrative dashboards. Prometheus metrics and audit logs provide continuous observability.
How to Use
1. Clone the repository and configure environment variables (.env). 2. Spin up services using docker-compose up -d (starts PostgreSQL 16, Redis 7, SMTP receiver, analysis worker, FastAPI backend, ML classifier, SpamAssassin, and React 19 frontend dashboard). 3. Send emails via SMTP port 25 or run test injectors. 4. Access the end-user webmail at http://localhost:3000 to manage inbox/compose/sent mail. 5. Access administrator review dashboards to inspect flagged phishing, spam, or warning emails, review XAI feature contributions, and trigger automated ML model retraining.
Impact & Results
Provides enterprise-grade, multi-layered email security in a self-hosted deployment. The weighted decision fusion engine evaluates signals across 5 detection layers, dramatically reducing false positives and zero-day phishing bypasses. Specialized evasion detection modules catch homograph/IDN spoofing, zero-width character injection, Base64/RTL obfuscation, and brand imposter attacks. The automated continuous retraining system allows models to adapt to new phishing campaigns without catastrophic forgetting, maintaining >85% validation threshold before live model deployment.
