An end-to-end Machine Learning platform for detecting phishing websites and network intrusions, powered by XGBoost and a modular MLOps pipeline.
Features • Tech Stack • Project Structure • Getting Started • API Reference • Deployment • Contributing • License
| Feature | Description |
|---|---|
| 🔗 Real-Time URL Scanning | Paste any URL and get an instant Phishing / Legitimate verdict using a trained ML model + PhishTank API cross-check. |
| 📄 Batch CSV Analysis | Upload a CSV of network traffic features to get bulk predictions with downloadable results. |
| 🤖 RAG Chatbot | An AI assistant powered by Google Gemini 1.5 Flash that answers questions about the platform, grounded in project context. |
| 📊 MLflow Experiment Tracking | Every training run logs F1, Precision, and Recall metrics to DagsHub/MLflow for experiment comparison. |
| ☁️ AWS S3 Artifact Storage | Training artifacts and final models are automatically synced to an S3 bucket after each pipeline run. |
| 🔐 User Authentication | Session-based login/register system backed by MongoDB. |
| 🏗️ CI/CD Pipeline | GitHub Actions workflow for linting, Docker build, ECR push, and self-hosted deployment. |
| ⚡ Optimised Inference | Model is loaded once at startup (~27 MB) and reused across all requests; URL scan results are cached in-memory. |
| Layer | Technologies |
|---|---|
| ML / Data Science | Scikit-Learn, XGBoost, KNN Imputer, Pandas, NumPy |
| Experiment Tracking | MLflow, DagsHub |
| Backend / API | FastAPI, Uvicorn, Mangum (AWS Lambda adapter) |
| Frontend | Jinja2 Templates, Bootstrap 5, Animate.css, Google Fonts (Outfit) |
| Database | MongoDB Atlas (PyMongo) |
| Cloud & Infra | AWS S3, AWS ECR, Docker, GitHub Actions |
| AI / Chatbot | Google Gemini 1.5 Flash (google-generativeai SDK) |
| Deployment | Netlify (serverless), Vercel, Self-hosted Docker |
.
├── .github/
│ └── workflows/
│ └── main.yml # CI/CD — Build → ECR → Deploy
│
├── data_schema/
│ └── schema.yaml # 30-feature schema (int64 columns)
│
├── final_model/
│ ├── model.pkl # Trained XGBoost/best model (~27 MB)
│ └── preprocessor.pkl # KNN Imputer pipeline (~2.3 MB)
│
├── Network_Data/
│ └── phisingData.csv # Raw phishing dataset (11,055 rows)
│
├── networksecurity/ # Core Python package
│ ├── chatbot/
│ │ └── rag_factory.py # Gemini-powered RAG chatbot
│ ├── cloud/
│ │ └── s3_syncer.py # AWS S3 sync utility
│ ├── components/
│ │ ├── data_ingestion.py # MongoDB → DataFrame → Train/Test
│ │ ├── data_validation.py # Schema validation & KS drift test
│ │ ├── data_transformation.py # KNN Imputation pipeline
│ │ └── model_trainer.py # Multi-model training + MLflow
│ ├── constant/
│ │ └── training_pipeline/
│ │ └── __init__.py # All pipeline constants
│ ├── entity/
│ │ ├── config_entity.py # Config dataclasses for each stage
│ │ └── artifact_entity.py # Artifact dataclasses for each stage
│ ├── exception/
│ │ └── exception.py # Custom exception with file/line info
│ ├── logging/
│ │ └── logger.py # Timestamped file-based logging
│ ├── pipeline/
│ │ └── training_pipeline.py # Orchestrates the full training run
│ └── utils/
│ ├── main_utils/
│ │ └── utils.py # save/load object, YAML, numpy I/O
│ └── ml_utils/
│ ├── feature_extractor.py # URL → 30-feature vector extraction
│ ├── metric/ # Classification metric helper
│ └── model/
│ └── estimator.py # NetworkModel (preprocessor + model)
│
├── static/
│ ├── css/ # style.css, premium.css
│ ├── js/ # dashboard.js
│ └── images/ # hero_banner.png, login_bg.png, etc.
│
├── templates/
│ ├── landing.html # Public marketing landing page
│ ├── login.html # Auth page (login + register)
│ ├── index.html # Main dashboard (URL scan, CSV upload, chatbot)
│ └── table.html # Prediction results table
│
├── app.py # FastAPI application entry point
├── main.py # CLI entry for training pipeline
├── push_data.py # Script to seed MongoDB from CSV
├── check_preprocessor.py # Debug utility for preprocessor inspection
├── setup.py # Package metadata & dependency installer
├── requirements.txt # Python dependencies
├── Dockerfile # Production container (Python 3.10-slim)
├── run.sh # Local deployment helper script
├── netlify.toml # Netlify serverless configuration
├── vercel.json # Vercel deployment configuration
├── AGENTS.md # AI agent rules & project conventions
├── CODE_OF_CONDUCT.md # Contributor Code of Conduct
├── CONTRIBUTING.md # Contribution guidelines
└── SECURITY.md # Security policy & vulnerability reporting
┌──────────────────────────────────────────────┐
│ Training Pipeline │
│ │
MongoDB Atlas ──▶ │ Ingestion → Validation → Transformation │
(or local CSV) │ │ │ │ │
│ ▼ ▼ ▼ │
│ Train/Test Drift KNN Imputer │
│ Split Report Pipeline │
│ │
│ Model Training │
│ ┌────────────────────────────────────┐ │
│ │ Random Forest │ XGBoost │ AdaBoost │ │
│ │ Decision Tree │ GBM │ Logistic │ │
│ └────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ Best Model ──▶ MLflow ──▶ DagsHub │
│ │ │
│ ▼ │
│ final_model/ ──▶ AWS S3 │
└──────────────────────────────────────────────┘
┌──────────────────────────────────────────────┐
│ Inference (FastAPI) │
│ │
│ /predict_url ──▶ Feature Extractor │
│ │ │ │
│ │ PhishTank API │
│ │ │ │
│ ▼ ▼ │
│ NetworkModel.predict() │
│ │ │
│ ▼ │
│ { prediction: "Phishing" | "Legitimate" } │
│ │
│ /predict ──▶ CSV → NumPy → predict() │
│ /api/chat ──▶ Gemini RAG chatbot │
└──────────────────────────────────────────────┘
- Python 3.10+
- MongoDB Atlas account (or local MongoDB instance)
- AWS CLI configured (for S3 artifact storage)
- Google API Key (for the RAG chatbot — Gemini 1.5 Flash)
git clone https://github.com/prashantshukla01/Network_Security.git
cd Network_Security# Using conda
conda create -n networkml python=3.10 -y
conda activate networkml
# Or using venv
python3 -m venv venv
source venv/bin/activate # macOS/Linux
venv\Scripts\activate # Windowspip install -r requirements.txt
pip install python-multipart # Required for file uploadsCreate a .env file in the project root:
MONGO_DB_URL=mongodb+srv://<username>:<password>@<cluster>.mongodb.net/?appName=<appName>
GOOGLE_API_KEY=<your-google-gemini-api-key>
AWS_ACCESS_KEY_ID=<your-aws-access-key>
AWS_SECRET_ACCESS_KEY=<your-aws-secret-key>
SECRET_KEY=<random-session-secret-key>
⚠️ Important: Never commit your.envfile. It is already listed in.gitignore.
If starting fresh, push the phishing dataset to MongoDB:
python push_data.pypython app.py
# Server starts at http://0.0.0.0:8000Or use the helper script:
chmod +x run.sh
./run.shRun the full ML training pipeline (Ingestion → Validation → Transformation → Training → S3 Sync):
python main.pyOr trigger via the API:
GET http://127.0.0.1:8000/train
| Model | Hyperparameter Grid |
|---|---|
| Random Forest | n_estimators: [8, 16, 32, 128, 256] |
| Decision Tree | criterion: [gini, entropy, log_loss] |
| Gradient Boosting | learning_rate, subsample, n_estimators |
| XGBoost | learning_rate, n_estimators, max_depth |
| AdaBoost | learning_rate, n_estimators |
| Logistic Regression | Default |
The best model is automatically selected based on test score and pushed to final_model/.
Metrics are logged to DagsHub:
# Set tracking (already configured in model_trainer.py via dagshub.init)
export MLFLOW_TRACKING_URI=https://dagshub.com/prashantshukla01/Network_Security.mlflow
export MLFLOW_TRACKING_USERNAME=<your-username>
export MLFLOW_TRACKING_PASSWORD=<your-token>
# View locally
mlflow ui| Method | Endpoint | Description |
|---|---|---|
GET |
/ |
Landing page |
GET |
/login |
Login / Register page |
POST |
/login |
Submit login credentials |
POST |
/register |
Register a new user |
GET |
/logout |
Clear session and redirect |
GET |
/dashboard |
Main dashboard (auth required) |
GET |
/train |
Trigger full training pipeline |
POST |
/predict |
Upload CSV for batch prediction |
POST |
/predict_url |
Scan a single URL (JSON body: {"url": "..."}) |
GET |
/api/history |
Get last 10 prediction history records |
POST |
/api/chat |
Chat with the RAG assistant (JSON body: {"message": "..."}) |
curl -X POST http://127.0.0.1:8000/predict_url \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'Response:
{
"url": "https://example.com",
"prediction": "Legitimate",
"raw_result": 1
}# Build
docker build -t netguard-ai .
# Run
docker run -d -p 8000:8000 \
-e MONGO_DB_URL="<your-mongo-url>" \
-e GOOGLE_API_KEY="<your-key>" \
--name netguard \
netguard-aiThe GitHub Actions workflow (.github/workflows/main.yml) handles:
- Continuous Integration — Lint + unit test stubs
- Continuous Delivery — Docker build → push to AWS ECR
- Continuous Deployment — Pull from ECR → run on self-hosted runner (port
8080:8000)
Required GitHub Secrets:
| Secret | Description |
|---|---|
AWS_ACCESS_KEY_ID |
AWS IAM access key |
AWS_SECRET_ACCESS_KEY |
AWS IAM secret key |
AWS_REGION |
e.g. ap-south-1 |
AWS_ECR_LOGIN_URI |
ECR registry URI |
ECR_REPOSITORY_NAME |
ECR repository name |
MONGO_DB_URL |
MongoDB Atlas connection string |
Configured via netlify.toml — uses Mangum adapter to wrap FastAPI as a serverless function.
Configured via vercel.json — routes all traffic to app.py.
The model uses 30 features extracted from URLs and web page attributes:
Click to expand full feature list
| # | Feature | Type |
|---|---|---|
| 1 | having_IP_Address |
int64 |
| 2 | URL_Length |
int64 |
| 3 | Shortining_Service |
int64 |
| 4 | having_At_Symbol |
int64 |
| 5 | double_slash_redirecting |
int64 |
| 6 | Prefix_Suffix |
int64 |
| 7 | having_Sub_Domain |
int64 |
| 8 | SSLfinal_State |
int64 |
| 9 | Domain_registeration_length |
int64 |
| 10 | Favicon |
int64 |
| 11 | port |
int64 |
| 12 | HTTPS_token |
int64 |
| 13 | Request_URL |
int64 |
| 14 | URL_of_Anchor |
int64 |
| 15 | Links_in_tags |
int64 |
| 16 | SFH |
int64 |
| 17 | Submitting_to_email |
int64 |
| 18 | Abnormal_URL |
int64 |
| 19 | Redirect |
int64 |
| 20 | on_mouseover |
int64 |
| 21 | RightClick |
int64 |
| 22 | popUpWidnow |
int64 |
| 23 | Iframe |
int64 |
| 24 | age_of_domain |
int64 |
| 25 | DNSRecord |
int64 |
| 26 | web_traffic |
int64 |
| 27 | Page_Rank |
int64 |
| 28 | Google_Index |
int64 |
| 29 | Links_pointing_to_page |
int64 |
| 30 | Statistical_report |
int64 |
Values: 1 (Legitimate), 0 (Suspicious), -1 (Phishing)
Contributions are welcome! Please see CONTRIBUTING.md for guidelines.
This project adheres to the Contributor Covenant Code of Conduct. By participating, you are expected to uphold this code.
For reporting vulnerabilities, see SECURITY.md.
Prashant Shukla
- Email: prashantshukla9812@gmail.com
- GitHub: @prashantshukla01
This project is open source and available under the MIT License.
Made with ❤️ for a safer internet.