Note: This document has been prepared to present a public-facing technical overview of a closed-source, commercial codebase. The system's architecture, decision-making mechanisms, and layers are detailed below.
- Overview
- High-Level Architecture
- Multi-LLM and Intelligence Orchestration
- Universal Autonomous Web Scraper
- Data and Infrastructure Layer
- Background and Scheduled Jobs
- Presentation Layer
- Security and Fault Tolerance
- Installation and Getting Started
Aegis (formerly EmailAgent) goes beyond the scope of an ordinary chatbot. It is an autonomous system built on Clean Architecture principles and powered by Semantic Kernel. Aegis conducts research on the user's behalf, resolves the structure of arbitrary e-commerce or classifieds websites through AI-driven discovery, tracks deals and price movements, reads and responds to email, and allows the entire workflow to be managed through Telegram or a modern web interface.
The system manages its external interfaces (Telegram, Web UI) while processing asynchronous tasks in the background through Hangfire and executing complex web automation through Playwright.
graph TD
U1[Telegram Bot] -->|Webhook / Long Polling| API(EmailAgent.API)
U2[React Web Panel] -->|REST and SignalR| API
API --> Agent[EmailAgent.Agent <br> Semantic Kernel]
API --> Infra[EmailAgent.Infrastructure]
API --> Core[EmailAgent.Core]
Agent -->|LLM API Calls| LLM((Multi-LLM: <br>Gemini, GPT-4, Claude, Groq))
Agent -->|Plugin Routing| Plugins[Aegis Plugins <br> Scraper, Shopping, Email]
Infra --> DB[(PostgreSQL)]
Infra --> HF{Hangfire Background Jobs}
Infra --> PW[Playwright Browser Engine]
HF -->|Scheduled Scraping| PW
PW -.->|Dynamic Selector Persistence| DB
The system is not dependent on a single model. Different large language models are invoked based on the nature of the task:
- Fast inference (Groq LLaMA 3): Telegram voice command transcription (Whisper) and low-latency responses for simple queries.
- Complex reasoning (Claude 3.5 Sonnet / GPT-4o): Email analysis, code generation, and structured JSON extraction.
- Default multimodal operations (Gemini): Visual analysis and content extraction from web pages.
When a user issues a request such as "Find me the cheapest RTX 4090 on sahibinden.com," the internal processing flow is as follows:
sequenceDiagram
participant User as User (Telegram)
participant Core as Semantic Kernel Core
participant Planner as AI Planner
participant Plugin as WebSearch and Scraper Plugins
participant LLM as Language Model
User->>Core: Natural language request
Core->>Planner: Which tools are required?
Planner->>LLM: Context analysis
LLM-->>Planner: Plan: 1. Perform search, 2. Scrape target site
Planner->>Plugin: Execute: identify URL
Plugin->>LLM: Analyze site HTML, determine CSS selectors
LLM-->>Plugin: JSON strategy (price, title selectors)
Plugin->>User: Products found, added to tracking list
Conventional web scraping breaks whenever a target site (for example Amazon, Sahibinden, or Hepsiburada) changes its layout. Aegis addresses this problem with an autonomous discovery algorithm.
Dynamic discovery system. When Aegis encounters a site for the first time, the SiteDiscoveryPlugin is triggered. The site's minified HTML is sent to the language model, which infers the required CSS selectors. The resulting strategy is persisted to PostgreSQL. On subsequent scans, the strategy is retrieved directly from the database, enabling fast scraping through Playwright within milliseconds.
flowchart TD
A([New URL Request]) --> B{Strategy exists in DB?}
B -- Yes --> C[Load dynamic strategy]
B -- No --> D[SiteDiscoveryPlugin triggered]
D --> E[Page HTML fetched via Playwright]
E --> F[DOM minified and cleaned]
F --> G[Request sent to LLM: identify title, price, image selectors]
G --> H[LLM returns JSON]
H --> I[(Persisted to SiteStrategyDefinitions table)]
I --> C
C --> J[Elements extracted via Playwright]
J --> K[Price and product object constructed]
K --> L(((Result: added to tracking list)))
The foundation of the platform is a PostgreSQL database modeled through Entity Framework Core. The relationships between the core tables are shown below:
erDiagram
USERS ||--o{ USER_PREFERENCES : has
USERS ||--o{ NOTIFICATION_LOGS : receives
USERS ||--o{ TRACKED_PRODUCTS : tracks
USERS ||--o{ CATEGORY_TRACKERS : tracks
TRACKED_PRODUCTS ||--o{ PRICE_HISTORY : logs
SITE_STRATEGIES ||--o{ TRACKED_PRODUCTS : applies_to
USERS {
uuid Id PK
string TelegramChatId
string Email
datetime CreatedAt
}
SITE_STRATEGIES {
int Id PK
string DomainName
jsonb Selectors "Title, Price, Image, Stock"
bool IsAI_Discovered
}
TRACKED_PRODUCTS {
int Id PK
string Url
decimal TargetPrice
decimal CurrentPrice
int SiteStrategyId FK
}
PRICE_HISTORY {
int Id PK
int TrackedProductId FK
decimal OldPrice
decimal NewPrice
datetime ChangeDate
}
Aegis is designed to operate continuously without user intervention. This autonomy is achieved through Hangfire background jobs.
- ShoppingTrackerJob: Periodically revisits products tracked by the user and triggers a notification when a price drop is detected.
- CategoryWatcherJob: Detects newly listed items or sharply discounted deals within a tracked category.
- DailyBriefingJob / MorningBriefingJob: Every morning, summarizes the user's email, calendar, and relevant market data, and delivers a daily briefing message through Telegram.
Users interact with Aegis through real-time messaging.
- Secure pairing: The user links their Telegram account using a PIN code obtained from the web panel.
- Natural language processing: Requests such as "Notify me when this product's price drops below 5000" are routed directly to Semantic Kernel.
- Voice and document handling: Voice messages are transcribed via
ISpeechToTextService(Whisper); uploaded PDF documents are parsed and summarized.
A modern, dark-mode dashboard. Real-time logs from background jobs, such as the scraper, are streamed to the interface via SignalR as a live ticker.
Large e-commerce platforms employ anti-bot mechanisms such as Cloudflare or reCAPTCHA to block automated scraping. Aegis applies the following countermeasures:
- Circuit breaker: When a site repeatedly returns
403 Forbiddenor429 Too Many Requests, the corresponding domain is placed into quarantine for a defined period (for example, 30 minutes). - Concurrency control via SemaphoreSlim: The number of requests per second directed at a single domain is throttled, minimizing the risk of IP bans.
- Session and cookie management: For sites requiring authentication, Playwright persists existing cookies to disk (
session_state_*.json) and reuses these sessions instead of re-authenticating on every run.
- .NET 10 SDK
- Node.js v18 or later (for the frontend)
- PostgreSQL 16 or later
# 1. Clone the repository (for authorized users)
git clone https://github.com/your-username/aegis-core.git
# 2. Install Playwright browsers
cd EmailAgent.API
pwsh bin/Debug/net10.0/playwright.ps1 install
# 3. Configure and run the database and API
# Add database connection and API keys (Gemini, Groq, etc.) to appsettings.json
dotnet run
# 4. Start the web interface
cd ../EmailAgent.Web
npm install
npm run devThe API is served at localhost:5209, and the web interface at localhost:5173. Swagger documentation is available at the /swagger route.
Aegis hands the operational load of everyday life over to the speed of artificial intelligence.