diff --git a/scheme.md b/scheme.md
index 9864a1e..fb3fce9 100644
--- a/scheme.md
+++ b/scheme.md
@@ -4,7 +4,7 @@ Sistema automatizzato per la generazione e mantenimento della documentazione tec
[](https://opensource.org/licenses/MIT)
[](https://www.python.org/downloads/)
-[](https://kafka.apache.org/)
+[](https://redis.io/)
## 📋 Indice
@@ -22,7 +22,8 @@ Sistema progettato per **automatizzare la creazione e l'aggiornamento della docu
- ✅ **Raccolta dati asincrona** da molteplici sistemi infrastrutturali
- ✅ **Isolamento di sicurezza**: LLM non accede mai ai sistemi live
-- ✅ **Event-driven architecture** con Apache Kafka
+- ✅ **Change Detection**: Documentazione generata solo su modifiche rilevate
+- ✅ **Redis Cache** per storage dati e performance
- ✅ **LLM locale on-premise** (Qwen) tramite MCP Server
- ✅ **Human-in-the-loop validation** con workflow GitOps
- ✅ **CI/CD automatizzato** per pubblicazione
@@ -31,11 +32,13 @@ Sistema progettato per **automatizzare la creazione e l'aggiornamento della docu
Il sistema è suddiviso in **3 flussi principali**:
-1. **Raccolta Dati (Background)**: Connettori interrogano periodicamente i sistemi infrastrutturali tramite API e pubblicano i dati su Kafka
-2. **Generazione Documentazione (On-Demand)**: LLM locale (Qwen) genera markdown interrogando Kafka/Redis tramite MCP Server
-3. **Validazione e Pubblicazione (GitOps)**: Review umana su Pull Request e deploy automatico via CI/CD
+1. **Raccolta Dati (Background)**: Connettori interrogano periodicamente i sistemi infrastrutturali tramite API e aggiornano Redis
+2. **Change Detection**: Sistema di rilevamento modifiche che attiva la generazione documentazione solo quando necessario
+3. **Generazione e Pubblicazione (Triggered)**: LLM locale (Qwen) genera markdown leggendo da Redis, seguito da review umana e deploy automatico
-> **Principio di Sicurezza**: L'LLM non ha mai accesso diretto ai sistemi infrastrutturali. Tutti i dati passano attraverso Kafka/Redis.
+> **Principio di Sicurezza**: L'LLM non ha mai accesso diretto ai sistemi infrastrutturali. Tutti i dati sono letti da Redis.
+
+> **Principio di Efficienza**: La documentazione viene generata solo quando il sistema rileva modifiche nella configurazione infrastrutturale.
---
@@ -56,8 +59,8 @@ INCOLLA QUI LO SCHEMA ARCHITETTURALE
graph TB
%% Styling
classDef infrastructure fill:#e1f5ff,stroke:#01579b,stroke-width:3px,color:#333
- classDef kafka fill:#fff3e0,stroke:#e65100,stroke-width:3px,color:#333
classDef cache fill:#f3e5f5,stroke:#4a148c,stroke-width:3px,color:#333
+ classDef change fill:#fff3e0,stroke:#e65100,stroke-width:3px,color:#333
classDef llm fill:#e8f5e9,stroke:#1b5e20,stroke-width:3px,color:#333
classDef git fill:#fce4ec,stroke:#880e4f,stroke-width:3px,color:#333
classDef human fill:#fff9c4,stroke:#f57f17,stroke-width:3px,color:#333
@@ -70,37 +73,42 @@ graph TB
CONN["🔌 CONNETTORI
Polling Automatico"]:::infrastructure
- KAFKA[("📨 APACHE KAFKA
Message Broker
+ Persistenza")]:::kafka
-
- CONSUMER["⚙️ KAFKA CONSUMER
Processor Service"]:::kafka
-
- REDIS[("💾 REDIS CACHE
(Opzionale)
Performance Layer")]:::cache
+ REDIS[("💾 REDIS CACHE
Configurazione
Infrastruttura")]:::cache
INFRA -->|"API Polling
Continuo"| CONN
- CONN -->|"Publish
Eventi"| KAFKA
- KAFKA -->|"Consume
Stream"| CONSUMER
- CONSUMER -.->|"Update
Opzionale"| REDIS
+ CONN -->|"Update
Configurazione"| REDIS
%% ========================================
- %% FLUSSO 2: GENERAZIONE DOCUMENTAZIONE
+ %% CHANGE DETECTION
%% ========================================
- USER["👤 UTENTE
Richiesta Doc"]:::human
+ CHANGE["🔍 CHANGE DETECTOR
Rileva Modifiche
Configurazione"]:::change
- LLM["🤖 LLM ENGINE
Claude / GPT"]:::llm
+ REDIS -->|"Monitor
Changes"| CHANGE
+
+ %% ========================================
+ %% FLUSSO 2: GENERAZIONE DOCUMENTAZIONE (Triggered)
+ %% ========================================
+
+ TRIGGER["⚡ TRIGGER
Solo se modifiche"]:::change
+
+ USER["👤 UTENTE
Richiesta Manuale"]:::human
+
+ LLM["🤖 LLM ENGINE
Qwen (Locale)"]:::llm
MCP["🔧 MCP SERVER
API Control Platform"]:::llm
DOC["📄 DOCUMENTO
Markdown Generato"]:::llm
- USER -->|"1. Prompt"| LLM
- LLM -->|"2. Tool Call"| MCP
- MCP -->|"3a. Query"| KAFKA
- MCP -.->|"3b. Query
Fast"| REDIS
- KAFKA -->|"4a. Dati"| MCP
- REDIS -.->|"4b. Dati"| MCP
- MCP -->|"5. Context"| LLM
- LLM -->|"6. Genera"| DOC
+ CHANGE -->|"Modifiche
Rilevate"| TRIGGER
+ USER -.->|"Opzionale"| TRIGGER
+
+ TRIGGER -->|"Avvia
Generazione"| LLM
+ LLM -->|"Tool Call"| MCP
+ MCP -->|"Query"| REDIS
+ REDIS -->|"Dati Config"| MCP
+ MCP -->|"Context"| LLM
+ LLM -->|"Genera"| DOC
%% ========================================
%% FLUSSO 3: VALIDAZIONE E PUBBLICAZIONE
@@ -127,16 +135,16 @@ graph TB
MKDOCS -->|"Deploy"| WEB
%% ========================================
- %% ANNOTAZIONI SICUREZZA
+ %% ANNOTAZIONI
%% ========================================
SECURITY["🔒 SICUREZZA
LLM isolato dai sistemi live"]:::human
- PERF["⚡ PERFORMANCE
Cache Redis opzionale"]:::cache
+ EFFICIENCY["⚡ EFFICIENZA
Doc generata solo
su modifiche"]:::change
LLM -.->|"NESSUN
ACCESSO"| INFRA
SECURITY -.-> LLM
- PERF -.-> REDIS
+ EFFICIENCY -.-> CHANGE
```
---
@@ -159,8 +167,8 @@ graph TB
%% Styling tecnico
classDef infra fill:#e1f5ff,stroke:#01579b,stroke-width:2px,color:#333,font-size:11px
classDef connector fill:#e3f2fd,stroke:#1565c0,stroke-width:2px,color:#333,font-size:11px
- classDef kafka fill:#fff3e0,stroke:#e65100,stroke-width:2px,color:#333,font-size:11px
classDef cache fill:#f3e5f5,stroke:#4a148c,stroke-width:2px,color:#333,font-size:11px
+ classDef change fill:#fff3e0,stroke:#e65100,stroke-width:2px,color:#333,font-size:11px
classDef llm fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px,color:#333,font-size:11px
classDef git fill:#fce4ec,stroke:#880e4f,stroke-width:2px,color:#333,font-size:11px
classDef monitor fill:#fff8e1,stroke:#f57f17,stroke-width:2px,color:#333,font-size:11px
@@ -181,7 +189,7 @@ graph TB
%% =====================================
subgraph CONNECTORS["🔌 DATA COLLECTORS (Python/Go)"]
- CONN_VM["VMware Collector
Lang: Python 3.11
Lib: pyvmomi
Schedule: */15 * * * *
Output: JSON"]:::connector
+ CONN_VM["VMware Collector
Lang: Python 3.11
Lib: pyvmomi
Schedule: */15 * * * *
Output: JSON → Redis"]:::connector
CONN_K8S["K8s Collector
Lang: Python 3.11
Lib: kubernetes-client
Schedule: */5 * * * *
Resources: pods,svc,ing,deploy"]:::connector
@@ -196,73 +204,78 @@ graph TB
CISCO -->|"NETCONF
get-config"| CONN_CSC
%% =====================================
- %% LAYER 3: MESSAGE BROKER
+ %% LAYER 3: REDIS STORAGE
%% =====================================
- subgraph MESSAGING["📨 KAFKA CLUSTER (3 brokers)"]
- KAFKA_TOPICS["Kafka Topics:
• vmware.inventory (P:6, R:3)
• k8s.resources (P:12, R:3)
• linux.systems (P:3, R:3)
• cisco.network (P:3, R:3)
Retention: 7 days
Format: JSON + Schema Registry"]:::kafka
-
- SCHEMA["Schema Registry
Avro Schemas
Versioning enabled
Port: 8081"]:::kafka
- end
-
- CONN_VM -->|"Producer
Batch 100 msg"| KAFKA_TOPICS
- CONN_K8S -->|"Producer
Batch 100 msg"| KAFKA_TOPICS
- CONN_LNX -->|"Producer
Batch 50 msg"| KAFKA_TOPICS
- CONN_CSC -->|"Producer
Batch 50 msg"| KAFKA_TOPICS
-
- KAFKA_TOPICS <--> SCHEMA
-
- %% =====================================
- %% LAYER 4: PROCESSING & CACHE
- %% =====================================
-
- subgraph PROCESSING["⚙️ STREAM PROCESSING"]
- CONSUMER_GRP["Kafka Consumer Group
Group ID: doc-consumers
Lang: Python 3.11
Lib: kafka-python
Workers: 6
Commit: auto (5s)"]:::kafka
-
- PROCESSOR["Data Processor
• Validation
• Transformation
• Enrichment
• Deduplication"]:::kafka
- end
-
- KAFKA_TOPICS -->|"Subscribe
offset management"| CONSUMER_GRP
- CONSUMER_GRP --> PROCESSOR
-
- subgraph STORAGE["💾 CACHE LAYER (Optional)"]
+ subgraph STORAGE["💾 REDIS CLUSTER"]
REDIS_CLUSTER["Redis Cluster
Mode: Cluster (6 nodes)
Port: 6379
Persistence: RDB + AOF
Memory: 64GB
Eviction: allkeys-lru"]:::cache
- REDIS_KEYS["Key Structure:
• vmware:vcenter-id:vms
• k8s:cluster:namespace:resource
• linux:hostname:info
• cisco:device-id:config
TTL: 1-24h based on type"]:::cache
+ REDIS_KEYS["Key Structure:
• vmware:vcenter-id:vms:hash
• k8s:cluster:namespace:resource:hash
• linux:hostname:info:hash
• cisco:device-id:config:hash
• changelog:timestamp:diff
TTL: 30d for data, 90d for changelog"]:::cache
end
- PROCESSOR -.->|"SET/HSET
Pipeline batch"| REDIS_CLUSTER
+ CONN_VM -->|"HSET/HMSET
+ Hash Storage"| REDIS_CLUSTER
+ CONN_K8S -->|"HSET/HMSET
+ Hash Storage"| REDIS_CLUSTER
+ CONN_LNX -->|"HSET/HMSET
+ Hash Storage"| REDIS_CLUSTER
+ CONN_CSC -->|"HSET/HMSET
+ Hash Storage"| REDIS_CLUSTER
+
REDIS_CLUSTER --> REDIS_KEYS
%% =====================================
- %% LAYER 5: LLM & MCP
+ %% LAYER 4: CHANGE DETECTION
%% =====================================
+ subgraph CHANGE_DETECTION["🔍 CHANGE DETECTION SYSTEM"]
+ DETECTOR["Change Detector Service
Lang: Python 3.11
Lib: redis-py
Algorithm: Hash comparison
Check interval: */5 * * * *"]:::change
+
+ DIFF_ENGINE["Diff Engine
• Deep object comparison
• JSON diff generation
• Change classification
• Severity assessment"]:::change
+
+ CHANGE_LOG["Change Log Store
Key: changelog:*
Data: diff JSON + metadata
Indexed by: timestamp, resource"]:::change
+
+ NOTIFIER["Change Notifier
• Webhook triggers
• Slack notifications
• Event emission
Target: LLM trigger"]:::change
+ end
+
+ REDIS_CLUSTER -->|"Monitor
key changes"| DETECTOR
+ DETECTOR --> DIFF_ENGINE
+ DIFF_ENGINE -->|"Store diff"| CHANGE_LOG
+ CHANGE_LOG --> REDIS_CLUSTER
+ DIFF_ENGINE -->|"Notify if
significant"| NOTIFIER
+
+ %% =====================================
+ %% LAYER 5: LLM TRIGGER & GENERATION
+ %% =====================================
+
+ subgraph TRIGGER_SYSTEM["⚡ TRIGGER SYSTEM"]
+ TRIGGER_SVC["Trigger Service
Lang: Python 3.11
Listen: Webhook + Redis Pub/Sub
Debounce: 5 min
Batch: multiple changes"]:::change
+
+ QUEUE["Generation Queue
Type: Redis List
Priority: High/Medium/Low
Processing: FIFO"]:::change
+ end
+
+ NOTIFIER -->|"Trigger event"| TRIGGER_SVC
+ TRIGGER_SVC -->|"Enqueue
generation task"| QUEUE
+
subgraph LLM_LAYER["🤖 AI GENERATION LAYER"]
- LLM_ENGINE["LLM Engine
Model: Claude Sonnet 4 / GPT-4
API: Anthropic/OpenAI
Temp: 0.3
Max Tokens: 4096
Timeout: 120s"]:::llm
+ LLM_ENGINE["LLM Engine
Model: Qwen (Locale)
API: Ollama/vLLM/LM Studio
Port: 11434
Temp: 0.3
Max Tokens: 4096
Timeout: 120s"]:::llm
MCP_SERVER["MCP Server
Lang: TypeScript/Node.js
Port: 3000
Protocol: JSON-RPC 2.0
Auth: JWT tokens"]:::llm
- MCP_TOOLS["MCP Tools:
• getVMwareInventory(vcenter)
• getK8sResources(cluster,ns,type)
• getLinuxSystemInfo(hostname)
• getCiscoConfig(device,section)
• queryTimeRange(start,end)
Return: JSON + Metadata"]:::llm
+ MCP_TOOLS["MCP Tools:
• getVMwareInventory(vcenter)
• getK8sResources(cluster,ns,type)
• getLinuxSystemInfo(hostname)
• getCiscoConfig(device,section)
• getChangelog(start,end,resource)
Return: JSON + Metadata"]:::llm
end
+ QUEUE -->|"Dequeue
task"| LLM_ENGINE
+
LLM_ENGINE <-->|"Tool calls
JSON-RPC"| MCP_SERVER
MCP_SERVER --> MCP_TOOLS
- MCP_TOOLS -->|"1. Query Kafka Consumer API
GET /api/v1/data"| CONSUMER_GRP
- MCP_TOOLS -.->|"2. Fallback Redis
MGET/HGETALL"| REDIS_CLUSTER
-
- CONSUMER_GRP -->|"JSON Response
+ Timestamps"| MCP_TOOLS
- REDIS_CLUSTER -.->|"Cached JSON
Fast response"| MCP_TOOLS
-
+ MCP_TOOLS -->|"HGETALL/MGET
Read data"| REDIS_CLUSTER
+ REDIS_CLUSTER -->|"Config data
+ Changelog"| MCP_TOOLS
MCP_TOOLS -->|"Structured Data
+ Context"| LLM_ENGINE
subgraph OUTPUT["📝 DOCUMENT GENERATION"]
TEMPLATE["Template Engine
Format: Jinja2
Templates: markdown/*.j2
Variables: from LLM"]:::llm
- MARKDOWN["Markdown Output
Format: CommonMark
Metadata: YAML frontmatter
Assets: diagrams in mermaid"]:::llm
+ MARKDOWN["Markdown Output
Format: CommonMark
Metadata: YAML frontmatter
Change summary included
Assets: diagrams in mermaid"]:::llm
- VALIDATOR["Doc Validator
• Markdown linting
• Link checking
• Schema validation"]:::llm
+ VALIDATOR["Doc Validator
• Markdown linting
• Link checking
• Schema validation
• Change verification"]:::llm
end
LLM_ENGINE --> TEMPLATE
@@ -278,14 +291,14 @@ graph TB
GIT_API["GitLab API
API: v4
Auth: Project Access Token
Permissions: api, write_repo"]:::git
- PR_AUTO["Automated PR Creator
Lang: Python 3.11
Lib: python-gitlab
Template: .gitlab/merge_request.md"]:::git
+ PR_AUTO["Automated PR Creator
Lang: Python 3.11
Lib: python-gitlab
Template: .gitlab/merge_request.md
Include: change summary"]:::git
end
VALIDATOR -->|"git add/commit/push"| GIT_REPO
GIT_REPO <--> GIT_API
GIT_API --> PR_AUTO
- REVIEWER["👨💼 Technical Reviewer
Role: Maintainer/Owner
Review: diff + validation
Approve: required (min 1)"]:::monitor
+ REVIEWER["👨💼 Technical Reviewer
Role: Maintainer/Owner
Review: diff + validation
Check: change correlation
Approve: required (min 1)"]:::monitor
PR_AUTO -->|"Notification
Email + Slack"| REVIEWER
REVIEWER -->|"Merge to main"| GIT_REPO
@@ -323,41 +336,44 @@ graph TB
%% =====================================
subgraph OBSERVABILITY["📊 MONITORING & LOGGING"]
- PROMETHEUS["Prometheus
Metrics: collector lag, cache hit/miss
Scrape: 30s
Retention: 15d"]:::monitor
+ PROMETHEUS["Prometheus
Metrics: collector updates, changes detected
Scrape: 30s
Retention: 15d"]:::monitor
- GRAFANA["Grafana Dashboards
• Kafka metrics
• Redis performance
• LLM response times
• Pipeline success rate"]:::monitor
+ GRAFANA["Grafana Dashboards
• Collector status
• Redis performance
• Change detection rate
• LLM response times
• Pipeline success rate"]:::monitor
ELK["ELK Stack
Logs: all components
Index: daily rotation
Retention: 30d"]:::monitor
- ALERTS["Alerting
• Connector failures
• Kafka lag > 10k
• Redis OOM
• Pipeline failures
Channel: Slack + PagerDuty"]:::monitor
+ ALERTS["Alerting
• Collector failures
• Redis issues
• Change detection errors
• Pipeline failures
Channel: Slack + PagerDuty"]:::monitor
end
CONN_VM -.->|"metrics"| PROMETHEUS
CONN_K8S -.->|"metrics"| PROMETHEUS
- KAFKA_TOPICS -.->|"metrics"| PROMETHEUS
REDIS_CLUSTER -.->|"metrics"| PROMETHEUS
+ DETECTOR -.->|"metrics"| PROMETHEUS
MCP_SERVER -.->|"metrics"| PROMETHEUS
GITLAB_CI -.->|"metrics"| PROMETHEUS
PROMETHEUS --> GRAFANA
CONN_VM -.->|"logs"| ELK
- CONSUMER_GRP -.->|"logs"| ELK
+ DETECTOR -.->|"logs"| ELK
MCP_SERVER -.->|"logs"| ELK
GITLAB_CI -.->|"logs"| ELK
GRAFANA --> ALERTS
%% =====================================
- %% SECURITY ANNOTATIONS
+ %% SECURITY & EFFICIENCY ANNOTATIONS
%% =====================================
SEC1["🔒 SECURITY:
• All APIs use TLS 1.3
• Secrets in Vault/K8s Secrets
• Network: private VPC
• LLM has NO direct access"]:::monitor
SEC2["🔐 AUTHENTICATION:
• API Tokens rotated 90d
• RBAC enforced
• Audit logs enabled
• MFA required for Git"]:::monitor
+ EFF1["⚡ EFFICIENCY:
• Doc generation only on changes
• Debounce prevents spam
• Hash-based change detection
• Batch processing"]:::change
+
SEC1 -.-> MCP_SERVER
SEC2 -.-> GIT_REPO
+ EFF1 -.-> DETECTOR
```
---