Daniele Viti
5b94e0a046
Some checks failed
Build / Build & Push Docker Images (api) (push) Has been cancelled
Build / Build & Push Docker Images (chat) (push) Has been cancelled
Build / Build & Push Docker Images (frontend) (push) Has been cancelled
Build / Build & Push Docker Images (worker) (push) Has been cancelled
Build / Code Quality Checks (push) Has been cancelled
374 lines
15 KiB
Markdown
374 lines
15 KiB
Markdown
# 📚 Automated Infrastructure Documentation System
|
|
|
|
Sistema automatizzato per la generazione e mantenimento della documentazione tecnica dell'infrastruttura aziendale tramite LLM locale con validazione umana e pubblicazione GitOps.
|
|
|
|
[](https://opensource.org/licenses/MIT)
|
|
[](https://www.python.org/downloads/)
|
|
[](https://kafka.apache.org/)
|
|
|
|
## 📋 Indice
|
|
|
|
- [Overview](#overview)
|
|
- [Architettura](#architettura)
|
|
- [Schema Architetturale](#schema-architetturale)
|
|
- [Schema Tecnico](#schema-tecnico)
|
|
- [Contatti](#contatti)
|
|
|
|
## 🎯 Overview
|
|
|
|
Sistema progettato per **automatizzare la creazione e l'aggiornamento della documentazione tecnica** di sistemi infrastrutturali complessi (VMware, Kubernetes, Linux, Cisco, ecc.) utilizzando un Large Language Model locale (Qwen).
|
|
|
|
### Caratteristiche Principali
|
|
|
|
- ✅ **Raccolta dati asincrona** da molteplici sistemi infrastrutturali
|
|
- ✅ **Isolamento di sicurezza**: LLM non accede mai ai sistemi live
|
|
- ✅ **Event-driven architecture** con Apache Kafka
|
|
- ✅ **LLM locale on-premise** (Qwen) tramite MCP Server
|
|
- ✅ **Human-in-the-loop validation** con workflow GitOps
|
|
- ✅ **CI/CD automatizzato** per pubblicazione
|
|
|
|
## 🏗️ Architettura
|
|
|
|
Il sistema è suddiviso in **3 flussi principali**:
|
|
|
|
1. **Raccolta Dati (Background)**: Connettori interrogano periodicamente i sistemi infrastrutturali tramite API e pubblicano i dati su Kafka
|
|
2. **Generazione Documentazione (On-Demand)**: LLM locale (Qwen) genera markdown interrogando Kafka/Redis tramite MCP Server
|
|
3. **Validazione e Pubblicazione (GitOps)**: Review umana su Pull Request e deploy automatico via CI/CD
|
|
|
|
> **Principio di Sicurezza**: L'LLM non ha mai accesso diretto ai sistemi infrastrutturali. Tutti i dati passano attraverso Kafka/Redis.
|
|
|
|
---
|
|
|
|
## 📊 Schema Architetturale
|
|
|
|
### Management View
|
|
|
|
Schema semplificato per presentazioni executive e management.
|
|
|
|
<!--
|
|
===========================================
|
|
INCOLLA QUI LO SCHEMA ARCHITETTURALE
|
|
(quello per il management, più semplice)
|
|
===========================================
|
|
-->
|
|
|
|
```mermaid
|
|
graph TB
|
|
%% Styling
|
|
classDef infrastructure fill:#e1f5ff,stroke:#01579b,stroke-width:3px,color:#333
|
|
classDef kafka fill:#fff3e0,stroke:#e65100,stroke-width:3px,color:#333
|
|
classDef cache fill:#f3e5f5,stroke:#4a148c,stroke-width:3px,color:#333
|
|
classDef llm fill:#e8f5e9,stroke:#1b5e20,stroke-width:3px,color:#333
|
|
classDef git fill:#fce4ec,stroke:#880e4f,stroke-width:3px,color:#333
|
|
classDef human fill:#fff9c4,stroke:#f57f17,stroke-width:3px,color:#333
|
|
|
|
%% ========================================
|
|
%% FLUSSO 1: RACCOLTA DATI (Background)
|
|
%% ========================================
|
|
|
|
INFRA[("🏢 SISTEMI<br/>INFRASTRUTTURALI<br/><br/>VMware | K8s | Linux | Cisco")]:::infrastructure
|
|
|
|
CONN["🔌 CONNETTORI<br/>Polling Automatico"]:::infrastructure
|
|
|
|
KAFKA[("📨 APACHE KAFKA<br/>Message Broker<br/>+ Persistenza")]:::kafka
|
|
|
|
CONSUMER["⚙️ KAFKA CONSUMER<br/>Processor Service"]:::kafka
|
|
|
|
REDIS[("💾 REDIS CACHE<br/>(Opzionale)<br/>Performance Layer")]:::cache
|
|
|
|
INFRA -->|"API Polling<br/>Continuo"| CONN
|
|
CONN -->|"Publish<br/>Eventi"| KAFKA
|
|
KAFKA -->|"Consume<br/>Stream"| CONSUMER
|
|
CONSUMER -.->|"Update<br/>Opzionale"| REDIS
|
|
|
|
%% ========================================
|
|
%% FLUSSO 2: GENERAZIONE DOCUMENTAZIONE
|
|
%% ========================================
|
|
|
|
USER["👤 UTENTE<br/>Richiesta Doc"]:::human
|
|
|
|
LLM["🤖 LLM ENGINE<br/>Claude / GPT"]:::llm
|
|
|
|
MCP["🔧 MCP SERVER<br/>API Control Platform"]:::llm
|
|
|
|
DOC["📄 DOCUMENTO<br/>Markdown Generato"]:::llm
|
|
|
|
USER -->|"1. Prompt"| LLM
|
|
LLM -->|"2. Tool Call"| MCP
|
|
MCP -->|"3a. Query"| KAFKA
|
|
MCP -.->|"3b. Query<br/>Fast"| REDIS
|
|
KAFKA -->|"4a. Dati"| MCP
|
|
REDIS -.->|"4b. Dati"| MCP
|
|
MCP -->|"5. Context"| LLM
|
|
LLM -->|"6. Genera"| DOC
|
|
|
|
%% ========================================
|
|
%% FLUSSO 3: VALIDAZIONE E PUBBLICAZIONE
|
|
%% ========================================
|
|
|
|
GIT["📦 GITLAB<br/>Repository"]:::git
|
|
|
|
PR["🔀 PULL REQUEST<br/>Review Automatica"]:::git
|
|
|
|
TECH["👨💼 TEAM TECNICO<br/>Validazione Umana"]:::human
|
|
|
|
PIPELINE["⚡ CI/CD PIPELINE<br/>GitLab Runner"]:::git
|
|
|
|
MKDOCS["📚 MKDOCS<br/>Static Site Generator"]:::git
|
|
|
|
WEB["🌐 DOCUMENTAZIONE<br/>GitLab Pages<br/>(Pubblicata)"]:::git
|
|
|
|
DOC -->|"Push +<br/>Branch"| GIT
|
|
GIT -->|"Crea"| PR
|
|
PR -->|"Notifica"| TECH
|
|
TECH -->|"Approva +<br/>Merge"| GIT
|
|
GIT -->|"Trigger"| PIPELINE
|
|
PIPELINE -->|"Build"| MKDOCS
|
|
MKDOCS -->|"Deploy"| WEB
|
|
|
|
%% ========================================
|
|
%% ANNOTAZIONI SICUREZZA
|
|
%% ========================================
|
|
|
|
SECURITY["🔒 SICUREZZA<br/>LLM isolato dai sistemi live"]:::human
|
|
PERF["⚡ PERFORMANCE<br/>Cache Redis opzionale"]:::cache
|
|
|
|
LLM -.->|"NESSUN<br/>ACCESSO"| INFRA
|
|
|
|
SECURITY -.-> LLM
|
|
PERF -.-> REDIS
|
|
```
|
|
|
|
---
|
|
|
|
## 🔧 Schema Tecnico
|
|
|
|
### Implementation View
|
|
|
|
Schema dettagliato per il team tecnico con specifiche implementative.
|
|
|
|
<!--
|
|
===========================================
|
|
INCOLLA QUI LO SCHEMA TECNICO DETTAGLIATO
|
|
(quello con tutti i dettagli per gli sviluppatori)
|
|
===========================================
|
|
-->
|
|
|
|
```mermaid
|
|
graph TB
|
|
%% Styling tecnico
|
|
classDef infra fill:#e1f5ff,stroke:#01579b,stroke-width:2px,color:#333,font-size:11px
|
|
classDef connector fill:#e3f2fd,stroke:#1565c0,stroke-width:2px,color:#333,font-size:11px
|
|
classDef kafka fill:#fff3e0,stroke:#e65100,stroke-width:2px,color:#333,font-size:11px
|
|
classDef cache fill:#f3e5f5,stroke:#4a148c,stroke-width:2px,color:#333,font-size:11px
|
|
classDef llm fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px,color:#333,font-size:11px
|
|
classDef git fill:#fce4ec,stroke:#880e4f,stroke-width:2px,color:#333,font-size:11px
|
|
classDef monitor fill:#fff8e1,stroke:#f57f17,stroke-width:2px,color:#333,font-size:11px
|
|
|
|
%% =====================================
|
|
%% LAYER 1: SISTEMI SORGENTE
|
|
%% =====================================
|
|
|
|
subgraph SOURCES["🏢 INFRASTRUCTURE SOURCES"]
|
|
VCENTER["VMware vCenter<br/>API: vSphere REST 7.0+<br/>Port: 443/HTTPS<br/>Auth: API Token"]:::infra
|
|
K8S_API["Kubernetes API<br/>API: v1.28+<br/>Port: 6443/HTTPS<br/>Auth: ServiceAccount + RBAC"]:::infra
|
|
LINUX["Linux Servers<br/>Protocol: SSH/Ansible<br/>Port: 22<br/>Auth: SSH Keys"]:::infra
|
|
CISCO["Cisco Devices<br/>Protocol: NETCONF/RESTCONF<br/>Port: 830/443<br/>Auth: AAA"]:::infra
|
|
end
|
|
|
|
%% =====================================
|
|
%% LAYER 2: CONNETTORI
|
|
%% =====================================
|
|
|
|
subgraph CONNECTORS["🔌 DATA COLLECTORS (Python/Go)"]
|
|
CONN_VM["VMware Collector<br/>Lang: Python 3.11<br/>Lib: pyvmomi<br/>Schedule: */15 * * * *<br/>Output: JSON"]:::connector
|
|
|
|
CONN_K8S["K8s Collector<br/>Lang: Python 3.11<br/>Lib: kubernetes-client<br/>Schedule: */5 * * * *<br/>Resources: pods,svc,ing,deploy"]:::connector
|
|
|
|
CONN_LNX["Linux Collector<br/>Lang: Python 3.11<br/>Lib: paramiko/ansible<br/>Schedule: */30 * * * *<br/>Data: sysinfo,packages,services"]:::connector
|
|
|
|
CONN_CSC["Cisco Collector<br/>Lang: Python 3.11<br/>Lib: ncclient<br/>Schedule: */30 * * * *<br/>Data: interfaces,routing,vlans"]:::connector
|
|
end
|
|
|
|
VCENTER -->|"GET /api/vcenter/vm"| CONN_VM
|
|
K8S_API -->|"kubectl proxy<br/>API calls"| CONN_K8S
|
|
LINUX -->|"SSH batch<br/>commands"| CONN_LNX
|
|
CISCO -->|"NETCONF<br/>get-config"| CONN_CSC
|
|
|
|
%% =====================================
|
|
%% LAYER 3: MESSAGE BROKER
|
|
%% =====================================
|
|
|
|
subgraph MESSAGING["📨 KAFKA CLUSTER (3 brokers)"]
|
|
KAFKA_TOPICS["Kafka Topics:<br/>• vmware.inventory (P:6, R:3)<br/>• k8s.resources (P:12, R:3)<br/>• linux.systems (P:3, R:3)<br/>• cisco.network (P:3, R:3)<br/>Retention: 7 days<br/>Format: JSON + Schema Registry"]:::kafka
|
|
|
|
SCHEMA["Schema Registry<br/>Avro Schemas<br/>Versioning enabled<br/>Port: 8081"]:::kafka
|
|
end
|
|
|
|
CONN_VM -->|"Producer<br/>Batch 100 msg"| KAFKA_TOPICS
|
|
CONN_K8S -->|"Producer<br/>Batch 100 msg"| KAFKA_TOPICS
|
|
CONN_LNX -->|"Producer<br/>Batch 50 msg"| KAFKA_TOPICS
|
|
CONN_CSC -->|"Producer<br/>Batch 50 msg"| KAFKA_TOPICS
|
|
|
|
KAFKA_TOPICS <--> SCHEMA
|
|
|
|
%% =====================================
|
|
%% LAYER 4: PROCESSING & CACHE
|
|
%% =====================================
|
|
|
|
subgraph PROCESSING["⚙️ STREAM PROCESSING"]
|
|
CONSUMER_GRP["Kafka Consumer Group<br/>Group ID: doc-consumers<br/>Lang: Python 3.11<br/>Lib: kafka-python<br/>Workers: 6<br/>Commit: auto (5s)"]:::kafka
|
|
|
|
PROCESSOR["Data Processor<br/>• Validation<br/>• Transformation<br/>• Enrichment<br/>• Deduplication"]:::kafka
|
|
end
|
|
|
|
KAFKA_TOPICS -->|"Subscribe<br/>offset management"| CONSUMER_GRP
|
|
CONSUMER_GRP --> PROCESSOR
|
|
|
|
subgraph STORAGE["💾 CACHE LAYER (Optional)"]
|
|
REDIS_CLUSTER["Redis Cluster<br/>Mode: Cluster (6 nodes)<br/>Port: 6379<br/>Persistence: RDB + AOF<br/>Memory: 64GB<br/>Eviction: allkeys-lru"]:::cache
|
|
|
|
REDIS_KEYS["Key Structure:<br/>• vmware:vcenter-id:vms<br/>• k8s:cluster:namespace:resource<br/>• linux:hostname:info<br/>• cisco:device-id:config<br/>TTL: 1-24h based on type"]:::cache
|
|
end
|
|
|
|
PROCESSOR -.->|"SET/HSET<br/>Pipeline batch"| REDIS_CLUSTER
|
|
REDIS_CLUSTER --> REDIS_KEYS
|
|
|
|
%% =====================================
|
|
%% LAYER 5: LLM & MCP
|
|
%% =====================================
|
|
|
|
subgraph LLM_LAYER["🤖 AI GENERATION LAYER"]
|
|
LLM_ENGINE["LLM Engine<br/>Model: Claude Sonnet 4 / GPT-4<br/>API: Anthropic/OpenAI<br/>Temp: 0.3<br/>Max Tokens: 4096<br/>Timeout: 120s"]:::llm
|
|
|
|
MCP_SERVER["MCP Server<br/>Lang: TypeScript/Node.js<br/>Port: 3000<br/>Protocol: JSON-RPC 2.0<br/>Auth: JWT tokens"]:::llm
|
|
|
|
MCP_TOOLS["MCP Tools:<br/>• getVMwareInventory(vcenter)<br/>• getK8sResources(cluster,ns,type)<br/>• getLinuxSystemInfo(hostname)<br/>• getCiscoConfig(device,section)<br/>• queryTimeRange(start,end)<br/>Return: JSON + Metadata"]:::llm
|
|
end
|
|
|
|
LLM_ENGINE <-->|"Tool calls<br/>JSON-RPC"| MCP_SERVER
|
|
MCP_SERVER --> MCP_TOOLS
|
|
|
|
MCP_TOOLS -->|"1. Query Kafka Consumer API<br/>GET /api/v1/data"| CONSUMER_GRP
|
|
MCP_TOOLS -.->|"2. Fallback Redis<br/>MGET/HGETALL"| REDIS_CLUSTER
|
|
|
|
CONSUMER_GRP -->|"JSON Response<br/>+ Timestamps"| MCP_TOOLS
|
|
REDIS_CLUSTER -.->|"Cached JSON<br/>Fast response"| MCP_TOOLS
|
|
|
|
MCP_TOOLS -->|"Structured Data<br/>+ Context"| LLM_ENGINE
|
|
|
|
subgraph OUTPUT["📝 DOCUMENT GENERATION"]
|
|
TEMPLATE["Template Engine<br/>Format: Jinja2<br/>Templates: markdown/*.j2<br/>Variables: from LLM"]:::llm
|
|
|
|
MARKDOWN["Markdown Output<br/>Format: CommonMark<br/>Metadata: YAML frontmatter<br/>Assets: diagrams in mermaid"]:::llm
|
|
|
|
VALIDATOR["Doc Validator<br/>• Markdown linting<br/>• Link checking<br/>• Schema validation"]:::llm
|
|
end
|
|
|
|
LLM_ENGINE --> TEMPLATE
|
|
TEMPLATE --> MARKDOWN
|
|
MARKDOWN --> VALIDATOR
|
|
|
|
%% =====================================
|
|
%% LAYER 6: GITOPS
|
|
%% =====================================
|
|
|
|
subgraph GITOPS["🔄 GITOPS WORKFLOW"]
|
|
GIT_REPO["GitLab Repository<br/>URL: gitlab.com/docs/infra<br/>Branch strategy: main + feature/*<br/>Protected: main (require approval)"]:::git
|
|
|
|
GIT_API["GitLab API<br/>API: v4<br/>Auth: Project Access Token<br/>Permissions: api, write_repo"]:::git
|
|
|
|
PR_AUTO["Automated PR Creator<br/>Lang: Python 3.11<br/>Lib: python-gitlab<br/>Template: .gitlab/merge_request.md"]:::git
|
|
end
|
|
|
|
VALIDATOR -->|"git add/commit/push"| GIT_REPO
|
|
GIT_REPO <--> GIT_API
|
|
GIT_API --> PR_AUTO
|
|
|
|
REVIEWER["👨💼 Technical Reviewer<br/>Role: Maintainer/Owner<br/>Review: diff + validation<br/>Approve: required (min 1)"]:::monitor
|
|
|
|
PR_AUTO -->|"Notification<br/>Email + Slack"| REVIEWER
|
|
REVIEWER -->|"Merge to main"| GIT_REPO
|
|
|
|
%% =====================================
|
|
%% LAYER 7: CI/CD & PUBLISH
|
|
%% =====================================
|
|
|
|
subgraph CICD["⚡ CI/CD PIPELINE"]
|
|
GITLAB_CI["GitLab CI/CD<br/>Runner: docker<br/>Image: python:3.11-alpine<br/>Stages: build, test, deploy"]:::git
|
|
|
|
PIPELINE_JOBS["Pipeline Jobs:<br/>1. lint (markdownlint-cli)<br/>2. build (mkdocs build)<br/>3. test (link-checker)<br/>4. deploy (rsync/s3)"]:::git
|
|
|
|
MKDOCS_CFG["MkDocs Config<br/>Theme: material<br/>Plugins: search, tags, mermaid<br/>Extensions: admonition, codehilite"]:::git
|
|
end
|
|
|
|
GIT_REPO -->|"on: push to main<br/>Webhook trigger"| GITLAB_CI
|
|
GITLAB_CI --> PIPELINE_JOBS
|
|
PIPELINE_JOBS --> MKDOCS_CFG
|
|
|
|
subgraph PUBLISH["🌐 PUBLICATION"]
|
|
STATIC_SITE["Static Site<br/>Generator: MkDocs<br/>Output: HTML/CSS/JS<br/>Assets: optimized images"]:::git
|
|
|
|
CDN["GitLab Pages / S3 + CloudFront<br/>URL: docs.company.com<br/>SSL: Let's Encrypt<br/>Cache: 1h"]:::git
|
|
|
|
SEARCH["Search Index<br/>Engine: Algolia/Meilisearch<br/>Update: on publish<br/>API: REST"]:::git
|
|
end
|
|
|
|
MKDOCS_CFG -->|"mkdocs build<br/>--strict"| STATIC_SITE
|
|
STATIC_SITE --> CDN
|
|
STATIC_SITE --> SEARCH
|
|
|
|
%% =====================================
|
|
%% LAYER 8: MONITORING & OBSERVABILITY
|
|
%% =====================================
|
|
|
|
subgraph OBSERVABILITY["📊 MONITORING & LOGGING"]
|
|
PROMETHEUS["Prometheus<br/>Metrics: collector lag, cache hit/miss<br/>Scrape: 30s<br/>Retention: 15d"]:::monitor
|
|
|
|
GRAFANA["Grafana Dashboards<br/>• Kafka metrics<br/>• Redis performance<br/>• LLM response times<br/>• Pipeline success rate"]:::monitor
|
|
|
|
ELK["ELK Stack<br/>Logs: all components<br/>Index: daily rotation<br/>Retention: 30d"]:::monitor
|
|
|
|
ALERTS["Alerting<br/>• Connector failures<br/>• Kafka lag > 10k<br/>• Redis OOM<br/>• Pipeline failures<br/>Channel: Slack + PagerDuty"]:::monitor
|
|
end
|
|
|
|
CONN_VM -.->|"metrics"| PROMETHEUS
|
|
CONN_K8S -.->|"metrics"| PROMETHEUS
|
|
KAFKA_TOPICS -.->|"metrics"| PROMETHEUS
|
|
REDIS_CLUSTER -.->|"metrics"| PROMETHEUS
|
|
MCP_SERVER -.->|"metrics"| PROMETHEUS
|
|
GITLAB_CI -.->|"metrics"| PROMETHEUS
|
|
|
|
PROMETHEUS --> GRAFANA
|
|
|
|
CONN_VM -.->|"logs"| ELK
|
|
CONSUMER_GRP -.->|"logs"| ELK
|
|
MCP_SERVER -.->|"logs"| ELK
|
|
GITLAB_CI -.->|"logs"| ELK
|
|
|
|
GRAFANA --> ALERTS
|
|
|
|
%% =====================================
|
|
%% SECURITY ANNOTATIONS
|
|
%% =====================================
|
|
|
|
SEC1["🔒 SECURITY:<br/>• All APIs use TLS 1.3<br/>• Secrets in Vault/K8s Secrets<br/>• Network: private VPC<br/>• LLM has NO direct access"]:::monitor
|
|
|
|
SEC2["🔐 AUTHENTICATION:<br/>• API Tokens rotated 90d<br/>• RBAC enforced<br/>• Audit logs enabled<br/>• MFA required for Git"]:::monitor
|
|
|
|
SEC1 -.-> MCP_SERVER
|
|
SEC2 -.-> GIT_REPO
|
|
```
|
|
|
|
---
|
|
|
|
## 📧 Contatti
|
|
|
|
- **Team**: Infrastructure Documentation Team
|
|
- **Email**: infra-docs@company.com
|
|
- **GitLab**: https://gitlab.com/company/infra-docs-automation
|
|
|
|
---
|
|
|
|
**Versione**: 1.0.0
|
|
**Ultimo aggiornamento**: 2025-10-28 |