it-ops/llm-automation-docs-and-remediation-engine
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
40824d991f1ffa22d58a50f96600122a3388af90
llm-automation-docs-and-rem…/scheme.md
Daniele Viti 40824d991f
Some checks failed
Build / Build & Push Docker Images (api) (push) Has been cancelled
Details
Build / Build & Push Docker Images (chat) (push) Has been cancelled
Details
Build / Build & Push Docker Images (frontend) (push) Has been cancelled
Details
Build / Build & Push Docker Images (worker) (push) Has been cancelled
Details
Build / Code Quality Checks (push) Has started running
Details
Update scheme.md
2025-10-28 11:27:44 +00:00

390 lines
16 KiB
Markdown
Raw Blame History

# 📚 Automated Infrastructure Documentation System
Sistema automatizzato per la generazione e mantenimento della documentazione tecnica dell'infrastruttura aziendale tramite LLM locale con validazione umana e pubblicazione GitOps.
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![Redis](https://img.shields.io/badge/Redis-7.2+-red.svg)](https://redis.io/)
## 📋 Indice
- [Overview](#overview)
- [Architettura](#architettura)
- [Schema Architetturale](#schema-architetturale)
- [Schema Tecnico](#schema-tecnico)
- [Contatti](#contatti)
## 🎯 Overview
Sistema progettato per **automatizzare la creazione e l'aggiornamento della documentazione tecnica** di sistemi infrastrutturali complessi (VMware, Kubernetes, Linux, Cisco, ecc.) utilizzando un Large Language Model locale (Qwen).
### Caratteristiche Principali
- ✅ **Raccolta dati asincrona** da molteplici sistemi infrastrutturali
- ✅ **Isolamento di sicurezza**: LLM non accede mai ai sistemi live
- ✅ **Change Detection**: Documentazione generata solo su modifiche rilevate
- ✅ **Redis Cache** per storage dati e performance
- ✅ **LLM locale on-premise** (Qwen) tramite MCP Server
- ✅ **Human-in-the-loop validation** con workflow GitOps
- ✅ **CI/CD automatizzato** per pubblicazione
## 🏗️ Architettura
Il sistema è suddiviso in **3 flussi principali**:
1. **Raccolta Dati (Background)**: Connettori interrogano periodicamente i sistemi infrastrutturali tramite API e aggiornano Redis
2. **Change Detection**: Sistema di rilevamento modifiche che attiva la generazione documentazione solo quando necessario
3. **Generazione e Pubblicazione (Triggered)**: LLM locale (Qwen) genera markdown leggendo da Redis, seguito da review umana e deploy automatico
> **Principio di Sicurezza**: L'LLM non ha mai accesso diretto ai sistemi infrastrutturali. Tutti i dati sono letti da Redis.
> **Principio di Efficienza**: La documentazione viene generata solo quando il sistema rileva modifiche nella configurazione infrastrutturale.
---
## 📊 Schema Architetturale
### Management View
Schema semplificato per presentazioni executive e management.
<!--
===========================================
INCOLLA QUI LO SCHEMA ARCHITETTURALE
(quello per il management, più semplice)
===========================================
-->
```mermaid
graph TB
%% Styling
classDef infrastructure fill:#e1f5ff,stroke:#01579b,stroke-width:3px,color:#333
classDef cache fill:#f3e5f5,stroke:#4a148c,stroke-width:3px,color:#333
classDef change fill:#fff3e0,stroke:#e65100,stroke-width:3px,color:#333
classDef llm fill:#e8f5e9,stroke:#1b5e20,stroke-width:3px,color:#333
classDef git fill:#fce4ec,stroke:#880e4f,stroke-width:3px,color:#333
classDef human fill:#fff9c4,stroke:#f57f17,stroke-width:3px,color:#333
%% ========================================
%% FLUSSO 1: RACCOLTA DATI (Background)
%% ========================================
INFRA[("🏢 SISTEMI<br/>INFRASTRUTTURALI<br/><br/>VMware | K8s | Linux | Cisco")]:::infrastructure
CONN["🔌 CONNETTORI<br/>Polling Automatico"]:::infrastructure
REDIS[("💾 REDIS CACHE<br/>Configurazione<br/>Infrastruttura")]:::cache
INFRA -->|"API Polling<br/>Continuo"| CONN
CONN -->|"Update<br/>Configurazione"| REDIS
%% ========================================
%% CHANGE DETECTION
%% ========================================
CHANGE["🔍 CHANGE DETECTOR<br/>Rileva Modifiche<br/>Configurazione"]:::change
REDIS -->|"Monitor<br/>Changes"| CHANGE
%% ========================================
%% FLUSSO 2: GENERAZIONE DOCUMENTAZIONE (Triggered)
%% ========================================
TRIGGER["⚡ TRIGGER<br/>Solo se modifiche"]:::change
USER["👤 UTENTE<br/>Richiesta Manuale"]:::human
LLM["🤖 LLM ENGINE<br/>Qwen (Locale)"]:::llm
MCP["🔧 MCP SERVER<br/>API Control Platform"]:::llm
DOC["📄 DOCUMENTO<br/>Markdown Generato"]:::llm
CHANGE -->|"Modifiche<br/>Rilevate"| TRIGGER
USER -.->|"Opzionale"| TRIGGER
TRIGGER -->|"Avvia<br/>Generazione"| LLM
LLM -->|"Tool Call"| MCP
MCP -->|"Query"| REDIS
REDIS -->|"Dati Config"| MCP
MCP -->|"Context"| LLM
LLM -->|"Genera"| DOC
%% ========================================
%% FLUSSO 3: VALIDAZIONE E PUBBLICAZIONE
%% ========================================
GIT["📦 GITLAB<br/>Repository"]:::git
PR["🔀 PULL REQUEST<br/>Review Automatica"]:::git
TECH["👨‍💼 TEAM TECNICO<br/>Validazione Umana"]:::human
PIPELINE["⚡ CI/CD PIPELINE<br/>GitLab Runner"]:::git
MKDOCS["📚 MKDOCS<br/>Static Site Generator"]:::git
WEB["🌐 DOCUMENTAZIONE<br/>GitLab Pages<br/>(Pubblicata)"]:::git
DOC -->|"Push +<br/>Branch"| GIT
GIT -->|"Crea"| PR
PR -->|"Notifica"| TECH
TECH -->|"Approva +<br/>Merge"| GIT
GIT -->|"Trigger"| PIPELINE
PIPELINE -->|"Build"| MKDOCS
MKDOCS -->|"Deploy"| WEB
%% ========================================
%% ANNOTAZIONI
%% ========================================
SECURITY["🔒 SICUREZZA<br/>LLM isolato dai sistemi live"]:::human
EFFICIENCY["⚡ EFFICIENZA<br/>Doc generata solo<br/>su modifiche"]:::change
LLM -.->|"NESSUN<br/>ACCESSO"| INFRA
SECURITY -.-> LLM
EFFICIENCY -.-> CHANGE
```
---
## 🔧 Schema Tecnico
### Implementation View
Schema dettagliato per il team tecnico con specifiche implementative.
<!--
===========================================
INCOLLA QUI LO SCHEMA TECNICO DETTAGLIATO
(quello con tutti i dettagli per gli sviluppatori)
===========================================
-->
```mermaid
graph TB
%% Styling tecnico
classDef infra fill:#e1f5ff,stroke:#01579b,stroke-width:2px,color:#333,font-size:11px
classDef connector fill:#e3f2fd,stroke:#1565c0,stroke-width:2px,color:#333,font-size:11px
classDef cache fill:#f3e5f5,stroke:#4a148c,stroke-width:2px,color:#333,font-size:11px
classDef change fill:#fff3e0,stroke:#e65100,stroke-width:2px,color:#333,font-size:11px
classDef llm fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px,color:#333,font-size:11px
classDef git fill:#fce4ec,stroke:#880e4f,stroke-width:2px,color:#333,font-size:11px
classDef monitor fill:#fff8e1,stroke:#f57f17,stroke-width:2px,color:#333,font-size:11px
%% =====================================
%% LAYER 1: SISTEMI SORGENTE
%% =====================================
subgraph SOURCES["🏢 INFRASTRUCTURE SOURCES"]
VCENTER["VMware vCenter<br/>API: vSphere REST 7.0+<br/>Port: 443/HTTPS<br/>Auth: API Token"]:::infra
K8S_API["Kubernetes API<br/>API: v1.28+<br/>Port: 6443/HTTPS<br/>Auth: ServiceAccount + RBAC"]:::infra
LINUX["Linux Servers<br/>Protocol: SSH/Ansible<br/>Port: 22<br/>Auth: SSH Keys"]:::infra
CISCO["Cisco Devices<br/>Protocol: NETCONF/RESTCONF<br/>Port: 830/443<br/>Auth: AAA"]:::infra
end
%% =====================================
%% LAYER 2: CONNETTORI
%% =====================================
subgraph CONNECTORS["🔌 DATA COLLECTORS (Python/Go)"]
CONN_VM["VMware Collector<br/>Lang: Python 3.11<br/>Lib: pyvmomi<br/>Schedule: */15 * * * *<br/>Output: JSON → Redis"]:::connector
CONN_K8S["K8s Collector<br/>Lang: Python 3.11<br/>Lib: kubernetes-client<br/>Schedule: */5 * * * *<br/>Resources: pods,svc,ing,deploy"]:::connector
CONN_LNX["Linux Collector<br/>Lang: Python 3.11<br/>Lib: paramiko/ansible<br/>Schedule: */30 * * * *<br/>Data: sysinfo,packages,services"]:::connector
CONN_CSC["Cisco Collector<br/>Lang: Python 3.11<br/>Lib: ncclient<br/>Schedule: */30 * * * *<br/>Data: interfaces,routing,vlans"]:::connector
end
VCENTER -->|"GET /api/vcenter/vm"| CONN_VM
K8S_API -->|"kubectl proxy<br/>API calls"| CONN_K8S
LINUX -->|"SSH batch<br/>commands"| CONN_LNX
CISCO -->|"NETCONF<br/>get-config"| CONN_CSC
%% =====================================
%% LAYER 3: REDIS STORAGE
%% =====================================
subgraph STORAGE["💾 REDIS CLUSTER"]
REDIS_CLUSTER["Redis Cluster<br/>Mode: Cluster (6 nodes)<br/>Port: 6379<br/>Persistence: RDB + AOF<br/>Memory: 64GB<br/>Eviction: allkeys-lru"]:::cache
REDIS_KEYS["Key Structure:<br/>• vmware:vcenter-id:vms:hash<br/>• k8s:cluster:namespace:resource:hash<br/>• linux:hostname:info:hash<br/>• cisco:device-id:config:hash<br/>• changelog:timestamp:diff<br/>TTL: 30d for data, 90d for changelog"]:::cache
end
CONN_VM -->|"HSET/HMSET<br/>+ Hash Storage"| REDIS_CLUSTER
CONN_K8S -->|"HSET/HMSET<br/>+ Hash Storage"| REDIS_CLUSTER
CONN_LNX -->|"HSET/HMSET<br/>+ Hash Storage"| REDIS_CLUSTER
CONN_CSC -->|"HSET/HMSET<br/>+ Hash Storage"| REDIS_CLUSTER
REDIS_CLUSTER --> REDIS_KEYS
%% =====================================
%% LAYER 4: CHANGE DETECTION
%% =====================================
subgraph CHANGE_DETECTION["🔍 CHANGE DETECTION SYSTEM"]
DETECTOR["Change Detector Service<br/>Lang: Python 3.11<br/>Lib: redis-py<br/>Algorithm: Hash comparison<br/>Check interval: */5 * * * *"]:::change
DIFF_ENGINE["Diff Engine<br/>• Deep object comparison<br/>• JSON diff generation<br/>• Change classification<br/>• Severity assessment"]:::change
CHANGE_LOG["Change Log Store<br/>Key: changelog:*<br/>Data: diff JSON + metadata<br/>Indexed by: timestamp, resource"]:::change
NOTIFIER["Change Notifier<br/>• Webhook triggers<br/>• Slack notifications<br/>• Event emission<br/>Target: LLM trigger"]:::change
end
REDIS_CLUSTER -->|"Monitor<br/>key changes"| DETECTOR
DETECTOR --> DIFF_ENGINE
DIFF_ENGINE -->|"Store diff"| CHANGE_LOG
CHANGE_LOG --> REDIS_CLUSTER
DIFF_ENGINE -->|"Notify if<br/>significant"| NOTIFIER
%% =====================================
%% LAYER 5: LLM TRIGGER & GENERATION
%% =====================================
subgraph TRIGGER_SYSTEM["⚡ TRIGGER SYSTEM"]
TRIGGER_SVC["Trigger Service<br/>Lang: Python 3.11<br/>Listen: Webhook + Redis Pub/Sub<br/>Debounce: 5 min<br/>Batch: multiple changes"]:::change
QUEUE["Generation Queue<br/>Type: Redis List<br/>Priority: High/Medium/Low<br/>Processing: FIFO"]:::change
end
NOTIFIER -->|"Trigger event"| TRIGGER_SVC
TRIGGER_SVC -->|"Enqueue<br/>generation task"| QUEUE
subgraph LLM_LAYER["🤖 AI GENERATION LAYER"]
LLM_ENGINE["LLM Engine<br/>Model: Qwen (Locale)<br/>API: Ollama/vLLM/LM Studio<br/>Port: 11434<br/>Temp: 0.3<br/>Max Tokens: 4096<br/>Timeout: 120s"]:::llm
MCP_SERVER["MCP Server<br/>Lang: TypeScript/Node.js<br/>Port: 3000<br/>Protocol: JSON-RPC 2.0<br/>Auth: JWT tokens"]:::llm
MCP_TOOLS["MCP Tools:<br/>• getVMwareInventory(vcenter)<br/>• getK8sResources(cluster,ns,type)<br/>• getLinuxSystemInfo(hostname)<br/>• getCiscoConfig(device,section)<br/>• getChangelog(start,end,resource)<br/>Return: JSON + Metadata"]:::llm
end
QUEUE -->|"Dequeue<br/>task"| LLM_ENGINE
LLM_ENGINE <-->|"Tool calls<br/>JSON-RPC"| MCP_SERVER
MCP_SERVER --> MCP_TOOLS
MCP_TOOLS -->|"HGETALL/MGET<br/>Read data"| REDIS_CLUSTER
REDIS_CLUSTER -->|"Config data<br/>+ Changelog"| MCP_TOOLS
MCP_TOOLS -->|"Structured Data<br/>+ Context"| LLM_ENGINE
subgraph OUTPUT["📝 DOCUMENT GENERATION"]
TEMPLATE["Template Engine<br/>Format: Jinja2<br/>Templates: markdown/*.j2<br/>Variables: from LLM"]:::llm
MARKDOWN["Markdown Output<br/>Format: CommonMark<br/>Metadata: YAML frontmatter<br/>Change summary included<br/>Assets: diagrams in mermaid"]:::llm
VALIDATOR["Doc Validator<br/>• Markdown linting<br/>• Link checking<br/>• Schema validation<br/>• Change verification"]:::llm
end
LLM_ENGINE --> TEMPLATE
TEMPLATE --> MARKDOWN
MARKDOWN --> VALIDATOR
%% =====================================
%% LAYER 6: GITOPS
%% =====================================
subgraph GITOPS["🔄 GITOPS WORKFLOW"]
GIT_REPO["GitLab Repository<br/>URL: gitlab.com/docs/infra<br/>Branch strategy: main + feature/*<br/>Protected: main (require approval)"]:::git
GIT_API["GitLab API<br/>API: v4<br/>Auth: Project Access Token<br/>Permissions: api, write_repo"]:::git
PR_AUTO["Automated PR Creator<br/>Lang: Python 3.11<br/>Lib: python-gitlab<br/>Template: .gitlab/merge_request.md<br/>Include: change summary"]:::git
end
VALIDATOR -->|"git add/commit/push"| GIT_REPO
GIT_REPO <--> GIT_API
GIT_API --> PR_AUTO
REVIEWER["👨‍💼 Technical Reviewer<br/>Role: Maintainer/Owner<br/>Review: diff + validation<br/>Check: change correlation<br/>Approve: required (min 1)"]:::monitor
PR_AUTO -->|"Notification<br/>Email + Slack"| REVIEWER
REVIEWER -->|"Merge to main"| GIT_REPO
%% =====================================
%% LAYER 7: CI/CD & PUBLISH
%% =====================================
subgraph CICD["⚡ CI/CD PIPELINE"]
GITLAB_CI["GitLab CI/CD<br/>Runner: docker<br/>Image: python:3.11-alpine<br/>Stages: build, test, deploy"]:::git
PIPELINE_JOBS["Pipeline Jobs:<br/>1. lint (markdownlint-cli)<br/>2. build (mkdocs build)<br/>3. test (link-checker)<br/>4. deploy (rsync/s3)"]:::git
MKDOCS_CFG["MkDocs Config<br/>Theme: material<br/>Plugins: search, tags, mermaid<br/>Extensions: admonition, codehilite"]:::git
end
GIT_REPO -->|"on: push to main<br/>Webhook trigger"| GITLAB_CI
GITLAB_CI --> PIPELINE_JOBS
PIPELINE_JOBS --> MKDOCS_CFG
subgraph PUBLISH["🌐 PUBLICATION"]
STATIC_SITE["Static Site<br/>Generator: MkDocs<br/>Output: HTML/CSS/JS<br/>Assets: optimized images"]:::git
CDN["GitLab Pages / S3 + CloudFront<br/>URL: docs.company.com<br/>SSL: Let's Encrypt<br/>Cache: 1h"]:::git
SEARCH["Search Index<br/>Engine: Algolia/Meilisearch<br/>Update: on publish<br/>API: REST"]:::git
end
MKDOCS_CFG -->|"mkdocs build<br/>--strict"| STATIC_SITE
STATIC_SITE --> CDN
STATIC_SITE --> SEARCH
%% =====================================
%% LAYER 8: MONITORING & OBSERVABILITY
%% =====================================
subgraph OBSERVABILITY["📊 MONITORING & LOGGING"]
PROMETHEUS["Prometheus<br/>Metrics: collector updates, changes detected<br/>Scrape: 30s<br/>Retention: 15d"]:::monitor
GRAFANA["Grafana Dashboards<br/>• Collector status<br/>• Redis performance<br/>• Change detection rate<br/>• LLM response times<br/>• Pipeline success rate"]:::monitor
ELK["ELK Stack<br/>Logs: all components<br/>Index: daily rotation<br/>Retention: 30d"]:::monitor
ALERTS["Alerting<br/>• Collector failures<br/>• Redis issues<br/>• Change detection errors<br/>• Pipeline failures<br/>Channel: Slack + PagerDuty"]:::monitor
end
CONN_VM -.->|"metrics"| PROMETHEUS
CONN_K8S -.->|"metrics"| PROMETHEUS
REDIS_CLUSTER -.->|"metrics"| PROMETHEUS
DETECTOR -.->|"metrics"| PROMETHEUS
MCP_SERVER -.->|"metrics"| PROMETHEUS
GITLAB_CI -.->|"metrics"| PROMETHEUS
PROMETHEUS --> GRAFANA
CONN_VM -.->|"logs"| ELK
DETECTOR -.->|"logs"| ELK
MCP_SERVER -.->|"logs"| ELK
GITLAB_CI -.->|"logs"| ELK
GRAFANA --> ALERTS
%% =====================================
%% SECURITY & EFFICIENCY ANNOTATIONS
%% =====================================
SEC1["🔒 SECURITY:<br/>• All APIs use TLS 1.3<br/>• Secrets in Vault/K8s Secrets<br/>• Network: private VPC<br/>• LLM has NO direct access"]:::monitor
SEC2["🔐 AUTHENTICATION:<br/>• API Tokens rotated 90d<br/>• RBAC enforced<br/>• Audit logs enabled<br/>• MFA required for Git"]:::monitor
EFF1["⚡ EFFICIENCY:<br/>• Doc generation only on changes<br/>• Debounce prevents spam<br/>• Hash-based change detection<br/>• Batch processing"]:::change
SEC1 -.-> MCP_SERVER
SEC2 -.-> GIT_REPO
EFF1 -.-> DETECTOR
```
---
## 📧 Contatti
- **Team**: Infrastructure Documentation Team
- **Email**: infra-docs@company.com
- **GitLab**: https://gitlab.com/company/infra-docs-automation
---
**Versione**: 1.0.0
**Ultimo aggiornamento**: 2025-10-28
Reference in New Issue View Git Blame Copy Permalink