feat: implement template-based documentation generation system for Proxmox
Some checks failed
CI/CD Pipeline / Run Tests (push) Has been cancelled
CI/CD Pipeline / Security Scanning (push) Has been cancelled
CI/CD Pipeline / Build and Push Docker Images (api) (push) Has been cancelled
CI/CD Pipeline / Build and Push Docker Images (chat) (push) Has been cancelled
CI/CD Pipeline / Build and Push Docker Images (frontend) (push) Has been cancelled
CI/CD Pipeline / Build and Push Docker Images (worker) (push) Has been cancelled
CI/CD Pipeline / Deploy to Staging (push) Has been cancelled
CI/CD Pipeline / Deploy to Production (push) Has been cancelled
CI/CD Pipeline / Generate Documentation (push) Has started running
CI/CD Pipeline / Lint Code (push) Has started running

Implement a scalable system for automatic documentation generation from infrastructure
systems, preventing LLM context overload through template-driven sectioning.

**New Features:**

1. **YAML Template System** (`templates/documentation/proxmox.yaml`)
   - Define documentation sections independently
   - Specify data requirements per section
   - Configure prompts, generation settings, and scheduling
   - Prevents LLM context overflow by sectioning data

2. **Template-Based Generator** (`src/datacenter_docs/generators/template_generator.py`)
   - Load and parse YAML templates
   - Generate documentation sections independently
   - Extract only required data for each section
   - Save sections individually to files and database
   - Combine sections with table of contents

3. **Celery Tasks** (`src/datacenter_docs/workers/documentation_tasks.py`)
   - `collect_and_generate_docs`: Collect data and generate docs
   - `generate_proxmox_docs`: Scheduled Proxmox documentation (daily at 2 AM)
   - `generate_all_docs`: Generate docs for all systems in parallel
   - `index_generated_docs`: Index generated docs into vector store for RAG
   - `full_docs_pipeline`: Complete workflow (collect → generate → index)

4. **Scheduled Jobs** (updated `celery_app.py`)
   - Daily Proxmox documentation generation
   - Every 6 hours: all systems documentation
   - Weekly: full pipeline with indexing
   - Proper task routing and rate limiting

5. **Test Script** (`scripts/test_proxmox_docs.py`)
   - End-to-end testing of documentation generation
   - Mock data collection from Proxmox
   - Template-based generation
   - File and database storage

6. **Configuration Updates** (`src/datacenter_docs/utils/config.py`)
   - Add port configuration fields for Docker services
   - Add MongoDB and Redis credentials
   - Support all required environment variables

**Proxmox Documentation Sections:**
- Infrastructure Overview (cluster, nodes, stats)
- Virtual Machines Inventory
- LXC Containers Inventory
- Storage Configuration
- Network Configuration
- Maintenance Procedures

**Benefits:**
- Scalable to multiple infrastructure systems
- Prevents LLM context window overflow
- Independent section generation
- Scheduled automatic updates
- Vector store integration for RAG chat
- Template-driven approach for consistency

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2025-10-20 19:23:30 +02:00
parent 27dd9e00b6
commit 16fc8e2659
6 changed files with 1178 additions and 1 deletions

View File

@@ -0,0 +1,221 @@
# Proxmox Documentation Template
# Defines documentation sections to generate from Proxmox data
# Each section is generated independently to avoid LLM context overload
metadata:
name: "Proxmox Virtual Environment"
collector: "proxmox"
version: "1.0.0"
description: "Documentation template for Proxmox VE infrastructure"
# Documentation sections - each generates a separate markdown file
sections:
- id: "proxmox_overview"
title: "Proxmox Infrastructure Overview"
category: "infrastructure"
priority: 1
description: "High-level overview of Proxmox cluster and resources"
data_requirements:
- "cluster"
- "statistics"
- "nodes"
prompt_template: |
Generate comprehensive documentation for our Proxmox Virtual Environment cluster.
**Cluster Information:**
{cluster}
**Infrastructure Statistics:**
{statistics}
**Nodes:**
{nodes}
Create a well-structured markdown document that includes:
1. Cluster overview with key statistics
2. Node inventory and status
3. Resource allocation summary (CPU, RAM, Storage)
4. High availability status
5. Capacity planning insights
Use tables, bullet points, and clear sections. Include actual values from the data.
- id: "proxmox_vms"
title: "Virtual Machines Inventory"
category: "virtualization"
priority: 2
description: "Complete inventory of QEMU virtual machines"
data_requirements:
- "vms"
- "nodes"
prompt_template: |
Generate detailed documentation for all virtual machines in the Proxmox cluster.
**Virtual Machines:**
{vms}
**Nodes:**
{nodes}
Create documentation that includes:
1. VM inventory table (VMID, Name, Node, Status, vCPU, RAM, Disk)
2. VMs grouped by node
3. VMs grouped by status (running/stopped)
4. Resource allocation per VM
5. Naming conventions and patterns observed
6. Recommendations for VM placement and balancing
Use markdown tables and organize information clearly.
- id: "proxmox_containers"
title: "LXC Containers Inventory"
category: "virtualization"
priority: 3
description: "Complete inventory of LXC containers"
data_requirements:
- "containers"
- "nodes"
prompt_template: |
Generate detailed documentation for all LXC containers in the Proxmox cluster.
**Containers:**
{containers}
**Nodes:**
{nodes}
Create documentation that includes:
1. Container inventory table (VMID, Name, Node, Status, vCPU, RAM, Disk)
2. Containers grouped by node
3. Containers grouped by status (running/stopped)
4. Resource allocation per container
5. Use cases and patterns for containers vs VMs
6. Recommendations for container management
Use markdown tables and clear organization.
- id: "proxmox_storage"
title: "Storage Configuration"
category: "storage"
priority: 4
description: "Storage pools and allocation"
data_requirements:
- "storage"
- "statistics"
prompt_template: |
Generate comprehensive storage documentation for the Proxmox cluster.
**Storage Pools:**
{storage}
**Overall Statistics:**
{statistics}
Create documentation that includes:
1. Storage inventory table (Name, Type, Total, Used, Available, Usage %)
2. Storage types explained (local, NFS, Ceph, etc.)
3. Content types per storage (images, ISO, containers)
4. Storage capacity analysis
5. Performance considerations
6. Backup storage recommendations
7. Capacity planning and alerts
Use markdown tables, charts (if possible), and clear sections.
- id: "proxmox_networking"
title: "Network Configuration"
category: "network"
priority: 5
description: "Network bridges and configuration"
data_requirements:
- "networks"
- "nodes"
prompt_template: |
Generate network configuration documentation for the Proxmox cluster.
**Network Interfaces:**
{networks}
**Nodes:**
{nodes}
Create documentation that includes:
1. Network bridges inventory (Bridge, Type, CIDR, Ports, Purpose)
2. Network topology diagram (text-based or description)
3. VLAN configuration if present
4. Network purposes (management, VM, storage, etc.)
5. Best practices for network separation
6. Troubleshooting guides for common network issues
Use markdown tables and clear explanations.
- id: "proxmox_maintenance"
title: "Maintenance Procedures"
category: "operations"
priority: 6
description: "Standard maintenance and operational procedures"
data_requirements:
- "nodes"
- "cluster"
- "vms"
- "containers"
prompt_template: |
Generate operational and maintenance documentation for the Proxmox cluster.
**Cluster Info:**
{cluster}
**Nodes:**
{nodes}
Based on the cluster configuration, create documentation that includes:
1. **Backup Procedures**
- VM/Container backup strategies
- Configuration backup
- Retention policies
2. **Update Procedures**
- Proxmox VE updates
- Kernel updates
- Rolling updates for HA clusters
3. **Monitoring**
- Key metrics to monitor
- Alert thresholds
- Dashboard recommendations
4. **Common Tasks**
- Creating VMs/Containers
- Migration procedures
- Storage management
- Snapshot management
5. **Troubleshooting**
- Common issues and solutions
- Log locations
- Recovery procedures
6. **Emergency Contacts**
- Escalation procedures
- Vendor support information
Make it practical and actionable for operations team.
# Generation settings
generation:
max_tokens: 4000
temperature: 0.7
language: "en" # Default language, can be overridden
# Output configuration
output:
directory: "output/proxmox"
filename_pattern: "{section_id}.md"
save_to_database: true
save_to_file: true
# Scheduling (for Celery tasks)
schedule:
enabled: true
cron: "0 2 * * *" # Daily at 2 AM
timezone: "UTC"