feat: implement template-based documentation generation system for Proxmox
Some checks failed
CI/CD Pipeline / Run Tests (push) Has been cancelled
CI/CD Pipeline / Security Scanning (push) Has been cancelled
CI/CD Pipeline / Build and Push Docker Images (api) (push) Has been cancelled
CI/CD Pipeline / Build and Push Docker Images (chat) (push) Has been cancelled
CI/CD Pipeline / Build and Push Docker Images (frontend) (push) Has been cancelled
CI/CD Pipeline / Build and Push Docker Images (worker) (push) Has been cancelled
CI/CD Pipeline / Deploy to Staging (push) Has been cancelled
CI/CD Pipeline / Deploy to Production (push) Has been cancelled
CI/CD Pipeline / Generate Documentation (push) Has started running
CI/CD Pipeline / Lint Code (push) Has started running
Some checks failed
CI/CD Pipeline / Run Tests (push) Has been cancelled
CI/CD Pipeline / Security Scanning (push) Has been cancelled
CI/CD Pipeline / Build and Push Docker Images (api) (push) Has been cancelled
CI/CD Pipeline / Build and Push Docker Images (chat) (push) Has been cancelled
CI/CD Pipeline / Build and Push Docker Images (frontend) (push) Has been cancelled
CI/CD Pipeline / Build and Push Docker Images (worker) (push) Has been cancelled
CI/CD Pipeline / Deploy to Staging (push) Has been cancelled
CI/CD Pipeline / Deploy to Production (push) Has been cancelled
CI/CD Pipeline / Generate Documentation (push) Has started running
CI/CD Pipeline / Lint Code (push) Has started running
Implement a scalable system for automatic documentation generation from infrastructure systems, preventing LLM context overload through template-driven sectioning. **New Features:** 1. **YAML Template System** (`templates/documentation/proxmox.yaml`) - Define documentation sections independently - Specify data requirements per section - Configure prompts, generation settings, and scheduling - Prevents LLM context overflow by sectioning data 2. **Template-Based Generator** (`src/datacenter_docs/generators/template_generator.py`) - Load and parse YAML templates - Generate documentation sections independently - Extract only required data for each section - Save sections individually to files and database - Combine sections with table of contents 3. **Celery Tasks** (`src/datacenter_docs/workers/documentation_tasks.py`) - `collect_and_generate_docs`: Collect data and generate docs - `generate_proxmox_docs`: Scheduled Proxmox documentation (daily at 2 AM) - `generate_all_docs`: Generate docs for all systems in parallel - `index_generated_docs`: Index generated docs into vector store for RAG - `full_docs_pipeline`: Complete workflow (collect → generate → index) 4. **Scheduled Jobs** (updated `celery_app.py`) - Daily Proxmox documentation generation - Every 6 hours: all systems documentation - Weekly: full pipeline with indexing - Proper task routing and rate limiting 5. **Test Script** (`scripts/test_proxmox_docs.py`) - End-to-end testing of documentation generation - Mock data collection from Proxmox - Template-based generation - File and database storage 6. **Configuration Updates** (`src/datacenter_docs/utils/config.py`) - Add port configuration fields for Docker services - Add MongoDB and Redis credentials - Support all required environment variables **Proxmox Documentation Sections:** - Infrastructure Overview (cluster, nodes, stats) - Virtual Machines Inventory - LXC Containers Inventory - Storage Configuration - Network Configuration - Maintenance Procedures **Benefits:** - Scalable to multiple infrastructure systems - Prevents LLM context window overflow - Independent section generation - Scheduled automatic updates - Vector store integration for RAG chat - Template-driven approach for consistency 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
221
templates/documentation/proxmox.yaml
Normal file
221
templates/documentation/proxmox.yaml
Normal file
@@ -0,0 +1,221 @@
|
||||
# Proxmox Documentation Template
|
||||
# Defines documentation sections to generate from Proxmox data
|
||||
# Each section is generated independently to avoid LLM context overload
|
||||
|
||||
metadata:
|
||||
name: "Proxmox Virtual Environment"
|
||||
collector: "proxmox"
|
||||
version: "1.0.0"
|
||||
description: "Documentation template for Proxmox VE infrastructure"
|
||||
|
||||
# Documentation sections - each generates a separate markdown file
|
||||
sections:
|
||||
- id: "proxmox_overview"
|
||||
title: "Proxmox Infrastructure Overview"
|
||||
category: "infrastructure"
|
||||
priority: 1
|
||||
description: "High-level overview of Proxmox cluster and resources"
|
||||
data_requirements:
|
||||
- "cluster"
|
||||
- "statistics"
|
||||
- "nodes"
|
||||
prompt_template: |
|
||||
Generate comprehensive documentation for our Proxmox Virtual Environment cluster.
|
||||
|
||||
**Cluster Information:**
|
||||
{cluster}
|
||||
|
||||
**Infrastructure Statistics:**
|
||||
{statistics}
|
||||
|
||||
**Nodes:**
|
||||
{nodes}
|
||||
|
||||
Create a well-structured markdown document that includes:
|
||||
1. Cluster overview with key statistics
|
||||
2. Node inventory and status
|
||||
3. Resource allocation summary (CPU, RAM, Storage)
|
||||
4. High availability status
|
||||
5. Capacity planning insights
|
||||
|
||||
Use tables, bullet points, and clear sections. Include actual values from the data.
|
||||
|
||||
- id: "proxmox_vms"
|
||||
title: "Virtual Machines Inventory"
|
||||
category: "virtualization"
|
||||
priority: 2
|
||||
description: "Complete inventory of QEMU virtual machines"
|
||||
data_requirements:
|
||||
- "vms"
|
||||
- "nodes"
|
||||
prompt_template: |
|
||||
Generate detailed documentation for all virtual machines in the Proxmox cluster.
|
||||
|
||||
**Virtual Machines:**
|
||||
{vms}
|
||||
|
||||
**Nodes:**
|
||||
{nodes}
|
||||
|
||||
Create documentation that includes:
|
||||
1. VM inventory table (VMID, Name, Node, Status, vCPU, RAM, Disk)
|
||||
2. VMs grouped by node
|
||||
3. VMs grouped by status (running/stopped)
|
||||
4. Resource allocation per VM
|
||||
5. Naming conventions and patterns observed
|
||||
6. Recommendations for VM placement and balancing
|
||||
|
||||
Use markdown tables and organize information clearly.
|
||||
|
||||
- id: "proxmox_containers"
|
||||
title: "LXC Containers Inventory"
|
||||
category: "virtualization"
|
||||
priority: 3
|
||||
description: "Complete inventory of LXC containers"
|
||||
data_requirements:
|
||||
- "containers"
|
||||
- "nodes"
|
||||
prompt_template: |
|
||||
Generate detailed documentation for all LXC containers in the Proxmox cluster.
|
||||
|
||||
**Containers:**
|
||||
{containers}
|
||||
|
||||
**Nodes:**
|
||||
{nodes}
|
||||
|
||||
Create documentation that includes:
|
||||
1. Container inventory table (VMID, Name, Node, Status, vCPU, RAM, Disk)
|
||||
2. Containers grouped by node
|
||||
3. Containers grouped by status (running/stopped)
|
||||
4. Resource allocation per container
|
||||
5. Use cases and patterns for containers vs VMs
|
||||
6. Recommendations for container management
|
||||
|
||||
Use markdown tables and clear organization.
|
||||
|
||||
- id: "proxmox_storage"
|
||||
title: "Storage Configuration"
|
||||
category: "storage"
|
||||
priority: 4
|
||||
description: "Storage pools and allocation"
|
||||
data_requirements:
|
||||
- "storage"
|
||||
- "statistics"
|
||||
prompt_template: |
|
||||
Generate comprehensive storage documentation for the Proxmox cluster.
|
||||
|
||||
**Storage Pools:**
|
||||
{storage}
|
||||
|
||||
**Overall Statistics:**
|
||||
{statistics}
|
||||
|
||||
Create documentation that includes:
|
||||
1. Storage inventory table (Name, Type, Total, Used, Available, Usage %)
|
||||
2. Storage types explained (local, NFS, Ceph, etc.)
|
||||
3. Content types per storage (images, ISO, containers)
|
||||
4. Storage capacity analysis
|
||||
5. Performance considerations
|
||||
6. Backup storage recommendations
|
||||
7. Capacity planning and alerts
|
||||
|
||||
Use markdown tables, charts (if possible), and clear sections.
|
||||
|
||||
- id: "proxmox_networking"
|
||||
title: "Network Configuration"
|
||||
category: "network"
|
||||
priority: 5
|
||||
description: "Network bridges and configuration"
|
||||
data_requirements:
|
||||
- "networks"
|
||||
- "nodes"
|
||||
prompt_template: |
|
||||
Generate network configuration documentation for the Proxmox cluster.
|
||||
|
||||
**Network Interfaces:**
|
||||
{networks}
|
||||
|
||||
**Nodes:**
|
||||
{nodes}
|
||||
|
||||
Create documentation that includes:
|
||||
1. Network bridges inventory (Bridge, Type, CIDR, Ports, Purpose)
|
||||
2. Network topology diagram (text-based or description)
|
||||
3. VLAN configuration if present
|
||||
4. Network purposes (management, VM, storage, etc.)
|
||||
5. Best practices for network separation
|
||||
6. Troubleshooting guides for common network issues
|
||||
|
||||
Use markdown tables and clear explanations.
|
||||
|
||||
- id: "proxmox_maintenance"
|
||||
title: "Maintenance Procedures"
|
||||
category: "operations"
|
||||
priority: 6
|
||||
description: "Standard maintenance and operational procedures"
|
||||
data_requirements:
|
||||
- "nodes"
|
||||
- "cluster"
|
||||
- "vms"
|
||||
- "containers"
|
||||
prompt_template: |
|
||||
Generate operational and maintenance documentation for the Proxmox cluster.
|
||||
|
||||
**Cluster Info:**
|
||||
{cluster}
|
||||
|
||||
**Nodes:**
|
||||
{nodes}
|
||||
|
||||
Based on the cluster configuration, create documentation that includes:
|
||||
1. **Backup Procedures**
|
||||
- VM/Container backup strategies
|
||||
- Configuration backup
|
||||
- Retention policies
|
||||
|
||||
2. **Update Procedures**
|
||||
- Proxmox VE updates
|
||||
- Kernel updates
|
||||
- Rolling updates for HA clusters
|
||||
|
||||
3. **Monitoring**
|
||||
- Key metrics to monitor
|
||||
- Alert thresholds
|
||||
- Dashboard recommendations
|
||||
|
||||
4. **Common Tasks**
|
||||
- Creating VMs/Containers
|
||||
- Migration procedures
|
||||
- Storage management
|
||||
- Snapshot management
|
||||
|
||||
5. **Troubleshooting**
|
||||
- Common issues and solutions
|
||||
- Log locations
|
||||
- Recovery procedures
|
||||
|
||||
6. **Emergency Contacts**
|
||||
- Escalation procedures
|
||||
- Vendor support information
|
||||
|
||||
Make it practical and actionable for operations team.
|
||||
|
||||
# Generation settings
|
||||
generation:
|
||||
max_tokens: 4000
|
||||
temperature: 0.7
|
||||
language: "en" # Default language, can be overridden
|
||||
|
||||
# Output configuration
|
||||
output:
|
||||
directory: "output/proxmox"
|
||||
filename_pattern: "{section_id}.md"
|
||||
save_to_database: true
|
||||
save_to_file: true
|
||||
|
||||
# Scheduling (for Celery tasks)
|
||||
schedule:
|
||||
enabled: true
|
||||
cron: "0 2 * * *" # Daily at 2 AM
|
||||
timezone: "UTC"
|
||||
Reference in New Issue
Block a user