feat: implement template-based documentation generation system for Proxmox
Some checks failed
CI/CD Pipeline / Run Tests (push) Has been cancelled
CI/CD Pipeline / Security Scanning (push) Has been cancelled
CI/CD Pipeline / Build and Push Docker Images (api) (push) Has been cancelled
CI/CD Pipeline / Build and Push Docker Images (chat) (push) Has been cancelled
CI/CD Pipeline / Build and Push Docker Images (frontend) (push) Has been cancelled
CI/CD Pipeline / Build and Push Docker Images (worker) (push) Has been cancelled
CI/CD Pipeline / Deploy to Staging (push) Has been cancelled
CI/CD Pipeline / Deploy to Production (push) Has been cancelled
CI/CD Pipeline / Generate Documentation (push) Has started running
CI/CD Pipeline / Lint Code (push) Has started running

Implement a scalable system for automatic documentation generation from infrastructure
systems, preventing LLM context overload through template-driven sectioning.

**New Features:**

1. **YAML Template System** (`templates/documentation/proxmox.yaml`)
   - Define documentation sections independently
   - Specify data requirements per section
   - Configure prompts, generation settings, and scheduling
   - Prevents LLM context overflow by sectioning data

2. **Template-Based Generator** (`src/datacenter_docs/generators/template_generator.py`)
   - Load and parse YAML templates
   - Generate documentation sections independently
   - Extract only required data for each section
   - Save sections individually to files and database
   - Combine sections with table of contents

3. **Celery Tasks** (`src/datacenter_docs/workers/documentation_tasks.py`)
   - `collect_and_generate_docs`: Collect data and generate docs
   - `generate_proxmox_docs`: Scheduled Proxmox documentation (daily at 2 AM)
   - `generate_all_docs`: Generate docs for all systems in parallel
   - `index_generated_docs`: Index generated docs into vector store for RAG
   - `full_docs_pipeline`: Complete workflow (collect → generate → index)

4. **Scheduled Jobs** (updated `celery_app.py`)
   - Daily Proxmox documentation generation
   - Every 6 hours: all systems documentation
   - Weekly: full pipeline with indexing
   - Proper task routing and rate limiting

5. **Test Script** (`scripts/test_proxmox_docs.py`)
   - End-to-end testing of documentation generation
   - Mock data collection from Proxmox
   - Template-based generation
   - File and database storage

6. **Configuration Updates** (`src/datacenter_docs/utils/config.py`)
   - Add port configuration fields for Docker services
   - Add MongoDB and Redis credentials
   - Support all required environment variables

**Proxmox Documentation Sections:**
- Infrastructure Overview (cluster, nodes, stats)
- Virtual Machines Inventory
- LXC Containers Inventory
- Storage Configuration
- Network Configuration
- Maintenance Procedures

**Benefits:**
- Scalable to multiple infrastructure systems
- Prevents LLM context window overflow
- Independent section generation
- Scheduled automatic updates
- Vector store integration for RAG chat
- Template-driven approach for consistency

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2025-10-20 19:23:30 +02:00
parent 27dd9e00b6
commit 16fc8e2659
6 changed files with 1178 additions and 1 deletions

View File

@@ -72,6 +72,20 @@ class Settings(BaseSettings):
CELERY_BROKER_URL: str = "redis://localhost:6379/0"
CELERY_RESULT_BACKEND: str = "redis://localhost:6379/0"
# Additional Port Configuration (for Docker services)
MONGODB_PORT: int = 27017
REDIS_PORT: int = 6379
CHAT_PORT: int = 8001
FLOWER_PORT: int = 5555
FRONTEND_PORT: int = 8080
# MongoDB Root Credentials (for Docker initialization)
MONGO_ROOT_USER: str = "admin"
MONGO_ROOT_PASSWORD: str = "admin123"
# Redis Password
REDIS_PASSWORD: str = ""
@model_validator(mode="before")
@classmethod
def set_celery_defaults(cls, values: Dict[str, Any]) -> Dict[str, Any]: