feat: Implement CLI tool, Celery workers, and VMware collector
Some checks failed
CI/CD Pipeline / Generate Documentation (push) Successful in 4m57s
CI/CD Pipeline / Lint Code (push) Successful in 5m33s
CI/CD Pipeline / Run Tests (push) Successful in 4m20s
CI/CD Pipeline / Security Scanning (push) Successful in 4m32s
CI/CD Pipeline / Build and Push Docker Images (chat) (push) Failing after 49s
CI/CD Pipeline / Build and Push Docker Images (frontend) (push) Failing after 48s
CI/CD Pipeline / Build and Push Docker Images (worker) (push) Failing after 46s
CI/CD Pipeline / Build and Push Docker Images (api) (push) Failing after 40s
CI/CD Pipeline / Deploy to Staging (push) Has been skipped
CI/CD Pipeline / Deploy to Production (push) Has been skipped

Complete implementation of core MVP components:

CLI Tool (src/datacenter_docs/cli.py):
- 11 commands for system management (serve, worker, init-db, generate, etc.)
- Auto-remediation policy management (enable/disable/status)
- System statistics and monitoring
- Rich formatted output with tables and panels

Celery Workers (src/datacenter_docs/workers/):
- celery_app.py with 4 specialized queues (documentation, auto_remediation, data_collection, maintenance)
- tasks.py with 8 async tasks integrated with MongoDB/Beanie
- Celery Beat scheduling (6h docs, 1h data collection, 15m metrics, 2am cleanup)
- Rate limiting (10 auto-remediation/h) and timeout configuration
- Task lifecycle signals and comprehensive logging

VMware Collector (src/datacenter_docs/collectors/):
- BaseCollector abstract class with full workflow (connect/collect/validate/store/disconnect)
- VMwareCollector for vSphere infrastructure data collection
- Collects VMs, ESXi hosts, clusters, datastores, networks with statistics
- MCP client integration with mock data fallback for development
- MongoDB storage via AuditLog and data validation

Documentation & Configuration:
- Updated README.md with CLI commands and Workers sections
- Updated TODO.md with project status (55% completion)
- Added CLAUDE.md with comprehensive project instructions
- Added Docker compose setup for development environment

Project Status:
- Completion: 50% -> 55%
- MVP Milestone: 80% complete (only Infrastructure Generator remaining)
- Estimated time to MVP: 1-2 days

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2025-10-19 22:29:59 +02:00
parent 541222ad68
commit 52655e9eee
34 changed files with 5246 additions and 456 deletions

152
README.md
View File

@@ -5,7 +5,7 @@
> AI-powered infrastructure documentation generation with autonomous problem resolution capabilities.
[![Version](https://img.shields.io/badge/version-2.0.0-blue.svg)](https://github.com/yourusername/datacenter-docs)
[![Python](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
[![Python](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
---
@@ -85,7 +85,7 @@
## 🚀 Quick Start
### Prerequisites
- Python 3.10+
- Python 3.12+
- Poetry 1.7+
- Docker & Docker Compose
- MCP Server running
@@ -141,6 +141,154 @@ kubectl apply -f deploy/kubernetes/
---
## 💻 CLI Tool
The system includes a comprehensive command-line tool for managing all aspects of the documentation and remediation engine.
### Available Commands
```bash
# Initialize database with collections and default data
datacenter-docs init-db
# Start API server
datacenter-docs serve # Production
datacenter-docs serve --reload # Development with auto-reload
# Start Celery worker for background tasks
datacenter-docs worker # All queues (default)
datacenter-docs worker --queue documentation # Documentation queue only
datacenter-docs worker --concurrency 8 # Custom concurrency
# Documentation generation
datacenter-docs generate vmware # Generate specific section
datacenter-docs generate-all # Generate all sections
datacenter-docs list-sections # List available sections
# System statistics and monitoring
datacenter-docs stats # Last 24 hours
datacenter-docs stats --period 7d # Last 7 days
# Auto-remediation management
datacenter-docs remediation status # Show all policies
datacenter-docs remediation enable # Enable globally
datacenter-docs remediation disable # Disable globally
datacenter-docs remediation enable --category network # Enable for category
datacenter-docs remediation disable --category network # Disable for category
# System information
datacenter-docs version # Show version info
datacenter-docs --help # Show help
```
### Example Workflow
```bash
# 1. Setup database
datacenter-docs init-db
# 2. Start services
datacenter-docs serve --reload & # API in background
datacenter-docs worker & # Worker in background
# 3. Generate documentation
datacenter-docs list-sections # See available sections
datacenter-docs generate vmware # Generate VMware docs
datacenter-docs generate-all # Generate everything
# 4. Monitor system
datacenter-docs stats --period 24h # Check statistics
# 5. Enable auto-remediation for safe categories
datacenter-docs remediation enable --category network
datacenter-docs remediation status # Verify
```
### Section IDs
The following documentation sections are available:
- `vmware` - VMware Infrastructure (vCenter, ESXi)
- `kubernetes` - Kubernetes Clusters
- `network` - Network Infrastructure (switches, routers)
- `storage` - Storage Systems (SAN, NAS)
- `database` - Database Servers
- `monitoring` - Monitoring Systems (Zabbix, Prometheus)
- `security` - Security & Compliance
---
## ⚙️ Background Workers (Celery)
The system uses **Celery** for asynchronous task processing with **4 specialized queues** and **8 task types**.
### Worker Queues
1. **documentation** - Documentation generation tasks
2. **auto_remediation** - Auto-remediation execution tasks
3. **data_collection** - Infrastructure data collection
4. **maintenance** - System cleanup and metrics
### Available Tasks
| Task | Queue | Schedule | Description |
|------|-------|----------|-------------|
| `generate_documentation_task` | documentation | Every 6 hours | Full documentation regeneration |
| `generate_section_task` | documentation | On-demand | Single section generation |
| `execute_auto_remediation_task` | auto_remediation | On-demand | Execute remediation actions (rate limit: 10/h) |
| `process_ticket_task` | auto_remediation | On-demand | AI ticket analysis and resolution |
| `collect_infrastructure_data_task` | data_collection | Every 1 hour | Collect infrastructure state |
| `cleanup_old_data_task` | maintenance | Daily 2 AM | Remove old records (90 days) |
| `update_system_metrics_task` | maintenance | Every 15 minutes | Calculate system metrics |
### Worker Management
```bash
# Start worker with all queues
datacenter-docs worker
# Start worker for specific queue only
datacenter-docs worker --queue documentation
datacenter-docs worker --queue auto_remediation
datacenter-docs worker --queue data_collection
datacenter-docs worker --queue maintenance
# Custom concurrency (default: 4)
datacenter-docs worker --concurrency 8
# Custom log level
datacenter-docs worker --log-level DEBUG
```
### Celery Beat (Scheduler)
The system includes **Celery Beat** for periodic task execution:
```bash
# Start beat scheduler (runs alongside worker)
celery -A datacenter_docs.workers.celery_app beat --loglevel=INFO
```
### Monitoring with Flower
Monitor Celery workers in real-time:
```bash
# Start Flower web UI (port 5555)
celery -A datacenter_docs.workers.celery_app flower
```
Access at: http://localhost:5555
### Task Configuration
- **Timeout**: 1 hour hard limit, 50 minutes soft limit
- **Retry**: Up to 3 retries for failed tasks
- **Prefetch**: 1 task per worker (prevents overload)
- **Max tasks per child**: 1000 (automatic worker restart)
- **Serialization**: JSON (secure and portable)
---
## 📖 Documentation
### Core Documentation