Initial commit: LLM Automation Docs & Remediation Engine v2.0
Features: - Automated datacenter documentation generation - MCP integration for device connectivity - Auto-remediation engine with safety checks - Multi-factor reliability scoring (0-100%) - Human feedback learning loop - Pattern recognition and continuous improvement - Agentic chat support with AI - API for ticket resolution - Frontend React with Material-UI - CI/CD pipelines (GitLab + Gitea) - Docker & Kubernetes deployment - Complete documentation and guides v2.0 Highlights: - Auto-remediation with write operations (disabled by default) - Reliability calculator with 4-factor scoring - Human feedback system for continuous learning - Pattern-based progressive automation - Approval workflow for critical actions - Full audit trail and rollback capability
This commit is contained in:
LLM Automation System
459
MONGODB_GUIDE.md
Normal file
459
MONGODB_GUIDE.md
Normal file
@@ -0,0 +1,459 @@
|
||||
# 🍃 MongoDB Migration Guide
|
||||
|
||||
## Perché MongoDB?
|
||||
|
||||
Il sistema è stato aggiornato per utilizzare **MongoDB 7.0** invece di PostgreSQL per i seguenti motivi:
|
||||
|
||||
### ✅ Vantaggi per questo Use Case
|
||||
|
||||
1. **Schema Flessibile**
|
||||
- Ticket metadata variabili senza migration
|
||||
- Facile aggiunta di nuovi campi
|
||||
- Supporto nativo per documenti JSON complessi
|
||||
|
||||
2. **Performance**
|
||||
- Ottime performance per operazioni di lettura
|
||||
- Aggregation pipeline potente per analytics
|
||||
- Indexing flessibile su campi nested
|
||||
|
||||
3. **Scalabilità**
|
||||
- Horizontal scaling nativo (sharding)
|
||||
- Replica set per high availability
|
||||
- Auto-failover integrato
|
||||
|
||||
4. **Document-Oriented**
|
||||
- Match perfetto per ticket system
|
||||
- Metadata JSON nativi
|
||||
- Embedding di related docs senza JOIN
|
||||
|
||||
5. **Vector Search (Future)**
|
||||
- MongoDB Atlas Vector Search integrato
|
||||
- Possibilità di sostituire ChromaDB
|
||||
- Unified database per docs + vectors
|
||||
|
||||
6. **Developer Experience**
|
||||
- Beanie ODM moderno con Pydantic
|
||||
- Async/await nativo con Motor
|
||||
- Type hints e validazione
|
||||
|
||||
## 🔄 Architettura Database
|
||||
|
||||
### Collezioni Principali
|
||||
|
||||
```
|
||||
datacenter_docs/
|
||||
├── tickets # Ticket e risoluzioni
|
||||
├── documentation_sections # Metadata sezioni doc
|
||||
├── chat_sessions # Conversazioni chat
|
||||
├── system_metrics # Metriche sistema
|
||||
└── audit_logs # Audit trail
|
||||
```
|
||||
|
||||
### Schema Ticket (Example)
|
||||
|
||||
```json
|
||||
{
|
||||
"_id": ObjectId("..."),
|
||||
"ticket_id": "INC-12345",
|
||||
"title": "Network connectivity issue",
|
||||
"description": "Cannot ping 10.0.20.5 from VLAN 100",
|
||||
"priority": "high",
|
||||
"category": "network",
|
||||
"status": "resolved",
|
||||
"resolution": "Check VLAN configuration...",
|
||||
"suggested_actions": [
|
||||
"Verify VLAN 100 on switch",
|
||||
"Check inter-VLAN routing"
|
||||
],
|
||||
"related_docs": [
|
||||
{
|
||||
"section": "networking",
|
||||
"content": "VLAN configuration...",
|
||||
"source": "/docs/02_networking.md"
|
||||
}
|
||||
],
|
||||
"confidence_score": 0.92,
|
||||
"processing_time": 2.34,
|
||||
"metadata": {
|
||||
"source_system": "ServiceNow",
|
||||
"tags": ["network", "vlan", "connectivity"],
|
||||
"custom_field": "any value"
|
||||
},
|
||||
"created_at": ISODate("2025-01-15T10:30:00Z"),
|
||||
"updated_at": ISODate("2025-01-15T10:30:02Z")
|
||||
}
|
||||
```
|
||||
|
||||
## 🚀 Migration da PostgreSQL
|
||||
|
||||
### Step 1: Export dati esistenti (se presenti)
|
||||
|
||||
```bash
|
||||
# Export tickets da PostgreSQL
|
||||
psql -U docs_user -d datacenter_docs -c \
|
||||
"COPY (SELECT * FROM tickets) TO '/tmp/tickets.csv' CSV HEADER"
|
||||
```
|
||||
|
||||
### Step 2: Import in MongoDB
|
||||
|
||||
```python
|
||||
import pandas as pd
|
||||
from motor.motor_asyncio import AsyncIOMotorClient
|
||||
import asyncio
|
||||
|
||||
async def migrate():
|
||||
# Leggi CSV
|
||||
df = pd.read_csv('/tmp/tickets.csv')
|
||||
|
||||
# Connetti MongoDB
|
||||
client = AsyncIOMotorClient('mongodb://admin:password@localhost:27017')
|
||||
db = client.datacenter_docs
|
||||
|
||||
# Insert documents
|
||||
tickets = df.to_dict('records')
|
||||
await db.tickets.insert_many(tickets)
|
||||
|
||||
print(f"Migrated {len(tickets)} tickets")
|
||||
|
||||
asyncio.run(migrate())
|
||||
```
|
||||
|
||||
### Step 3: Verifica
|
||||
|
||||
```bash
|
||||
# Connetti a MongoDB
|
||||
mongosh mongodb://admin:password@localhost:27017
|
||||
|
||||
use datacenter_docs
|
||||
|
||||
# Conta documenti
|
||||
db.tickets.countDocuments()
|
||||
|
||||
# Query esempio
|
||||
db.tickets.find({status: "resolved"}).limit(5)
|
||||
```
|
||||
|
||||
## 📦 Setup Locale
|
||||
|
||||
### Docker Compose
|
||||
|
||||
```bash
|
||||
# Start MongoDB
|
||||
docker-compose up -d mongodb redis
|
||||
|
||||
# Verifica connessione
|
||||
docker-compose exec mongodb mongosh \
|
||||
-u admin -p password --authenticationDatabase admin
|
||||
|
||||
# Test query
|
||||
use datacenter_docs
|
||||
db.tickets.find().limit(1)
|
||||
```
|
||||
|
||||
### Kubernetes
|
||||
|
||||
```bash
|
||||
# Deploy MongoDB StatefulSet
|
||||
kubectl apply -f deploy/kubernetes/mongodb.yaml
|
||||
|
||||
# Wait for pods
|
||||
kubectl get pods -n datacenter-docs -w
|
||||
|
||||
# Initialize replica set
|
||||
kubectl apply -f deploy/kubernetes/mongodb.yaml
|
||||
|
||||
# Verify
|
||||
kubectl exec -n datacenter-docs mongodb-0 -- \
|
||||
mongosh -u admin -p password --authenticationDatabase admin \
|
||||
--eval "rs.status()"
|
||||
```
|
||||
|
||||
## 🔧 Configurazione
|
||||
|
||||
### Connection String
|
||||
|
||||
```bash
|
||||
# Development (local)
|
||||
MONGODB_URL=mongodb://admin:password@localhost:27017
|
||||
|
||||
# Docker Compose
|
||||
MONGODB_URL=mongodb://admin:password@mongodb:27017
|
||||
|
||||
# Kubernetes (single node)
|
||||
MONGODB_URL=mongodb://admin:password@mongodb.datacenter-docs.svc.cluster.local:27017
|
||||
|
||||
# Kubernetes (replica set)
|
||||
MONGODB_URL=mongodb://admin:password@mongodb-0.mongodb.datacenter-docs.svc.cluster.local:27017,mongodb-1.mongodb.datacenter-docs.svc.cluster.local:27017,mongodb-2.mongodb.datacenter-docs.svc.cluster.local:27017/?replicaSet=rs0
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
# MongoDB
|
||||
MONGODB_URL=mongodb://admin:password@mongodb:27017
|
||||
MONGODB_DATABASE=datacenter_docs
|
||||
|
||||
# MongoDB Root (for admin operations)
|
||||
MONGO_ROOT_USER=admin
|
||||
MONGO_ROOT_PASSWORD=secure_password
|
||||
```
|
||||
|
||||
## 🔐 Security
|
||||
|
||||
### Authentication
|
||||
|
||||
```bash
|
||||
# Create application user
|
||||
mongosh -u admin -p password --authenticationDatabase admin
|
||||
|
||||
use datacenter_docs
|
||||
|
||||
db.createUser({
|
||||
user: "docs_app",
|
||||
pwd: "app_password",
|
||||
roles: [
|
||||
{ role: "readWrite", db: "datacenter_docs" }
|
||||
]
|
||||
})
|
||||
|
||||
# Use app user in connection string
|
||||
MONGODB_URL=mongodb://docs_app:app_password@mongodb:27017/datacenter_docs
|
||||
```
|
||||
|
||||
### Encryption at Rest
|
||||
|
||||
```yaml
|
||||
# docker-compose.yml
|
||||
mongodb:
|
||||
command:
|
||||
- --enableEncryption
|
||||
- --encryptionKeyFile=/data/mongodb-keyfile
|
||||
volumes:
|
||||
- ./mongodb-keyfile:/data/mongodb-keyfile:ro
|
||||
```
|
||||
|
||||
### TLS/SSL
|
||||
|
||||
```bash
|
||||
# Generate certificates
|
||||
openssl req -newkey rsa:2048 -nodes -keyout mongodb.key \
|
||||
-x509 -days 365 -out mongodb.crt
|
||||
|
||||
# Configure MongoDB
|
||||
mongodb:
|
||||
command:
|
||||
- --tlsMode=requireTLS
|
||||
- --tlsCertificateKeyFile=/etc/ssl/mongodb.pem
|
||||
```
|
||||
|
||||
## 📊 Indexing Strategy
|
||||
|
||||
### Automatic Indexes (via Beanie)
|
||||
|
||||
```python
|
||||
class Ticket(Document):
|
||||
ticket_id: Indexed(str, unique=True) # Unique index
|
||||
status: str # Indexed in Settings
|
||||
|
||||
class Settings:
|
||||
indexes = [
|
||||
"status",
|
||||
"category",
|
||||
[("status", 1), ("created_at", -1)], # Compound
|
||||
]
|
||||
```
|
||||
|
||||
### Custom Indexes
|
||||
|
||||
```javascript
|
||||
// Text search
|
||||
db.tickets.createIndex({
|
||||
title: "text",
|
||||
description: "text",
|
||||
resolution: "text"
|
||||
})
|
||||
|
||||
// Geospatial (future use)
|
||||
db.locations.createIndex({ location: "2dsphere" })
|
||||
|
||||
// TTL index (auto-delete old docs)
|
||||
db.chat_sessions.createIndex(
|
||||
{ last_activity: 1 },
|
||||
{ expireAfterSeconds: 2592000 } // 30 days
|
||||
)
|
||||
```
|
||||
|
||||
## 🔍 Query Examples
|
||||
|
||||
### Python (Beanie)
|
||||
|
||||
```python
|
||||
from datacenter_docs.api.models import Ticket
|
||||
|
||||
# Find by status
|
||||
tickets = await Ticket.find(Ticket.status == "resolved").to_list()
|
||||
|
||||
# Complex query
|
||||
from datetime import datetime, timedelta
|
||||
recent = datetime.now() - timedelta(days=7)
|
||||
|
||||
tickets = await Ticket.find(
|
||||
Ticket.status == "resolved",
|
||||
Ticket.confidence_score > 0.8,
|
||||
Ticket.created_at > recent
|
||||
).sort(-Ticket.created_at).to_list()
|
||||
|
||||
# Aggregation
|
||||
pipeline = [
|
||||
{"$group": {
|
||||
"_id": "$category",
|
||||
"count": {"$sum": 1},
|
||||
"avg_confidence": {"$avg": "$confidence_score"}
|
||||
}},
|
||||
{"$sort": {"count": -1}}
|
||||
]
|
||||
result = await Ticket.aggregate(pipeline).to_list()
|
||||
```
|
||||
|
||||
### MongoDB Shell
|
||||
|
||||
```javascript
|
||||
// Find resolved tickets
|
||||
db.tickets.find({ status: "resolved" })
|
||||
|
||||
// Complex aggregation
|
||||
db.tickets.aggregate([
|
||||
{ $match: { status: "resolved" } },
|
||||
{ $group: {
|
||||
_id: "$category",
|
||||
total: { $sum: 1 },
|
||||
avg_confidence: { $avg: "$confidence_score" },
|
||||
avg_time: { $avg: "$processing_time" }
|
||||
}},
|
||||
{ $sort: { total: -1 } }
|
||||
])
|
||||
|
||||
// Text search
|
||||
db.tickets.find({
|
||||
$text: { $search: "network connectivity" }
|
||||
})
|
||||
```
|
||||
|
||||
## 📈 Performance Optimization
|
||||
|
||||
### Indexes
|
||||
|
||||
```javascript
|
||||
// Explain query
|
||||
db.tickets.find({ status: "resolved" }).explain("executionStats")
|
||||
|
||||
// Check index usage
|
||||
db.tickets.aggregate([
|
||||
{ $indexStats: {} }
|
||||
])
|
||||
```
|
||||
|
||||
### Connection Pooling
|
||||
|
||||
```python
|
||||
# config.py
|
||||
MONGODB_URL = "mongodb://user:pass@host:27017/?maxPoolSize=50"
|
||||
```
|
||||
|
||||
### Read Preference
|
||||
|
||||
```python
|
||||
# For read-heavy workloads with replica set
|
||||
from pymongo import ReadPreference
|
||||
|
||||
client = AsyncIOMotorClient(
|
||||
MONGODB_URL,
|
||||
readPreference=ReadPreference.SECONDARY_PREFERRED
|
||||
)
|
||||
```
|
||||
|
||||
## 🛠️ Maintenance
|
||||
|
||||
### Backup
|
||||
|
||||
```bash
|
||||
# Full backup
|
||||
mongodump --uri="mongodb://admin:password@localhost:27017" \
|
||||
--authenticationDatabase=admin \
|
||||
--out=/backup/$(date +%Y%m%d)
|
||||
|
||||
# Restore
|
||||
mongorestore --uri="mongodb://admin:password@localhost:27017" \
|
||||
--authenticationDatabase=admin \
|
||||
/backup/20250115
|
||||
```
|
||||
|
||||
### Monitoring
|
||||
|
||||
```javascript
|
||||
// Database stats
|
||||
db.stats()
|
||||
|
||||
// Collection stats
|
||||
db.tickets.stats()
|
||||
|
||||
// Current operations
|
||||
db.currentOp()
|
||||
|
||||
// Server status
|
||||
db.serverStatus()
|
||||
```
|
||||
|
||||
### Cleanup
|
||||
|
||||
```javascript
|
||||
// Remove old chat sessions
|
||||
db.chat_sessions.deleteMany({
|
||||
last_activity: { $lt: new Date(Date.now() - 30*24*60*60*1000) }
|
||||
})
|
||||
|
||||
// Compact collection
|
||||
db.runCommand({ compact: "tickets" })
|
||||
```
|
||||
|
||||
## 🔄 Replica Set (Production)
|
||||
|
||||
### Setup
|
||||
|
||||
```bash
|
||||
# Initialize replica set
|
||||
rs.initiate({
|
||||
_id: "rs0",
|
||||
members: [
|
||||
{ _id: 0, host: "mongodb-0:27017" },
|
||||
{ _id: 1, host: "mongodb-1:27017" },
|
||||
{ _id: 2, host: "mongodb-2:27017" }
|
||||
]
|
||||
})
|
||||
|
||||
# Check status
|
||||
rs.status()
|
||||
|
||||
# Add member
|
||||
rs.add("mongodb-3:27017")
|
||||
```
|
||||
|
||||
### Connection String
|
||||
|
||||
```bash
|
||||
MONGODB_URL=mongodb://user:pass@mongodb-0:27017,mongodb-1:27017,mongodb-2:27017/?replicaSet=rs0&w=majority
|
||||
```
|
||||
|
||||
## 📚 References
|
||||
|
||||
- [MongoDB Manual](https://docs.mongodb.com/manual/)
|
||||
- [Motor Documentation](https://motor.readthedocs.io/)
|
||||
- [Beanie ODM](https://beanie-odm.dev/)
|
||||
- [MongoDB Best Practices](https://docs.mongodb.com/manual/administration/production-notes/)
|
||||
|
||||
---
|
||||
|
||||
**MongoDB Version**: 7.0
|
||||
**Driver**: Motor (Async)
|
||||
**ODM**: Beanie
|
||||
**Python**: 3.10+
|
||||
Reference in New Issue
Block a user