llm-automation-docs-and-remediation-engine/scripts

Scripts Directory

This directory contains utility scripts for the Datacenter Documentation project.

---

🔍 test-ci-pipeline.sh

Local CI/CD Pipeline Validation Script

Description

Simulates the complete GitLab CI/CD pipeline locally before pushing code to the repository. This script runs all the same checks that would run in GitHub Actions, GitLab CI, or Gitea Actions.

Usage

# Run from project root
bash scripts/test-ci-pipeline.sh

# Or make it executable and run directly
chmod +x scripts/test-ci-pipeline.sh
./scripts/test-ci-pipeline.sh

Pipeline Stages

The script executes the following stages in order:

1. LINT Stage

• Black: Code formatting check
• Ruff: Linting and code quality
• MyPy: Type checking (strict mode)

2. TEST Stage

• Unit Tests: Runs pytest with coverage
• Security Scan: Bandit (if installed)

3. BUILD Stage

• Poetry Check: Validates pyproject.toml configuration
• Dependency Resolution: Tests if all dependencies can be installed
• Docker Validation: Checks Dockerfile syntax

4. INTEGRATION Stage (Optional)

• API Health Check: Tests if local API is running

Output

The script provides:

• ✅ Color-coded output for easy readability
• 📊 Real-time progress for each job
• 📄 Summary report at the end
• 📝 Written report saved to ci-pipeline-report-TIMESTAMP.txt

Example Output

╔═══════════════════════════════════════════════════════╗
║       LOCAL CI/CD PIPELINE SIMULATION                ║
║       GitLab CI Pipeline                             ║
╚═══════════════════════════════════════════════════════╝

=====================================
STAGE: LINT
=====================================

>>> JOB: lint:black
Running: poetry run black --check src/ tests/
✅ PASSED: Black code formatting

>>> JOB: lint:ruff
Running: poetry run ruff check src/ tests/
✅ PASSED: Ruff linting

>>> JOB: lint:mypy
Running: poetry run mypy src/
✅ PASSED: MyPy type checking

...

╔═══════════════════════════════════════════════════════╗
║          ✅ PIPELINE PASSED SUCCESSFULLY ✅           ║
╚═══════════════════════════════════════════════════════╝

Total Tests:  8
Passed:       8
Failed:       0
Duration:     6s

Exit Codes

• 0: All checks passed ✅
• 1: One or more checks failed ❌

Requirements

• Poetry: For dependency management
• Python 3.12+: As specified in pyproject.toml
• Docker/Podman (optional): For Docker validation stage
• MongoDB (optional): For integration tests

When to Run

Run this script:

• ✅ Before every commit to ensure code quality
• ✅ Before creating a pull request
• ✅ After making significant changes
• ✅ To verify CI/CD pipeline compatibility

Integration with Git

You can add this as a Git pre-push hook:

#!/bin/bash
# .git/hooks/pre-push

echo "Running CI pipeline validation..."
bash scripts/test-ci-pipeline.sh

if [ $? -ne 0 ]; then
    echo "❌ CI pipeline validation failed. Push aborted."
    exit 1
fi

echo "✅ CI pipeline validation passed. Proceeding with push..."
exit 0

Continuous Integration Compatibility

This script simulates:

• ✅ GitHub Actions (.github/workflows/build-deploy.yml)
• ✅ GitLab CI (.gitlab-ci.yml)
• ✅ Gitea Actions (.gitea/workflows/ci.yml)

All checks performed locally will also pass in the actual CI/CD platforms.

---

📝 Report Files

After running the validation script, you'll find:

• ci-pipeline-report-TIMESTAMP.txt: Plain text summary
• CI_VALIDATION_REPORT.md: Comprehensive markdown report with details

---

🔄 convert_config.py

Configuration Format Converter

Description

Converts between .env and values.yaml configuration formats, making it easy to switch between Docker Compose and Helm deployments.

Usage

Prerequisites

pip install pyyaml

Convert .env to values.yaml

./scripts/convert_config.py env-to-yaml .env values.yaml

Convert values.yaml to .env

./scripts/convert_config.py yaml-to-env values.yaml .env

Examples

Example 1: Create values.yaml from existing .env

# You have an existing .env file from Docker development
./scripts/convert_config.py env-to-yaml .env my-values.yaml

# Use the generated values.yaml with Helm
helm install my-release deploy/helm/datacenter-docs -f my-values.yaml

Example 2: Generate .env from values.yaml

# You have a values.yaml from Kubernetes deployment
./scripts/convert_config.py yaml-to-env values.yaml .env

# Use the generated .env with Docker Compose
cd deploy/docker
docker-compose -f docker-compose.dev.yml up -d

Example 3: Environment migration

# Convert development .env to staging values.yaml
./scripts/convert_config.py env-to-yaml .env.development values-staging.yaml

# Manually adjust staging-specific settings
nano values-staging.yaml

# Deploy to staging Kubernetes cluster
helm install staging deploy/helm/datacenter-docs -f values-staging.yaml

Supported Configuration

The script converts:

• MongoDB: Connection settings and authentication
• Redis: Connection and authentication
• MCP Server: URL and API key
• Proxmox: Host, authentication, SSL settings
• LLM: Provider settings (OpenAI, Anthropic, Ollama, etc.)
• API: Server configuration and workers
• CORS: Allowed origins
• Application: Logging and debug settings
• Celery: Broker and result backend
• Vector Store: ChromaDB and embedding model

Output

Reading .env file: .env
Converting to values.yaml format...
Writing values.yaml: my-values.yaml
✓ Conversion completed successfully!

Output written to: my-values.yaml

Limitations

• Converts common configuration options only
• Complex nested structures may require manual adjustment
• Helm-specific values (resource limits, replicas) not included in .env conversion
• Always review and test converted configuration

Tips

1. Review output: Always check converted files for accuracy
2. Test first: Validate in development before production
3. Keep secrets secure: Use proper secret management tools
4. Version control: Track configuration changes

See Also

• CONFIGURATION.md - Complete configuration guide
• .env.example - Environment variable template
• values.yaml - YAML configuration template

---

🚀 Quick Start

# First time setup
poetry install

# Run validation
bash scripts/test-ci-pipeline.sh

# If all passes, commit and push
git add .
git commit -m "your commit message"
git push

---

🔧 Troubleshooting

"poetry: command not found"

Install Poetry: https://python-poetry.org/docs/#installation

"Black would reformat X files"

Run: poetry run black src/ tests/

"Ruff found X errors"

Run: poetry run ruff check --fix src/ tests/

"MyPy found X errors"

Fix type errors or add type ignores where appropriate.

Docker validation fails

Ensure Docker or Podman is installed:

• Ubuntu/Debian: sudo apt install docker.io
• Fedora: sudo dnf install podman podman-compose

---

📚 Additional Resources

• CLAUDE.md - Project documentation for AI assistants
• README.md - Project overview
• TODO.md - Development roadmap
• CI_VALIDATION_REPORT.md - Latest validation report