api7-demo/web/docs/service-discovery.md

# Service Discovery

Learn how API7 Gateway dynamically discovers backend services using Kubernetes service discovery.

## Overview

Service Discovery enables API7 Gateway to automatically discover and track backend Pod endpoints instead of using static upstream node configuration. This provides a dynamic, scalable, and resilient architecture.

## Why Use Service Discovery?

### Traditional Static Configuration

```yaml
upstream:
  nodes:
    - host: 10.0.1.23
      port: 8080
      weight: 100
    - host: 10.0.1.24
      port: 8080
      weight: 100
```

**Problems:**
- ❌ Manual updates required when Pods scale
- ❌ Stale Pod IPs after deployments
- ❌ No automatic health checks
- ❌ Downtime during updates

### With Kubernetes Service Discovery

```yaml
upstream:
  discovery_type: kubernetes
  service_name: my-service
  namespace_id: default
```

**Benefits:**
- ✅ Automatic Pod discovery
- ✅ Real-time updates during scaling
- ✅ Health checks built-in
- ✅ Zero downtime deployments

## How It Works

```mermaid
graph LR
    A[API7 Gateway] -->|Query Endpoints| B[Kubernetes API]
    B -->|Return Pod IPs| A
    A -->|Route Traffic| C[Pod 1]
    A -->|Route Traffic| D[Pod 2]
    A -->|Route Traffic| E[Pod 3]

    F[HPA/Deploy] -->|Scale/Update| B
    B -->|Notify| A
```

1. **API7 Gateway** watches Kubernetes Endpoints API
2. **Kubernetes API** returns list of healthy Pod IPs
3. **Gateway** automatically adds/removes Pods from upstream
4. **Traffic** is load-balanced across discovered Pods

## Configuration

### Enable in Helm Chart

Edit `values.yaml`:

```yaml
api7:
  serviceDiscovery:
    enabled: true  # Enable Kubernetes service discovery
    namespace: ""  # Leave empty to use release namespace
```

### ADC Configuration

The Helm chart automatically generates ADC configuration with service discovery:

```yaml
services:
  - name: my-service
    upstream:
      name: my-upstream
      type: roundrobin
      discovery_type: kubernetes
      service_name: my-service  # Kubernetes Service name
      namespace_id: default     # Kubernetes namespace
```

### RBAC Requirements

Service Discovery requires permissions to watch Kubernetes resources. These are automatically configured in `rbac-adc.yaml`:

```yaml
rules:
  # Allow reading services and endpoints for service discovery
  - apiGroups: [""]
    resources: ["services", "endpoints"]
    verbs: ["get", "list", "watch"]
```

## Use Cases

### 1. Auto-Scaling Applications

When your application scales:

```bash
# Scale deployment
kubectl scale deployment my-app --replicas=10

# API7 Gateway automatically discovers new Pods
# No configuration changes needed!
```

### 2. Rolling Updates

During deployments:

```bash
# Start rolling update
kubectl rollout restart deployment my-app

# Gateway automatically:
# 1. Discovers new Pods
# 2. Adds them to upstream
# 3. Removes old Pods
# 4. Zero downtime!
```

### 3. Multi-Service Architecture

Discover multiple backend services:

```yaml
services:
  - name: web-frontend
    upstream:
      discovery_type: kubernetes
      service_name: web-frontend
      namespace_id: production

  - name: api-backend
    upstream:
      discovery_type: kubernetes
      service_name: api-backend
      namespace_id: production

  - name: admin-api
    upstream:
      discovery_type: kubernetes
      service_name: admin-api
      namespace_id: admin
```

## Advanced Configuration

### Cross-Namespace Discovery

Discover services in different namespaces:

```yaml
upstream:
  discovery_type: kubernetes
  service_name: shared-service
  namespace_id: shared-services  # Different namespace
```

!!! warning "RBAC Permissions"
    Ensure the ServiceAccount has permissions to watch Endpoints in the target namespace.

### Load Balancing Algorithms

Combine service discovery with load balancing:

```yaml
upstream:
  discovery_type: kubernetes
  service_name: my-service
  namespace_id: default
  type: roundrobin  # or chash, least_conn, ewma
```

**Available algorithms:**
- `roundrobin`: Distribute requests evenly (default)
- `chash`: Consistent hashing (sticky sessions)
- `least_conn`: Send to Pod with fewest connections
- `ewma`: Exponentially weighted moving average

### Health Checks

API7 Gateway respects Kubernetes readiness probes:

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: app
    readinessProbe:
      httpGet:
        path: /health
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 3
```

Only Pods with passing readiness checks receive traffic.

## Monitoring

### Check Discovered Endpoints

View upstream nodes in API7 Dashboard:

```bash
# Port-forward to dashboard
kubectl port-forward -n api7ee svc/api7ee3-0-1759339083-dashboard 7443:7443

# Open browser
open https://localhost:7443
```

Navigate to: **Gateway Groups → Upstreams → View Nodes**

### Debug Service Discovery

Check ADC sync logs:

```bash
# View ADC sync job logs
kubectl logs -n api7ee job/api7ee-demo-api7ee-demo-k8s-adc-sync

# Look for service discovery messages
# [INFO] Discovered 3 endpoints for service my-service
```

## Troubleshooting

### No Endpoints Discovered

**Symptoms:** Gateway shows 0 upstream nodes

**Causes:**
1. Service doesn't exist
2. No Pods match Service selector
3. All Pods are not ready

**Solution:**
```bash
# Check Service exists
kubectl get service my-service -n default

# Check Endpoints
kubectl get endpoints my-service -n default

# Check Pod readiness
kubectl get pods -l app=my-service -n default
```

### RBAC Permission Denied

**Error:** `forbidden: User "system:serviceaccount:..." cannot list endpoints`

**Solution:**
Verify RBAC permissions:

```bash
# Check Role binding
kubectl get rolebinding -n api7ee | grep adc

# Describe permissions
kubectl describe role api7ee-demo-api7ee-demo-k8s-adc -n api7ee
```

### Stale Endpoints

**Symptoms:** Traffic sent to terminated Pods

**Causes:** Kubernetes API not updating fast enough

**Solution:**
Check Kubernetes API health:

```bash
# Verify API server is responsive
kubectl get --raw /healthz

# Check endpoint update timestamps
kubectl get endpoints my-service -o yaml | grep -i time
```

## Best Practices

### 1. Use Service Discovery for All Dynamic Workloads

✅ **Do:**
```yaml
upstream:
  discovery_type: kubernetes
  service_name: my-dynamic-app
```

❌ **Don't:**
```yaml
upstream:
  nodes:
    - host: pod-ip  # Brittle, breaks on restart
```

### 2. Configure Readiness Probes

Always configure readiness probes on Pods:

```yaml
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5
```

### 3. Use Namespaces for Isolation

Separate environments by namespace:

```yaml
# Production
namespace_id: production

# Staging
namespace_id: staging

# Development
namespace_id: development
```

### 4. Monitor Discovery Performance

Set up alerts for:
- Number of discovered endpoints
- Service discovery latency
- RBAC permission errors

### 5. Test Failover Scenarios

Regularly test:
- Pod termination
- Rolling updates
- Scale up/down events

## Comparison: Static vs Service Discovery

| Feature | Static Nodes | Service Discovery |
|---------|--------------|-------------------|
| Setup | Manual | Automatic |
| Scaling | Manual updates | Automatic |
| Health checks | Manual | Built-in |
| Deployments | Downtime risk | Zero downtime |
| Maintenance | High | Low |
| Flexibility | Low | High |
| Production-ready | ❌ | ✅ |

## Next Steps

- [Configure Secret Management](secret-management.md)
- [Set up Ingress Routing](kubernetes-resources.md#ingress)
- [Monitor Gateway Metrics](troubleshooting.md#monitoring)

## References

- [API7 Service Discovery Documentation](https://docs.api7.ai/enterprise/3.2.14.6/best-practice/service-discovery/)
- [Kubernetes Endpoints API](https://kubernetes.io/docs/concepts/services-networking/service/#endpoints)
- [Kubernetes Service Discovery](https://kubernetes.io/docs/concepts/services-networking/service/)