# Mini-PaaS Deployment System

A pull-based deployment system for deploying Nix-built services without full NixOS rebuilds.

## Quick Start

### Deploy a Service

```bash
# Build, cache to S3, and update manifest
Omni/Ide/push.sh Biz/PodcastItLater/Web.py

# The deployer on the target host polls every 5 minutes
# To force immediate deployment, SSH to host and run:
ssh biz sudo systemctl start biz-deployer
```

### View Current State

```bash
# Show current manifest
deploy-manifest show

# List archived manifests (for rollback)
deploy-manifest list

# Check deployer status on target
ssh biz sudo systemctl status biz-deployer
ssh biz cat /var/lib/biz-deployer/state.json
```

## Deployment Workflow

```
Developer Machine                    S3 Cache                    Target Host (biz)
       │                                │                              │
       │ push.sh Biz/App.py             │                              │
       ├───────────────────────────────►│                              │
       │  1. bild builds + caches       │                              │
       │  2. deploy-manifest update     │                              │
       │                                │   poll every 5 min           │
       │                                │◄─────────────────────────────┤
       │                                │                              │
       │                                │   manifest changed?          │
       │                                │   - pull closure             │
       │                                │   - generate systemd unit    │
       │                                │   - update Caddy route       │
       │                                │   - restart service          │
       │                                │─────────────────────────────►│
       │                                │                              │
```

## Adding a New Service

### 1. Create the Service Definition

```bash
deploy-manifest add-service '{
  "name": "my-new-service",
  "artifact": {"storePath": "/nix/store/placeholder"},
  "hosts": ["biz"],
  "exec": {"command": null, "user": "root", "group": "root"},
  "env": {"PORT": "8080", "AREA": "Live"},
  "envFile": "/var/lib/biz-secrets/my-new-service.env",
  "http": {"domain": "myservice.bensima.com", "path": "/", "internalPort": 8080}
}'
```

### 2. Create Secrets File on Target

```bash
ssh biz
sudo mkdir -p /var/lib/biz-secrets
sudo tee /var/lib/biz-secrets/my-new-service.env << 'EOF'
SECRET_KEY=your-secret-here
DATABASE_URL=postgres://...
EOF
sudo chmod 600 /var/lib/biz-secrets/my-new-service.env
```

### 3. Deploy the Service

```bash
Omni/Ide/push.sh Biz/MyService.py
```

## Secrets Management

Secrets are stored out-of-band on the target host, never in S3 or the manifest.

**Location**: `/var/lib/biz-secrets/<service-name>.env`

**Format**: Standard environment file
```
SECRET_KEY=abc123
DATABASE_URL=postgres://user:pass@localhost/db
STRIPE_API_KEY=sk_live_...
```

**Permissions**: `chmod 600`, owned by root

**Referencing in manifest**: Set `envFile` field to the path

## Rollback

### List Available Versions

```bash
deploy-manifest list
# Output:
# manifest-20251216T033000Z.json
# manifest-20251216T045211.json
# manifest-20251215T120000Z.json
```

### Rollback to Previous Version

```bash
# Restore a specific archived manifest
deploy-manifest rollback manifest-20251215T120000Z.json

# Force immediate deployment
ssh biz sudo systemctl start biz-deployer
```

The rollback archives the current manifest before restoring, so you can always rollback the rollback.

## Troubleshooting

### Service Not Starting

```bash
# Check deployer logs
ssh biz sudo journalctl -u biz-deployer -f

# Check service logs
ssh biz sudo journalctl -u <service-name> -f

# Check deployer state
ssh biz cat /var/lib/biz-deployer/state.json
```

### Manifest Update Failed

```bash
# Verify AWS credentials
aws s3 ls s3://omni-nix-cache/ --endpoint-url https://nyc3.digitaloceanspaces.com --profile digitalocean

# Check manifest exists
deploy-manifest show
```

### Closure Not Pulling

```bash
# Check if store path exists in cache
aws s3 ls s3://omni-nix-cache/<hash>.narinfo --endpoint-url https://nyc3.digitaloceanspaces.com --profile digitalocean

# Check target can access cache
ssh biz nix copy --from 's3://omni-nix-cache?profile=digitalocean&scheme=https&endpoint=nyc3.digitaloceanspaces.com' /nix/store/<path>
```

### Caddy Route Issues

```bash
# Check Caddy config
ssh biz curl -s localhost:2019/config/ | jq .

# Check Caddy logs
ssh biz sudo journalctl -u caddy -f
```

## Architecture

| Component | Location | Purpose |
|-----------|----------|---------|
| `bild` | Dev machine | Build tool, caches to S3 |
| `push.sh` | Dev machine | Orchestrates deploys |
| `deploy-manifest` | Dev machine | Manage manifest in S3 |
| `biz-deployer` | Target host | Polls manifest, deploys services |
| Caddy | Target host | Reverse proxy with auto-HTTPS |

### File Locations on Target

| Path | Purpose |
|------|---------|
| `/var/lib/biz-deployer/state.json` | Local deployer state |
| `/var/lib/biz-deployer/services/` | Generated systemd units |
| `/var/lib/biz-secrets/` | Service secret env files |
| `/nix/var/nix/gcroots/biz/` | GC roots for deployed closures |
| `/root/.aws/credentials` | S3 credentials |

## Scale-Up Path

| Stage | Hosts | Changes Needed |
|-------|-------|----------------|
| Current | 1 | Full architecture as described |
| 2-3 hosts | 2-3 | Add `hosts` filtering (already supported) |
| 4+ hosts | 4+ | Consider migrating to Nomad + nix-nomad |

The manifest already supports multi-host deployments via the `hosts` array. Each host runs its own deployer and only deploys services where its hostname appears in the `hosts` list.

## Related Files

- [Omni/Deploy/Manifest.hs](Manifest.hs) - Manifest CLI and schema
- [Omni/Deploy/Deployer.hs](Deployer.hs) - Deployer service
- [Omni/Deploy/Deployer.nix](Deployer.nix) - NixOS module
- [Omni/Deploy/Systemd.hs](Systemd.hs) - Systemd unit generation
- [Omni/Deploy/Caddy.hs](Caddy.hs) - Caddy API integration
- [Omni/Ide/push.sh](../Ide/push.sh) - Deploy script
- [Omni/Deploy/PLAN.md](PLAN.md) - Original design document