> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/ComposioHQ/agent-orchestrator/llms.txt
> Use this file to discover all available pages before exploring further.

# Custom Workflows

> Build advanced automation with CI/CD integration, custom notification channels, and orchestrator-driven workflows

Agent Orchestrator's plugin architecture and event system enable powerful custom workflows beyond the default CI/PR automation.

## Orchestrator Agent Workflows

The orchestrator agent is a special session that manages worker agents. It can:

* Spawn multiple agents for a batch of issues
* Monitor agent progress and intervene when stuck
* Coordinate multi-agent workflows
* Handle escalations and notifications

### Starting the Orchestrator Agent

```bash theme={null}
ao start my-project
```

This:

1. Generates an orchestrator prompt with project context
2. Launches a Claude Code session in the project directory
3. Provides the orchestrator with CLI access
4. Opens the dashboard for monitoring

**Orchestrator capabilities:**

```bash theme={null}
# Inside orchestrator session
ao status                        # Monitor all sessions
ao batch-spawn my-app 1 2 3 4 5  # Spawn multiple agents
ao send my-app-1 "Fix the tests" # Send instructions
ao session cleanup               # Clean up completed work
```

### Orchestrator Prompt

The orchestrator receives a comprehensive prompt (from `orchestrator-prompt.ts`):

<CodeGroup>
  ```markdown Quick Start theme={null}
  # See all sessions at a glance
  ao status

  # Spawn sessions for issues
  ao spawn my-project INT-1234
  ao batch-spawn my-project INT-1 INT-2 INT-3

  # Send message to a session
  ao send my-app-1 "Your message here"
  ```

  ```markdown Session Management theme={null}
  When you spawn a session:
  1. A git worktree is created from `main`
  2. A feature branch is created (e.g., `feat/INT-1234`)
  3. A tmux session is started (e.g., `app-1`)
  4. The agent is launched with context about the issue
  5. Metadata is written to the sessions directory
  ```

  ```markdown Workflows theme={null}
  ### Bulk Issue Processing
  1. Get list of issues from tracker
  2. Use `ao batch-spawn` to spawn sessions
  3. Monitor with `ao status`
  4. Agents fetch, implement, test, PR, respond to reviews
  5. Use `ao session cleanup` when PRs are merged
  ```
</CodeGroup>

### Custom Orchestrator Rules

Provide project-specific orchestrator instructions:

```yaml theme={null}
projects:
  my-app:
    orchestratorRules: |
      Prefer to batch-spawn related issues together.
      Monitor for stuck agents every 10 minutes.
      If more than 3 agents are stuck, escalate to human.
      When CI fails twice, attach to the session to investigate.
      Clean up merged sessions daily at 6pm.
```

These rules are injected into the orchestrator prompt and guide its behavior.

## CI/CD Integration

### GitHub Actions Integration

Trigger agent spawning from CI:

**`.github/workflows/agent-spawn.yml`:**

```yaml theme={null}
name: Spawn Agent for Issue

on:
  issues:
    types: [labeled]

jobs:
  spawn-agent:
    if: github.event.label.name == 'agent-task'
    runs-on: ubuntu-latest
    
    steps:
      - name: Checkout agent-orchestrator
        uses: actions/checkout@v3
        with:
          repository: your-org/agent-orchestrator
          path: agent-orchestrator
      
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '20'
      
      - name: Install dependencies
        run: |
          cd agent-orchestrator
          pnpm install
          pnpm build
      
      - name: Spawn agent
        run: |
          cd agent-orchestrator
          ao spawn my-project ${{ github.event.issue.number }}
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
```

**Trigger:** Label an issue with `agent-task` → agent spawns automatically.

### GitLab CI Integration

**`.gitlab-ci.yml`:**

```yaml theme={null}
spawn-agent:
  stage: deploy
  only:
    - issues
  script:
    - ao spawn my-project $CI_ISSUE_IID
  when: manual
```

**Trigger:** Click "Run Pipeline" on an issue → spawns agent.

### Webhook-Triggered Spawning

Create a webhook server that spawns agents:

```typescript theme={null}
// webhook-server.ts
import express from "express";
import { exec } from "child_process";
import { promisify } from "util";

const execAsync = promisify(exec);
const app = express();

app.post("/webhook/spawn", async (req, res) => {
  const { project, issue } = req.body;
  
  try {
    await execAsync(`ao spawn ${project} ${issue}`);
    res.json({ success: true, message: `Spawned agent for ${project}:${issue}` });
  } catch (err) {
    res.status(500).json({ error: err.message });
  }
});

app.listen(4000, () => console.log("Webhook server listening on :4000"));
```

**Trigger from anywhere:**

```bash theme={null}
curl -X POST http://localhost:4000/webhook/spawn \
  -H "Content-Type: application/json" \
  -d '{"project": "my-app", "issue": "123"}'
```

## Custom Notification Channels

### Slack Notifications

Configure Slack webhook:

```yaml theme={null}
notifiers:
  slack:
    plugin: slack
    webhook: ${SLACK_WEBHOOK_URL}
    channel: "#agent-updates"

notificationRouting:
  urgent: [slack]
  action: [slack]
  warning: [slack]
  info: [slack]
```

**Get webhook URL:**

1. Go to [https://api.slack.com/messaging/webhooks](https://api.slack.com/messaging/webhooks)
2. Create incoming webhook
3. Copy URL and set environment variable:
   ```bash theme={null}
   export SLACK_WEBHOOK_URL="https://hooks.slack.com/services/..."
   ```

### Discord Notifications

Create a custom Discord notifier:

```yaml theme={null}
notifiers:
  discord:
    plugin: webhook
    url: ${DISCORD_WEBHOOK_URL}
    method: POST
    headers:
      Content-Type: "application/json"
```

**Discord webhook format:**

```json theme={null}
{
  "content": "Agent fe-1 stuck on issue #123",
  "embeds": [
    {
      "title": "Session fe-1",
      "description": "Agent has been inactive for 10 minutes",
      "color": 15158332,
      "fields": [
        { "name": "Project", "value": "frontend", "inline": true },
        { "name": "Issue", "value": "#123", "inline": true },
        { "name": "Status", "value": "stuck", "inline": true }
      ]
    }
  ]
}
```

### Email Notifications

Create a custom email notifier plugin:

```typescript theme={null}
// packages/plugins/notifier-email/src/index.ts
import type { Notifier, PluginModule } from "@composio/ao-core";
import nodemailer from "nodemailer";

export const manifest = {
  name: "email",
  slot: "notifier" as const,
  description: "Email notifications via SMTP",
  version: "0.1.0",
};

export function create(config: { smtp: string; from: string; to: string }): Notifier {
  const transporter = nodemailer.createTransport(config.smtp);
  
  return {
    name: "email",
    async notify(notification) {
      await transporter.sendMail({
        from: config.from,
        to: config.to,
        subject: `[Agent Orchestrator] ${notification.title}`,
        text: notification.message,
      });
    },
  };
}

export default { manifest, create } satisfies PluginModule<Notifier>;
```

**Configure in YAML:**

```yaml theme={null}
notifiers:
  email:
    plugin: email
    smtp: "smtp://user:pass@smtp.gmail.com:587"
    from: "orchestrator@company.com"
    to: "team@company.com"
```

### PagerDuty Integration

For urgent escalations:

```yaml theme={null}
notifiers:
  pagerduty:
    plugin: webhook
    url: https://events.pagerduty.com/v2/enqueue
    method: POST
    headers:
      Authorization: "Token token=${PAGERDUTY_TOKEN}"
      Content-Type: "application/json"

notificationRouting:
  urgent: [pagerduty]  # Only page for urgent issues
  action: [slack]
  warning: [slack]
  info: []
```

## Advanced Use Cases

### 1. Coordinated Multi-Agent Workflow

**Scenario:** Frontend and backend changes for a feature.

```bash theme={null}
# Orchestrator spawns both
ao spawn frontend FEAT-123
ao spawn backend FEAT-123

# Monitor both sessions
ao status

# Coordinate: Wait for backend PR to merge before frontend
ao send fe-123 "Wait for backend PR #456 to merge before creating your PR"
```

### 2. Scheduled Batch Processing

**Scenario:** Process all new issues daily.

**Cron job:**

```bash theme={null}
# crontab -e
0 9 * * * /home/user/scripts/daily-agent-spawn.sh
```

**daily-agent-spawn.sh:**

```bash theme={null}
#!/bin/bash
set -e

cd ~/agent-orchestrator

# Get new issues from Linear
ISSUES=$(linear issue list --filter 'status:backlog AND label:ready' --format json | jq -r '.[].identifier')

if [ -z "$ISSUES" ]; then
  echo "No new issues"
  exit 0
fi

# Spawn agents for all issues
echo "$ISSUES" | xargs ao batch-spawn my-project

# Send Slack notification
curl -X POST $SLACK_WEBHOOK_URL \
  -H 'Content-Type: application/json' \
  -d "{\"text\": \"Spawned agents for $(echo $ISSUES | wc -w) issues\"}"
```

### 3. Auto-Retry with Escalation

**Scenario:** Retry failed agents with different prompts.

```yaml theme={null}
reactions:
  agent-stuck:
    threshold: 10m
    action: notify
    priority: urgent
```

**Custom escalation script:**

```bash theme={null}
#!/bin/bash
# retry-stuck-agent.sh

SESSION=$1
RETRY_COUNT=$(cat ~/.agent-orchestrator/sessions/my-app/$SESSION/retry_count || echo 0)

if [ $RETRY_COUNT -lt 2 ]; then
  # Retry with hint
  ao send $SESSION "You seem stuck. Try breaking the task into smaller steps."
  echo $((RETRY_COUNT + 1)) > ~/.agent-orchestrator/sessions/my-app/$SESSION/retry_count
else
  # Escalate to human
  curl -X POST $SLACK_WEBHOOK_URL \
    -H 'Content-Type: application/json' \
    -d "{\"text\": \"Session $SESSION stuck after 2 retries. Manual intervention needed.\"}"
fi
```

### 4. Continuous Monitoring Dashboard

**Scenario:** Real-time monitoring with Grafana.

**Export metrics:**

```bash theme={null}
#!/bin/bash
# export-metrics.sh

while true; do
  ao status --json > /tmp/agent-status.json
  
  # Parse and send to Prometheus pushgateway
  ACTIVE=$(jq '[.[] | select(.activity == "active")] | length' /tmp/agent-status.json)
  STUCK=$(jq '[.[] | select(.activity == "blocked" or .status == "stuck")] | length' /tmp/agent-status.json)
  
  echo "agent_active $ACTIVE" | curl --data-binary @- http://pushgateway:9091/metrics/job/agent_orchestrator
  echo "agent_stuck $STUCK" | curl --data-binary @- http://pushgateway:9091/metrics/job/agent_orchestrator
  
  sleep 30
done
```

**Grafana alerts:**

* Alert if `agent_stuck > 3`
* Alert if `agent_active == 0` during business hours

### 5. PR Review Automation

**Scenario:** Auto-approve low-risk PRs.

```yaml theme={null}
reactions:
  approved-and-green:
    auto: true
    action: auto-merge
```

**Custom review script:**

```bash theme={null}
#!/bin/bash
# auto-review.sh

PR_NUMBER=$1
PR_DATA=$(gh pr view $PR_NUMBER --json files,additions,deletions)

ADDITIONS=$(echo $PR_DATA | jq '.additions')
DELETIONS=$(echo $PR_DATA | jq '.deletions')
FILES=$(echo $PR_DATA | jq '.files | length')

# Auto-approve if small change
if [ $ADDITIONS -lt 50 ] && [ $DELETIONS -lt 50 ] && [ $FILES -lt 3 ]; then
  gh pr review $PR_NUMBER --approve --body "Auto-approved: Low-risk change"
fi
```

### 6. Multi-Repo Coordination

**Scenario:** Update multiple repos for a breaking API change.

```yaml theme={null}
# multi-repo-config.yaml
projects:
  api:
    repo: org/api
    path: ~/api
  
  frontend:
    repo: org/frontend
    path: ~/frontend
  
  mobile:
    repo: org/mobile
    path: ~/mobile
```

**Orchestrator workflow:**

```bash theme={null}
# 1. Update API
ao spawn api BREAKING-123
ao send api-1 "Implement breaking change to /auth endpoint"

# 2. Wait for API PR to merge
# (Manual or auto-merge)

# 3. Update dependents
ao spawn frontend BREAKING-123
ao spawn mobile BREAKING-123

ao send fe-1 "Update frontend to use new /auth endpoint"
ao send mob-1 "Update mobile to use new /auth endpoint"

# 4. Coordinate PRs
# Link PRs in GitHub: "Depends on #456"
```

## Event Log Analysis

All events are logged to `~/.agent-orchestrator/events.jsonl`:

```jsonl theme={null}
{"timestamp":"2024-03-04T10:30:00Z","type":"session.spawned","sessionId":"fe-1","projectId":"frontend","issueId":"123"}
{"timestamp":"2024-03-04T10:32:00Z","type":"pr.created","sessionId":"fe-1","prNumber":42}
{"timestamp":"2024-03-04T10:35:00Z","type":"ci.failed","sessionId":"fe-1","prNumber":42}
{"timestamp":"2024-03-04T10:36:00Z","type":"reaction.triggered","reaction":"ci-failed","action":"send-to-agent"}
```

**Query events:**

```bash theme={null}
# Count CI failures
jq 'select(.type == "ci.failed") | .sessionId' ~/.agent-orchestrator/events.jsonl | sort | uniq -c

# Find stuck sessions
jq 'select(.type == "session.stuck")' ~/.agent-orchestrator/events.jsonl

# PR creation rate
jq 'select(.type == "pr.created")' ~/.agent-orchestrator/events.jsonl | wc -l
```

## Custom Plugins

### Runtime Plugin (Docker Example)

Run agents in Docker containers:

```typescript theme={null}
// packages/plugins/runtime-docker/src/index.ts
import type { Runtime, RuntimeHandle, PluginModule } from "@composio/ao-core";
import Docker from "dockerode";

const docker = new Docker();

export const manifest = {
  name: "docker",
  slot: "runtime" as const,
  description: "Runtime plugin: Docker containers",
  version: "0.1.0",
};

export function create(): Runtime {
  return {
    name: "docker",
    
    async create(config) {
      const container = await docker.createContainer({
        Image: "agent-orchestrator:latest",
        Cmd: ["bash"],
        Tty: true,
        WorkingDir: config.workingDir,
      });
      
      await container.start();
      
      return {
        id: container.id,
        runtimeName: "docker",
        data: { containerId: container.id },
      };
    },
    
    async destroy(handle) {
      const container = docker.getContainer(handle.id);
      await container.stop();
      await container.remove();
    },
    
    async sendCommand(handle, command) {
      const container = docker.getContainer(handle.id);
      const exec = await container.exec({
        Cmd: ["bash", "-c", command],
        AttachStdout: true,
        AttachStderr: true,
      });
      await exec.start({});
    },
  };
}

export default { manifest, create } satisfies PluginModule<Runtime>;
```

**Use in config:**

```yaml theme={null}
defaults:
  runtime: docker
```

## Best Practices

<Steps>
  <Step title="Start simple">
    Use default reactions and workflows before building custom automation.
  </Step>

  <Step title="Monitor events">
    Tail the event log to understand system behavior:

    ```bash theme={null}
    tail -f ~/.agent-orchestrator/events.jsonl | jq
    ```
  </Step>

  <Step title="Test reactions in isolation">
    Use a test project to validate custom workflows before production.
  </Step>

  <Step title="Gradual escalation">
    Start with notifications, add automation as confidence grows.
  </Step>

  <Step title="Preserve audit trail">
    Never delete event logs or session metadata (disk is cheap).
  </Step>
</Steps>

## Next Steps

<CardGroup cols={2}>
  <Card title="Plugin Development" icon="plug" href="/advanced/plugin-development">
    Build custom Runtime, Agent, or Notifier plugins
  </Card>

  <Card title="API Reference" icon="code" href="/api-reference/core-types">
    Explore the full type definitions and interfaces
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.