@admin
Tạo pipeline CI/CD thực dụng từ tín hiệu công nghệ của dự án, gồm kiểm tra lặp lại và giai đoạn triển khai theo môi trường.
---
name: "ci-cd-pipeline-builder"
description: "Generate pragmatic CI/CD pipelines from detected project stack signals — fast baseline generation, repeatable checks, environment-aware deployment stages. Use when setting up CI for a new project, refactoring existing pipelines, or standardizing deployment workflows across multiple repos."
---
# CI/CD Pipeline Builder
**Tier:** POWERFUL
**Category:** Engineering
**Domain:** DevOps / Automation
## Overview
Use this skill to generate pragmatic CI/CD pipelines from detected project stack signals, not guesswork. It focuses on fast baseline generation, repeatable checks, and environment-aware deployment stages.
## Core Capabilities
- Detect language/runtime/tooling from repository files
- Recommend CI stages (`lint`, `test`, `build`, `deploy`)
- Generate GitHub Actions or GitLab CI starter pipelines
- Include caching and matrix strategy based on detected stack
- Emit machine-readable detection output for automation
- Keep pipeline logic aligned with project lockfiles and build commands
## When to Use
- Bootstrapping CI for a new repository
- Replacing brittle copied pipeline files
- Migrating between GitHub Actions and GitLab CI
- Auditing whether pipeline steps match actual stack
- Creating a reproducible baseline before custom hardening
## Key Workflows
### 1. Detect Stack
```bash
python3 scripts/stack_detector.py --repo . --format text
python3 scripts/stack_detector.py --repo . --format json > detected-stack.json
```
Supports input via stdin or `--input` file for offline analysis payloads.
### 2. Generate Pipeline From Detection
```bash
python3 scripts/pipeline_generator.py \
--input detected-stack.json \
--platform github \
--output .github/workflows/ci.yml \
--format text
```
Or end-to-end from repo directly:
```bash
python3 scripts/pipeline_generator.py --repo . --platform gitlab --output .gitlab-ci.yml
```
### 3. Validate Before Merge
1. Confirm commands exist in project (`test`, `lint`, `build`).
2. Run generated pipeline locally where possible.
3. Ensure required secrets/env vars are documented.
4. Keep deploy jobs gated by protected branches/environments.
### 4. Add Deployment Stages Safely
- Start with CI-only (`lint/test/build`).
- Add staging deploy with explicit environment context.
- Add production deploy with manual gate/approval.
- Keep rollout/rollback commands explicit and auditable.
## Script Interfaces
- `python3 scripts/stack_detector.py --help`
- Detects stack signals from repository files
- Reads optional JSON input from stdin/`--input`
- `python3 scripts/pipeline_generator.py --help`
- Generates GitHub/GitLab YAML from detection payload
- Writes to stdout or `--output`
## Common Pitfalls
1. Copying a Node pipeline into Python/Go repos
2. Enabling deploy jobs before stable tests
3. Forgetting dependency cache keys
4. Running expensive matrix builds for every trivial branch
5. Missing branch protections around prod deploy jobs
6. Hardcoding secrets in YAML instead of CI secret stores
## Best Practices
1. Detect stack first, then generate pipeline.
2. Keep generated baseline under version control.
3. Add one optimization at a time (cache, matrix, split jobs).
4. Require green CI before deployment jobs.
5. Use protected environments for production credentials.
6. Regenerate pipeline when stack changes significantly.
## References
- [references/github-actions-templates.md](references/github-actions-templates.md)
- [references/gitlab-ci-templates.md](references/gitlab-ci-templates.md)
- [references/deployment-gates.md](references/deployment-gates.md)
- [README.md](README.md)
## Detection Heuristics
The stack detector prioritizes deterministic file signals over heuristics:
- Lockfiles determine package manager preference
- Language manifests determine runtime families
- Script commands (if present) drive lint/test/build commands
- Missing scripts trigger conservative placeholder commands
## Generation Strategy
Start with a minimal, reliable pipeline:
1. Checkout and setup runtime
2. Install dependencies with cache strategy
3. Run lint, test, build in separate steps
4. Publish artifacts only after passing checks
Then layer advanced behavior (matrix builds, security scans, deploy gates).
## Platform Decision Notes
- GitHub Actions for tight GitHub ecosystem integration
- GitLab CI for integrated SCM + CI in self-hosted environments
- Keep one canonical pipeline source per repo to reduce drift
## Validation Checklist
1. Generated YAML parses successfully.
2. All referenced commands exist in the repo.
3. Cache strategy matches package manager.
4. Required secrets are documented, not embedded.
5. Branch/protected-environment rules match org policy.
## Scaling Guidance
- Split long jobs by stage when runtime exceeds 10 minutes.
- Introduce test matrix only when compatibility truly requires it.
- Separate deploy jobs from CI jobs to keep feedback fast.
- Track pipeline duration and flakiness as first-class metrics.
FILE:README.md
# CI/CD Pipeline Builder
Detects your repository stack and generates practical CI pipeline templates for GitHub Actions and GitLab CI. Designed as a fast baseline you can extend with deployment controls.
## Quick Start
```bash
# Detect stack
python3 scripts/stack_detector.py --repo . --format json > stack.json
# Generate GitHub Actions workflow
python3 scripts/pipeline_generator.py \
--input stack.json \
--platform github \
--output .github/workflows/ci.yml \
--format text
```
## Included Tools
- `scripts/stack_detector.py`: repository signal detection with JSON/text output
- `scripts/pipeline_generator.py`: generate GitHub/GitLab CI YAML from detection payload
## References
- `references/github-actions-templates.md`
- `references/gitlab-ci-templates.md`
- `references/deployment-gates.md`
## Installation
### Claude Code
```bash
cp -R engineering/ci-cd-pipeline-builder ~/.claude/skills/ci-cd-pipeline-builder
```
### OpenAI Codex
```bash
cp -R engineering/ci-cd-pipeline-builder ~/.codex/skills/ci-cd-pipeline-builder
```
### OpenClaw
```bash
cp -R engineering/ci-cd-pipeline-builder ~/.openclaw/skills/ci-cd-pipeline-builder
```
FILE:references/deployment-gates.md
# Deployment Gates
## Minimum Gate Policy
- `lint` must pass before `test`.
- `test` must pass before `build`.
- `build` artifact required for deploy jobs.
- Production deploy requires manual approval and protected branch.
## Environment Pattern
- `develop` -> auto deploy to staging
- `main` -> manual promote to production
## Rollback Requirement
Every deploy job should define a rollback command or procedure reference.
FILE:references/github-actions-templates.md
# GitHub Actions Templates
## Node.js Baseline
```yaml
name: Node CI
on: [push, pull_request]
jobs:
ci:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
- run: npm ci
- run: npm run lint
- run: npm test
- run: npm run build
```
## Python Baseline
```yaml
name: Python CI
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- run: python3 -m pip install -U pip
- run: python3 -m pip install -r requirements.txt
- run: python3 -m pytest
```
FILE:references/gitlab-ci-templates.md
# GitLab CI Templates
## Node.js Baseline
```yaml
stages:
- lint
- test
- build
node_lint:
image: node:20
stage: lint
script:
- npm ci
- npm run lint
node_test:
image: node:20
stage: test
script:
- npm ci
- npm test
```
## Python Baseline
```yaml
stages:
- test
python_test:
image: python:3.12
stage: test
script:
- python3 -m pip install -U pip
- python3 -m pip install -r requirements.txt
- python3 -m pytest
```
FILE:scripts/pipeline_generator.py
#!/usr/bin/env python3
"""Generate CI pipeline YAML from detected stack data.
Input sources:
- --input stack report JSON file
- stdin stack report JSON
- --repo path (auto-detect stack)
Output:
- text/json summary
- pipeline YAML written via --output or printed to stdout
"""
import argparse
import json
import sys
from dataclasses import dataclass, asdict
from pathlib import Path
from typing import Any, Dict, List, Optional
class CLIError(Exception):
"""Raised for expected CLI failures."""
@dataclass
class PipelineSummary:
platform: str
output: str
stages: List[str]
uses_cache: bool
languages: List[str]
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Generate CI/CD pipeline YAML from detected stack.")
parser.add_argument("--input", help="Stack report JSON file. If omitted, can read stdin JSON.")
parser.add_argument("--repo", help="Repository path for auto-detection fallback.")
parser.add_argument("--platform", choices=["github", "gitlab"], required=True, help="Target CI platform.")
parser.add_argument("--output", help="Write YAML to this file; otherwise print to stdout.")
parser.add_argument("--format", choices=["text", "json"], default="text", help="Summary output format.")
return parser.parse_args()
def load_json_input(input_path: Optional[str]) -> Optional[Dict[str, Any]]:
if input_path:
try:
return json.loads(Path(input_path).read_text(encoding="utf-8"))
except Exception as exc:
raise CLIError(f"Failed reading --input: {exc}") from exc
if not sys.stdin.isatty():
raw = sys.stdin.read().strip()
if raw:
try:
return json.loads(raw)
except json.JSONDecodeError as exc:
raise CLIError(f"Invalid JSON from stdin: {exc}") from exc
return None
def detect_stack(repo: Path) -> Dict[str, Any]:
scripts = {}
pkg_file = repo / "package.json"
if pkg_file.exists():
try:
pkg = json.loads(pkg_file.read_text(encoding="utf-8"))
raw_scripts = pkg.get("scripts", {})
if isinstance(raw_scripts, dict):
scripts = raw_scripts
except Exception:
scripts = {}
languages: List[str] = []
if pkg_file.exists():
languages.append("node")
if (repo / "pyproject.toml").exists() or (repo / "requirements.txt").exists():
languages.append("python")
if (repo / "go.mod").exists():
languages.append("go")
return {
"languages": sorted(set(languages)),
"signals": {
"pnpm_lock": (repo / "pnpm-lock.yaml").exists(),
"yarn_lock": (repo / "yarn.lock").exists(),
"npm_lock": (repo / "package-lock.json").exists(),
"dockerfile": (repo / "Dockerfile").exists(),
},
"lint_commands": ["npm run lint"] if "lint" in scripts else [],
"test_commands": ["npm test"] if "test" in scripts else [],
"build_commands": ["npm run build"] if "build" in scripts else [],
}
def select_node_install(signals: Dict[str, Any]) -> str:
if signals.get("pnpm_lock"):
return "pnpm install --frozen-lockfile"
if signals.get("yarn_lock"):
return "yarn install --frozen-lockfile"
return "npm ci"
def github_yaml(stack: Dict[str, Any]) -> str:
langs = stack.get("languages", [])
signals = stack.get("signals", {})
lint_cmds = stack.get("lint_commands", []) or ["echo 'No lint command configured'"]
test_cmds = stack.get("test_commands", []) or ["echo 'No test command configured'"]
build_cmds = stack.get("build_commands", []) or ["echo 'No build command configured'"]
lines: List[str] = [
"name: CI",
"on:",
" push:",
" branches: [main, develop]",
" pull_request:",
" branches: [main, develop]",
"",
"jobs:",
]
if "node" in langs:
lines.extend(
[
" node-ci:",
" runs-on: ubuntu-latest",
" steps:",
" - uses: actions/checkout@v4",
" - uses: actions/setup-node@v4",
" with:",
" node-version: '20'",
" cache: 'npm'",
f" - run: {select_node_install(signals)}",
]
)
for cmd in lint_cmds + test_cmds + build_cmds:
lines.append(f" - run: {cmd}")
if "python" in langs:
lines.extend(
[
" python-ci:",
" runs-on: ubuntu-latest",
" steps:",
" - uses: actions/checkout@v4",
" - uses: actions/setup-python@v5",
" with:",
" python-version: '3.12'",
" - run: python3 -m pip install -U pip",
" - run: python3 -m pip install -r requirements.txt || true",
" - run: python3 -m pytest || true",
]
)
if "go" in langs:
lines.extend(
[
" go-ci:",
" runs-on: ubuntu-latest",
" steps:",
" - uses: actions/checkout@v4",
" - uses: actions/setup-go@v5",
" with:",
" go-version: '1.22'",
" - run: go test ./...",
" - run: go build ./...",
]
)
return "\n".join(lines) + "\n"
def gitlab_yaml(stack: Dict[str, Any]) -> str:
langs = stack.get("languages", [])
signals = stack.get("signals", {})
lint_cmds = stack.get("lint_commands", []) or ["echo 'No lint command configured'"]
test_cmds = stack.get("test_commands", []) or ["echo 'No test command configured'"]
build_cmds = stack.get("build_commands", []) or ["echo 'No build command configured'"]
lines: List[str] = [
"stages:",
" - lint",
" - test",
" - build",
"",
]
if "node" in langs:
install_cmd = select_node_install(signals)
lines.extend(
[
"node_lint:",
" image: node:20",
" stage: lint",
" script:",
f" - {install_cmd}",
]
)
for cmd in lint_cmds:
lines.append(f" - {cmd}")
lines.extend(
[
"",
"node_test:",
" image: node:20",
" stage: test",
" script:",
f" - {install_cmd}",
]
)
for cmd in test_cmds:
lines.append(f" - {cmd}")
lines.extend(
[
"",
"node_build:",
" image: node:20",
" stage: build",
" script:",
f" - {install_cmd}",
]
)
for cmd in build_cmds:
lines.append(f" - {cmd}")
if "python" in langs:
lines.extend(
[
"",
"python_test:",
" image: python:3.12",
" stage: test",
" script:",
" - python3 -m pip install -U pip",
" - python3 -m pip install -r requirements.txt || true",
" - python3 -m pytest || true",
]
)
if "go" in langs:
lines.extend(
[
"",
"go_test:",
" image: golang:1.22",
" stage: test",
" script:",
" - go test ./...",
" - go build ./...",
]
)
return "\n".join(lines) + "\n"
def main() -> int:
args = parse_args()
stack = load_json_input(args.input)
if stack is None:
if not args.repo:
raise CLIError("Provide stack input via --input/stdin or set --repo for auto-detection.")
repo = Path(args.repo).resolve()
if not repo.exists() or not repo.is_dir():
raise CLIError(f"Invalid repo path: {repo}")
stack = detect_stack(repo)
if args.platform == "github":
yaml_content = github_yaml(stack)
else:
yaml_content = gitlab_yaml(stack)
output_path = args.output or "stdout"
if args.output:
out = Path(args.output)
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(yaml_content, encoding="utf-8")
else:
print(yaml_content, end="")
summary = PipelineSummary(
platform=args.platform,
output=output_path,
stages=["lint", "test", "build"],
uses_cache=True,
languages=stack.get("languages", []),
)
if args.format == "json":
print(json.dumps(asdict(summary), indent=2), file=sys.stderr if not args.output else sys.stdout)
else:
text = (
"Pipeline generated\n"
f"- platform: {summary.platform}\n"
f"- output: {summary.output}\n"
f"- stages: {', '.join(summary.stages)}\n"
f"- languages: {', '.join(summary.languages) if summary.languages else 'none'}"
)
print(text, file=sys.stderr if not args.output else sys.stdout)
return 0
if __name__ == "__main__":
try:
raise SystemExit(main())
except CLIError as exc:
print(f"ERROR: {exc}", file=sys.stderr)
raise SystemExit(2)
FILE:scripts/stack_detector.py
#!/usr/bin/env python3
"""Detect project stack/tooling signals for CI/CD pipeline generation.
Input sources:
- repository scan via --repo
- JSON via --input file
- JSON via stdin
Output:
- text summary or JSON payload
"""
import argparse
import json
import sys
from dataclasses import dataclass, asdict
from pathlib import Path
from typing import Dict, List, Optional
class CLIError(Exception):
"""Raised for expected CLI failures."""
@dataclass
class StackReport:
repo: str
languages: List[str]
package_managers: List[str]
ci_targets: List[str]
test_commands: List[str]
build_commands: List[str]
lint_commands: List[str]
signals: Dict[str, bool]
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Detect stack/tooling from a repository.")
parser.add_argument("--input", help="JSON input file (precomputed signal payload).")
parser.add_argument("--repo", default=".", help="Repository path to scan.")
parser.add_argument("--format", choices=["text", "json"], default="text", help="Output format.")
return parser.parse_args()
def load_payload(input_path: Optional[str]) -> Optional[dict]:
if input_path:
try:
return json.loads(Path(input_path).read_text(encoding="utf-8"))
except Exception as exc:
raise CLIError(f"Failed reading --input file: {exc}") from exc
if not sys.stdin.isatty():
raw = sys.stdin.read().strip()
if raw:
try:
return json.loads(raw)
except json.JSONDecodeError as exc:
raise CLIError(f"Invalid JSON from stdin: {exc}") from exc
return None
def read_package_scripts(repo: Path) -> Dict[str, str]:
pkg = repo / "package.json"
if not pkg.exists():
return {}
try:
data = json.loads(pkg.read_text(encoding="utf-8"))
except Exception:
return {}
scripts = data.get("scripts", {})
return scripts if isinstance(scripts, dict) else {}
def detect(repo: Path) -> StackReport:
signals = {
"package_json": (repo / "package.json").exists(),
"pnpm_lock": (repo / "pnpm-lock.yaml").exists(),
"yarn_lock": (repo / "yarn.lock").exists(),
"npm_lock": (repo / "package-lock.json").exists(),
"pyproject": (repo / "pyproject.toml").exists(),
"requirements": (repo / "requirements.txt").exists(),
"go_mod": (repo / "go.mod").exists(),
"dockerfile": (repo / "Dockerfile").exists(),
"vercel": (repo / "vercel.json").exists(),
"helm": (repo / "helm").exists() or (repo / "charts").exists(),
"k8s": (repo / "k8s").exists() or (repo / "kubernetes").exists(),
}
languages: List[str] = []
package_managers: List[str] = []
ci_targets: List[str] = ["github", "gitlab"]
if signals["package_json"]:
languages.append("node")
if signals["pnpm_lock"]:
package_managers.append("pnpm")
elif signals["yarn_lock"]:
package_managers.append("yarn")
else:
package_managers.append("npm")
if signals["pyproject"] or signals["requirements"]:
languages.append("python")
package_managers.append("pip")
if signals["go_mod"]:
languages.append("go")
scripts = read_package_scripts(repo)
lint_commands: List[str] = []
test_commands: List[str] = []
build_commands: List[str] = []
if "lint" in scripts:
lint_commands.append("npm run lint")
if "test" in scripts:
test_commands.append("npm test")
if "build" in scripts:
build_commands.append("npm run build")
if "python" in languages:
lint_commands.append("python3 -m ruff check .")
test_commands.append("python3 -m pytest")
if "go" in languages:
lint_commands.append("go vet ./...")
test_commands.append("go test ./...")
build_commands.append("go build ./...")
return StackReport(
repo=str(repo.resolve()),
languages=sorted(set(languages)),
package_managers=sorted(set(package_managers)),
ci_targets=ci_targets,
test_commands=sorted(set(test_commands)),
build_commands=sorted(set(build_commands)),
lint_commands=sorted(set(lint_commands)),
signals=signals,
)
def format_text(report: StackReport) -> str:
lines = [
"Detected stack",
f"- repo: {report.repo}",
f"- languages: {', '.join(report.languages) if report.languages else 'none'}",
f"- package managers: {', '.join(report.package_managers) if report.package_managers else 'none'}",
f"- lint commands: {', '.join(report.lint_commands) if report.lint_commands else 'none'}",
f"- test commands: {', '.join(report.test_commands) if report.test_commands else 'none'}",
f"- build commands: {', '.join(report.build_commands) if report.build_commands else 'none'}",
]
return "\n".join(lines)
def main() -> int:
args = parse_args()
payload = load_payload(args.input)
if payload:
try:
report = StackReport(**payload)
except TypeError as exc:
raise CLIError(f"Invalid input payload for StackReport: {exc}") from exc
else:
repo = Path(args.repo).resolve()
if not repo.exists() or not repo.is_dir():
raise CLIError(f"Invalid repo path: {repo}")
report = detect(repo)
if args.format == "json":
print(json.dumps(asdict(report), indent=2))
else:
print(format_text(report))
return 0
if __name__ == "__main__":
try:
raise SystemExit(main())
except CLIError as exc:
print(f"ERROR: {exc}", file=sys.stderr)
raise SystemExit(2)
Viết, viết lại và cải thiện nội dung marketing cho trang chủ, trang đích, trang giá, trang tính năng và giới thiệu.
---
name: copywriting
description: When the user wants to write, rewrite, or improve marketing copy for any page — including homepage, landing pages, pricing pages, feature pages, about pages, or product pages. Also use when the user says "write copy for," "improve this copy," "rewrite this page," "marketing copy," "headline help," "CTA copy," "value proposition," "tagline," "subheadline," "hero section copy," "above the fold," "this copy is weak," "make this more compelling," or "help me describe my product." Use this whenever someone is working on website text that needs to persuade or convert. For email copy, see emails. For popup copy, see popups. For editing existing copy, see copy-editing. For the offer underneath the copy (bonuses, guarantees, value framing), see offers.
metadata:
version: 2.0.2
---
# Copywriting
You are an expert conversion copywriter. Your goal is to write marketing copy that is clear, compelling, and drives action.
## Before Writing
**Check for product marketing context first:**
If `.agents/product-marketing.md` exists (or `.claude/product-marketing.md`, or the legacy `product-marketing-context.md` filename, in older setups), read it before asking questions. Use that context and only ask for information not already covered or specific to this task.
Gather this context (ask if not provided):
### 1. Page Purpose
- What type of page? (homepage, landing page, pricing, feature, about)
- What is the ONE primary action you want visitors to take?
### 2. Audience
- Who is the ideal customer?
- What problem are they trying to solve?
- What objections or hesitations do they have?
- What language do they use to describe their problem?
### 3. Product/Offer
- What are you selling or offering?
- What makes it different from alternatives?
- What's the key transformation or outcome?
- Any proof points (numbers, testimonials, case studies)?
### 4. Context
- Where is traffic coming from? (ads, organic, email)
- What do visitors already know before arriving?
---
## Copywriting Principles
### Clarity Over Cleverness
If you have to choose between clear and creative, choose clear. Clarity is not just tidier — it converts: clearer positioning and copy is associated with +81% conversions, a 38% shorter sales cycle, 28% lower CAC, and 175% more referrals. When a reader has to decode your line, you've lost them.
**For message-market fit tools** — the "Now you can" test, the Human Action Model (discomfort → vision → path), the Perception Gap, and the clarity metrics: See [references/copy-frameworks.md](references/copy-frameworks.md#clarity--message-market-fit)
### Benefits Over Features
Features: What it does. Benefits: What that means for the customer.
### Specificity Over Vagueness
- Vague: "Save time on your workflow"
- Specific: "Cut your weekly reporting from 4 hours to 15 minutes"
### Customer Language Over Company Language
Use words your customers use. Mirror voice-of-customer from reviews, interviews, support tickets.
### One Idea Per Section
Each section should advance one argument. Build a logical flow down the page.
---
## Writing Style Rules
### Core Principles
1. **Simple over complex** — "Use" not "utilize," "help" not "facilitate"
2. **Specific over vague** — Avoid "streamline," "optimize," "innovative"
3. **Active over passive** — "We generate reports" not "Reports are generated"
4. **Confident over qualified** — Remove "almost," "very," "really"
5. **Show over tell** — Describe the outcome instead of using adverbs
6. **Honest over sensational** — Fabricated statistics or testimonials erode trust and create legal liability
### Quick Quality Check
- Jargon that could confuse outsiders?
- Sentences trying to do too much?
- Passive voice constructions?
- Exclamation points? (remove them)
- Marketing buzzwords without substance?
For thorough line-by-line review, use the **copy-editing** skill after your draft.
---
## Best Practices
### Be Direct
Get to the point. Don't bury the value in qualifications.
❌ Slack lets you share files instantly, from documents to images, directly in your conversations
✅ Need to share a screenshot? Send as many documents, images, and audio files as your heart desires.
### Use Rhetorical Questions
Questions engage readers and make them think about their own situation.
- "Hate returning stuff to Amazon?"
- "Tired of chasing approvals?"
### Use Analogies When Helpful
Analogies make abstract concepts concrete and memorable.
### Pepper in Humor (When Appropriate)
Puns and wit make copy memorable—but only if it fits the brand and doesn't undermine clarity.
---
## Page Structure Framework
### Above the Fold
**Headline**
- Your single most important message
- Communicate core value proposition
- Specific > generic
**Example formulas:**
- "{Achieve outcome} without {pain point}"
- "The {category} for {audience}"
- "Never {unpleasant event} again"
- "{Question highlighting main pain point}"
**For comprehensive headline formulas**: See [references/copy-frameworks.md](references/copy-frameworks.md)
**Structure the hero as a transformation** — current discomfort → better vision → path to action (the Human Action Model), then run every headline through the "Now you can" test. See [references/copy-frameworks.md](references/copy-frameworks.md#clarity--message-market-fit)
**For natural transition phrases**: See [references/natural-transitions.md](references/natural-transitions.md)
**Subheadline**
- Expands on headline
- Adds specificity
- 1-2 sentences max
**Primary CTA**
- Action-oriented button text
- Communicate what they get: "Start Free Trial" > "Sign Up"
### Core Sections
| Section | Purpose |
|---------|---------|
| Social Proof | Build credibility (logos, stats, testimonials) |
| Problem/Pain | Show you understand their situation |
| Solution/Benefits | Connect to outcomes (3-5 key benefits) |
| How It Works | Reduce perceived complexity (3-4 steps) |
| Objection Handling | FAQ, comparisons, guarantees |
| Final CTA | Recap value, repeat CTA, risk reversal |
**For detailed section types and page templates**: See [references/copy-frameworks.md](references/copy-frameworks.md)
---
## CTA Copy Guidelines
**Weak CTAs (avoid):**
- Submit, Sign Up, Learn More, Click Here, Get Started
**Strong CTAs (use):**
- Start Free Trial
- Get [Specific Thing]
- See [Product] in Action
- Create Your First [Thing]
- Download the Guide
**Formula:** [Action Verb] + [What They Get] + [Qualifier if needed]
Examples:
- "Start My Free Trial"
- "Get the Complete Checklist"
- "See Pricing for My Team"
---
## Page-Specific Guidance
### Homepage
- Serve multiple audiences without being generic
- Lead with broadest value proposition
- Provide clear paths for different visitor intents
### Landing Page
- Single message, single CTA
- Match headline to ad/traffic source
- Complete argument on one page
### Pricing Page
- Help visitors choose the right plan
- Address "which is right for me?" anxiety
- Make recommended plan obvious
### Feature Page
- Connect feature → benefit → outcome
- Show use cases and examples
- Clear path to try or buy
### About Page
- Tell the story of why you exist
- Connect mission to customer benefit
- Still include a CTA
---
## Voice and Tone
Before writing, establish:
**Formality level:**
- Casual/conversational
- Professional but friendly
- Formal/enterprise
**Brand personality:**
- Playful or serious?
- Bold or understated?
- Technical or accessible?
Maintain consistency, but adjust intensity:
- Headlines can be bolder
- Body copy should be clearer
- CTAs should be action-oriented
---
## Output Format
When writing copy, provide:
### Page Copy
Organized by section:
- Headline, Subheadline, CTA
- Section headers and body copy
- Secondary CTAs
### Annotations
For key elements, explain:
- Why you made this choice
- What principle it applies
### Alternatives
For headlines and CTAs, provide 2-3 options:
- Option A: [copy] — [rationale]
- Option B: [copy] — [rationale]
### Meta Content (if relevant)
- Page title (for SEO)
- Meta description
---
## Related Skills
- **copy-editing**: For polishing existing copy (use after your draft)
- **cro**: If page structure/strategy needs work, not just copy
- **emails**: For email copywriting
- **popups**: For popup and modal copy
- **ab-testing**: To test copy variations
FILE:evals/evals.json
{
"skill_name": "copywriting",
"evals": [
{
"id": 1,
"prompt": "Write homepage copy for a SaaS tool that automates employee onboarding. Target audience is HR directors at mid-size companies (200-2000 employees). Main differentiator is that it integrates with all major HRIS systems and cuts onboarding time from 2 weeks to 2 days.",
"expected_output": "Should check for product-marketing.md first. Should write full page copy organized by section: Headline, Subheadline, CTA (above the fold), then Social Proof, Problem/Pain, Solution/Benefits, How It Works, Objection Handling, and Final CTA. Should follow copywriting principles: clarity over cleverness, benefits over features, specificity (use the '2 weeks to 2 days' stat), customer language. Headline should communicate core value proposition. CTAs should be action-oriented ('Start Free Trial' not 'Submit'). Should provide 2-3 headline alternatives with rationale. Should include annotations explaining key copy choices. Should include meta content (SEO page title and meta description).",
"assertions": [
"Checks for product-marketing.md",
"Writes full page copy organized by section",
"Includes Headline, Subheadline, and CTA above the fold",
"Includes Social Proof, Problem/Pain, Solution/Benefits, How It Works sections",
"Uses the '2 weeks to 2 days' specificity in copy",
"CTAs are action-oriented, not generic",
"Provides 2-3 headline alternatives with rationale",
"Includes annotations explaining copy choices",
"Includes meta content (SEO title and meta description)"
],
"files": []
},
{
"id": 2,
"prompt": "Rewrite this headline: 'An Innovative AI-Powered Platform for Streamlined Business Operations' — it's for a B2B SaaS tool that helps small businesses manage invoicing and payments.",
"expected_output": "Should identify problems: jargon ('innovative,' 'AI-powered,' 'streamlined,' 'business operations'), too vague, company language not customer language. Should apply copywriting principles — specificity over vagueness, benefits over features, customer language over company language. Should provide 2-3 alternative headlines using formulas like '{Achieve outcome} without {pain point}' or 'The {category} for {audience}'. Each alternative should include rationale. Should also suggest a subheadline that adds specificity.",
"assertions": [
"Identifies jargon in original headline",
"Identifies vagueness as a problem",
"Identifies company language vs customer language issue",
"Provides 2-3 alternative headlines",
"Alternatives use headline formulas from the skill",
"Each alternative includes rationale",
"Suggests a subheadline"
],
"files": []
},
{
"id": 3,
"prompt": "i need copy for my pricing page. we have three plans: starter ($29/mo), pro ($79/mo), business ($199/mo). it's a social media scheduling tool for marketers",
"expected_output": "Should trigger on the casual phrasing. Should ask or infer audience context. Should apply Pricing Page guidance: help visitors choose the right plan, address 'which is right for me?' anxiety, make recommended plan obvious. Should write plan names, descriptions, feature lists with benefit-oriented copy (not just feature names). Should include a page headline that addresses the pricing decision. CTAs should be specific per plan. Should handle objection handling (FAQ copy). Should provide alternatives for key elements.",
"assertions": [
"Triggers on casual phrasing",
"Applies Pricing Page guidance",
"Addresses 'which plan is right for me' anxiety",
"Makes recommended plan obvious",
"Writes benefit-oriented feature copy, not just feature names",
"Includes page headline",
"CTAs are specific per plan",
"Includes FAQ or objection handling copy",
"Provides alternatives for key elements"
],
"files": []
},
{
"id": 4,
"prompt": "Write copy for our About page. We're a 3-person startup that built a developer tool for database migrations. Founded because we kept losing data during migrations at our last jobs. Tone should be professional but human.",
"expected_output": "Should apply About Page guidance: tell the story of why you exist, connect mission to customer benefit, still include a CTA. Should adapt voice and tone to 'professional but human' as specified. Should tell the founder origin story authentically. Should connect the personal pain to the customer's pain. Should include a CTA even on the About page. Copy should follow style rules: active voice, confident, specific. Should NOT be overly corporate or generic.",
"assertions": [
"Applies About Page guidance",
"Tells the story of why the company exists",
"Connects mission to customer benefit",
"Includes a CTA",
"Adapts tone to professional but human",
"Uses the founder origin story",
"Connects personal pain to customer pain",
"Uses active voice",
"Avoids corporate jargon"
],
"files": []
},
{
"id": 5,
"prompt": "Can you improve this CTA? We currently have 'Learn More' on our feature page for our analytics dashboard product.",
"expected_output": "Should immediately identify 'Learn More' as a weak CTA per the guidelines. Should apply the CTA formula: [Action Verb] + [What They Get] + [Qualifier]. Should provide 2-3 strong alternatives like 'See the Dashboard in Action,' 'Start Your Free Trial,' or 'Explore Analytics Features.' Each alternative should include rationale and context for when it works best. Should also consider CTA hierarchy — whether this is a primary or secondary CTA, and suggest complementary CTAs if relevant.",
"assertions": [
"Identifies 'Learn More' as a weak CTA",
"Applies the CTA formula from the skill",
"Provides 2-3 strong alternatives",
"Each alternative includes rationale",
"Considers CTA hierarchy (primary vs secondary)",
"Suggests complementary CTAs"
],
"files": []
},
{
"id": 6,
"prompt": "Write me a 5-email welcome sequence for new trial users of our project management tool.",
"expected_output": "Should recognize this is an email copywriting task, not page copywriting. Should defer to or cross-reference the emails skill, which specifically handles email sequences, drip campaigns, and lifecycle emails. May provide brief general guidance but should make clear that emails is the right skill for this task.",
"assertions": [
"Recognizes this as email sequence work",
"References or defers to emails skill",
"Does not attempt to write a full email sequence using page copywriting patterns"
],
"files": []
},
{
"id": 7,
"prompt": "Review this copy and tell me what's wrong: 'We are extremely excited to announce our revolutionary, cutting-edge platform that will totally transform how businesses optimize their workflows! Sign up now!!'",
"expected_output": "Should apply the Quick Quality Check. Should identify: exclamation points (remove them), marketing buzzwords without substance ('revolutionary,' 'cutting-edge,' 'totally transform,' 'optimize'), passive/weak constructions ('we are excited to announce'), vague language ('workflows'). Should apply writing style rules: simple over complex, specific over vague, confident over qualified, show over tell. Should rewrite the copy following these principles. Should provide 2-3 alternatives.",
"assertions": [
"Identifies exclamation point overuse",
"Identifies marketing buzzwords without substance",
"Identifies vague language",
"Applies writing style rules",
"Rewrites the copy following principles",
"Provides alternatives",
"Result is specific, clear, and jargon-free"
],
"files": []
},
{
"id": 8,
"prompt": "Write above-the-fold copy for a calendar scheduling tool. Our differentiator is that the recipient gets to overlay their own calendar on the invite, so picking a time feels fair to both people instead of one-sided. Same product needs to work for indie founders AND for enterprise ops teams.",
"expected_output": "Should structure the hero using the Human Action Model transformation spine: current discomfort (the awkwardness of sending a one-sided scheduling link), better vision (scheduling that feels considerate to both people), and path to action (the overlay mechanic + a specific CTA). Should run headline candidates through the 'Now you can' test and prefer lines that are compelling and true. Should reference or echo the SavvyCal awkward-link insight ('You shouldn't have to feel awkward sending out your scheduling link') as the message-market-fit model. Should surface the Perception Gap: the same benefit reads differently by risk tolerance, so it should provide a value-prop swap — a founder-facing framing (speed, no sales calls) and an enterprise-facing framing (security, SLAs, reliability) rather than one averaged, mushy message. Should favor clarity over cleverness and provide 2-3 headline alternatives with rationale.",
"assertions": [
"Structures the hero as discomfort -> vision -> path (Human Action Model)",
"Applies the 'Now you can' test to headline candidates",
"References the SavvyCal awkward-link message-market-fit insight",
"Surfaces the Perception Gap between segments",
"Provides a value-prop swap: founder framing vs enterprise framing",
"Favors clarity over cleverness",
"Provides 2-3 headline alternatives with rationale"
],
"files": []
}
]
}
FILE:references/copy-frameworks.md
# Copy Frameworks Reference
Headline formulas, page section types, and structural templates.
## Contents
- Headline Formulas (outcome-focused, problem-focused, audience-focused, differentiation-focused, proof-focused, additional formulas)
- Landing Page Section Types (core sections, supporting sections)
- Page Structure Templates (feature-heavy page, varied engaging page, compact landing page, enterprise/B2B landing page, product launch page)
- Section Writing Tips (problem section, benefits section, how it works section, testimonial selection)
- Clarity & Message-Market Fit (the "Now you can" test, Human Action Model, the Perception Gap, the SavvyCal case, clarity metrics)
## Headline Formulas
### Outcome-Focused
**{Achieve desirable outcome} without {pain point}**
> Understand how users are really experiencing your site without drowning in numbers
**{Achieve desirable outcome} by {how product makes it possible}**
> Generate more leads by seeing which companies visit your site
**Turn {input} into {outcome}**
> Turn your hard-earned sales into repeat customers
**[Achieve outcome] in [timeframe]**
> Get your tax refund in 10 days
---
### Problem-Focused
**Never {unpleasant event} again**
> Never miss a sales opportunity again
**{Question highlighting the main pain point}**
> Hate returning stuff to Amazon?
**Stop [pain]. Start [pleasure].**
> Stop chasing invoices. Start getting paid on time.
---
### Audience-Focused
**{Key feature/product type} for {target audience}**
> Advanced analytics for Shopify e-commerce
**{Key feature/product type} for {target audience} to {what it's used for}**
> An online whiteboard for teams to ideate and brainstorm together
**You don't have to {skills or resources} to {achieve desirable outcome}**
> With Ahrefs, you don't have to be an SEO pro to rank higher and get more traffic
---
### Differentiation-Focused
**The {opposite of usual process} way to {achieve desirable outcome}**
> The easiest way to turn your passion into income
**The [category] that [key differentiator]**
> The CRM that updates itself
---
### Proof-Focused
**[Number] [people] use [product] to [outcome]**
> 50,000 marketers use Drip to send better emails
**{Key benefit of your product}**
> Sound clear in online meetings
---
### Additional Formulas
**The simple way to {outcome}**
> The simple way to track your time
**Finally, {category} that {benefit}**
> Finally, accounting software that doesn't suck
**{Outcome} without {common pain}**
> Build your website without writing code
**Get {benefit} from your {thing}**
> Get more revenue from your existing traffic
**{Action verb} your {thing} like {admirable example}**
> Market your SaaS like a Fortune 500
**What if you could {desirable outcome}?**
> What if you could close deals 30% faster?
**Everything you need to {outcome}**
> Everything you need to launch your course
**The {adjective} {category} built for {audience}**
> The lightweight CRM built for startups
---
## Landing Page Section Types
### Core Sections
**Hero (Above the Fold)**
- Headline + subheadline
- Primary CTA
- Supporting visual (product screenshot, hero image)
- Optional: Social proof bar
**Social Proof Bar**
- Customer logos (recognizable > many)
- Key metric ("10,000+ teams")
- Star rating with review count
- Short testimonial snippet
**Problem/Pain Section**
- Articulate their problem better than they can
- Create recognition ("that's exactly my situation")
- Hint at cost of not solving it
**Solution/Benefits Section**
- Bridge from problem to your solution
- 3-5 key benefits (not 10)
- Each: headline + explanation + proof if available
**How It Works**
- 3-4 numbered steps
- Reduces perceived complexity
- Each step: action + outcome
**Final CTA Section**
- Recap value proposition
- Repeat primary CTA
- Risk reversal (guarantee, free trial)
---
### Supporting Sections
**Testimonials**
- Full quotes with names, roles, companies
- Photos when possible
- Specific results over vague praise
- Formats: quote cards, video, tweet embeds
**Case Studies**
- Problem → Solution → Results
- Specific metrics and outcomes
- Customer name and context
- Can be snippets with "Read more" links
**Use Cases**
- Different ways product is used
- Helps visitors self-identify
- "For marketers who need X" format
**Personas / "Built For" Sections**
- Explicitly call out target audience
- "Perfect for [role]" blocks
- Addresses "Is this for me?" question
**FAQ Section**
- Address common objections
- Good for SEO
- Reduces support burden
- 5-10 most common questions
**Comparison Section**
- vs. competitors (name them or don't)
- vs. status quo (spreadsheets, manual processes)
- Tables or side-by-side format
**Integrations / Partners**
- Logos of tools you connect with
- "Works with your stack" messaging
- Builds credibility
**Founder Story / Manifesto**
- Why you built this
- What you believe
- Emotional connection
- Differentiates from faceless competitors
**Demo / Product Tour**
- Interactive demos
- Video walkthroughs
- GIF previews
- Shows product in action
**Pricing Preview**
- Teaser even on non-pricing pages
- Starting price or "from $X/mo"
- Moves decision-makers forward
**Guarantee / Risk Reversal**
- Money-back guarantee
- Free trial terms
- "Cancel anytime"
- Reduces friction
**Stats Section**
- Key metrics that build credibility
- "10,000+ customers"
- "4.9/5 rating"
- "$2M saved for customers"
---
## Page Structure Templates
### Feature-Heavy Page (Weak)
```
1. Hero
2. Feature 1
3. Feature 2
4. Feature 3
5. Feature 4
6. CTA
```
This is a list, not a persuasive narrative.
---
### Varied, Engaging Page (Strong)
```
1. Hero with clear value prop
2. Social proof bar (logos or stats)
3. Problem/pain section
4. How it works (3 steps)
5. Key benefits (2-3, not 10)
6. Testimonial
7. Use cases or personas
8. Comparison to alternatives
9. Case study snippet
10. FAQ
11. Final CTA with guarantee
```
This tells a story and addresses objections.
---
### Compact Landing Page
```
1. Hero (headline, subhead, CTA, image)
2. Social proof bar
3. 3 key benefits with icons
4. Testimonial
5. How it works (3 steps)
6. Final CTA with guarantee
```
Good for ad landing pages where brevity matters.
---
### Enterprise/B2B Landing Page
```
1. Hero (outcome-focused headline)
2. Logo bar (recognizable companies)
3. Problem section (business pain)
4. Solution overview
5. Use cases by role/department
6. Security/compliance section
7. Integration logos
8. Case study with metrics
9. ROI/value section
10. Contact/demo CTA
```
Addresses enterprise buyer concerns.
---
### Product Launch Page
```
1. Hero with launch announcement
2. Video demo or walkthrough
3. Feature highlights (3-5)
4. Before/after comparison
5. Early testimonials
6. Launch pricing or early access offer
7. CTA with urgency
```
Good for ProductHunt, launches, or announcements.
---
## Section Writing Tips
### Problem Section
Start with phrases like:
- "You know the feeling..."
- "If you're like most [role]..."
- "Every day, [audience] struggles with..."
- "We've all been there..."
Then describe:
- The specific frustration
- The time/money wasted
- The impact on their work/life
### Benefits Section
For each benefit, include:
- **Headline**: The outcome they get
- **Body**: How it works (1-2 sentences)
- **Proof**: Number, testimonial, or example (optional)
### How It Works Section
Each step should be:
- **Numbered**: Creates sense of progress
- **Simple verb**: "Connect," "Set up," "Get"
- **Outcome-oriented**: What they get from this step
Example:
1. Connect your tools (takes 2 minutes)
2. Set your preferences
3. Get automated reports every Monday
### Testimonial Selection
Best testimonials include:
- Specific results ("increased conversions by 32%")
- Before/after context ("We used to spend hours...")
- Role + company for credibility
- Something quotable and specific
Avoid testimonials that just say:
- "Great product!"
- "Love it!"
- "Easy to use!"
---
## Clarity & Message-Market Fit
Headline formulas give you the shape of a line. These tools tell you whether the line is actually *working* — whether it's clear, whether it maps to how the reader already thinks, and whether it lands with the right person. Positioning is the prologue to your novel: it sets up everything that follows. Get it clear and the rest of the page writes itself.
### The "Now you can" Test
A fast gut-check for any headline or benefit line. Mentally prefix it with **"Now you can…"**. If the result is both **compelling** and **true**, the line is doing its job. If it reads as vague, obvious, or a stretch, rewrite it.
The test works because "Now you can…" forces the copy into the reader's world — it has to name a concrete new ability they didn't have before. Feature-speak and buzzwords collapse under it.
| Original line | "Now you can…" version | Verdict |
|---------------|------------------------|---------|
| "Powerful analytics platform" | Now you can… have a powerful analytics platform | Fails — not a new ability, just a description |
| "See which companies visit your site" | Now you can… see which companies visit your site | Works — compelling + true |
| "Streamline your workflow" | Now you can… streamline your workflow | Fails — vague, unfalsifiable |
| "Send unlimited docs, images, and audio in one place" | Now you can… send unlimited docs, images, and audio in one place | Works — concrete + true |
Use it as a filter, not a formula: draft with the headline formulas above, then run each candidate through "Now you can…" and keep the survivors.
### The Human Action Model (landing-page narrative spine)
Ludwig von Mises' Human Action Model explains *why* anyone acts: a person acts only when three things line up. Every above-the-fold that converts follows the same three-beat spine:
1. **Current discomfort** — the felt problem, named in the reader's own words. They have to recognize their situation ("that's exactly me").
2. **Better vision** — a clearly imagined, more satisfying state. What life looks like once the discomfort is gone.
3. **Path to action** — the belief that *this specific step* closes the gap between the two. The product is the bridge, and the CTA is how they cross it.
Miss any beat and the reader stalls. No discomfort = no reason to move. No vision = no destination. No path = no reason to believe *you're* the way there.
**Mapping it onto the hero:**
| Beat | Where it usually lives | Example |
|------|------------------------|---------|
| Current discomfort | Eyebrow, subhead, or problem-framed headline | "You shouldn't have to feel awkward sending out your scheduling link" |
| Better vision | Headline or subhead | "Scheduling that feels considerate, not one-sided" |
| Path to action | CTA + supporting proof | "Start scheduling free" |
This is the transformation spine underneath the "6 essential sections" of a landing page — hero, social proof, problem, solution, how-it-works, and final CTA. The hero states the transformation; the rest of the page substantiates each beat.
### The Perception Gap
The same benefit can read as a **selling point to one segment and a red flag to another**. The gap is between what *you* think you're saying and what a given reader hears through their own risk tolerance.
The fix isn't softer copy — it's **matching the value prop to the reader's risk tolerance**. Segment first, then swap the framing.
| Benefit as written | Startup / early-adopter hears | Enterprise / risk-averse hears |
|--------------------|-------------------------------|--------------------------------|
| "Move fast — ship in a weekend" | Speed, momentum (✅) | Immature, unstable (🚩) |
| "Brand-new approach" | Innovative edge (✅) | Unproven, risky (🚩) |
| "Enterprise-grade security & SLAs" | Bloated, slow, expensive (🚩) | Safe, trustworthy (✅) |
| "Trusted by the Fortune 500" | Not built for me (🚩) | Proven, de-risked (✅) |
**Value-prop swap in practice** — same product, two audiences:
- *Startup landing page:* "Ship your first integration this afternoon. No sales calls, no procurement."
- *Enterprise landing page:* "SOC 2 Type II, 99.99% uptime SLA, and a named implementation lead. Roll out with confidence."
When a page has to serve both, don't average them into mush — segment the traffic (separate pages, or a persona split) and let each read its own version of the truth.
### Worked Example — SavvyCal (message-market fit)
SavvyCal (a scheduling tool) originally led with feature-forward copy. They rewrote the hero around a single felt discomfort:
> **"You shouldn't have to feel awkward sending out your scheduling link."**
That one line **roughly tripled (3×) conversions**. It works because it hits all three beats of the Human Action Model at once:
- **Discomfort:** the small social awkwardness of "here's my link, pick a time" — named exactly as users feel it.
- **Vision:** scheduling that feels considerate to *both* people.
- **Path:** SavvyCal's overlay-your-calendar mechanic is the bridge, so the CTA feels like the obvious next step.
The lesson: message-market fit beats feature lists. The winning line wasn't cleverer — it named a real feeling the reader hadn't heard a scheduling tool acknowledge before. Run your own hero through "Now you can…" and the Human Action Model to find that line.
### Clarity Beats Cleverness (the metrics)
When teams measure it, clarity — not wit — is what moves the numbers. Clearer positioning and copy is associated with:
- **+81% conversions**
- **−38% sales cycle** (shorter time to close)
- **−28% CAC** (lower customer acquisition cost)
- **+175% referrals**
The mechanism: clear copy lets the *right* buyer self-qualify fast and the wrong one bounce early, so every downstream metric improves. Clever copy that requires decoding does the opposite — it adds a comprehension tax at the exact moment attention is scarcest.
**Practical rule:** if a reader has to pause to figure out what you mean, you've already lost. When forced to choose between a clever line and a clear one, ship the clear one — then use the tests above ("Now you can…", the Human Action Model, the Perception Gap) to make the clear line compelling too.
FILE:references/natural-transitions.md
# Natural Transitions
Transitional phrases to guide readers through your content. Good signposting improves readability, user engagement, and helps search engines understand content structure.
Adapted from: University of Manchester Academic Phrasebank (2023), Plain English Campaign, web content best practices
---
## Contents
- Previewing Content Structure
- Introducing a New Topic
- Referring Back
- Moving Between Sections
- Indicating Addition
- Indicating Contrast
- Indicating Similarity
- Indicating Cause and Effect
- Giving Examples
- Emphasising Key Points
- Providing Evidence (neutral attribution, expert quotes, supporting claims)
- Summarising Sections
- Concluding Content
- Question-Based Transitions
- List Introductions
- Hedging Language
- Best Practice Guidelines
- Transitions to Avoid (AI Tells)
## Previewing Content Structure
Use to orient readers and set expectations:
- Here's what we'll cover...
- This guide walks you through...
- Below, you'll find...
- We'll start with X, then move to Y...
- First, let's look at...
- Let's break this down step by step.
- The sections below explain...
---
## Introducing a New Topic
- When it comes to X,...
- Regarding X,...
- Speaking of X,...
- Now let's talk about X.
- Another key factor is...
- X is worth exploring because...
---
## Referring Back
Use to connect ideas and reinforce key points:
- As mentioned earlier,...
- As we covered above,...
- Remember when we discussed X?
- Building on that point,...
- Going back to X,...
- Earlier, we explained that...
---
## Moving Between Sections
- Now let's look at...
- Next up:...
- Moving on to...
- With that covered, let's turn to...
- Now that you understand X, here's Y.
- That brings us to...
---
## Indicating Addition
- Also,...
- Plus,...
- On top of that,...
- What's more,...
- Another benefit is...
- Beyond that,...
- In addition,...
- There's also...
**Note:** Use "moreover" and "furthermore" sparingly. They can sound AI-generated when overused.
---
## Indicating Contrast
- However,...
- But,...
- That said,...
- On the flip side,...
- In contrast,...
- Unlike X, Y...
- While X is true, Y...
- Despite this,...
---
## Indicating Similarity
- Similarly,...
- Likewise,...
- In the same way,...
- Just like X, Y also...
- This mirrors...
- The same applies to...
---
## Indicating Cause and Effect
- So,...
- This means...
- As a result,...
- That's why...
- Because of this,...
- This leads to...
- The outcome?...
- Here's what happens:...
---
## Giving Examples
- For example,...
- For instance,...
- Here's an example:...
- Take X, for instance.
- Consider this:...
- A good example is...
- To illustrate,...
- Like when...
- Say you want to...
---
## Emphasising Key Points
- Here's the key takeaway:...
- The important thing is...
- What matters most is...
- Don't miss this:...
- Pay attention to...
- This is critical:...
- The bottom line?...
---
## Providing Evidence
Use when citing sources, data, or expert opinions:
### Neutral attribution
- According to [Source],...
- [Source] reports that...
- Research shows that...
- Data from [Source] indicates...
- A study by [Source] found...
### Expert quotes
- As [Expert] puts it,...
- [Expert] explains,...
- In the words of [Expert],...
- [Expert] notes that...
### Supporting claims
- This is backed by...
- Evidence suggests...
- The numbers confirm...
- This aligns with findings from...
---
## Summarising Sections
- To recap,...
- Here's the short version:...
- In short,...
- The takeaway?...
- So what does this mean?...
- Let's pull this together:...
- Quick summary:...
---
## Concluding Content
- Wrapping up,...
- The bottom line is...
- Here's what to do next:...
- To sum up,...
- Final thoughts:...
- Ready to get started?...
- Now it's your turn.
**Note:** Avoid "In conclusion" at the start of a paragraph. It's overused and signals AI writing.
---
## Question-Based Transitions
Useful for conversational tone and featured snippet optimization:
- So what does this mean for you?
- But why does this matter?
- How do you actually do this?
- What's the catch?
- Sound complicated? It's not.
- Wondering where to start?
- Still not sure? Here's the breakdown.
---
## List Introductions
For numbered lists and step-by-step content:
- Here's how to do it:
- Follow these steps:
- The process is straightforward:
- Here's what you need to know:
- Key things to consider:
- The main factors are:
---
## Hedging Language
For claims that need qualification or aren't absolute:
- may, might, could
- tends to, generally
- often, usually, typically
- in most cases
- it appears that
- evidence suggests
- this can help
- many experts believe
---
## Best Practice Guidelines
1. **Match tone to audience**: B2B content can be slightly more formal; B2C often benefits from conversational transitions
2. **Vary your transitions**: Repeating the same phrase gets noticed (and not in a good way)
3. **Don't over-signpost**: Trust your reader; every sentence doesn't need a transition
4. **Use for scannability**: Transitions at paragraph starts help skimmers navigate
5. **Keep it natural**: Read aloud; if it sounds forced, simplify
6. **Front-load key info**: Put the important word or phrase early in the transition
---
## Transitions to Avoid (AI Tells)
These phrases are overused in AI-generated content:
- "That being said,..."
- "It's worth noting that..."
- "At its core,..."
- "In today's digital landscape,..."
- "When it comes to the realm of..."
- "This begs the question..."
- "Let's delve into..."
See the seo-audit skill's `references/ai-writing-detection.md` for a complete list of AI writing tells.
Phân tích sản phẩm đối thủ từ trang giá, đánh giá ứng dụng, tin tuyển dụng, SEO, tạo ma trận tính năng và SWOT.
---
name: "competitive-teardown"
description: "Analyzes competitor products and companies by synthesizing data from pricing pages, app store reviews, job postings, SEO signals, and social media into structured competitive intelligence. Produces feature comparison matrices scored across 12 dimensions, SWOT analyses, positioning maps, UX audits, pricing model breakdowns, action item roadmaps, and stakeholder presentation templates. Use when conducting competitor analysis, comparing products against competitors, researching the competitive landscape, building battle cards for sales, preparing for a product strategy or roadmap session, responding to a competitor's new feature or pricing change, or performing a quarterly competitive review."
---
# Competitive Teardown
**Tier:** POWERFUL
**Category:** Product Team
**Domain:** Competitive Intelligence, Product Strategy, Market Analysis
---
## When to Use
- Before a product strategy or roadmap session
- When a competitor launches a major feature or pricing change
- Quarterly competitive review
- Before a sales pitch where you need battle card data
- When entering a new market segment
---
## Teardown Workflow
Follow these steps in sequence to produce a complete teardown:
1. **Define competitors** — List 2–4 competitors to analyze. Confirm which is the primary focus.
2. **Collect data** — Use `references/data-collection-guide.md` to gather raw signals from at least 3 sources per competitor (website, reviews, job postings, SEO, social).
_Validation checkpoint: Before proceeding, confirm you have pricing data, at least 20 reviews, and job posting counts for each competitor._
3. **Score using rubric** — Apply the 12-dimension rubric below to produce a numeric scorecard for each competitor and your own product.
_Validation checkpoint: Every dimension should have a score and at least one supporting evidence note._
4. **Generate outputs** — Populate the templates in `references/analysis-templates.md` (Feature Matrix, Pricing Analysis, SWOT, Positioning Map, UX Audit).
5. **Build action plan** — Translate findings into the Action Items template (quick wins / medium-term / strategic).
6. **Package for stakeholders** — Assemble the Stakeholder Presentation using outputs from steps 3–5.
---
## Data Collection Guide
> Full executable scripts for each source are in `references/data-collection-guide.md`. Summaries of what to capture are below.
### 1. Website Analysis
Key things to capture:
- Pricing tiers and price points
- Feature lists per tier
- Primary CTA and messaging
- Case studies / customer logos (signals ICP)
- Integration logos
- Trust signals (certifications, compliance badges)
### 2. App Store Reviews
Review sentiment categories:
- **Praise** → what users love (defend / strengthen these)
- **Feature requests** → unmet needs (opportunity gaps)
- **Bugs** → quality signals
- **UX complaints** → friction points you can beat them on
**Sample App Store query (iTunes Search API):**
```
GET https://itunes.apple.com/search?term=<competitor_name>&entity=software&limit=1
# Extract trackId, then:
GET https://itunes.apple.com/rss/customerreviews/id=<trackId>/sortBy=mostRecent/json?l=en&limit=50
```
Parse `entry[].content.label` for review text and `entry[].im:rating.label` for star rating.
### 3. Job Postings (Team Size & Tech Stack Signals)
Signals from job postings:
- **Engineering volume** → scaling vs. consolidating
- **Specific tech mentions** → stack (React/Vue, Postgres/Mongo, AWS/GCP)
- **Sales/CS ratio** → product-led vs. sales-led motion
- **Data/ML roles** → upcoming AI features
- **Compliance roles** → regulatory expansion
### 4. SEO Analysis
SEO signals to capture:
- Top 20 organic keywords (intent: informational / navigational / commercial)
- Domain Authority / backlink count
- Blog publishing cadence and topics
- Which pages rank (product pages vs. blog vs. docs)
### 5. Social Media Sentiment
Capture recent mentions via Twitter/X API v2, Reddit, or LinkedIn. Look for recurring praise, complaints, and feature requests. See `references/data-collection-guide.md` for API query examples.
---
## Scoring Rubric (12 Dimensions, 1-5)
| # | Dimension | 1 (Weak) | 3 (Average) | 5 (Best-in-class) |
|---|-----------|----------|-------------|-------------------|
| 1 | **Features** | Core only, many gaps | Solid coverage | Comprehensive + unique |
| 2 | **Pricing** | Confusing / overpriced | Market-rate, clear | Transparent, flexible, fair |
| 3 | **UX** | Confusing, high friction | Functional | Delightful, minimal friction |
| 4 | **Performance** | Slow, unreliable | Acceptable | Fast, high uptime |
| 5 | **Docs** | Sparse, outdated | Decent coverage | Comprehensive, searchable |
| 6 | **Support** | Email only, slow | Chat + email | 24/7, great response |
| 7 | **Integrations** | 0-5 integrations | 6-25 | 26+ or deep ecosystem |
| 8 | **Security** | No mentions | SOC2 claimed | SOC2 Type II, ISO 27001 |
| 9 | **Scalability** | No enterprise tier | Mid-market ready | Enterprise-grade |
| 10 | **Brand** | Generic, unmemorable | Decent positioning | Strong, differentiated |
| 11 | **Community** | None | Forum / Slack | Active, vibrant community |
| 12 | **Innovation** | No recent releases | Quarterly | Frequent, meaningful |
**Example completed row** (Competitor: Acme Corp, Dimension 3 – UX):
| Dimension | Acme Corp Score | Evidence |
|-----------|----------------|---------|
| UX | 2 | App Store reviews cite "confusing navigation" (38 mentions); onboarding requires 7 steps before TTFV; no onboarding wizard; CC required at signup. |
Apply this pattern to all 12 dimensions for each competitor.
---
## Templates
> Full template markdown is in `references/analysis-templates.md`. Abbreviated reference below.
### Feature Comparison Matrix
Rows: core features, pricing tiers, platform capabilities (web, iOS, Android, API).
Columns: your product + up to 3 competitors.
Score each cell 1–5. Sum to get total out of 60.
**Score legend:** 5=Best-in-class, 4=Strong, 3=Average, 2=Below average, 1=Weak/Missing
### Pricing Analysis
Capture per competitor: model type (per-seat / usage-based / flat rate / freemium), entry/mid/enterprise price points, free trial length.
Summarize: price leader, value leader, premium positioning, your position, and 2–3 pricing opportunity bullets.
### SWOT Analysis
For each competitor: 3–5 bullets per quadrant (Strengths, Weaknesses, Opportunities for us, Threats to us). Anchor every bullet to a data signal (review quote, job posting count, pricing page, etc.).
### Positioning Map
2x2 axes (e.g., Simple ↔ Complex / Low Value ↔ High Value). Place each competitor and your product. Bubble size = market share or funding. See `references/analysis-templates.md` for ASCII and editable versions.
### UX Audit Checklist
Onboarding: TTFV (minutes), steps to activation, CC-required, onboarding wizard quality.
Key workflows: steps, friction points, comparative score (yours vs. theirs).
Mobile: iOS/Android ratings, feature parity, top complaint and praise.
Navigation: global search, keyboard shortcuts, in-app help.
### Action Items
| Horizon | Effort | Examples |
|---------|--------|---------|
| Quick wins (0–4 wks) | Low | Add review badges, publish comparison landing page |
| Medium-term (1–3 mo) | Moderate | Launch free tier, improve onboarding TTFV, add top-requested integration |
| Strategic (3–12 mo) | High | Enter new market, build API v2, achieve SOC2 Type II |
### Stakeholder Presentation (7 slides)
1. **Executive Summary** — Threat level (LOW/MEDIUM/HIGH/CRITICAL), top strength, top opportunity, recommended action
2. **Market Position** — 2x2 positioning map
3. **Feature Scorecard** — 12-dimension radar or table, total scores
4. **Pricing Analysis** — Comparison table + key insight
5. **UX Highlights** — What they do better (3 bullets) vs. where we win (3 bullets)
6. **Voice of Customer** — Top 3 review complaints (quoted or paraphrased)
7. **Our Action Plan** — Quick wins, medium-term, strategic priorities; Appendix with raw data
## Related Skills
- **Product Strategist** (`product-team/product-strategist/`) — Competitive insights feed OKR and strategy planning
- **Landing Page Generator** (`product-team/landing-page-generator/`) — Competitive positioning informs landing page messaging
FILE:references/analysis-templates.md
# Competitive Analysis Templates
## 1. SWOT Analysis Template
### Company/Product: [Competitor Name]
**Date:** [Analysis Date] | **Analyst:** [Name] | **Version:** [1.0]
#### Strengths (Internal Advantages)
| # | Strength | Evidence | Impact |
|---|----------|----------|--------|
| 1 | [e.g., Strong brand recognition] | [Source/data point] | High/Med/Low |
| 2 | | | |
| 3 | | | |
#### Weaknesses (Internal Limitations)
| # | Weakness | Evidence | Exploitability |
|---|----------|----------|---------------|
| 1 | [e.g., Limited API capabilities] | [Source/data point] | High/Med/Low |
| 2 | | | |
| 3 | | | |
#### Opportunities (External Favorable)
| # | Opportunity | Timeframe | Our Advantage |
|---|------------|-----------|---------------|
| 1 | [e.g., Competitor slow to adopt AI] | Short/Med/Long | [How we capitalize] |
| 2 | | | |
| 3 | | | |
#### Threats (External Unfavorable)
| # | Threat | Likelihood | Mitigation |
|---|--------|-----------|-----------|
| 1 | [e.g., Competitor acquired by larger company] | High/Med/Low | [Our response plan] |
| 2 | | | |
| 3 | | | |
---
## 2. Porter's Five Forces (Product Application)
### Market: [Your Product Category]
#### Force 1: Competitive Rivalry (Intensity: High/Med/Low)
- Number of direct competitors: ___
- Market growth rate: ___% annually
- Product differentiation level: High/Med/Low
- Switching costs for customers: High/Med/Low
- Exit barriers: High/Med/Low
- **Assessment:** [Summary of competitive rivalry intensity]
#### Force 2: Threat of New Entrants (Intensity: High/Med/Low)
- Capital requirements: High/Med/Low
- Technology barriers: High/Med/Low
- Network effects strength: Strong/Moderate/Weak
- Regulatory barriers: High/Med/Low
- Brand loyalty in market: Strong/Moderate/Weak
- **Assessment:** [Summary of new entrant threat]
#### Force 3: Threat of Substitutes (Intensity: High/Med/Low)
- Alternative solutions: [List substitutes]
- Price-performance of substitutes: Better/Same/Worse
- Switching costs to substitutes: High/Med/Low
- Customer propensity to switch: High/Med/Low
- **Assessment:** [Summary of substitute threat]
#### Force 4: Bargaining Power of Buyers (Power: High/Med/Low)
- Buyer concentration: Concentrated/Fragmented
- Price sensitivity: High/Med/Low
- Information availability: Full/Partial/Limited
- Switching costs: High/Med/Low
- Volume of purchases: High/Med/Low
- **Assessment:** [Summary of buyer power]
#### Force 5: Bargaining Power of Suppliers (Power: High/Med/Low)
- Key technology dependencies: [List]
- Cloud provider lock-in: High/Med/Low
- Talent market tightness: Tight/Balanced/Loose
- Data source dependencies: Critical/Important/Optional
- **Assessment:** [Summary of supplier power]
#### Overall Industry Attractiveness: [Score 1-10]
---
## 3. Competitive Positioning Map
### Axis Definitions
- **X-Axis:** [e.g., Ease of Use] (Low to High)
- **Y-Axis:** [e.g., Feature Completeness] (Low to High)
### Competitor Positions
| Competitor | X Score (1-10) | Y Score (1-10) | Quadrant |
|-----------|---------------|---------------|----------|
| Your Product | ___ | ___ | ___ |
| Competitor A | ___ | ___ | ___ |
| Competitor B | ___ | ___ | ___ |
| Competitor C | ___ | ___ | ___ |
| Competitor D | ___ | ___ | ___ |
### Quadrant Definitions
- **Top-Right (Leaders):** High on both axes - market leaders
- **Top-Left (Feature-Rich):** High features, lower ease of use - complex tools
- **Bottom-Right (Simple):** Easy to use, fewer features - niche players
- **Bottom-Left (Laggards):** Low on both axes - disruption candidates
### Positioning Insights
- **White space opportunities:** [Areas with no competitor presence]
- **Crowded areas:** [Where competition is fiercest]
- **Our trajectory:** [Direction we're moving on the map]
---
## 4. Win/Loss Analysis Template
### Deal: [Opportunity Name]
**Date:** [Close Date] | **Result:** Won / Lost | **Competitor:** [Name]
#### Deal Context
- **Deal Size:** $___
- **Sales Cycle:** ___ days
- **Segment:** SMB / Mid-Market / Enterprise
- **Industry:** ___
- **Decision Makers:** [Roles involved]
- **Evaluation Criteria:** [What mattered most to buyer]
#### Competitive Comparison (Buyer Perspective)
| Factor | Us (Score 1-5) | Competitor (Score 1-5) | Decisive? |
|--------|---------------|----------------------|-----------|
| Product Fit | | | Yes/No |
| Pricing | | | Yes/No |
| Ease of Use | | | Yes/No |
| Support Quality | | | Yes/No |
| Integration | | | Yes/No |
| Brand/Trust | | | Yes/No |
| Implementation | | | Yes/No |
#### Win/Loss Factors
- **Primary reason for outcome:** [Single most important factor]
- **Secondary factors:** [Supporting reasons]
- **Buyer quotes:** ["Direct quotes from debrief"]
#### Action Items
| # | Action | Owner | Due Date |
|---|--------|-------|----------|
| 1 | [e.g., Improve onboarding flow] | [Name] | [Date] |
| 2 | | | |
---
## 5. Battle Card Template
### Competitor: [Name]
**Last Updated:** [Date] | **Confidence:** High/Med/Low
#### Quick Facts
- **Founded:** ___
- **Funding:** $___
- **Employees:** ___
- **Customers:** ___
- **HQ:** ___
#### Elevator Pitch (Their Positioning)
> [How the competitor describes themselves in one sentence]
#### Our Positioning Against Them
> [How we differentiate - our one-liner against this competitor]
#### Where They Win
| Strength | Our Counter |
|----------|------------|
| [e.g., Lower price point] | [e.g., Emphasize TCO including implementation costs] |
| [e.g., Larger integration marketplace] | [e.g., Highlight quality over quantity, key integrations] |
| | |
#### Where We Win
| Our Strength | Evidence |
|-------------|----------|
| [e.g., Superior onboarding experience] | [Metric or customer quote] |
| [e.g., Better enterprise security] | [Certification or feature] |
| | |
#### Landmines to Set
Questions to ask prospects that expose competitor weaknesses:
1. "Have you evaluated how [specific capability] scales beyond [threshold]?"
2. "What's their approach to [area where competitor is weak]?"
3. "Can you share their uptime SLA and historical performance?"
#### Objection Handling
| Objection | Response |
|-----------|----------|
| "[Competitor] is cheaper" | [Value-based response] |
| "[Competitor] has more features" | [Quality/relevance response] |
| "We already use [Competitor]" | [Migration/coexistence story] |
#### Trap Questions They Set
Questions competitors ask about us, and how to respond:
1. **Q:** "[Our known weakness]?" **A:** [Honest, redirect response]
2. **Q:** "[Feature gap]?" **A:** [Roadmap or alternative approach]
#### Recent Intel
- [Date]: [Notable change - pricing, feature, hire, funding]
- [Date]: [Notable change]
FILE:references/competitive-analysis-frameworks.md
# Competitive Analysis Frameworks
This reference provides practical frameworks for evaluating competitors and positioning decisions.
## Porter's Five Forces
Assess the competitive intensity of your market:
1. Threat of new entrants
- Barriers to entry (capital, regulation, network effects)
- Speed of competitor replication
2. Bargaining power of suppliers
- Dependency on core infrastructure vendors
- Concentration of key technical providers
3. Bargaining power of buyers
- Customer switching costs
- Procurement complexity and contract leverage
4. Threat of substitutes
- Adjacent alternatives solving the same job
- DIY and internal build options
5. Rivalry among existing competitors
- Number and similarity of competitors
- Price competition and differentiation pressure
### Five Forces Template
| Force | Current Pressure (Low/Med/High) | Evidence | Strategic Response |
|---|---|---|---|
| New Entrants | | | |
| Supplier Power | | | |
| Buyer Power | | | |
| Substitutes | | | |
| Rivalry | | | |
## SWOT Analysis
Use SWOT to map internal and external context quickly.
### SWOT Template
| Strengths (Internal) | Weaknesses (Internal) |
|---|---|
| What we do better than alternatives | Where competitors outperform us |
| Unique capabilities or assets | Known product or go-to-market gaps |
| Opportunities (External) | Threats (External) |
|---|---|
| Market trends we can exploit | Competitor moves or macro risks |
| Unserved segments and use cases | Regulatory, platform, or pricing pressure |
### SWOT Quality Checklist
- Base every point on evidence, not assumptions.
- Separate observations from conclusions.
- Prioritize top 3 items per quadrant.
## Feature Comparison Matrix
Compare products on meaningful buying criteria, not vanity features.
### Feature Matrix Template
| Dimension | Weight | Your Product | Competitor A | Competitor B | Notes |
|---|---:|---:|---:|---:|---|
| Core workflow coverage | 25% | | | | |
| Ease of implementation | 15% | | | | |
| Performance / reliability | 15% | | | | |
| Integrations / ecosystem | 15% | | | | |
| Security / compliance | 15% | | | | |
| Pricing / TCO | 15% | | | | |
Scoring scale recommendation: 1-5 (weak to strong).
## Competitive Positioning Map
Create a 2-axis map showing market whitespace and crowding.
### Positioning Map Steps
1. Select two high-signal dimensions customers care about.
2. Place each competitor based on evidence (pricing pages, reviews, demos).
3. Mark clusters where products are undifferentiated.
4. Identify white space where demand exists but options are weak.
Example axes:
- X-axis: Ease of use
- Y-axis: Enterprise readiness
## Blue Ocean Strategy Canvas
Use a strategy canvas to decide where to raise, reduce, eliminate, or create factors.
### ERRC Grid (Eliminate-Reduce-Raise-Create)
| Eliminate | Reduce | Raise | Create |
|---|---|---|---|
| Commodity table-stakes not valued by target users | Costly features with weak adoption | Differentiators tied to target job-to-be-done | New value dimensions competitors ignore |
### Strategy Canvas Checklist
- Compare value curves between your product and top competitors.
- Ensure target segment is explicit.
- Tie every strategic choice to measurable outcome.
FILE:references/data-collection-guide.md
# Competitive Data Collection Guide
## Overview
This guide outlines systematic approaches for gathering competitive intelligence from publicly available sources. All methods described here are ethical and rely on information that competitors have made publicly accessible.
## Public Data Sources
### Review Platforms
- **G2**: Enterprise software reviews, feature comparisons, satisfaction scores
- **Capterra**: SMB-focused reviews, pricing transparency, deployment details
- **TrustRadius**: In-depth reviews with verified users, TrustMaps
- **Product Hunt**: Launch positioning, early adopter sentiment, feature highlights
- **App Store / Google Play**: Mobile app ratings, review themes, update frequency
### Company Publications
- **Pricing Pages**: Tier structure, feature gating, enterprise vs self-serve
- **Changelogs / Release Notes**: Development velocity, feature priorities, tech direction
- **Blog Posts**: Strategic messaging, thought leadership topics, market positioning
- **Case Studies**: Target customer profiles, value propositions, success metrics
- **Help Documentation**: Feature depth, API capabilities, integration ecosystem
### Talent & Organization Signals
- **Job Postings**: Technology stack, team growth areas, strategic initiatives
- **LinkedIn**: Team size, org structure, key hires, department ratios
- **Glassdoor**: Company culture, internal challenges, growth trajectory
### Financial & Legal
- **Patent Filings**: Innovation direction, defensive IP, technology differentiation
- **SEC Filings (public companies)**: Revenue, growth rate, customer count, churn
- **Crunchbase / PitchBook**: Funding rounds, investors, valuation trends
### Technical Intelligence
- **BuiltWith / Wappalyzer**: Technology stack detection
- **GitHub**: Open-source contributions, SDK quality, developer engagement
- **API Documentation**: Integration capabilities, rate limits, data models
- **Status Pages**: Uptime history, incident frequency, infrastructure maturity
## Data Points to Collect Per Competitor
### Product
- Core features and capabilities (feature-by-feature matrix)
- Unique differentiators and proprietary technology
- Platform support (web, mobile, desktop, API)
- Integration ecosystem (number and quality of integrations)
- Performance benchmarks (if available from reviews)
### Business
- Pricing tiers and per-seat/usage costs
- Target customer segments (SMB, mid-market, enterprise)
- Estimated customer count and notable logos
- Geographic focus and localization
- Go-to-market model (PLG, sales-led, hybrid)
### Team & Technology
- Estimated team size and engineering ratio
- Technology stack and infrastructure choices
- Development velocity (release frequency)
- Open-source involvement and developer relations
### Market Position
- Market share estimates
- Brand perception and NPS (from reviews)
- Analyst coverage (Gartner, Forrester positioning)
- Partnership and channel strategy
## Ethical Guidelines
1. **Use only public information** - Never access private systems, NDA-protected content, or internal documents
2. **No deception** - Do not misrepresent yourself to obtain information (e.g., fake sales inquiries)
3. **Respect terms of service** - Follow scraping policies and API usage terms
4. **Attribute sources** - Document where each data point came from for verification
5. **No employee poaching for intelligence** - Hiring decisions should be talent-driven, not intelligence-driven
6. **Legal compliance** - Ensure data collection complies with local regulations
## Update Cadence Recommendations
| Data Type | Frequency | Trigger Events |
|-----------|-----------|---------------|
| Pricing | Monthly | Competitor pricing page changes |
| Features | Bi-weekly | Changelog updates, product launches |
| Reviews | Monthly | Batch review analysis |
| Job Postings | Monthly | Hiring surge detection |
| Financials | Quarterly | Earnings reports, funding rounds |
| Tech Stack | Quarterly | Major platform changes |
| Full Teardown | Quarterly | Strategic planning cycles |
## Collection Workflow
1. **Set up monitoring** - Google Alerts, competitor RSS feeds, social listening
2. **Schedule regular sweeps** - Calendar recurring data collection tasks
3. **Centralize data** - Use a shared competitive intelligence database or spreadsheet
4. **Validate findings** - Cross-reference multiple sources for accuracy
5. **Tag and categorize** - Apply consistent taxonomy for easy retrieval
6. **Share insights** - Distribute relevant findings to product, sales, and marketing teams
7. **Archive versions** - Maintain historical snapshots for trend analysis
## Tools for Automation
- **Google Alerts**: Free monitoring for competitor mentions
- **Visualping**: Website change detection (pricing pages, feature pages)
- **Feedly**: RSS aggregation for competitor blogs and news
- **SimilarWeb**: Traffic estimates and audience overlap
- **SEMrush / Ahrefs**: SEO positioning and content strategy analysis
FILE:references/scoring-rubric.md
# Competitive Scoring Rubric
## Overview
This rubric provides a standardized framework for evaluating competitors across key dimensions. Consistent scoring enables meaningful comparisons and tracks competitive position changes over time.
## Scoring Scale (1-10)
| Score | Label | Definition |
|-------|-------|-----------|
| 1-2 | Poor | Significant gaps, major usability issues, or missing capability |
| 3-4 | Below Average | Basic functionality with notable limitations |
| 5-6 | Average | Meets market expectations, no standout qualities |
| 7-8 | Above Average | Strong execution with clear advantages |
| 9-10 | Exceptional | Industry-leading, sets the standard for others |
## Dimension Categories
### 1. User Experience (UX) - Weight: 20%
- **Onboarding**: Time to first value, setup complexity, guided flows
- **Navigation**: Information architecture, discoverability, consistency
- **Visual Design**: Modern aesthetics, brand coherence, accessibility
- **Performance**: Page load times, responsiveness, offline capability
- **Mobile Experience**: Native app quality, responsive design, feature parity
### 2. Feature Completeness - Weight: 25%
- **Core Features**: Coverage of essential use cases
- **Advanced Features**: Power user capabilities, automation, customization
- **Workflow Support**: End-to-end process coverage without workarounds
- **API & Extensibility**: API coverage, webhook support, SDK quality
- **Innovation**: Unique capabilities not found in competitors
### 3. Pricing & Value - Weight: 15%
- **Transparency**: Clear pricing without hidden costs
- **Flexibility**: Plan options matching different customer sizes
- **Value-to-Cost Ratio**: Feature access relative to price point
- **Free Tier / Trial**: Quality of free offering for evaluation
- **Contract Terms**: Lock-in requirements, cancellation ease
### 4. Integrations - Weight: 10%
- **Native Integrations**: Number and quality of built-in connectors
- **Marketplace**: Third-party app ecosystem breadth
- **API Quality**: Documentation, reliability, rate limits
- **Data Import/Export**: Migration ease, format support
- **Workflow Automation**: Zapier, Make, native automation support
### 5. Support & Documentation - Weight: 10%
- **Documentation Quality**: Completeness, searchability, freshness
- **Support Channels**: Chat, email, phone, community availability
- **Response Time**: SLA adherence, resolution speed
- **Self-Service**: Knowledge base, video tutorials, community forums
- **Onboarding Support**: Dedicated CSM, implementation assistance
### 6. Performance & Reliability - Weight: 10%
- **Uptime**: Historical availability, SLA commitments
- **Speed**: Application responsiveness under normal load
- **Scalability**: Performance at high volume, enterprise readiness
- **Data Handling**: Large dataset support, bulk operations
- **Global Performance**: CDN, regional deployments, latency
### 7. Security & Compliance - Weight: 10%
- **Authentication**: SSO, MFA, RBAC granularity
- **Data Protection**: Encryption at rest and in transit, data residency
- **Certifications**: SOC 2, ISO 27001, GDPR, HIPAA compliance
- **Audit Trail**: Activity logging, access monitoring
- **Privacy Controls**: Data retention policies, right to deletion
## Weighting Guidelines
Default weights above suit most B2B SaaS evaluations. Adjust based on:
- **Enterprise buyers**: Increase Security (15%), Support (15%), reduce Pricing (10%)
- **Developer tools**: Increase Integrations (20%), Features (30%), reduce UX (10%)
- **SMB products**: Increase Pricing (25%), UX (25%), reduce Security (5%)
- **Regulated industries**: Increase Security (25%), reduce Features (15%)
## Calibration Process
1. **Anchor scoring** - Score your own product first to establish baseline
2. **Multiple scorers** - Have 2-3 team members score independently
3. **Discuss outliers** - Reconcile scores that differ by more than 2 points
4. **Document evidence** - Record specific examples justifying each score
5. **Normalize quarterly** - Re-calibrate as market expectations evolve
## Bias Mitigation
- **Avoid halo effect** - Score each dimension independently, not influenced by overall impression
- **Use evidence, not feelings** - Every score must link to observable data points
- **Include competitor strengths** - Resist tendency to under-score competitors
- **Rotate scorers** - Different team members bring fresh perspectives
- **Blind scoring** - When possible, evaluate features without knowing which competitor
- **Customer validation** - Compare internal scores against user review sentiment
## Composite Score Calculation
```
Weighted Score = SUM(Dimension Score x Dimension Weight)
Example:
UX(8) x 0.20 = 1.60
Features(7) x 0.25 = 1.75
Pricing(6) x 0.15 = 0.90
Integrations(8) x 0.10 = 0.80
Support(7) x 0.10 = 0.70
Performance(9) x 0.10 = 0.90
Security(8) x 0.10 = 0.80
---
Total = 7.45 / 10
```
## Output Format
Present results as a comparison matrix with color coding:
- Green (8-10): Competitive advantage
- Yellow (5-7): Market parity
- Red (1-4): Competitive gap
FILE:scripts/competitive_matrix_builder.py
#!/usr/bin/env python3
"""Competitive Matrix Builder — Analyze and score competitors across feature dimensions.
Generates weighted competitive matrices, gap analysis, and positioning insights
from structured competitor data.
Usage:
python competitive_matrix_builder.py competitors.json --format json
python competitive_matrix_builder.py competitors.json --format text
python competitive_matrix_builder.py competitors.json --format text --weights pricing=2,ux=1.5
"""
import argparse
import json
import sys
from typing import Dict, List, Any, Optional
from datetime import datetime
from statistics import mean, stdev
def load_competitors(path: str) -> Dict[str, Any]:
"""Load competitor data from JSON file."""
with open(path, "r") as f:
return json.load(f)
def normalize_score(value: float, min_val: float = 1.0, max_val: float = 10.0) -> float:
"""Normalize a score to 0-100 scale."""
return max(0.0, min(100.0, ((value - min_val) / (max_val - min_val)) * 100))
def calculate_weighted_scores(
competitors: List[Dict[str, Any]],
dimensions: List[str],
weights: Optional[Dict[str, float]] = None
) -> List[Dict[str, Any]]:
"""Calculate weighted scores for each competitor across dimensions."""
if weights is None:
weights = {d: 1.0 for d in dimensions}
results = []
for comp in competitors:
scores = comp.get("scores", {})
weighted_total = 0.0
weight_sum = 0.0
dimension_results = {}
for dim in dimensions:
raw = scores.get(dim, 0)
w = weights.get(dim, 1.0)
normalized = normalize_score(raw)
weighted = normalized * w
weighted_total += weighted
weight_sum += w
dimension_results[dim] = {
"raw": raw,
"normalized": round(normalized, 1),
"weight": w,
"weighted": round(weighted, 1)
}
overall = round(weighted_total / weight_sum, 1) if weight_sum > 0 else 0
results.append({
"name": comp["name"],
"overall_score": overall,
"dimensions": dimension_results,
"tier": classify_tier(overall),
"pricing": comp.get("pricing", {}),
"strengths": comp.get("strengths", []),
"weaknesses": comp.get("weaknesses", [])
})
results.sort(key=lambda x: x["overall_score"], reverse=True)
return results
def classify_tier(score: float) -> str:
"""Classify competitor into tier based on overall score."""
if score >= 80:
return "Leader"
elif score >= 60:
return "Strong Competitor"
elif score >= 40:
return "Viable Alternative"
elif score >= 20:
return "Niche Player"
else:
return "Weak"
def gap_analysis(
your_scores: Dict[str, float],
competitor_scores: List[Dict[str, Any]],
dimensions: List[str]
) -> Dict[str, Any]:
"""Identify gaps between your product and competitors."""
gaps = {}
for dim in dimensions:
your_val = your_scores.get(dim, 0)
comp_vals = [c["dimensions"][dim]["raw"] for c in competitor_scores if dim in c.get("dimensions", {})]
if not comp_vals:
continue
avg_comp = mean(comp_vals)
best_comp = max(comp_vals)
gap_to_avg = round(your_val - avg_comp, 1)
gap_to_best = round(your_val - best_comp, 1)
gaps[dim] = {
"your_score": your_val,
"competitor_avg": round(avg_comp, 1),
"competitor_best": best_comp,
"gap_to_avg": gap_to_avg,
"gap_to_best": gap_to_best,
"status": "ahead" if gap_to_avg > 0.5 else ("behind" if gap_to_avg < -0.5 else "parity"),
"priority": "high" if gap_to_best < -2 else ("medium" if gap_to_best < -1 else "low")
}
return {
"gaps": gaps,
"biggest_opportunities": sorted(
[{"dimension": k, **v} for k, v in gaps.items() if v["status"] == "behind"],
key=lambda x: x["gap_to_best"]
)[:5],
"competitive_advantages": sorted(
[{"dimension": k, **v} for k, v in gaps.items() if v["status"] == "ahead"],
key=lambda x: -x["gap_to_avg"]
)[:5]
}
def positioning_analysis(scored: List[Dict[str, Any]]) -> Dict[str, Any]:
"""Generate positioning insights from scored competitors."""
scores = [c["overall_score"] for c in scored]
return {
"market_leaders": [c["name"] for c in scored if c["tier"] == "Leader"],
"your_rank": next((i + 1 for i, c in enumerate(scored) if c.get("is_you")), None),
"total_competitors": len(scored),
"score_distribution": {
"mean": round(mean(scores), 1) if scores else 0,
"stdev": round(stdev(scores), 1) if len(scores) > 1 else 0,
"min": round(min(scores), 1) if scores else 0,
"max": round(max(scores), 1) if scores else 0
},
"tier_distribution": {
tier: len([c for c in scored if c["tier"] == tier])
for tier in ["Leader", "Strong Competitor", "Viable Alternative", "Niche Player", "Weak"]
}
}
def format_text(result: Dict[str, Any]) -> str:
"""Format results as human-readable text."""
lines = []
lines.append("=" * 70)
lines.append("COMPETITIVE MATRIX ANALYSIS")
lines.append(f"Generated: {result['generated_at']}")
lines.append("=" * 70)
# Ranking table
lines.append("\n## COMPETITIVE RANKING\n")
lines.append(f"{'Rank':<6}{'Competitor':<25}{'Score':<10}{'Tier':<20}")
lines.append("-" * 61)
for i, c in enumerate(result["scored_competitors"], 1):
marker = " ← YOU" if c.get("is_you") else ""
lines.append(f"{i:<6}{c['name']:<25}{c['overall_score']:<10}{c['tier']:<20}{marker}")
# Dimension breakdown
lines.append("\n## DIMENSION BREAKDOWN\n")
dims = result["dimensions"]
header = f"{'Dimension':<20}" + "".join(f"{c['name'][:12]:<14}" for c in result["scored_competitors"])
lines.append(header)
lines.append("-" * len(header))
for dim in dims:
row = f"{dim:<20}"
for c in result["scored_competitors"]:
val = c["dimensions"].get(dim, {}).get("raw", "N/A")
row += f"{val:<14}"
lines.append(row)
# Gap analysis
if result.get("gap_analysis"):
ga = result["gap_analysis"]
if ga["biggest_opportunities"]:
lines.append("\n## BIGGEST OPPORTUNITIES (where you're behind)\n")
for opp in ga["biggest_opportunities"]:
lines.append(f" • {opp['dimension']}: You={opp['your_score']}, "
f"Best={opp['competitor_best']}, Gap={opp['gap_to_best']} "
f"[{opp['priority'].upper()} priority]")
if ga["competitive_advantages"]:
lines.append("\n## COMPETITIVE ADVANTAGES (where you lead)\n")
for adv in ga["competitive_advantages"]:
lines.append(f" • {adv['dimension']}: You={adv['your_score']}, "
f"Avg={adv['competitor_avg']}, Lead=+{adv['gap_to_avg']}")
# Positioning
pos = result.get("positioning", {})
if pos:
lines.append("\n## MARKET POSITIONING\n")
lines.append(f" Market Leaders: {', '.join(pos.get('market_leaders', ['None']))}")
if pos.get("your_rank"):
lines.append(f" Your Rank: #{pos['your_rank']} of {pos['total_competitors']}")
dist = pos.get("score_distribution", {})
lines.append(f" Score Range: {dist.get('min', 0)} - {dist.get('max', 0)} "
f"(avg: {dist.get('mean', 0)}, stdev: {dist.get('stdev', 0)})")
lines.append("\n" + "=" * 70)
return "\n".join(lines)
def build_matrix(data: Dict[str, Any], weight_overrides: Optional[Dict[str, float]] = None) -> Dict[str, Any]:
"""Main entry: build competitive matrix from input data."""
competitors = data.get("competitors", [])
dimensions = data.get("dimensions", [])
your_product = data.get("your_product", {})
if not competitors:
return {"error": "No competitors provided"}
if not dimensions:
# Auto-detect from first competitor's scores
dimensions = list(competitors[0].get("scores", {}).keys())
weights = data.get("weights", {})
if weight_overrides:
weights.update(weight_overrides)
# Include your product in scoring if provided
all_entries = list(competitors)
if your_product:
your_product["is_you"] = True
all_entries.insert(0, your_product)
scored = calculate_weighted_scores(all_entries, dimensions, weights)
# Mark your product
for s in scored:
if any(c.get("is_you") and c["name"] == s["name"] for c in all_entries):
s["is_you"] = True
result = {
"generated_at": datetime.now().isoformat(),
"dimensions": dimensions,
"weights": weights if weights else {d: 1.0 for d in dimensions},
"scored_competitors": scored,
"positioning": positioning_analysis(scored)
}
if your_product:
result["gap_analysis"] = gap_analysis(
your_product.get("scores", {}), scored, dimensions
)
return result
def parse_weights(weight_str: str) -> Dict[str, float]:
"""Parse weight string like 'pricing=2,ux=1.5' into dict."""
weights = {}
for pair in weight_str.split(","):
if "=" in pair:
k, v = pair.split("=", 1)
weights[k.strip()] = float(v.strip())
return weights
def main():
parser = argparse.ArgumentParser(
description="Build competitive matrix with scoring and gap analysis"
)
parser.add_argument("input", help="Path to competitors JSON file")
parser.add_argument("--format", choices=["json", "text"], default="text",
help="Output format (default: text)")
parser.add_argument("--weights", type=str, default=None,
help="Weight overrides: 'dim1=2.0,dim2=1.5'")
parser.add_argument("--output", type=str, default=None,
help="Output file path (default: stdout)")
args = parser.parse_args()
data = load_competitors(args.input)
weight_overrides = parse_weights(args.weights) if args.weights else None
result = build_matrix(data, weight_overrides)
if args.format == "json":
output = json.dumps(result, indent=2)
else:
output = format_text(result)
if args.output:
with open(args.output, "w") as f:
f.write(output)
print(f"Output written to {args.output}")
else:
print(output)
if __name__ == "__main__":
main()
Đánh giá và so sánh tech stack với phân tích TCO, đánh giá bảo mật, chấm điểm hệ sinh thái và lộ trình di chuyển.
---
name: "tech-stack-evaluator"
description: Technology stack evaluation and comparison with TCO analysis, security assessment, and ecosystem health scoring. Use when comparing frameworks, evaluating technology stacks, calculating total cost of ownership, assessing migration paths, or analyzing ecosystem viability.
---
# Technology Stack Evaluator
Evaluate and compare technologies, frameworks, and cloud providers with data-driven analysis and actionable recommendations.
## Table of Contents
- [Capabilities](#capabilities)
- [Quick Start](#quick-start)
- [Input Formats](#input-formats)
- [Analysis Types](#analysis-types)
- [Scripts](#scripts)
- [References](#references)
---
## Capabilities
| Capability | Description |
|------------|-------------|
| Technology Comparison | Compare frameworks and libraries with weighted scoring |
| TCO Analysis | Calculate 5-year total cost including hidden costs |
| Ecosystem Health | Assess GitHub metrics, npm adoption, community strength |
| Security Assessment | Evaluate vulnerabilities and compliance readiness |
| Migration Analysis | Estimate effort, risks, and timeline for migrations |
| Cloud Comparison | Compare AWS, Azure, GCP for specific workloads |
---
## Quick Start
### Compare Two Technologies
```
Compare React vs Vue for a SaaS dashboard.
Priorities: developer productivity (40%), ecosystem (30%), performance (30%).
```
### Calculate TCO
```
Calculate 5-year TCO for Next.js on Vercel.
Team: 8 developers. Hosting: $2500/month. Growth: 40%/year.
```
### Assess Migration
```
Evaluate migrating from Angular.js to React.
Codebase: 50,000 lines, 200 components. Team: 6 developers.
```
---
## Input Formats
The evaluator accepts three input formats:
**Text** - Natural language queries
```
Compare PostgreSQL vs MongoDB for our e-commerce platform.
```
**YAML** - Structured input for automation
```yaml
comparison:
technologies: ["React", "Vue"]
use_case: "SaaS dashboard"
weights:
ecosystem: 30
performance: 25
developer_experience: 45
```
**JSON** - Programmatic integration
```json
{
"technologies": ["React", "Vue"],
"use_case": "SaaS dashboard"
}
```
---
## Analysis Types
### Quick Comparison (200-300 tokens)
- Weighted scores and recommendation
- Top 3 decision factors
- Confidence level
### Standard Analysis (500-800 tokens)
- Comparison matrix
- TCO overview
- Security summary
### Full Report (1200-1500 tokens)
- All metrics and calculations
- Migration analysis
- Detailed recommendations
---
## Scripts
### stack_comparator.py
Compare technologies with customizable weighted criteria.
```bash
python scripts/stack_comparator.py --help
```
### tco_calculator.py
Calculate total cost of ownership over multi-year projections.
```bash
python scripts/tco_calculator.py --input assets/sample_input_tco.json
```
### ecosystem_analyzer.py
Analyze ecosystem health from GitHub, npm, and community metrics.
```bash
python scripts/ecosystem_analyzer.py --technology react
```
### security_assessor.py
Evaluate security posture and compliance readiness.
```bash
python scripts/security_assessor.py --technology express --compliance soc2,gdpr
```
### migration_analyzer.py
Estimate migration complexity, effort, and risks.
```bash
python scripts/migration_analyzer.py --from angular-1.x --to react
```
---
## References
| Document | Content |
|----------|---------|
| `references/metrics.md` | Detailed scoring algorithms and calculation formulas |
| `references/examples.md` | Input/output examples for all analysis types |
| `references/workflows.md` | Step-by-step evaluation workflows |
---
## Confidence Levels
| Level | Score | Interpretation |
|-------|-------|----------------|
| High | 80-100% | Clear winner, strong data |
| Medium | 50-79% | Trade-offs present, moderate uncertainty |
| Low | < 50% | Close call, limited data |
---
## When to Use
- Comparing frontend/backend frameworks for new projects
- Evaluating cloud providers for specific workloads
- Planning technology migrations with risk assessment
- Calculating build vs. buy decisions with TCO
- Assessing open-source library viability
## When NOT to Use
- Trivial decisions between similar tools (use team preference)
- Mandated technology choices (decision already made)
- Emergency production issues (use monitoring tools)
FILE:assets/expected_output_comparison.json
{
"technologies": {
"PostgreSQL": {
"category_scores": {
"performance": 85.0,
"scalability": 90.0,
"developer_experience": 75.0,
"ecosystem": 95.0,
"learning_curve": 70.0,
"documentation": 90.0,
"community_support": 95.0,
"enterprise_readiness": 95.0
},
"weighted_total": 85.5,
"strengths": ["scalability", "ecosystem", "documentation", "community_support", "enterprise_readiness"],
"weaknesses": ["learning_curve"]
},
"MongoDB": {
"category_scores": {
"performance": 80.0,
"scalability": 95.0,
"developer_experience": 85.0,
"ecosystem": 85.0,
"learning_curve": 80.0,
"documentation": 85.0,
"community_support": 85.0,
"enterprise_readiness": 75.0
},
"weighted_total": 84.5,
"strengths": ["scalability", "developer_experience", "learning_curve"],
"weaknesses": []
}
},
"recommendation": "PostgreSQL",
"confidence": 52.0,
"decision_factors": [
{
"category": "performance",
"importance": "20.0%",
"best_performer": "PostgreSQL",
"score": 85.0
},
{
"category": "scalability",
"importance": "20.0%",
"best_performer": "MongoDB",
"score": 95.0
},
{
"category": "developer_experience",
"importance": "15.0%",
"best_performer": "MongoDB",
"score": 85.0
}
],
"comparison_matrix": [
{
"category": "Performance",
"weight": "20.0%",
"scores": {
"PostgreSQL": "85.0",
"MongoDB": "80.0"
}
},
{
"category": "Scalability",
"weight": "20.0%",
"scores": {
"PostgreSQL": "90.0",
"MongoDB": "95.0"
}
},
{
"category": "WEIGHTED TOTAL",
"weight": "100%",
"scores": {
"PostgreSQL": "85.5",
"MongoDB": "84.5"
}
}
]
}
FILE:assets/sample_input_structured.json
{
"comparison": {
"technologies": [
{
"name": "PostgreSQL",
"performance": {"score": 85},
"scalability": {"score": 90},
"developer_experience": {"score": 75},
"ecosystem": {"score": 95},
"learning_curve": {"score": 70},
"documentation": {"score": 90},
"community_support": {"score": 95},
"enterprise_readiness": {"score": 95}
},
{
"name": "MongoDB",
"performance": {"score": 80},
"scalability": {"score": 95},
"developer_experience": {"score": 85},
"ecosystem": {"score": 85},
"learning_curve": {"score": 80},
"documentation": {"score": 85},
"community_support": {"score": 85},
"enterprise_readiness": {"score": 75}
}
],
"use_case": "SaaS application with complex queries",
"weights": {
"performance": 20,
"scalability": 20,
"developer_experience": 15,
"ecosystem": 15,
"learning_curve": 10,
"documentation": 10,
"community_support": 5,
"enterprise_readiness": 5
}
}
}
FILE:assets/sample_input_tco.json
{
"tco_analysis": {
"technology": "AWS",
"team_size": 10,
"timeline_years": 5,
"initial_costs": {
"licensing": 0,
"training_hours_per_dev": 40,
"developer_hourly_rate": 100,
"training_materials": 1000,
"migration": 50000,
"setup": 10000,
"tooling": 5000
},
"operational_costs": {
"annual_licensing": 0,
"monthly_hosting": 5000,
"annual_support": 20000,
"maintenance_hours_per_dev_monthly": 20
},
"scaling_params": {
"initial_users": 5000,
"annual_growth_rate": 0.30,
"initial_servers": 10,
"cost_per_server_monthly": 300
},
"productivity_factors": {
"productivity_multiplier": 1.2,
"time_to_market_reduction_days": 15,
"avg_feature_time_days": 45,
"avg_feature_value": 15000,
"technical_debt_percentage": 0.12,
"vendor_lock_in_risk": "medium",
"security_incidents_per_year": 0.3,
"avg_security_incident_cost": 30000,
"downtime_hours_per_year": 4,
"downtime_cost_per_hour": 8000,
"annual_turnover_rate": 0.12,
"cost_per_new_hire": 35000
}
}
}
FILE:assets/sample_input_text.json
{
"format": "text",
"input": "Compare React vs Vue for building a SaaS dashboard with real-time collaboration features. Our team has 8 developers, and we need to consider developer experience, ecosystem maturity, and performance."
}
FILE:references/examples.md
# Technology Evaluation Examples
Concrete examples showing input formats and expected outputs.
---
## Table of Contents
- [Quick Comparison Example](#quick-comparison-example)
- [TCO Analysis Example](#tco-analysis-example)
- [Ecosystem Analysis Example](#ecosystem-analysis-example)
- [Migration Assessment Example](#migration-assessment-example)
- [Multi-Technology Comparison](#multi-technology-comparison)
---
## Quick Comparison Example
### Input (Text Format)
```
Compare React vs Vue for building a SaaS dashboard.
Focus on: developer productivity, ecosystem maturity, performance.
```
### Output
```
TECHNOLOGY COMPARISON: React vs Vue for SaaS Dashboard
=======================================================
RECOMMENDATION: React
Confidence: 78% (Medium-High)
COMPARISON MATRIX
-----------------
| Category | Weight | React | Vue |
|----------------------|--------|-------|------|
| Performance | 15% | 82.0 | 85.0 |
| Scalability | 15% | 88.0 | 80.0 |
| Developer Experience | 20% | 85.0 | 90.0 |
| Ecosystem | 15% | 92.0 | 78.0 |
| Learning Curve | 10% | 70.0 | 85.0 |
| Documentation | 10% | 88.0 | 82.0 |
| Community Support | 10% | 90.0 | 75.0 |
| Enterprise Readiness | 5% | 85.0 | 72.0 |
|----------------------|--------|-------|------|
| WEIGHTED TOTAL | 100% | 85.2 | 81.1 |
KEY DECISION FACTORS
--------------------
1. Ecosystem (15%): React leads with 92.0 - larger npm ecosystem
2. Developer Experience (20%): Vue leads with 90.0 - gentler learning curve
3. Community Support (10%): React leads with 90.0 - more Stack Overflow resources
PROS/CONS SUMMARY
-----------------
React:
✓ Excellent ecosystem (92.0/100)
✓ Strong community support (90.0/100)
✓ Excellent scalability (88.0/100)
✗ Steeper learning curve (70.0/100)
Vue:
✓ Excellent developer experience (90.0/100)
✓ Good performance (85.0/100)
✓ Easier learning curve (85.0/100)
✗ Smaller enterprise presence (72.0/100)
```
---
## TCO Analysis Example
### Input (JSON Format)
```json
{
"technology": "Next.js on Vercel",
"team_size": 8,
"timeline_years": 5,
"initial_costs": {
"licensing": 0,
"training_hours_per_dev": 24,
"developer_hourly_rate": 85,
"migration": 15000,
"setup": 5000
},
"operational_costs": {
"monthly_hosting": 2500,
"annual_support": 0,
"maintenance_hours_per_dev_monthly": 16
},
"scaling_params": {
"initial_users": 5000,
"annual_growth_rate": 0.40,
"initial_servers": 3,
"cost_per_server_monthly": 150
}
}
```
### Output
```
TCO ANALYSIS: Next.js on Vercel (5-Year Projection)
====================================================
EXECUTIVE SUMMARY
-----------------
Total TCO: $1,247,320
Net TCO (after productivity gains): $987,320
Average Yearly Cost: $249,464
INITIAL COSTS (One-Time)
------------------------
| Component | Cost |
|----------------|-----------|
| Licensing | $0 |
| Training | $16,820 |
| Migration | $15,000 |
| Setup | $5,000 |
|----------------|-----------|
| TOTAL INITIAL | $36,820 |
OPERATIONAL COSTS (Per Year)
----------------------------
| Year | Hosting | Maintenance | Total |
|------|----------|-------------|-----------|
| 1 | $30,000 | $130,560 | $160,560 |
| 2 | $42,000 | $130,560 | $172,560 |
| 3 | $58,800 | $130,560 | $189,360 |
| 4 | $82,320 | $130,560 | $212,880 |
| 5 | $115,248 | $130,560 | $245,808 |
SCALING ANALYSIS
----------------
User Projections: 5,000 → 7,000 → 9,800 → 13,720 → 19,208
Cost per User: $32.11 → $24.65 → $19.32 → $15.52 → $12.79
Scaling Efficiency: Excellent - economies of scale achieved
KEY COST DRIVERS
----------------
1. Developer maintenance time ($652,800 over 5 years)
2. Infrastructure/hosting ($328,368 over 5 years)
OPTIMIZATION OPPORTUNITIES
--------------------------
• Consider automation to reduce maintenance hours
• Evaluate reserved capacity pricing for hosting
```
---
## Ecosystem Analysis Example
### Input
```yaml
technology: "Svelte"
github:
stars: 78000
forks: 4100
contributors: 680
commits_last_month: 45
avg_issue_response_hours: 36
issue_resolution_rate: 0.72
releases_per_year: 8
active_maintainers: 5
npm:
weekly_downloads: 420000
version: "4.2.8"
dependencies_count: 0
days_since_last_publish: 21
community:
stackoverflow_questions: 8500
job_postings: 1200
tutorials_count: 350
forum_members: 25000
corporate_backing:
type: "community_led"
funding_millions: 0
```
### Output
```
ECOSYSTEM ANALYSIS: Svelte
==========================
OVERALL HEALTH SCORE: 71.5/100
HEALTH SCORE BREAKDOWN
----------------------
| Category | Score | Assessment |
|--------------------|-------|-------------------|
| GitHub Health | 78.0 | Strong |
| npm Health | 72.0 | Good |
| Community Health | 58.0 | Moderate |
| Corporate Backing | 40.0 | Community-led |
| Maintenance Health | 75.0 | Active |
VIABILITY ASSESSMENT
--------------------
Overall: Good - Solid viability with minor concerns
Risk Level: Low-Medium
Strengths:
• Strong GitHub presence with active community
• Active maintenance with responsive issue resolution
• High npm adoption with stable releases
Risks:
• Weak corporate backing - sustainability concerns
• Smaller community - limited enterprise resources
RECOMMENDATION
--------------
Suitable for adoption - monitor corporate backing situation.
Consider alternative with stronger backing for mission-critical enterprise apps.
KEY METRICS
-----------
GitHub: 78,000 stars, 680 contributors, 45 commits/month
npm: 420,000 weekly downloads, v4.2.8
Community: 8,500 SO questions, 1,200 job postings
```
---
## Migration Assessment Example
### Input
```
Assess migration from Angular.js (1.x) to React.
Application: 50,000 lines of code, 200 components.
Team: 6 developers, 2 have React experience.
Timeline: Need completion within 8 months.
```
### Output
```
MIGRATION ASSESSMENT: Angular.js 1.x → React
=============================================
COMPLEXITY SCORE: 7.8/10 (High)
COMPLEXITY BREAKDOWN
--------------------
| Factor | Score | Impact |
|---------------------|-------|-------------|
| Code Changes | 8/10 | 50K LOC |
| Architecture Impact | 9/10 | Complete |
| Data Migration | 5/10 | State only |
| Downtime Risk | 6/10 | Mitigatable |
EFFORT ESTIMATION
-----------------
| Phase | Hours | Duration |
|--------------------------|--------|----------|
| Planning & Prototyping | 240 | 3 weeks |
| Core Migration | 2,400 | 16 weeks |
| Testing & Validation | 480 | 5 weeks |
| Deployment & Monitoring | 160 | 2 weeks |
|--------------------------|--------|----------|
| TOTAL | 3,280 | 26 weeks |
Timeline Assessment: 26 weeks exceeds 8-month target (32 weeks)
Recommendation: Feasible with buffer
RISK ASSESSMENT
---------------
Technical Risks:
• High - Two-way binding to unidirectional data flow
• Medium - Dependency injection to hooks/context
• Medium - Performance regression during transition
Business Risks:
• Medium - Feature freeze during migration
• Low - Customer impact with phased approach
Team Risks:
• High - 4 of 6 developers need React training
• Medium - Learning curve affects velocity initially
MIGRATION STRATEGY
------------------
Recommended Approach: Strangler Fig Pattern
Phase 1 (Weeks 1-8): Strangler setup
• Create React shell application
• Set up micro-frontend routing
• Migrate shared utilities
Phase 2 (Weeks 9-18): Component migration
• Migrate components by feature area
• Maintain Angular.js for untouched areas
• Continuous integration testing
Phase 3 (Weeks 19-24): Completion
• Migrate remaining components
• Remove Angular.js dependencies
• Performance optimization
Phase 4 (Weeks 25-26): Stabilization
• Production deployment
• Monitoring and fixes
```
---
## Multi-Technology Comparison
### Input
```
Compare: React, Vue, Svelte, Angular for enterprise SaaS.
Use case: Large team (20+ developers), complex state management.
Generate comprehensive decision matrix.
```
### Output
```
MULTI-TECHNOLOGY COMPARISON: Enterprise SaaS Platform
======================================================
RECOMMENDATION: React
Confidence: 72% (Medium)
Runner-up: Angular (close second for enterprise)
DECISION MATRIX
---------------
| Category | Wt | React | Vue | Svelte | Angular |
|----------------------|------|-------|------|--------|---------|
| Performance | 15% | 82 | 85 | 95 | 78 |
| Scalability | 15% | 90 | 82 | 75 | 92 |
| Developer Experience | 20% | 85 | 90 | 88 | 75 |
| Ecosystem | 15% | 95 | 80 | 65 | 88 |
| Learning Curve | 10% | 70 | 85 | 80 | 60 |
| Documentation | 10% | 90 | 85 | 75 | 92 |
| Community Support | 10% | 92 | 78 | 55 | 85 |
| Enterprise Readiness | 5% | 88 | 72 | 50 | 95 |
|----------------------|------|-------|------|--------|---------|
| WEIGHTED TOTAL | 100% | 86.3 | 83.1 | 76.2 | 83.0 |
FRAMEWORK PROFILES
------------------
React: Best for large ecosystem, hiring pool
Angular: Best for enterprise structure, TypeScript-first
Vue: Best for developer experience, gradual adoption
Svelte: Best for performance, smaller bundles
RECOMMENDATION RATIONALE
------------------------
For 20+ developer team with complex state management:
1. React (Recommended)
• Largest talent pool for hiring
• Extensive enterprise libraries (Redux, React Query)
• Meta backing ensures long-term support
• Most Stack Overflow resources
2. Angular (Strong Alternative)
• Built-in structure for large teams
• TypeScript-first reduces bugs
• Comprehensive CLI and tooling
• Google enterprise backing
3. Vue (Consider for DX)
• Excellent documentation
• Easier onboarding
• Growing enterprise adoption
• Consider if DX is top priority
4. Svelte (Not Recommended for This Use Case)
• Smaller ecosystem for enterprise
• Limited hiring pool
• State management options less mature
• Better for smaller teams/projects
```
FILE:references/metrics.md
# Technology Evaluation Metrics
Detailed metrics and calculations used in technology stack evaluation.
---
## Table of Contents
- [Scoring and Comparison](#scoring-and-comparison)
- [Financial Calculations](#financial-calculations)
- [Ecosystem Health Metrics](#ecosystem-health-metrics)
- [Security Metrics](#security-metrics)
- [Migration Metrics](#migration-metrics)
- [Performance Benchmarks](#performance-benchmarks)
---
## Scoring and Comparison
### Technology Comparison Matrix
| Metric | Scale | Description |
|--------|-------|-------------|
| Feature Completeness | 0-100 | Coverage of required features |
| Learning Curve | Easy/Medium/Hard | Time to developer proficiency |
| Developer Experience | 0-100 | Tooling, debugging, workflow quality |
| Documentation Quality | 0-10 | Completeness, clarity, examples |
### Weighted Scoring Algorithm
The comparator uses normalized weighted scoring:
```python
# Default category weights (sum to 100%)
weights = {
"performance": 15,
"scalability": 15,
"developer_experience": 20,
"ecosystem": 15,
"learning_curve": 10,
"documentation": 10,
"community_support": 10,
"enterprise_readiness": 5
}
# Final score calculation
weighted_score = sum(category_score * weight / 100 for each category)
```
### Confidence Scoring
Confidence is calculated based on score gap between top options:
| Score Gap | Confidence Level |
|-----------|------------------|
| < 5 points | Low (40-50%) |
| 5-15 points | Medium (50-70%) |
| > 15 points | High (70-100%) |
---
## Financial Calculations
### TCO Components
**Initial Costs (One-Time)**
- Licensing fees
- Training: `team_size * hours_per_dev * hourly_rate + materials`
- Migration costs
- Setup and tooling
**Operational Costs (Annual)**
- Licensing renewals
- Hosting: `base_cost * (1 + growth_rate)^(year - 1)`
- Support contracts
- Maintenance: `team_size * hours_per_dev_monthly * hourly_rate * 12`
**Scaling Costs**
- Infrastructure: `servers * cost_per_server * 12`
- Cost per user: `total_yearly_cost / user_count`
### ROI Calculations
```
productivity_value = additional_features_per_year * avg_feature_value
net_tco = total_cost - (productivity_value * years)
roi_percentage = (benefits - costs) / costs * 100
```
### Cost Per Metric Reference
| Metric | Description |
|--------|-------------|
| Cost per user | Monthly or yearly per active user |
| Cost per API request | Average cost per 1000 requests |
| Cost per GB | Storage and transfer costs |
| Cost per compute hour | Processing time costs |
---
## Ecosystem Health Metrics
### GitHub Health Score (0-100)
| Metric | Max Points | Thresholds |
|--------|------------|------------|
| Stars | 30 | 50K+: 30, 20K+: 25, 10K+: 20, 5K+: 15, 1K+: 10 |
| Forks | 20 | 10K+: 20, 5K+: 15, 2K+: 12, 1K+: 10 |
| Contributors | 20 | 500+: 20, 200+: 15, 100+: 12, 50+: 10 |
| Commits/month | 30 | 100+: 30, 50+: 25, 25+: 20, 10+: 15 |
### npm Health Score (0-100)
| Metric | Max Points | Thresholds |
|--------|------------|------------|
| Weekly downloads | 40 | 1M+: 40, 500K+: 35, 100K+: 30, 50K+: 25, 10K+: 20 |
| Major version | 20 | v5+: 20, v3+: 15, v1+: 10 |
| Dependencies | 20 | ≤10: 20, ≤25: 15, ≤50: 10 (fewer is better) |
| Days since publish | 20 | ≤30: 20, ≤90: 15, ≤180: 10, ≤365: 5 |
### Community Health Score (0-100)
| Metric | Max Points | Thresholds |
|--------|------------|------------|
| Stack Overflow questions | 25 | 50K+: 25, 20K+: 20, 10K+: 15, 5K+: 10 |
| Job postings | 25 | 5K+: 25, 2K+: 20, 1K+: 15, 500+: 10 |
| Tutorials | 25 | 1K+: 25, 500+: 20, 200+: 15, 100+: 10 |
| Forum/Discord members | 25 | 50K+: 25, 20K+: 20, 10K+: 15, 5K+: 10 |
### Corporate Backing Score
| Backing Type | Score |
|--------------|-------|
| Major tech company (Google, Microsoft, Meta) | 100 |
| Established company (Vercel, HashiCorp) | 80 |
| Funded startup | 60 |
| Community-led (strong community) | 40 |
| Individual maintainers | 20 |
---
## Security Metrics
### Security Scoring Components
| Metric | Description |
|--------|-------------|
| CVE Count (12 months) | Known vulnerabilities in last year |
| CVE Count (3 years) | Longer-term vulnerability history |
| Severity Distribution | Critical/High/Medium/Low counts |
| Patch Frequency | Average days to patch vulnerabilities |
### Compliance Readiness Levels
| Level | Score Range | Description |
|-------|-------------|-------------|
| Ready | 90-100% | Meets compliance requirements |
| Mostly Ready | 70-89% | Minor gaps to address |
| Partial | 50-69% | Significant work needed |
| Not Ready | < 50% | Major gaps exist |
### Compliance Framework Coverage
**GDPR**
- Data privacy features
- Consent management
- Data portability
- Right to deletion
**SOC2**
- Access controls
- Encryption at rest/transit
- Audit logging
- Change management
**HIPAA**
- PHI handling
- Encryption standards
- Access controls
- Audit trails
---
## Migration Metrics
### Complexity Scoring (1-10 Scale)
| Factor | Weight | Description |
|--------|--------|-------------|
| Code Changes | 30% | Lines of code affected |
| Architecture Impact | 25% | Breaking changes, API compatibility |
| Data Migration | 25% | Schema changes, data transformation |
| Downtime Requirements | 20% | Zero-downtime possible vs planned outage |
### Effort Estimation
| Phase | Components |
|-------|------------|
| Development | Hours per component * complexity factor |
| Testing | Unit + integration + E2E hours |
| Training | Team size * learning curve hours |
| Buffer | 20-30% for unknowns |
### Risk Assessment Matrix
| Risk Category | Factors Evaluated |
|---------------|-------------------|
| Technical | API incompatibilities, performance regressions |
| Business | Downtime impact, feature parity gaps |
| Team | Learning curve, skill gaps |
---
## Performance Benchmarks
### Throughput/Latency Metrics
| Metric | Description |
|--------|-------------|
| RPS | Requests per second |
| Avg Response Time | Mean response latency (ms) |
| P95 Latency | 95th percentile response time |
| P99 Latency | 99th percentile response time |
| Concurrent Users | Maximum simultaneous connections |
### Resource Usage Metrics
| Metric | Unit |
|--------|------|
| Memory | MB/GB per instance |
| CPU | Utilization percentage |
| Storage | GB required |
| Network | Bandwidth MB/s |
### Scalability Characteristics
| Type | Description |
|------|-------------|
| Horizontal | Add more instances, efficiency factor |
| Vertical | CPU/memory limits per instance |
| Cost per Performance | Dollar per 1000 RPS |
| Scaling Inflection | Point where cost efficiency changes |
FILE:references/workflows.md
# Technology Evaluation Workflows
Step-by-step workflows for common evaluation scenarios.
---
## Table of Contents
- [Framework Comparison Workflow](#framework-comparison-workflow)
- [TCO Analysis Workflow](#tco-analysis-workflow)
- [Migration Assessment Workflow](#migration-assessment-workflow)
- [Security Evaluation Workflow](#security-evaluation-workflow)
- [Cloud Provider Selection Workflow](#cloud-provider-selection-workflow)
---
## Framework Comparison Workflow
Use this workflow when comparing frontend/backend frameworks or libraries.
### Step 1: Define Requirements
1. Identify the use case:
- What type of application? (SaaS, e-commerce, real-time, etc.)
- What scale? (users, requests, data volume)
- What team size and skill level?
2. Set priorities (weights must sum to 100%):
- Performance: ____%
- Scalability: ____%
- Developer Experience: ____%
- Ecosystem: ____%
- Learning Curve: ____%
- Other: ____%
3. List constraints:
- Budget limitations
- Timeline requirements
- Compliance needs
- Existing infrastructure
### Step 2: Run Comparison
```bash
python scripts/stack_comparator.py \
--technologies "React,Vue,Angular" \
--use-case "enterprise-saas" \
--weights "performance:20,ecosystem:25,scalability:20,developer_experience:35"
```
### Step 3: Analyze Results
1. Review weighted total scores
2. Check confidence level (High/Medium/Low)
3. Examine strengths and weaknesses for each option
4. Review decision factors
### Step 4: Validate Recommendation
1. Match recommendation to your constraints
2. Consider team skills and hiring market
3. Evaluate ecosystem for your specific needs
4. Check corporate backing and long-term viability
### Step 5: Document Decision
Record:
- Final selection with rationale
- Trade-offs accepted
- Risks identified
- Mitigation strategies
---
## TCO Analysis Workflow
Use this workflow for comprehensive cost analysis over multiple years.
### Step 1: Gather Cost Data
**Initial Costs:**
- [ ] Licensing fees (if any)
- [ ] Training hours per developer
- [ ] Developer hourly rate
- [ ] Migration costs
- [ ] Setup and tooling costs
**Operational Costs:**
- [ ] Monthly hosting costs
- [ ] Annual support contracts
- [ ] Maintenance hours per developer per month
**Scaling Parameters:**
- [ ] Initial user count
- [ ] Expected annual growth rate
- [ ] Infrastructure scaling approach
### Step 2: Run TCO Calculator
```bash
python scripts/tco_calculator.py \
--input assets/sample_input_tco.json \
--years 5 \
--output tco_report.json
```
### Step 3: Analyze Cost Breakdown
1. Review initial vs. operational costs ratio
2. Examine year-over-year cost growth
3. Check cost per user trends
4. Identify scaling efficiency
### Step 4: Identify Optimization Opportunities
Review:
- Can hosting costs be reduced with reserved pricing?
- Can automation reduce maintenance hours?
- Are there cheaper alternatives for specific components?
### Step 5: Compare Multiple Options
Run TCO analysis for each technology option:
1. Current state (baseline)
2. Option A
3. Option B
Compare:
- 5-year total cost
- Break-even point
- Risk-adjusted costs
---
## Migration Assessment Workflow
Use this workflow when planning technology migrations.
### Step 1: Document Current State
1. Count lines of code
2. List all components/modules
3. Identify dependencies
4. Document current architecture
5. Note existing pain points
### Step 2: Define Target State
1. Target technology/framework
2. Target architecture
3. Expected benefits
4. Success criteria
### Step 3: Assess Team Readiness
- How many developers have target technology experience?
- What training is needed?
- What is the team's capacity during migration?
### Step 4: Run Migration Analysis
```bash
python scripts/migration_analyzer.py \
--from "angular-1.x" \
--to "react" \
--codebase-size 50000 \
--components 200 \
--team-size 6
```
### Step 5: Review Risk Assessment
For each risk category:
1. Identify specific risks
2. Assess probability and impact
3. Define mitigation strategies
4. Assign risk owners
### Step 6: Plan Migration Phases
1. **Phase 1: Foundation**
- Setup new infrastructure
- Create migration utilities
- Train team
2. **Phase 2: Incremental Migration**
- Migrate by feature area
- Maintain parallel systems
- Continuous testing
3. **Phase 3: Completion**
- Remove legacy code
- Optimize performance
- Complete documentation
4. **Phase 4: Stabilization**
- Monitor production
- Address issues
- Gather metrics
### Step 7: Define Rollback Plan
Document:
- Trigger conditions for rollback
- Rollback procedure
- Data recovery steps
- Communication plan
---
## Security Evaluation Workflow
Use this workflow for security and compliance assessment.
### Step 1: Identify Requirements
1. List applicable compliance standards:
- [ ] GDPR
- [ ] SOC2
- [ ] HIPAA
- [ ] PCI-DSS
- [ ] Other: _____
2. Define security priorities:
- Data encryption requirements
- Access control needs
- Audit logging requirements
- Incident response expectations
### Step 2: Gather Security Data
For each technology:
- [ ] CVE count (last 12 months)
- [ ] CVE count (last 3 years)
- [ ] Severity distribution
- [ ] Average patch time
- [ ] Security features list
### Step 3: Run Security Assessment
```bash
python scripts/security_assessor.py \
--technology "express-js" \
--compliance "soc2,gdpr" \
--output security_report.json
```
### Step 4: Analyze Results
Review:
1. Overall security score
2. Vulnerability trends
3. Patch responsiveness
4. Compliance readiness per standard
### Step 5: Identify Gaps
For each compliance standard:
1. List missing requirements
2. Estimate remediation effort
3. Identify workarounds if available
4. Calculate compliance cost
### Step 6: Make Risk-Based Decision
Consider:
- Acceptable risk level
- Cost of remediation
- Alternative technologies
- Business impact of compliance gaps
---
## Cloud Provider Selection Workflow
Use this workflow for AWS vs Azure vs GCP decisions.
### Step 1: Define Workload Requirements
1. Workload type:
- [ ] Web application
- [ ] API services
- [ ] Data analytics
- [ ] Machine learning
- [ ] IoT
- [ ] Other: _____
2. Resource requirements:
- Compute: ____ instances, ____ cores, ____ GB RAM
- Storage: ____ TB, type (block/object/file)
- Database: ____ type, ____ size
- Network: ____ GB/month transfer
3. Special requirements:
- [ ] GPU/TPU for ML
- [ ] Edge computing
- [ ] Multi-region
- [ ] Specific compliance certifications
### Step 2: Evaluate Feature Availability
For each provider, verify:
- Required services exist
- Service maturity level
- Regional availability
- SLA guarantees
### Step 3: Run Cost Comparison
```bash
python scripts/tco_calculator.py \
--providers "aws,azure,gcp" \
--workload-config workload.json \
--years 3
```
### Step 4: Assess Ecosystem Fit
Consider:
- Team's existing expertise
- Development tooling preferences
- CI/CD integration
- Monitoring and observability tools
### Step 5: Evaluate Vendor Lock-in
For each provider:
1. List proprietary services you'll use
2. Estimate migration cost if switching
3. Identify portable alternatives
4. Calculate lock-in risk score
### Step 6: Make Final Selection
Weight factors:
- Cost: ____%
- Features: ____%
- Team expertise: ____%
- Lock-in risk: ____%
- Support quality: ____%
Select provider with highest weighted score.
---
## Best Practices
### For All Evaluations
1. **Document assumptions** - Make all assumptions explicit
2. **Validate data** - Verify metrics from multiple sources
3. **Consider context** - Generic scores may not apply to your situation
4. **Include stakeholders** - Get input from team members who will use the technology
5. **Plan for change** - Technology landscapes evolve; plan for flexibility
### Common Pitfalls to Avoid
1. Over-weighting recent popularity vs. long-term stability
2. Ignoring team learning curve in timeline estimates
3. Underestimating migration complexity
4. Assuming vendor claims are accurate
5. Not accounting for hidden costs (training, hiring, technical debt)
FILE:scripts/ecosystem_analyzer.py
"""
Ecosystem Health Analyzer.
Analyzes technology ecosystem health including community size, maintenance status,
GitHub metrics, npm downloads, and long-term viability assessment.
"""
from typing import Dict, List, Any, Optional
from datetime import datetime, timedelta
class EcosystemAnalyzer:
"""Analyze technology ecosystem health and viability."""
def __init__(self, ecosystem_data: Dict[str, Any]):
"""
Initialize analyzer with ecosystem data.
Args:
ecosystem_data: Dictionary containing GitHub, npm, and community metrics
"""
self.technology = ecosystem_data.get('technology', 'Unknown')
self.github_data = ecosystem_data.get('github', {})
self.npm_data = ecosystem_data.get('npm', {})
self.community_data = ecosystem_data.get('community', {})
self.corporate_backing = ecosystem_data.get('corporate_backing', {})
def calculate_health_score(self) -> Dict[str, float]:
"""
Calculate overall ecosystem health score (0-100).
Returns:
Dictionary of health score components
"""
scores = {
'github_health': self._score_github_health(),
'npm_health': self._score_npm_health(),
'community_health': self._score_community_health(),
'corporate_backing': self._score_corporate_backing(),
'maintenance_health': self._score_maintenance_health()
}
# Calculate weighted average
weights = {
'github_health': 0.25,
'npm_health': 0.20,
'community_health': 0.20,
'corporate_backing': 0.15,
'maintenance_health': 0.20
}
overall = sum(scores[k] * weights[k] for k in scores.keys())
scores['overall_health'] = overall
return scores
def _score_github_health(self) -> float:
"""
Score GitHub repository health.
Returns:
GitHub health score (0-100)
"""
score = 0.0
# Stars (0-30 points)
stars = self.github_data.get('stars', 0)
if stars >= 50000:
score += 30
elif stars >= 20000:
score += 25
elif stars >= 10000:
score += 20
elif stars >= 5000:
score += 15
elif stars >= 1000:
score += 10
else:
score += max(0, stars / 100) # 1 point per 100 stars
# Forks (0-20 points)
forks = self.github_data.get('forks', 0)
if forks >= 10000:
score += 20
elif forks >= 5000:
score += 15
elif forks >= 2000:
score += 12
elif forks >= 1000:
score += 10
else:
score += max(0, forks / 100)
# Contributors (0-20 points)
contributors = self.github_data.get('contributors', 0)
if contributors >= 500:
score += 20
elif contributors >= 200:
score += 15
elif contributors >= 100:
score += 12
elif contributors >= 50:
score += 10
else:
score += max(0, contributors / 5)
# Commit frequency (0-30 points)
commits_last_month = self.github_data.get('commits_last_month', 0)
if commits_last_month >= 100:
score += 30
elif commits_last_month >= 50:
score += 25
elif commits_last_month >= 25:
score += 20
elif commits_last_month >= 10:
score += 15
else:
score += max(0, commits_last_month * 1.5)
return min(100.0, score)
def _score_npm_health(self) -> float:
"""
Score npm package health (if applicable).
Returns:
npm health score (0-100)
"""
if not self.npm_data:
return 50.0 # Neutral score if not applicable
score = 0.0
# Weekly downloads (0-40 points)
weekly_downloads = self.npm_data.get('weekly_downloads', 0)
if weekly_downloads >= 1000000:
score += 40
elif weekly_downloads >= 500000:
score += 35
elif weekly_downloads >= 100000:
score += 30
elif weekly_downloads >= 50000:
score += 25
elif weekly_downloads >= 10000:
score += 20
else:
score += max(0, weekly_downloads / 500)
# Version stability (0-20 points)
version = self.npm_data.get('version', '0.0.1')
major_version = int(version.split('.')[0]) if version else 0
if major_version >= 5:
score += 20
elif major_version >= 3:
score += 15
elif major_version >= 1:
score += 10
else:
score += 5
# Dependencies count (0-20 points, fewer is better)
dependencies = self.npm_data.get('dependencies_count', 50)
if dependencies <= 10:
score += 20
elif dependencies <= 25:
score += 15
elif dependencies <= 50:
score += 10
else:
score += max(0, 20 - (dependencies - 50) / 10)
# Last publish date (0-20 points)
days_since_publish = self.npm_data.get('days_since_last_publish', 365)
if days_since_publish <= 30:
score += 20
elif days_since_publish <= 90:
score += 15
elif days_since_publish <= 180:
score += 10
elif days_since_publish <= 365:
score += 5
else:
score += 0
return min(100.0, score)
def _score_community_health(self) -> float:
"""
Score community health and engagement.
Returns:
Community health score (0-100)
"""
score = 0.0
# Stack Overflow questions (0-25 points)
so_questions = self.community_data.get('stackoverflow_questions', 0)
if so_questions >= 50000:
score += 25
elif so_questions >= 20000:
score += 20
elif so_questions >= 10000:
score += 15
elif so_questions >= 5000:
score += 10
else:
score += max(0, so_questions / 500)
# Job postings (0-25 points)
job_postings = self.community_data.get('job_postings', 0)
if job_postings >= 5000:
score += 25
elif job_postings >= 2000:
score += 20
elif job_postings >= 1000:
score += 15
elif job_postings >= 500:
score += 10
else:
score += max(0, job_postings / 50)
# Tutorials and resources (0-25 points)
tutorials = self.community_data.get('tutorials_count', 0)
if tutorials >= 1000:
score += 25
elif tutorials >= 500:
score += 20
elif tutorials >= 200:
score += 15
elif tutorials >= 100:
score += 10
else:
score += max(0, tutorials / 10)
# Active forums/Discord (0-25 points)
forum_members = self.community_data.get('forum_members', 0)
if forum_members >= 50000:
score += 25
elif forum_members >= 20000:
score += 20
elif forum_members >= 10000:
score += 15
elif forum_members >= 5000:
score += 10
else:
score += max(0, forum_members / 500)
return min(100.0, score)
def _score_corporate_backing(self) -> float:
"""
Score corporate backing strength.
Returns:
Corporate backing score (0-100)
"""
backing_type = self.corporate_backing.get('type', 'none')
scores = {
'major_tech_company': 100, # Google, Microsoft, Meta, etc.
'established_company': 80, # Dedicated company (Vercel, HashiCorp)
'startup_backed': 60, # Funded startup
'community_led': 40, # Strong community, no corporate backing
'none': 20 # Individual maintainers
}
base_score = scores.get(backing_type, 40)
# Adjust for funding
funding = self.corporate_backing.get('funding_millions', 0)
if funding >= 100:
base_score = min(100, base_score + 20)
elif funding >= 50:
base_score = min(100, base_score + 10)
elif funding >= 10:
base_score = min(100, base_score + 5)
return base_score
def _score_maintenance_health(self) -> float:
"""
Score maintenance activity and responsiveness.
Returns:
Maintenance health score (0-100)
"""
score = 0.0
# Issue response time (0-30 points)
avg_response_hours = self.github_data.get('avg_issue_response_hours', 168) # 7 days default
if avg_response_hours <= 24:
score += 30
elif avg_response_hours <= 48:
score += 25
elif avg_response_hours <= 168: # 1 week
score += 20
elif avg_response_hours <= 336: # 2 weeks
score += 10
else:
score += 5
# Issue resolution rate (0-30 points)
resolution_rate = self.github_data.get('issue_resolution_rate', 0.5)
score += resolution_rate * 30
# Release frequency (0-20 points)
releases_per_year = self.github_data.get('releases_per_year', 4)
if releases_per_year >= 12:
score += 20
elif releases_per_year >= 6:
score += 15
elif releases_per_year >= 4:
score += 10
elif releases_per_year >= 2:
score += 5
else:
score += 0
# Active maintainers (0-20 points)
active_maintainers = self.github_data.get('active_maintainers', 1)
if active_maintainers >= 10:
score += 20
elif active_maintainers >= 5:
score += 15
elif active_maintainers >= 3:
score += 10
elif active_maintainers >= 1:
score += 5
else:
score += 0
return min(100.0, score)
def assess_viability(self) -> Dict[str, Any]:
"""
Assess long-term viability of technology.
Returns:
Viability assessment with risk factors
"""
health = self.calculate_health_score()
overall_health = health['overall_health']
# Determine viability level
if overall_health >= 80:
viability = "Excellent - Strong long-term viability"
risk_level = "Low"
elif overall_health >= 65:
viability = "Good - Solid viability with minor concerns"
risk_level = "Low-Medium"
elif overall_health >= 50:
viability = "Moderate - Viable but with notable risks"
risk_level = "Medium"
elif overall_health >= 35:
viability = "Concerning - Significant viability risks"
risk_level = "Medium-High"
else:
viability = "Poor - High risk of abandonment"
risk_level = "High"
# Identify specific risks
risks = self._identify_viability_risks(health)
# Identify strengths
strengths = self._identify_viability_strengths(health)
return {
'overall_viability': viability,
'risk_level': risk_level,
'health_score': overall_health,
'risks': risks,
'strengths': strengths,
'recommendation': self._generate_viability_recommendation(overall_health, risks)
}
def _identify_viability_risks(self, health: Dict[str, float]) -> List[str]:
"""
Identify viability risks from health scores.
Args:
health: Health score components
Returns:
List of identified risks
"""
risks = []
if health['maintenance_health'] < 50:
risks.append("Low maintenance activity - slow issue resolution")
if health['github_health'] < 50:
risks.append("Limited GitHub activity - smaller community")
if health['corporate_backing'] < 40:
risks.append("Weak corporate backing - sustainability concerns")
if health['npm_health'] < 50 and self.npm_data:
risks.append("Low npm adoption - limited ecosystem")
if health['community_health'] < 50:
risks.append("Small community - limited resources and support")
return risks if risks else ["No significant risks identified"]
def _identify_viability_strengths(self, health: Dict[str, float]) -> List[str]:
"""
Identify viability strengths from health scores.
Args:
health: Health score components
Returns:
List of identified strengths
"""
strengths = []
if health['maintenance_health'] >= 70:
strengths.append("Active maintenance with responsive issue resolution")
if health['github_health'] >= 70:
strengths.append("Strong GitHub presence with active community")
if health['corporate_backing'] >= 70:
strengths.append("Strong corporate backing ensures sustainability")
if health['npm_health'] >= 70 and self.npm_data:
strengths.append("High npm adoption with stable releases")
if health['community_health'] >= 70:
strengths.append("Large, active community with extensive resources")
return strengths if strengths else ["Baseline viability maintained"]
def _generate_viability_recommendation(self, health_score: float, risks: List[str]) -> str:
"""
Generate viability recommendation.
Args:
health_score: Overall health score
risks: List of identified risks
Returns:
Recommendation string
"""
if health_score >= 80:
return "Recommended for long-term adoption - strong ecosystem support"
elif health_score >= 65:
return "Suitable for adoption - monitor identified risks"
elif health_score >= 50:
return "Proceed with caution - have contingency plans"
else:
return "Not recommended - consider alternatives with stronger ecosystems"
def generate_ecosystem_report(self) -> Dict[str, Any]:
"""
Generate comprehensive ecosystem report.
Returns:
Complete ecosystem analysis
"""
health = self.calculate_health_score()
viability = self.assess_viability()
return {
'technology': self.technology,
'health_scores': health,
'viability_assessment': viability,
'github_metrics': self._format_github_metrics(),
'npm_metrics': self._format_npm_metrics() if self.npm_data else None,
'community_metrics': self._format_community_metrics()
}
def _format_github_metrics(self) -> Dict[str, Any]:
"""Format GitHub metrics for reporting."""
return {
'stars': f"{self.github_data.get('stars', 0):,}",
'forks': f"{self.github_data.get('forks', 0):,}",
'contributors': f"{self.github_data.get('contributors', 0):,}",
'commits_last_month': self.github_data.get('commits_last_month', 0),
'open_issues': self.github_data.get('open_issues', 0),
'issue_resolution_rate': f"{self.github_data.get('issue_resolution_rate', 0) * 100:.1f}%"
}
def _format_npm_metrics(self) -> Dict[str, Any]:
"""Format npm metrics for reporting."""
return {
'weekly_downloads': f"{self.npm_data.get('weekly_downloads', 0):,}",
'version': self.npm_data.get('version', 'N/A'),
'dependencies': self.npm_data.get('dependencies_count', 0),
'days_since_publish': self.npm_data.get('days_since_last_publish', 0)
}
def _format_community_metrics(self) -> Dict[str, Any]:
"""Format community metrics for reporting."""
return {
'stackoverflow_questions': f"{self.community_data.get('stackoverflow_questions', 0):,}",
'job_postings': f"{self.community_data.get('job_postings', 0):,}",
'tutorials': self.community_data.get('tutorials_count', 0),
'forum_members': f"{self.community_data.get('forum_members', 0):,}"
}
FILE:scripts/format_detector.py
"""
Input Format Detector.
Automatically detects input format (text, YAML, JSON, URLs) and parses
accordingly for technology stack evaluation requests.
"""
from typing import Dict, Any, Optional, Tuple
import json
import re
class FormatDetector:
"""Detect and parse various input formats for stack evaluation."""
def __init__(self, input_data: str):
"""
Initialize format detector with raw input.
Args:
input_data: Raw input string from user
"""
self.raw_input = input_data.strip()
self.detected_format = None
self.parsed_data = None
def detect_format(self) -> str:
"""
Detect the input format.
Returns:
Format type: 'json', 'yaml', 'url', 'text'
"""
# Try JSON first
if self._is_json():
self.detected_format = 'json'
return 'json'
# Try YAML
if self._is_yaml():
self.detected_format = 'yaml'
return 'yaml'
# Check for URLs
if self._contains_urls():
self.detected_format = 'url'
return 'url'
# Default to conversational text
self.detected_format = 'text'
return 'text'
def _is_json(self) -> bool:
"""Check if input is valid JSON."""
try:
json.loads(self.raw_input)
return True
except (json.JSONDecodeError, ValueError):
return False
def _is_yaml(self) -> bool:
"""
Check if input looks like YAML.
Returns:
True if input appears to be YAML format
"""
# YAML indicators
yaml_patterns = [
r'^\s*[\w\-]+\s*:', # Key-value pairs
r'^\s*-\s+', # List items
r':\s*$', # Trailing colons
]
# Must not be JSON
if self._is_json():
return False
# Check for YAML patterns
lines = self.raw_input.split('\n')
yaml_line_count = 0
for line in lines:
for pattern in yaml_patterns:
if re.match(pattern, line):
yaml_line_count += 1
break
# If >50% of lines match YAML patterns, consider it YAML
if len(lines) > 0 and yaml_line_count / len(lines) > 0.5:
return True
return False
def _contains_urls(self) -> bool:
"""Check if input contains URLs."""
url_pattern = r'https?://[^\s]+'
return bool(re.search(url_pattern, self.raw_input))
def parse(self) -> Dict[str, Any]:
"""
Parse input based on detected format.
Returns:
Parsed data dictionary
"""
if self.detected_format is None:
self.detect_format()
if self.detected_format == 'json':
self.parsed_data = self._parse_json()
elif self.detected_format == 'yaml':
self.parsed_data = self._parse_yaml()
elif self.detected_format == 'url':
self.parsed_data = self._parse_urls()
else: # text
self.parsed_data = self._parse_text()
return self.parsed_data
def _parse_json(self) -> Dict[str, Any]:
"""Parse JSON input."""
try:
data = json.loads(self.raw_input)
return self._normalize_structure(data)
except json.JSONDecodeError:
return {'error': 'Invalid JSON', 'raw': self.raw_input}
def _parse_yaml(self) -> Dict[str, Any]:
"""
Parse YAML-like input (simplified, no external dependencies).
Returns:
Parsed dictionary
"""
result = {}
current_section = None
current_list = None
lines = self.raw_input.split('\n')
for line in lines:
stripped = line.strip()
if not stripped or stripped.startswith('#'):
continue
# Key-value pair
if ':' in stripped:
key, value = stripped.split(':', 1)
key = key.strip()
value = value.strip()
# Empty value might indicate nested structure
if not value:
current_section = key
result[current_section] = {}
current_list = None
else:
if current_section:
result[current_section][key] = self._parse_value(value)
else:
result[key] = self._parse_value(value)
# List item
elif stripped.startswith('-'):
item = stripped[1:].strip()
if current_section:
if current_list is None:
current_list = []
result[current_section] = current_list
current_list.append(self._parse_value(item))
return self._normalize_structure(result)
def _parse_value(self, value: str) -> Any:
"""
Parse a value string to appropriate type.
Args:
value: Value string
Returns:
Parsed value (str, int, float, bool)
"""
value = value.strip()
# Boolean
if value.lower() in ['true', 'yes']:
return True
if value.lower() in ['false', 'no']:
return False
# Number
try:
if '.' in value:
return float(value)
else:
return int(value)
except ValueError:
pass
# String (remove quotes if present)
if value.startswith('"') and value.endswith('"'):
return value[1:-1]
if value.startswith("'") and value.endswith("'"):
return value[1:-1]
return value
def _parse_urls(self) -> Dict[str, Any]:
"""Parse URLs from input."""
url_pattern = r'https?://[^\s]+'
urls = re.findall(url_pattern, self.raw_input)
# Categorize URLs
github_urls = [u for u in urls if 'github.com' in u]
npm_urls = [u for u in urls if 'npmjs.com' in u or 'npm.io' in u]
other_urls = [u for u in urls if u not in github_urls and u not in npm_urls]
# Also extract any text context
text_without_urls = re.sub(url_pattern, '', self.raw_input).strip()
result = {
'format': 'url',
'urls': {
'github': github_urls,
'npm': npm_urls,
'other': other_urls
},
'context': text_without_urls
}
return self._normalize_structure(result)
def _parse_text(self) -> Dict[str, Any]:
"""Parse conversational text input."""
text = self.raw_input.lower()
# Extract technologies being compared
technologies = self._extract_technologies(text)
# Extract use case
use_case = self._extract_use_case(text)
# Extract priorities
priorities = self._extract_priorities(text)
# Detect analysis type
analysis_type = self._detect_analysis_type(text)
result = {
'format': 'text',
'technologies': technologies,
'use_case': use_case,
'priorities': priorities,
'analysis_type': analysis_type,
'raw_text': self.raw_input
}
return self._normalize_structure(result)
def _extract_technologies(self, text: str) -> list:
"""
Extract technology names from text.
Args:
text: Lowercase text
Returns:
List of identified technologies
"""
# Common technologies pattern
tech_keywords = [
'react', 'vue', 'angular', 'svelte', 'next.js', 'nuxt.js',
'node.js', 'python', 'java', 'go', 'rust', 'ruby',
'postgresql', 'postgres', 'mysql', 'mongodb', 'redis',
'aws', 'azure', 'gcp', 'google cloud',
'docker', 'kubernetes', 'k8s',
'express', 'fastapi', 'django', 'flask', 'spring boot'
]
found = []
for tech in tech_keywords:
if tech in text:
# Normalize names
normalized = {
'postgres': 'PostgreSQL',
'next.js': 'Next.js',
'nuxt.js': 'Nuxt.js',
'node.js': 'Node.js',
'k8s': 'Kubernetes',
'gcp': 'Google Cloud Platform'
}.get(tech, tech.title())
if normalized not in found:
found.append(normalized)
return found if found else ['Unknown']
def _extract_use_case(self, text: str) -> str:
"""
Extract use case description from text.
Args:
text: Lowercase text
Returns:
Use case description
"""
use_case_keywords = {
'real-time': 'Real-time application',
'collaboration': 'Collaboration platform',
'saas': 'SaaS application',
'dashboard': 'Dashboard application',
'api': 'API-heavy application',
'data-intensive': 'Data-intensive application',
'e-commerce': 'E-commerce platform',
'enterprise': 'Enterprise application'
}
for keyword, description in use_case_keywords.items():
if keyword in text:
return description
return 'General purpose application'
def _extract_priorities(self, text: str) -> list:
"""
Extract priority criteria from text.
Args:
text: Lowercase text
Returns:
List of priorities
"""
priority_keywords = {
'performance': 'Performance',
'scalability': 'Scalability',
'developer experience': 'Developer experience',
'ecosystem': 'Ecosystem',
'learning curve': 'Learning curve',
'cost': 'Cost',
'security': 'Security',
'compliance': 'Compliance'
}
priorities = []
for keyword, priority in priority_keywords.items():
if keyword in text:
priorities.append(priority)
return priorities if priorities else ['Developer experience', 'Performance']
def _detect_analysis_type(self, text: str) -> str:
"""
Detect type of analysis requested.
Args:
text: Lowercase text
Returns:
Analysis type
"""
type_keywords = {
'migration': 'migration_analysis',
'migrate': 'migration_analysis',
'tco': 'tco_analysis',
'total cost': 'tco_analysis',
'security': 'security_analysis',
'compliance': 'security_analysis',
'compare': 'comparison',
'vs': 'comparison',
'evaluate': 'evaluation'
}
for keyword, analysis_type in type_keywords.items():
if keyword in text:
return analysis_type
return 'comparison' # Default
def _normalize_structure(self, data: Dict[str, Any]) -> Dict[str, Any]:
"""
Normalize parsed data to standard structure.
Args:
data: Parsed data dictionary
Returns:
Normalized data structure
"""
# Ensure standard keys exist
standard_keys = [
'technologies',
'use_case',
'priorities',
'analysis_type',
'format'
]
normalized = data.copy()
for key in standard_keys:
if key not in normalized:
# Set defaults
defaults = {
'technologies': [],
'use_case': 'general',
'priorities': [],
'analysis_type': 'comparison',
'format': self.detected_format or 'unknown'
}
normalized[key] = defaults.get(key)
return normalized
def get_format_info(self) -> Dict[str, Any]:
"""
Get information about detected format.
Returns:
Format detection metadata
"""
return {
'detected_format': self.detected_format,
'input_length': len(self.raw_input),
'line_count': len(self.raw_input.split('\n')),
'parsing_successful': self.parsed_data is not None
}
FILE:scripts/migration_analyzer.py
"""
Migration Path Analyzer.
Analyzes migration complexity, risks, timelines, and strategies for moving
from legacy technology stacks to modern alternatives.
"""
from typing import Dict, List, Any, Optional, Tuple
class MigrationAnalyzer:
"""Analyze migration paths and complexity for technology stack changes."""
# Migration complexity factors
COMPLEXITY_FACTORS = [
'code_volume',
'architecture_changes',
'data_migration',
'api_compatibility',
'dependency_changes',
'testing_requirements'
]
def __init__(self, migration_data: Dict[str, Any]):
"""
Initialize migration analyzer with migration parameters.
Args:
migration_data: Dictionary containing source/target technologies and constraints
"""
self.source_tech = migration_data.get('source_technology', 'Unknown')
self.target_tech = migration_data.get('target_technology', 'Unknown')
self.codebase_stats = migration_data.get('codebase_stats', {})
self.constraints = migration_data.get('constraints', {})
self.team_info = migration_data.get('team', {})
def calculate_complexity_score(self) -> Dict[str, Any]:
"""
Calculate overall migration complexity (1-10 scale).
Returns:
Dictionary with complexity scores by factor
"""
scores = {
'code_volume': self._score_code_volume(),
'architecture_changes': self._score_architecture_changes(),
'data_migration': self._score_data_migration(),
'api_compatibility': self._score_api_compatibility(),
'dependency_changes': self._score_dependency_changes(),
'testing_requirements': self._score_testing_requirements()
}
# Calculate weighted average
weights = {
'code_volume': 0.20,
'architecture_changes': 0.25,
'data_migration': 0.20,
'api_compatibility': 0.15,
'dependency_changes': 0.10,
'testing_requirements': 0.10
}
overall = sum(scores[k] * weights[k] for k in scores.keys())
scores['overall_complexity'] = overall
return scores
def _score_code_volume(self) -> float:
"""
Score complexity based on codebase size.
Returns:
Code volume complexity score (1-10)
"""
lines_of_code = self.codebase_stats.get('lines_of_code', 10000)
num_files = self.codebase_stats.get('num_files', 100)
num_components = self.codebase_stats.get('num_components', 50)
# Score based on lines of code (primary factor)
if lines_of_code < 5000:
base_score = 2
elif lines_of_code < 20000:
base_score = 4
elif lines_of_code < 50000:
base_score = 6
elif lines_of_code < 100000:
base_score = 8
else:
base_score = 10
# Adjust for component count
if num_components > 200:
base_score = min(10, base_score + 1)
elif num_components > 500:
base_score = min(10, base_score + 2)
return float(base_score)
def _score_architecture_changes(self) -> float:
"""
Score complexity based on architectural changes.
Returns:
Architecture complexity score (1-10)
"""
arch_change_level = self.codebase_stats.get('architecture_change_level', 'moderate')
scores = {
'minimal': 2, # Same patterns, just different framework
'moderate': 5, # Some pattern changes, similar concepts
'significant': 7, # Different patterns, major refactoring
'complete': 10 # Complete rewrite, different paradigm
}
return float(scores.get(arch_change_level, 5))
def _score_data_migration(self) -> float:
"""
Score complexity based on data migration requirements.
Returns:
Data migration complexity score (1-10)
"""
has_database = self.codebase_stats.get('has_database', True)
if not has_database:
return 1.0
database_size_gb = self.codebase_stats.get('database_size_gb', 10)
schema_changes = self.codebase_stats.get('schema_changes_required', 'minimal')
data_transformation = self.codebase_stats.get('data_transformation_required', False)
# Base score from database size
if database_size_gb < 1:
score = 2
elif database_size_gb < 10:
score = 3
elif database_size_gb < 100:
score = 5
elif database_size_gb < 1000:
score = 7
else:
score = 9
# Adjust for schema changes
schema_adjustments = {
'none': 0,
'minimal': 1,
'moderate': 2,
'significant': 3
}
score += schema_adjustments.get(schema_changes, 1)
# Adjust for data transformation
if data_transformation:
score += 2
return min(10.0, float(score))
def _score_api_compatibility(self) -> float:
"""
Score complexity based on API compatibility.
Returns:
API compatibility complexity score (1-10)
"""
breaking_api_changes = self.codebase_stats.get('breaking_api_changes', 'some')
scores = {
'none': 1, # Fully compatible
'minimal': 3, # Few breaking changes
'some': 5, # Moderate breaking changes
'many': 7, # Significant breaking changes
'complete': 10 # Complete API rewrite
}
return float(scores.get(breaking_api_changes, 5))
def _score_dependency_changes(self) -> float:
"""
Score complexity based on dependency changes.
Returns:
Dependency complexity score (1-10)
"""
num_dependencies = self.codebase_stats.get('num_dependencies', 20)
dependencies_to_replace = self.codebase_stats.get('dependencies_to_replace', 5)
# Score based on replacement percentage
if num_dependencies == 0:
return 1.0
replacement_pct = (dependencies_to_replace / num_dependencies) * 100
if replacement_pct < 10:
return 2.0
elif replacement_pct < 25:
return 4.0
elif replacement_pct < 50:
return 6.0
elif replacement_pct < 75:
return 8.0
else:
return 10.0
def _score_testing_requirements(self) -> float:
"""
Score complexity based on testing requirements.
Returns:
Testing complexity score (1-10)
"""
test_coverage = self.codebase_stats.get('current_test_coverage', 0.5) # 0-1 scale
num_tests = self.codebase_stats.get('num_tests', 100)
# If good test coverage, easier migration (can verify)
if test_coverage >= 0.8:
base_score = 3
elif test_coverage >= 0.6:
base_score = 5
elif test_coverage >= 0.4:
base_score = 7
else:
base_score = 9 # Poor coverage = hard to verify migration
# Large test suites need updates
if num_tests > 500:
base_score = min(10, base_score + 1)
return float(base_score)
def estimate_effort(self) -> Dict[str, Any]:
"""
Estimate migration effort in person-hours and timeline.
Returns:
Dictionary with effort estimates
"""
complexity = self.calculate_complexity_score()
overall_complexity = complexity['overall_complexity']
# Base hours estimation
lines_of_code = self.codebase_stats.get('lines_of_code', 10000)
base_hours = lines_of_code / 50 # 50 lines per hour baseline
# Complexity multiplier
complexity_multiplier = 1 + (overall_complexity / 10)
estimated_hours = base_hours * complexity_multiplier
# Break down by phase
phases = self._calculate_phase_breakdown(estimated_hours)
# Calculate timeline
team_size = self.team_info.get('team_size', 3)
hours_per_week_per_dev = self.team_info.get('hours_per_week', 30) # Account for other work
total_dev_weeks = estimated_hours / (team_size * hours_per_week_per_dev)
total_calendar_weeks = total_dev_weeks * 1.2 # Buffer for blockers
return {
'total_hours': estimated_hours,
'total_person_months': estimated_hours / 160, # 160 hours per person-month
'phases': phases,
'estimated_timeline': {
'dev_weeks': total_dev_weeks,
'calendar_weeks': total_calendar_weeks,
'calendar_months': total_calendar_weeks / 4.33
},
'team_assumptions': {
'team_size': team_size,
'hours_per_week_per_dev': hours_per_week_per_dev
}
}
def _calculate_phase_breakdown(self, total_hours: float) -> Dict[str, Dict[str, float]]:
"""
Calculate effort breakdown by migration phase.
Args:
total_hours: Total estimated hours
Returns:
Hours breakdown by phase
"""
# Standard phase percentages
phase_percentages = {
'planning_and_prototyping': 0.15,
'core_migration': 0.45,
'testing_and_validation': 0.25,
'deployment_and_monitoring': 0.10,
'buffer_and_contingency': 0.05
}
phases = {}
for phase, percentage in phase_percentages.items():
hours = total_hours * percentage
phases[phase] = {
'hours': hours,
'person_weeks': hours / 40,
'percentage': f"{percentage * 100:.0f}%"
}
return phases
def assess_risks(self) -> Dict[str, List[Dict[str, str]]]:
"""
Identify and assess migration risks.
Returns:
Categorized risks with mitigation strategies
"""
complexity = self.calculate_complexity_score()
risks = {
'technical_risks': self._identify_technical_risks(complexity),
'business_risks': self._identify_business_risks(),
'team_risks': self._identify_team_risks()
}
return risks
def _identify_technical_risks(self, complexity: Dict[str, float]) -> List[Dict[str, str]]:
"""
Identify technical risks.
Args:
complexity: Complexity scores
Returns:
List of technical risks with mitigations
"""
risks = []
# API compatibility risks
if complexity['api_compatibility'] >= 7:
risks.append({
'risk': 'Breaking API changes may cause integration failures',
'severity': 'High',
'mitigation': 'Create compatibility layer; implement feature flags for gradual rollout'
})
# Data migration risks
if complexity['data_migration'] >= 7:
risks.append({
'risk': 'Data migration could cause data loss or corruption',
'severity': 'Critical',
'mitigation': 'Implement robust backup strategy; run parallel systems during migration; extensive validation'
})
# Architecture risks
if complexity['architecture_changes'] >= 8:
risks.append({
'risk': 'Major architectural changes increase risk of performance regression',
'severity': 'High',
'mitigation': 'Extensive performance testing; staged rollout; monitoring and alerting'
})
# Testing risks
if complexity['testing_requirements'] >= 7:
risks.append({
'risk': 'Inadequate test coverage may miss critical bugs',
'severity': 'Medium',
'mitigation': 'Improve test coverage before migration; automated regression testing; user acceptance testing'
})
if not risks:
risks.append({
'risk': 'Standard technical risks (bugs, edge cases)',
'severity': 'Low',
'mitigation': 'Standard QA processes and staged rollout'
})
return risks
def _identify_business_risks(self) -> List[Dict[str, str]]:
"""
Identify business risks.
Returns:
List of business risks with mitigations
"""
risks = []
# Downtime risk
downtime_tolerance = self.constraints.get('downtime_tolerance', 'low')
if downtime_tolerance == 'none':
risks.append({
'risk': 'Zero-downtime migration increases complexity and risk',
'severity': 'High',
'mitigation': 'Blue-green deployment; feature flags; gradual traffic migration'
})
# Feature parity risk
risks.append({
'risk': 'New implementation may lack feature parity',
'severity': 'Medium',
'mitigation': 'Comprehensive feature audit; prioritized feature list; clear communication'
})
# Timeline risk
risks.append({
'risk': 'Migration may take longer than estimated',
'severity': 'Medium',
'mitigation': 'Build in 20% buffer; regular progress reviews; scope management'
})
return risks
def _identify_team_risks(self) -> List[Dict[str, str]]:
"""
Identify team-related risks.
Returns:
List of team risks with mitigations
"""
risks = []
# Learning curve
team_experience = self.team_info.get('target_tech_experience', 'low')
if team_experience in ['low', 'none']:
risks.append({
'risk': 'Team lacks experience with target technology',
'severity': 'High',
'mitigation': 'Training program; hire experienced developers; external consulting'
})
# Team size
team_size = self.team_info.get('team_size', 3)
if team_size < 3:
risks.append({
'risk': 'Small team size may extend timeline',
'severity': 'Medium',
'mitigation': 'Consider augmenting team; reduce scope; extend timeline'
})
# Knowledge retention
risks.append({
'risk': 'Loss of institutional knowledge during migration',
'severity': 'Medium',
'mitigation': 'Comprehensive documentation; knowledge sharing sessions; pair programming'
})
return risks
def generate_migration_plan(self) -> Dict[str, Any]:
"""
Generate comprehensive migration plan.
Returns:
Complete migration plan with timeline and recommendations
"""
complexity = self.calculate_complexity_score()
effort = self.estimate_effort()
risks = self.assess_risks()
# Generate phased approach
approach = self._recommend_migration_approach(complexity['overall_complexity'])
# Generate recommendation
recommendation = self._generate_migration_recommendation(complexity, effort, risks)
return {
'source_technology': self.source_tech,
'target_technology': self.target_tech,
'complexity_analysis': complexity,
'effort_estimation': effort,
'risk_assessment': risks,
'recommended_approach': approach,
'overall_recommendation': recommendation,
'success_criteria': self._define_success_criteria()
}
def _recommend_migration_approach(self, complexity_score: float) -> Dict[str, Any]:
"""
Recommend migration approach based on complexity.
Args:
complexity_score: Overall complexity score
Returns:
Recommended approach details
"""
if complexity_score <= 3:
approach = 'direct_migration'
description = 'Direct migration - low complexity allows straightforward migration'
timeline_multiplier = 1.0
elif complexity_score <= 6:
approach = 'phased_migration'
description = 'Phased migration - migrate components incrementally to manage risk'
timeline_multiplier = 1.3
else:
approach = 'strangler_pattern'
description = 'Strangler pattern - gradually replace old system while running in parallel'
timeline_multiplier = 1.5
return {
'approach': approach,
'description': description,
'timeline_multiplier': timeline_multiplier,
'phases': self._generate_approach_phases(approach)
}
def _generate_approach_phases(self, approach: str) -> List[str]:
"""
Generate phase descriptions for migration approach.
Args:
approach: Migration approach type
Returns:
List of phase descriptions
"""
phases = {
'direct_migration': [
'Phase 1: Set up target environment and migrate configuration',
'Phase 2: Migrate codebase and dependencies',
'Phase 3: Migrate data with validation',
'Phase 4: Comprehensive testing',
'Phase 5: Cutover and monitoring'
],
'phased_migration': [
'Phase 1: Identify and prioritize components for migration',
'Phase 2: Migrate non-critical components first',
'Phase 3: Migrate core components with parallel running',
'Phase 4: Migrate critical components with rollback plan',
'Phase 5: Decommission old system'
],
'strangler_pattern': [
'Phase 1: Set up routing layer between old and new systems',
'Phase 2: Implement new features in target technology only',
'Phase 3: Gradually migrate existing features (lowest risk first)',
'Phase 4: Migrate high-risk components last with extensive testing',
'Phase 5: Complete migration and remove routing layer'
]
}
return phases.get(approach, phases['phased_migration'])
def _generate_migration_recommendation(
self,
complexity: Dict[str, float],
effort: Dict[str, Any],
risks: Dict[str, List[Dict[str, str]]]
) -> str:
"""
Generate overall migration recommendation.
Args:
complexity: Complexity analysis
effort: Effort estimation
risks: Risk assessment
Returns:
Recommendation string
"""
overall_complexity = complexity['overall_complexity']
timeline_months = effort['estimated_timeline']['calendar_months']
# Count high/critical severity risks
high_risk_count = sum(
1 for risk_list in risks.values()
for risk in risk_list
if risk['severity'] in ['High', 'Critical']
)
if overall_complexity <= 4 and high_risk_count <= 2:
return f"Recommended - Low complexity migration achievable in {timeline_months:.1f} months with manageable risks"
elif overall_complexity <= 7 and high_risk_count <= 4:
return f"Proceed with caution - Moderate complexity migration requiring {timeline_months:.1f} months and careful risk management"
else:
return f"High risk - Complex migration requiring {timeline_months:.1f} months. Consider: incremental approach, additional resources, or alternative solutions"
def _define_success_criteria(self) -> List[str]:
"""
Define success criteria for migration.
Returns:
List of success criteria
"""
return [
'Feature parity with current system',
'Performance equal or better than current system',
'Zero data loss or corruption',
'All tests passing (unit, integration, E2E)',
'Successful production deployment with <1% error rate',
'Team trained and comfortable with new technology',
'Documentation complete and up-to-date'
]
FILE:scripts/report_generator.py
"""
Report Generator - Context-aware report generation with progressive disclosure.
Generates reports adapted for Claude Desktop (rich markdown) or CLI (terminal-friendly),
with executive summaries and detailed breakdowns on demand.
"""
from typing import Dict, List, Any, Optional
import os
import platform
class ReportGenerator:
"""Generate context-aware technology evaluation reports."""
def __init__(self, report_data: Dict[str, Any], output_context: Optional[str] = None):
"""
Initialize report generator.
Args:
report_data: Complete evaluation data
output_context: 'desktop', 'cli', or None for auto-detect
"""
self.report_data = report_data
self.output_context = output_context or self._detect_context()
def _detect_context(self) -> str:
"""
Detect output context (Desktop vs CLI).
Returns:
Context type: 'desktop' or 'cli'
"""
# Check for Claude Desktop environment variables or indicators
# This is a simplified detection - actual implementation would check for
# Claude Desktop-specific environment variables
if os.getenv('CLAUDE_DESKTOP'):
return 'desktop'
# Check if running in terminal
if os.isatty(1): # stdout is a terminal
return 'cli'
# Default to desktop for rich formatting
return 'desktop'
def generate_executive_summary(self, max_tokens: int = 300) -> str:
"""
Generate executive summary (200-300 tokens).
Args:
max_tokens: Maximum tokens for summary
Returns:
Executive summary markdown
"""
summary_parts = []
# Title
technologies = self.report_data.get('technologies', [])
tech_names = ', '.join(technologies[:3]) # First 3
summary_parts.append(f"# Technology Evaluation: {tech_names}\n")
# Recommendation
recommendation = self.report_data.get('recommendation', {})
rec_text = recommendation.get('text', 'No recommendation available')
confidence = recommendation.get('confidence', 0)
summary_parts.append(f"## Recommendation\n")
summary_parts.append(f"**{rec_text}**\n")
summary_parts.append(f"*Confidence: {confidence:.0f}%*\n")
# Top 3 Pros
pros = recommendation.get('pros', [])[:3]
if pros:
summary_parts.append(f"\n### Top Strengths\n")
for pro in pros:
summary_parts.append(f"- {pro}\n")
# Top 3 Cons
cons = recommendation.get('cons', [])[:3]
if cons:
summary_parts.append(f"\n### Key Concerns\n")
for con in cons:
summary_parts.append(f"- {con}\n")
# Key Decision Factors
decision_factors = self.report_data.get('decision_factors', [])[:3]
if decision_factors:
summary_parts.append(f"\n### Decision Factors\n")
for factor in decision_factors:
category = factor.get('category', 'Unknown')
best = factor.get('best_performer', 'Unknown')
summary_parts.append(f"- **{category.replace('_', ' ').title()}**: {best}\n")
summary_parts.append(f"\n---\n")
summary_parts.append(f"*For detailed analysis, request full report sections*\n")
return ''.join(summary_parts)
def generate_full_report(self, sections: Optional[List[str]] = None) -> str:
"""
Generate complete report with selected sections.
Args:
sections: List of sections to include, or None for all
Returns:
Complete report markdown
"""
if sections is None:
sections = self._get_available_sections()
report_parts = []
# Title and metadata
report_parts.append(self._generate_title())
# Generate each requested section
for section in sections:
section_content = self._generate_section(section)
if section_content:
report_parts.append(section_content)
return '\n\n'.join(report_parts)
def _get_available_sections(self) -> List[str]:
"""
Get list of available report sections.
Returns:
List of section names
"""
sections = ['executive_summary']
if 'comparison_matrix' in self.report_data:
sections.append('comparison_matrix')
if 'tco_analysis' in self.report_data:
sections.append('tco_analysis')
if 'ecosystem_health' in self.report_data:
sections.append('ecosystem_health')
if 'security_assessment' in self.report_data:
sections.append('security_assessment')
if 'migration_analysis' in self.report_data:
sections.append('migration_analysis')
if 'performance_benchmarks' in self.report_data:
sections.append('performance_benchmarks')
return sections
def _generate_title(self) -> str:
"""Generate report title section."""
technologies = self.report_data.get('technologies', [])
tech_names = ' vs '.join(technologies)
use_case = self.report_data.get('use_case', 'General Purpose')
if self.output_context == 'desktop':
return f"""# Technology Stack Evaluation Report
**Technologies**: {tech_names}
**Use Case**: {use_case}
**Generated**: {self._get_timestamp()}
---
"""
else: # CLI
return f"""================================================================================
TECHNOLOGY STACK EVALUATION REPORT
================================================================================
Technologies: {tech_names}
Use Case: {use_case}
Generated: {self._get_timestamp()}
================================================================================
"""
def _generate_section(self, section_name: str) -> Optional[str]:
"""
Generate specific report section.
Args:
section_name: Name of section to generate
Returns:
Section markdown or None
"""
generators = {
'executive_summary': self._section_executive_summary,
'comparison_matrix': self._section_comparison_matrix,
'tco_analysis': self._section_tco_analysis,
'ecosystem_health': self._section_ecosystem_health,
'security_assessment': self._section_security_assessment,
'migration_analysis': self._section_migration_analysis,
'performance_benchmarks': self._section_performance_benchmarks
}
generator = generators.get(section_name)
if generator:
return generator()
return None
def _section_executive_summary(self) -> str:
"""Generate executive summary section."""
return self.generate_executive_summary()
def _section_comparison_matrix(self) -> str:
"""Generate comparison matrix section."""
matrix_data = self.report_data.get('comparison_matrix', [])
if not matrix_data:
return ""
if self.output_context == 'desktop':
return self._render_matrix_desktop(matrix_data)
else:
return self._render_matrix_cli(matrix_data)
def _render_matrix_desktop(self, matrix_data: List[Dict[str, Any]]) -> str:
"""Render comparison matrix for desktop (rich markdown table)."""
parts = ["## Comparison Matrix\n"]
if not matrix_data:
return ""
# Get technology names from first row
tech_names = list(matrix_data[0].get('scores', {}).keys())
# Build table header
header = "| Category | Weight |"
for tech in tech_names:
header += f" {tech} |"
parts.append(header)
# Separator
separator = "|----------|--------|"
separator += "--------|" * len(tech_names)
parts.append(separator)
# Rows
for row in matrix_data:
category = row.get('category', '').replace('_', ' ').title()
weight = row.get('weight', '')
scores = row.get('scores', {})
row_str = f"| {category} | {weight} |"
for tech in tech_names:
score = scores.get(tech, '0.0')
row_str += f" {score} |"
parts.append(row_str)
return '\n'.join(parts)
def _render_matrix_cli(self, matrix_data: List[Dict[str, Any]]) -> str:
"""Render comparison matrix for CLI (ASCII table)."""
parts = ["COMPARISON MATRIX", "=" * 80, ""]
if not matrix_data:
return ""
# Get technology names
tech_names = list(matrix_data[0].get('scores', {}).keys())
# Calculate column widths
category_width = 25
weight_width = 8
score_width = 10
# Header
header = f"{'Category':<{category_width}} {'Weight':<{weight_width}}"
for tech in tech_names:
header += f" {tech[:score_width-1]:<{score_width}}"
parts.append(header)
parts.append("-" * 80)
# Rows
for row in matrix_data:
category = row.get('category', '').replace('_', ' ').title()[:category_width-1]
weight = row.get('weight', '')
scores = row.get('scores', {})
row_str = f"{category:<{category_width}} {weight:<{weight_width}}"
for tech in tech_names:
score = scores.get(tech, '0.0')
row_str += f" {score:<{score_width}}"
parts.append(row_str)
return '\n'.join(parts)
def _section_tco_analysis(self) -> str:
"""Generate TCO analysis section."""
tco_data = self.report_data.get('tco_analysis', {})
if not tco_data:
return ""
parts = ["## Total Cost of Ownership Analysis\n"]
# Summary
total_tco = tco_data.get('total_tco', 0)
timeline = tco_data.get('timeline_years', 5)
avg_yearly = tco_data.get('average_yearly_cost', 0)
parts.append(f"**{timeline}-Year Total**: ,.2f")
parts.append(f"**Average Yearly**: ,.2f\n")
# Cost breakdown
initial = tco_data.get('initial_costs', {})
parts.append(f"### Initial Costs: ,.2f")
# Operational costs
operational = tco_data.get('operational_costs', {})
if operational:
parts.append(f"\n### Operational Costs (Yearly)")
yearly_totals = operational.get('total_yearly', [])
for year, cost in enumerate(yearly_totals, 1):
parts.append(f"- Year {year}: ,.2f")
return '\n'.join(parts)
def _section_ecosystem_health(self) -> str:
"""Generate ecosystem health section."""
ecosystem_data = self.report_data.get('ecosystem_health', {})
if not ecosystem_data:
return ""
parts = ["## Ecosystem Health Analysis\n"]
# Overall score
overall_score = ecosystem_data.get('overall_health', 0)
parts.append(f"**Overall Health Score**: {overall_score:.1f}/100\n")
# Component scores
scores = ecosystem_data.get('health_scores', {})
parts.append("### Health Metrics")
for metric, score in scores.items():
if metric != 'overall_health':
metric_name = metric.replace('_', ' ').title()
parts.append(f"- {metric_name}: {score:.1f}/100")
# Viability assessment
viability = ecosystem_data.get('viability_assessment', {})
if viability:
parts.append(f"\n### Viability: {viability.get('overall_viability', 'Unknown')}")
parts.append(f"**Risk Level**: {viability.get('risk_level', 'Unknown')}")
return '\n'.join(parts)
def _section_security_assessment(self) -> str:
"""Generate security assessment section."""
security_data = self.report_data.get('security_assessment', {})
if not security_data:
return ""
parts = ["## Security & Compliance Assessment\n"]
# Security score
security_score = security_data.get('security_score', {})
overall = security_score.get('overall_security_score', 0)
grade = security_score.get('security_grade', 'N/A')
parts.append(f"**Security Score**: {overall:.1f}/100 (Grade: {grade})\n")
# Compliance
compliance = security_data.get('compliance_assessment', {})
if compliance:
parts.append("### Compliance Readiness")
for standard, assessment in compliance.items():
level = assessment.get('readiness_level', 'Unknown')
pct = assessment.get('readiness_percentage', 0)
parts.append(f"- **{standard}**: {level} ({pct:.0f}%)")
return '\n'.join(parts)
def _section_migration_analysis(self) -> str:
"""Generate migration analysis section."""
migration_data = self.report_data.get('migration_analysis', {})
if not migration_data:
return ""
parts = ["## Migration Path Analysis\n"]
# Complexity
complexity = migration_data.get('complexity_analysis', {})
overall_complexity = complexity.get('overall_complexity', 0)
parts.append(f"**Migration Complexity**: {overall_complexity:.1f}/10\n")
# Effort estimation
effort = migration_data.get('effort_estimation', {})
if effort:
total_hours = effort.get('total_hours', 0)
person_months = effort.get('total_person_months', 0)
timeline = effort.get('estimated_timeline', {})
calendar_months = timeline.get('calendar_months', 0)
parts.append(f"### Effort Estimate")
parts.append(f"- Total Effort: {person_months:.1f} person-months ({total_hours:.0f} hours)")
parts.append(f"- Timeline: {calendar_months:.1f} calendar months")
# Recommended approach
approach = migration_data.get('recommended_approach', {})
if approach:
parts.append(f"\n### Recommended Approach: {approach.get('approach', 'Unknown').replace('_', ' ').title()}")
parts.append(f"{approach.get('description', '')}")
return '\n'.join(parts)
def _section_performance_benchmarks(self) -> str:
"""Generate performance benchmarks section."""
benchmark_data = self.report_data.get('performance_benchmarks', {})
if not benchmark_data:
return ""
parts = ["## Performance Benchmarks\n"]
# Throughput
throughput = benchmark_data.get('throughput', {})
if throughput:
parts.append("### Throughput")
for tech, rps in throughput.items():
parts.append(f"- {tech}: {rps:,} requests/sec")
# Latency
latency = benchmark_data.get('latency', {})
if latency:
parts.append("\n### Latency (P95)")
for tech, ms in latency.items():
parts.append(f"- {tech}: {ms}ms")
return '\n'.join(parts)
def _get_timestamp(self) -> str:
"""Get current timestamp."""
from datetime import datetime
return datetime.now().strftime("%Y-%m-%d %H:%M")
def export_to_file(self, filename: str, sections: Optional[List[str]] = None) -> str:
"""
Export report to file.
Args:
filename: Output filename
sections: Sections to include
Returns:
Path to exported file
"""
report = self.generate_full_report(sections)
with open(filename, 'w', encoding='utf-8') as f:
f.write(report)
return filename
FILE:scripts/security_assessor.py
"""
Security and Compliance Assessor.
Analyzes security vulnerabilities, compliance readiness (GDPR, SOC2, HIPAA),
and overall security posture of technology stacks.
"""
from typing import Dict, List, Any, Optional
from datetime import datetime, timedelta
class SecurityAssessor:
"""Assess security and compliance readiness of technology stacks."""
# Compliance standards mapping
COMPLIANCE_STANDARDS = {
'GDPR': ['data_privacy', 'consent_management', 'data_portability', 'right_to_deletion', 'audit_logging'],
'SOC2': ['access_controls', 'encryption_at_rest', 'encryption_in_transit', 'audit_logging', 'backup_recovery'],
'HIPAA': ['phi_protection', 'encryption_at_rest', 'encryption_in_transit', 'access_controls', 'audit_logging'],
'PCI_DSS': ['payment_data_encryption', 'access_controls', 'network_security', 'vulnerability_management']
}
def __init__(self, security_data: Dict[str, Any]):
"""
Initialize security assessor with security data.
Args:
security_data: Dictionary containing vulnerability and compliance data
"""
self.technology = security_data.get('technology', 'Unknown')
self.vulnerabilities = security_data.get('vulnerabilities', {})
self.security_features = security_data.get('security_features', {})
self.compliance_requirements = security_data.get('compliance_requirements', [])
def calculate_security_score(self) -> Dict[str, Any]:
"""
Calculate overall security score (0-100).
Returns:
Dictionary with security score components
"""
# Component scores
vuln_score = self._score_vulnerabilities()
patch_score = self._score_patch_responsiveness()
features_score = self._score_security_features()
track_record_score = self._score_track_record()
# Weighted average
weights = {
'vulnerability_score': 0.30,
'patch_responsiveness': 0.25,
'security_features': 0.30,
'track_record': 0.15
}
overall = (
vuln_score * weights['vulnerability_score'] +
patch_score * weights['patch_responsiveness'] +
features_score * weights['security_features'] +
track_record_score * weights['track_record']
)
return {
'overall_security_score': overall,
'vulnerability_score': vuln_score,
'patch_responsiveness': patch_score,
'security_features_score': features_score,
'track_record_score': track_record_score,
'security_grade': self._calculate_grade(overall)
}
def _score_vulnerabilities(self) -> float:
"""
Score based on vulnerability count and severity.
Returns:
Vulnerability score (0-100, higher is better)
"""
# Get vulnerability counts by severity (last 12 months)
critical = self.vulnerabilities.get('critical_last_12m', 0)
high = self.vulnerabilities.get('high_last_12m', 0)
medium = self.vulnerabilities.get('medium_last_12m', 0)
low = self.vulnerabilities.get('low_last_12m', 0)
# Calculate weighted vulnerability count
weighted_vulns = (critical * 4) + (high * 2) + (medium * 1) + (low * 0.5)
# Score based on weighted count (fewer is better)
if weighted_vulns == 0:
score = 100
elif weighted_vulns <= 5:
score = 90
elif weighted_vulns <= 10:
score = 80
elif weighted_vulns <= 20:
score = 70
elif weighted_vulns <= 30:
score = 60
elif weighted_vulns <= 50:
score = 50
else:
score = max(0, 50 - (weighted_vulns - 50) / 2)
# Penalty for critical vulnerabilities
if critical > 0:
score = max(0, score - (critical * 10))
return max(0.0, min(100.0, score))
def _score_patch_responsiveness(self) -> float:
"""
Score based on patch response time.
Returns:
Patch responsiveness score (0-100)
"""
# Average days to patch critical vulnerabilities
critical_patch_days = self.vulnerabilities.get('avg_critical_patch_days', 30)
high_patch_days = self.vulnerabilities.get('avg_high_patch_days', 60)
# Score critical patch time (most important)
if critical_patch_days <= 7:
critical_score = 50
elif critical_patch_days <= 14:
critical_score = 40
elif critical_patch_days <= 30:
critical_score = 30
elif critical_patch_days <= 60:
critical_score = 20
else:
critical_score = 10
# Score high severity patch time
if high_patch_days <= 14:
high_score = 30
elif high_patch_days <= 30:
high_score = 25
elif high_patch_days <= 60:
high_score = 20
elif high_patch_days <= 90:
high_score = 15
else:
high_score = 10
# Has active security team
has_security_team = self.vulnerabilities.get('has_security_team', False)
team_score = 20 if has_security_team else 0
total_score = critical_score + high_score + team_score
return min(100.0, total_score)
def _score_security_features(self) -> float:
"""
Score based on built-in security features.
Returns:
Security features score (0-100)
"""
score = 0.0
# Essential features (10 points each)
essential_features = [
'encryption_at_rest',
'encryption_in_transit',
'authentication',
'authorization',
'input_validation'
]
for feature in essential_features:
if self.security_features.get(feature, False):
score += 10
# Advanced features (5 points each)
advanced_features = [
'rate_limiting',
'csrf_protection',
'xss_protection',
'sql_injection_protection',
'audit_logging',
'mfa_support',
'rbac',
'secrets_management',
'security_headers',
'cors_configuration'
]
for feature in advanced_features:
if self.security_features.get(feature, False):
score += 5
return min(100.0, score)
def _score_track_record(self) -> float:
"""
Score based on historical security track record.
Returns:
Track record score (0-100)
"""
score = 50.0 # Start at neutral
# Years since major security incident
years_since_major = self.vulnerabilities.get('years_since_major_incident', 5)
if years_since_major >= 3:
score += 30
elif years_since_major >= 1:
score += 15
else:
score -= 10
# Security certifications
has_certifications = self.vulnerabilities.get('has_security_certifications', False)
if has_certifications:
score += 20
# Bug bounty program
has_bug_bounty = self.vulnerabilities.get('has_bug_bounty_program', False)
if has_bug_bounty:
score += 10
# Security audits
security_audits = self.vulnerabilities.get('security_audits_per_year', 0)
score += min(20, security_audits * 10)
return min(100.0, max(0.0, score))
def _calculate_grade(self, score: float) -> str:
"""
Convert score to letter grade.
Args:
score: Security score (0-100)
Returns:
Letter grade
"""
if score >= 90:
return "A"
elif score >= 80:
return "B"
elif score >= 70:
return "C"
elif score >= 60:
return "D"
else:
return "F"
def assess_compliance(self, standards: List[str] = None) -> Dict[str, Dict[str, Any]]:
"""
Assess compliance readiness for specified standards.
Args:
standards: List of compliance standards to assess (defaults to all required)
Returns:
Dictionary of compliance assessments by standard
"""
if standards is None:
standards = self.compliance_requirements
results = {}
for standard in standards:
if standard not in self.COMPLIANCE_STANDARDS:
results[standard] = {
'readiness': 'Unknown',
'score': 0,
'status': 'Unknown standard'
}
continue
readiness = self._assess_standard_readiness(standard)
results[standard] = readiness
return results
def _assess_standard_readiness(self, standard: str) -> Dict[str, Any]:
"""
Assess readiness for a specific compliance standard.
Args:
standard: Compliance standard name
Returns:
Readiness assessment
"""
required_features = self.COMPLIANCE_STANDARDS[standard]
met_count = 0
total_count = len(required_features)
missing_features = []
for feature in required_features:
if self.security_features.get(feature, False):
met_count += 1
else:
missing_features.append(feature)
# Calculate readiness percentage
readiness_pct = (met_count / total_count * 100) if total_count > 0 else 0
# Determine readiness level
if readiness_pct >= 90:
readiness_level = "Ready"
status = "Compliant - meets all requirements"
elif readiness_pct >= 70:
readiness_level = "Mostly Ready"
status = "Minor gaps - additional configuration needed"
elif readiness_pct >= 50:
readiness_level = "Partial"
status = "Significant work required"
else:
readiness_level = "Not Ready"
status = "Major gaps - extensive implementation needed"
return {
'readiness_level': readiness_level,
'readiness_percentage': readiness_pct,
'status': status,
'features_met': met_count,
'features_required': total_count,
'missing_features': missing_features,
'recommendation': self._generate_compliance_recommendation(readiness_level, missing_features)
}
def _generate_compliance_recommendation(self, readiness_level: str, missing_features: List[str]) -> str:
"""
Generate compliance recommendation.
Args:
readiness_level: Current readiness level
missing_features: List of missing features
Returns:
Recommendation string
"""
if readiness_level == "Ready":
return "Proceed with compliance audit and certification"
elif readiness_level == "Mostly Ready":
return f"Implement missing features: {', '.join(missing_features[:3])}"
elif readiness_level == "Partial":
return f"Significant implementation needed. Start with: {', '.join(missing_features[:3])}"
else:
return "Not recommended without major security enhancements"
def identify_vulnerabilities(self) -> Dict[str, Any]:
"""
Identify and categorize vulnerabilities.
Returns:
Categorized vulnerability report
"""
# Current vulnerabilities
current = {
'critical': self.vulnerabilities.get('critical_last_12m', 0),
'high': self.vulnerabilities.get('high_last_12m', 0),
'medium': self.vulnerabilities.get('medium_last_12m', 0),
'low': self.vulnerabilities.get('low_last_12m', 0)
}
# Historical vulnerabilities (last 3 years)
historical = {
'critical': self.vulnerabilities.get('critical_last_3y', 0),
'high': self.vulnerabilities.get('high_last_3y', 0),
'medium': self.vulnerabilities.get('medium_last_3y', 0),
'low': self.vulnerabilities.get('low_last_3y', 0)
}
# Common vulnerability types
common_types = self.vulnerabilities.get('common_vulnerability_types', [
'SQL Injection',
'XSS',
'CSRF',
'Authentication Issues'
])
return {
'current_vulnerabilities': current,
'total_current': sum(current.values()),
'historical_vulnerabilities': historical,
'total_historical': sum(historical.values()),
'common_types': common_types,
'severity_distribution': self._calculate_severity_distribution(current),
'trend': self._analyze_vulnerability_trend(current, historical)
}
def _calculate_severity_distribution(self, vulnerabilities: Dict[str, int]) -> Dict[str, str]:
"""
Calculate percentage distribution of vulnerability severities.
Args:
vulnerabilities: Vulnerability counts by severity
Returns:
Percentage distribution
"""
total = sum(vulnerabilities.values())
if total == 0:
return {k: "0%" for k in vulnerabilities.keys()}
return {
severity: f"{(count / total * 100):.1f}%"
for severity, count in vulnerabilities.items()
}
def _analyze_vulnerability_trend(self, current: Dict[str, int], historical: Dict[str, int]) -> str:
"""
Analyze vulnerability trend.
Args:
current: Current vulnerabilities
historical: Historical vulnerabilities
Returns:
Trend description
"""
current_total = sum(current.values())
historical_avg = sum(historical.values()) / 3 # 3-year average
if current_total < historical_avg * 0.7:
return "Improving - fewer vulnerabilities than historical average"
elif current_total < historical_avg * 1.2:
return "Stable - consistent with historical average"
else:
return "Concerning - more vulnerabilities than historical average"
def generate_security_report(self) -> Dict[str, Any]:
"""
Generate comprehensive security assessment report.
Returns:
Complete security analysis
"""
security_score = self.calculate_security_score()
compliance = self.assess_compliance()
vulnerabilities = self.identify_vulnerabilities()
# Generate recommendations
recommendations = self._generate_security_recommendations(
security_score,
compliance,
vulnerabilities
)
return {
'technology': self.technology,
'security_score': security_score,
'compliance_assessment': compliance,
'vulnerability_analysis': vulnerabilities,
'recommendations': recommendations,
'overall_risk_level': self._determine_risk_level(security_score['overall_security_score'])
}
def _generate_security_recommendations(
self,
security_score: Dict[str, Any],
compliance: Dict[str, Dict[str, Any]],
vulnerabilities: Dict[str, Any]
) -> List[str]:
"""
Generate security recommendations.
Args:
security_score: Security score data
compliance: Compliance assessment
vulnerabilities: Vulnerability analysis
Returns:
List of recommendations
"""
recommendations = []
# Security score recommendations
if security_score['overall_security_score'] < 70:
recommendations.append("Improve overall security posture - score below acceptable threshold")
# Vulnerability recommendations
current_critical = vulnerabilities['current_vulnerabilities']['critical']
if current_critical > 0:
recommendations.append(f"Address {current_critical} critical vulnerabilities immediately")
# Patch responsiveness
if security_score['patch_responsiveness'] < 60:
recommendations.append("Improve vulnerability patch response time")
# Security features
if security_score['security_features_score'] < 70:
recommendations.append("Implement additional security features (MFA, audit logging, RBAC)")
# Compliance recommendations
for standard, assessment in compliance.items():
if assessment['readiness_level'] == "Not Ready":
recommendations.append(f"{standard}: {assessment['recommendation']}")
if not recommendations:
recommendations.append("Security posture is strong - continue monitoring and maintenance")
return recommendations
def _determine_risk_level(self, security_score: float) -> str:
"""
Determine overall risk level.
Args:
security_score: Overall security score
Returns:
Risk level description
"""
if security_score >= 85:
return "Low Risk - Strong security posture"
elif security_score >= 70:
return "Medium Risk - Acceptable with monitoring"
elif security_score >= 55:
return "High Risk - Security improvements needed"
else:
return "Critical Risk - Not recommended for production use"
FILE:scripts/stack_comparator.py
"""
Technology Stack Comparator - Main comparison engine with weighted scoring.
Provides comprehensive technology comparison with customizable weighted criteria,
feature matrices, and intelligent recommendation generation.
"""
from typing import Dict, List, Any, Optional, Tuple
import json
class StackComparator:
"""Main comparison engine for technology stack evaluation."""
# Feature categories for evaluation
FEATURE_CATEGORIES = [
"performance",
"scalability",
"developer_experience",
"ecosystem",
"learning_curve",
"documentation",
"community_support",
"enterprise_readiness"
]
# Default weights if not provided
DEFAULT_WEIGHTS = {
"performance": 15,
"scalability": 15,
"developer_experience": 20,
"ecosystem": 15,
"learning_curve": 10,
"documentation": 10,
"community_support": 10,
"enterprise_readiness": 5
}
def __init__(self, comparison_data: Dict[str, Any]):
"""
Initialize comparator with comparison data.
Args:
comparison_data: Dictionary containing technologies to compare and criteria
"""
self.technologies = comparison_data.get('technologies', [])
self.use_case = comparison_data.get('use_case', 'general')
self.priorities = comparison_data.get('priorities', {})
self.weights = self._normalize_weights(comparison_data.get('weights', {}))
self.scores = {}
def _normalize_weights(self, custom_weights: Dict[str, float]) -> Dict[str, float]:
"""
Normalize weights to sum to 100.
Args:
custom_weights: User-provided weights
Returns:
Normalized weights dictionary
"""
# Start with defaults
weights = self.DEFAULT_WEIGHTS.copy()
# Override with custom weights
weights.update(custom_weights)
# Normalize to 100
total = sum(weights.values())
if total == 0:
return self.DEFAULT_WEIGHTS
return {k: (v / total) * 100 for k, v in weights.items()}
def score_technology(self, tech_name: str, tech_data: Dict[str, Any]) -> Dict[str, float]:
"""
Score a single technology across all criteria.
Args:
tech_name: Name of technology
tech_data: Technology feature and metric data
Returns:
Dictionary of category scores (0-100 scale)
"""
scores = {}
for category in self.FEATURE_CATEGORIES:
# Get raw score from tech data (0-100 scale)
raw_score = tech_data.get(category, {}).get('score', 50.0)
# Apply use-case specific adjustments
adjusted_score = self._adjust_for_use_case(category, raw_score, tech_name)
scores[category] = min(100.0, max(0.0, adjusted_score))
return scores
def _adjust_for_use_case(self, category: str, score: float, tech_name: str) -> float:
"""
Apply use-case specific adjustments to scores.
Args:
category: Feature category
score: Raw score
tech_name: Technology name
Returns:
Adjusted score
"""
# Use case specific bonuses/penalties
adjustments = {
'real-time': {
'performance': 1.1, # 10% bonus for real-time use cases
'scalability': 1.1
},
'enterprise': {
'enterprise_readiness': 1.2, # 20% bonus
'documentation': 1.1
},
'startup': {
'developer_experience': 1.15,
'learning_curve': 1.1
}
}
# Determine use case type
use_case_lower = self.use_case.lower()
use_case_type = None
for uc_key in adjustments.keys():
if uc_key in use_case_lower:
use_case_type = uc_key
break
# Apply adjustment if applicable
if use_case_type and category in adjustments[use_case_type]:
multiplier = adjustments[use_case_type][category]
return score * multiplier
return score
def calculate_weighted_score(self, category_scores: Dict[str, float]) -> float:
"""
Calculate weighted total score.
Args:
category_scores: Dictionary of category scores
Returns:
Weighted total score (0-100 scale)
"""
total = 0.0
for category, score in category_scores.items():
weight = self.weights.get(category, 0.0) / 100.0 # Convert to decimal
total += score * weight
return total
def compare_technologies(self, tech_data_list: List[Dict[str, Any]]) -> Dict[str, Any]:
"""
Compare multiple technologies and generate recommendation.
Args:
tech_data_list: List of technology data dictionaries
Returns:
Comparison results with scores and recommendation
"""
results = {
'technologies': {},
'recommendation': None,
'confidence': 0.0,
'decision_factors': [],
'comparison_matrix': []
}
# Score each technology
tech_scores = {}
for tech_data in tech_data_list:
tech_name = tech_data.get('name', 'Unknown')
category_scores = self.score_technology(tech_name, tech_data)
weighted_score = self.calculate_weighted_score(category_scores)
tech_scores[tech_name] = {
'category_scores': category_scores,
'weighted_total': weighted_score,
'strengths': self._identify_strengths(category_scores),
'weaknesses': self._identify_weaknesses(category_scores)
}
results['technologies'] = tech_scores
# Generate recommendation
results['recommendation'], results['confidence'] = self._generate_recommendation(tech_scores)
results['decision_factors'] = self._extract_decision_factors(tech_scores)
results['comparison_matrix'] = self._build_comparison_matrix(tech_scores)
return results
def _identify_strengths(self, category_scores: Dict[str, float], threshold: float = 75.0) -> List[str]:
"""
Identify strength categories (scores above threshold).
Args:
category_scores: Category scores dictionary
threshold: Score threshold for strength identification
Returns:
List of strength categories
"""
return [
category for category, score in category_scores.items()
if score >= threshold
]
def _identify_weaknesses(self, category_scores: Dict[str, float], threshold: float = 50.0) -> List[str]:
"""
Identify weakness categories (scores below threshold).
Args:
category_scores: Category scores dictionary
threshold: Score threshold for weakness identification
Returns:
List of weakness categories
"""
return [
category for category, score in category_scores.items()
if score < threshold
]
def _generate_recommendation(self, tech_scores: Dict[str, Dict[str, Any]]) -> Tuple[str, float]:
"""
Generate recommendation and confidence level.
Args:
tech_scores: Technology scores dictionary
Returns:
Tuple of (recommended_technology, confidence_score)
"""
if not tech_scores:
return "Insufficient data", 0.0
# Sort by weighted total score
sorted_techs = sorted(
tech_scores.items(),
key=lambda x: x[1]['weighted_total'],
reverse=True
)
top_tech = sorted_techs[0][0]
top_score = sorted_techs[0][1]['weighted_total']
# Calculate confidence based on score gap
if len(sorted_techs) > 1:
second_score = sorted_techs[1][1]['weighted_total']
score_gap = top_score - second_score
# Confidence increases with score gap
# 0-5 gap: low confidence
# 5-15 gap: medium confidence
# 15+ gap: high confidence
if score_gap < 5:
confidence = 40.0 + (score_gap * 2) # 40-50%
elif score_gap < 15:
confidence = 50.0 + (score_gap - 5) * 2 # 50-70%
else:
confidence = 70.0 + min(score_gap - 15, 30) # 70-100%
else:
confidence = 100.0 # Only one option
return top_tech, min(100.0, confidence)
def _extract_decision_factors(self, tech_scores: Dict[str, Dict[str, Any]]) -> List[Dict[str, Any]]:
"""
Extract key decision factors from comparison.
Args:
tech_scores: Technology scores dictionary
Returns:
List of decision factors with importance weights
"""
factors = []
# Get top weighted categories
sorted_weights = sorted(
self.weights.items(),
key=lambda x: x[1],
reverse=True
)[:3] # Top 3 factors
for category, weight in sorted_weights:
# Get scores for this category across all techs
category_scores = {
tech: scores['category_scores'].get(category, 0.0)
for tech, scores in tech_scores.items()
}
# Find best performer
best_tech = max(category_scores.items(), key=lambda x: x[1])
factors.append({
'category': category,
'importance': f"{weight:.1f}%",
'best_performer': best_tech[0],
'score': best_tech[1]
})
return factors
def _build_comparison_matrix(self, tech_scores: Dict[str, Dict[str, Any]]) -> List[Dict[str, Any]]:
"""
Build comparison matrix for display.
Args:
tech_scores: Technology scores dictionary
Returns:
List of comparison matrix rows
"""
matrix = []
for category in self.FEATURE_CATEGORIES:
row = {
'category': category,
'weight': f"{self.weights.get(category, 0):.1f}%",
'scores': {}
}
for tech_name, scores in tech_scores.items():
category_score = scores['category_scores'].get(category, 0.0)
row['scores'][tech_name] = f"{category_score:.1f}"
matrix.append(row)
# Add weighted totals row
totals_row = {
'category': 'WEIGHTED TOTAL',
'weight': '100%',
'scores': {}
}
for tech_name, scores in tech_scores.items():
totals_row['scores'][tech_name] = f"{scores['weighted_total']:.1f}"
matrix.append(totals_row)
return matrix
def generate_pros_cons(self, tech_name: str, tech_scores: Dict[str, Any]) -> Dict[str, List[str]]:
"""
Generate pros and cons for a technology.
Args:
tech_name: Technology name
tech_scores: Technology scores dictionary
Returns:
Dictionary with 'pros' and 'cons' lists
"""
category_scores = tech_scores['category_scores']
strengths = tech_scores['strengths']
weaknesses = tech_scores['weaknesses']
pros = []
cons = []
# Generate pros from strengths
for strength in strengths[:3]: # Top 3
score = category_scores[strength]
pros.append(f"Excellent {strength.replace('_', ' ')} (score: {score:.1f}/100)")
# Generate cons from weaknesses
for weakness in weaknesses[:3]: # Top 3
score = category_scores[weakness]
cons.append(f"Weaker {weakness.replace('_', ' ')} (score: {score:.1f}/100)")
# Add generic pros/cons if not enough specific ones
if len(pros) == 0:
pros.append(f"Balanced performance across all categories")
if len(cons) == 0:
cons.append(f"No significant weaknesses identified")
return {'pros': pros, 'cons': cons}
FILE:scripts/tco_calculator.py
"""
Total Cost of Ownership (TCO) Calculator.
Calculates comprehensive TCO including licensing, hosting, developer productivity,
scaling costs, and hidden costs over multi-year projections.
"""
from typing import Dict, List, Any, Optional
import json
class TCOCalculator:
"""Calculate Total Cost of Ownership for technology stacks."""
def __init__(self, tco_data: Dict[str, Any]):
"""
Initialize TCO calculator with cost parameters.
Args:
tco_data: Dictionary containing cost parameters and projections
"""
self.technology = tco_data.get('technology', 'Unknown')
self.team_size = tco_data.get('team_size', 5)
self.timeline_years = tco_data.get('timeline_years', 5)
self.initial_costs = tco_data.get('initial_costs', {})
self.operational_costs = tco_data.get('operational_costs', {})
self.scaling_params = tco_data.get('scaling_params', {})
self.productivity_factors = tco_data.get('productivity_factors', {})
def calculate_initial_costs(self) -> Dict[str, float]:
"""
Calculate one-time initial costs.
Returns:
Dictionary of initial cost components
"""
costs = {
'licensing': self.initial_costs.get('licensing', 0.0),
'training': self._calculate_training_costs(),
'migration': self.initial_costs.get('migration', 0.0),
'setup': self.initial_costs.get('setup', 0.0),
'tooling': self.initial_costs.get('tooling', 0.0)
}
costs['total_initial'] = sum(costs.values())
return costs
def _calculate_training_costs(self) -> float:
"""
Calculate training costs based on team size and learning curve.
Returns:
Total training cost
"""
# Default training assumptions
hours_per_developer = self.initial_costs.get('training_hours_per_dev', 40)
avg_hourly_rate = self.initial_costs.get('developer_hourly_rate', 100)
training_materials = self.initial_costs.get('training_materials', 500)
total_hours = self.team_size * hours_per_developer
total_cost = (total_hours * avg_hourly_rate) + training_materials
return total_cost
def calculate_operational_costs(self) -> Dict[str, List[float]]:
"""
Calculate ongoing operational costs per year.
Returns:
Dictionary with yearly cost projections
"""
yearly_costs = {
'licensing': [],
'hosting': [],
'support': [],
'maintenance': [],
'total_yearly': []
}
for year in range(1, self.timeline_years + 1):
# Licensing costs (may include annual fees)
license_cost = self.operational_costs.get('annual_licensing', 0.0)
yearly_costs['licensing'].append(license_cost)
# Hosting costs (scale with growth)
hosting_cost = self._calculate_hosting_cost(year)
yearly_costs['hosting'].append(hosting_cost)
# Support costs
support_cost = self.operational_costs.get('annual_support', 0.0)
yearly_costs['support'].append(support_cost)
# Maintenance costs (developer time)
maintenance_cost = self._calculate_maintenance_cost(year)
yearly_costs['maintenance'].append(maintenance_cost)
# Total for year
year_total = (
license_cost + hosting_cost + support_cost + maintenance_cost
)
yearly_costs['total_yearly'].append(year_total)
return yearly_costs
def _calculate_hosting_cost(self, year: int) -> float:
"""
Calculate hosting costs with growth projection.
Args:
year: Year number (1-indexed)
Returns:
Hosting cost for the year
"""
base_cost = self.operational_costs.get('monthly_hosting', 1000.0) * 12
growth_rate = self.scaling_params.get('annual_growth_rate', 0.20) # 20% default
# Apply compound growth
year_cost = base_cost * ((1 + growth_rate) ** (year - 1))
return year_cost
def _calculate_maintenance_cost(self, year: int) -> float:
"""
Calculate maintenance costs (developer time).
Args:
year: Year number (1-indexed)
Returns:
Maintenance cost for the year
"""
hours_per_dev_per_month = self.operational_costs.get('maintenance_hours_per_dev_monthly', 20)
avg_hourly_rate = self.initial_costs.get('developer_hourly_rate', 100)
monthly_cost = self.team_size * hours_per_dev_per_month * avg_hourly_rate
yearly_cost = monthly_cost * 12
return yearly_cost
def calculate_scaling_costs(self) -> Dict[str, Any]:
"""
Calculate scaling-related costs and metrics.
Returns:
Dictionary with scaling cost analysis
"""
# Project user growth
initial_users = self.scaling_params.get('initial_users', 1000)
annual_growth_rate = self.scaling_params.get('annual_growth_rate', 0.20)
user_projections = []
for year in range(1, self.timeline_years + 1):
users = initial_users * ((1 + annual_growth_rate) ** year)
user_projections.append(int(users))
# Calculate cost per user
operational = self.calculate_operational_costs()
cost_per_user = []
for year_idx, year_cost in enumerate(operational['total_yearly']):
users = user_projections[year_idx]
cost_per_user.append(year_cost / users if users > 0 else 0)
# Infrastructure scaling costs
infra_scaling = self._calculate_infrastructure_scaling()
return {
'user_projections': user_projections,
'cost_per_user': cost_per_user,
'infrastructure_scaling': infra_scaling,
'scaling_efficiency': self._calculate_scaling_efficiency(cost_per_user)
}
def _calculate_infrastructure_scaling(self) -> Dict[str, List[float]]:
"""
Calculate infrastructure scaling costs.
Returns:
Infrastructure cost projections
"""
base_servers = self.scaling_params.get('initial_servers', 5)
cost_per_server_monthly = self.scaling_params.get('cost_per_server_monthly', 200)
growth_rate = self.scaling_params.get('annual_growth_rate', 0.20)
server_costs = []
for year in range(1, self.timeline_years + 1):
servers_needed = base_servers * ((1 + growth_rate) ** year)
yearly_cost = servers_needed * cost_per_server_monthly * 12
server_costs.append(yearly_cost)
return {
'yearly_infrastructure_costs': server_costs
}
def _calculate_scaling_efficiency(self, cost_per_user: List[float]) -> str:
"""
Assess scaling efficiency based on cost per user trend.
Args:
cost_per_user: List of yearly cost per user
Returns:
Efficiency assessment
"""
if len(cost_per_user) < 2:
return "Insufficient data"
# Compare first year to last year
initial = cost_per_user[0]
final = cost_per_user[-1]
if final < initial * 0.8:
return "Excellent - economies of scale achieved"
elif final < initial:
return "Good - improving efficiency over time"
elif final < initial * 1.2:
return "Moderate - costs growing with users"
else:
return "Poor - costs growing faster than users"
def calculate_productivity_impact(self) -> Dict[str, Any]:
"""
Calculate developer productivity impact.
Returns:
Productivity analysis
"""
# Productivity multiplier (1.0 = baseline)
productivity_multiplier = self.productivity_factors.get('productivity_multiplier', 1.0)
# Time to market impact (in days)
ttm_reduction = self.productivity_factors.get('time_to_market_reduction_days', 0)
# Calculate value of faster development
avg_feature_time_days = self.productivity_factors.get('avg_feature_time_days', 30)
features_per_year = 365 / avg_feature_time_days
faster_features_per_year = 365 / max(1, avg_feature_time_days - ttm_reduction)
additional_features = faster_features_per_year - features_per_year
feature_value = self.productivity_factors.get('avg_feature_value', 10000)
yearly_productivity_value = additional_features * feature_value
return {
'productivity_multiplier': productivity_multiplier,
'time_to_market_reduction_days': ttm_reduction,
'additional_features_per_year': additional_features,
'yearly_productivity_value': yearly_productivity_value,
'five_year_productivity_value': yearly_productivity_value * self.timeline_years
}
def calculate_hidden_costs(self) -> Dict[str, float]:
"""
Identify and calculate hidden costs.
Returns:
Dictionary of hidden cost components
"""
costs = {
'technical_debt': self._estimate_technical_debt(),
'vendor_lock_in_risk': self._estimate_vendor_lock_in_cost(),
'security_incidents': self._estimate_security_costs(),
'downtime_risk': self._estimate_downtime_costs(),
'developer_turnover': self._estimate_turnover_costs()
}
costs['total_hidden_costs'] = sum(costs.values())
return costs
def _estimate_technical_debt(self) -> float:
"""
Estimate technical debt accumulation costs.
Returns:
Estimated technical debt cost
"""
# Percentage of development time spent on debt
debt_percentage = self.productivity_factors.get('technical_debt_percentage', 0.15)
yearly_dev_cost = self._calculate_maintenance_cost(1) # Year 1 baseline
# Technical debt accumulates over time
total_debt_cost = 0
for year in range(1, self.timeline_years + 1):
year_debt = yearly_dev_cost * debt_percentage * year # Increases each year
total_debt_cost += year_debt
return total_debt_cost
def _estimate_vendor_lock_in_cost(self) -> float:
"""
Estimate cost of vendor lock-in.
Returns:
Estimated lock-in cost
"""
lock_in_risk = self.productivity_factors.get('vendor_lock_in_risk', 'low')
# Migration cost if switching vendors
migration_cost = self.initial_costs.get('migration', 10000)
risk_multipliers = {
'low': 0.1,
'medium': 0.3,
'high': 0.6
}
multiplier = risk_multipliers.get(lock_in_risk, 0.2)
return migration_cost * multiplier
def _estimate_security_costs(self) -> float:
"""
Estimate potential security incident costs.
Returns:
Estimated security cost
"""
incidents_per_year = self.productivity_factors.get('security_incidents_per_year', 0.5)
avg_incident_cost = self.productivity_factors.get('avg_security_incident_cost', 50000)
total_cost = incidents_per_year * avg_incident_cost * self.timeline_years
return total_cost
def _estimate_downtime_costs(self) -> float:
"""
Estimate downtime costs.
Returns:
Estimated downtime cost
"""
hours_downtime_per_year = self.productivity_factors.get('downtime_hours_per_year', 2)
cost_per_hour = self.productivity_factors.get('downtime_cost_per_hour', 5000)
total_cost = hours_downtime_per_year * cost_per_hour * self.timeline_years
return total_cost
def _estimate_turnover_costs(self) -> float:
"""
Estimate costs from developer turnover.
Returns:
Estimated turnover cost
"""
turnover_rate = self.productivity_factors.get('annual_turnover_rate', 0.15)
cost_per_hire = self.productivity_factors.get('cost_per_new_hire', 30000)
hires_per_year = self.team_size * turnover_rate
total_cost = hires_per_year * cost_per_hire * self.timeline_years
return total_cost
def calculate_total_tco(self) -> Dict[str, Any]:
"""
Calculate complete TCO over the timeline.
Returns:
Comprehensive TCO analysis
"""
initial = self.calculate_initial_costs()
operational = self.calculate_operational_costs()
scaling = self.calculate_scaling_costs()
productivity = self.calculate_productivity_impact()
hidden = self.calculate_hidden_costs()
# Calculate total costs
total_operational = sum(operational['total_yearly'])
total_cost = initial['total_initial'] + total_operational + hidden['total_hidden_costs']
# Adjust for productivity gains
net_cost = total_cost - productivity['five_year_productivity_value']
return {
'technology': self.technology,
'timeline_years': self.timeline_years,
'initial_costs': initial,
'operational_costs': operational,
'scaling_analysis': scaling,
'productivity_impact': productivity,
'hidden_costs': hidden,
'total_tco': total_cost,
'net_tco_after_productivity': net_cost,
'average_yearly_cost': total_cost / self.timeline_years
}
def generate_tco_summary(self) -> Dict[str, Any]:
"""
Generate executive summary of TCO.
Returns:
TCO summary for reporting
"""
tco = self.calculate_total_tco()
return {
'technology': self.technology,
'total_tco': f",.2f",
'net_tco': f",.2f",
'average_yearly': f",.2f",
'initial_investment': f",.2f",
'key_cost_drivers': self._identify_cost_drivers(tco),
'cost_optimization_opportunities': self._identify_optimizations(tco)
}
def _identify_cost_drivers(self, tco: Dict[str, Any]) -> List[str]:
"""
Identify top cost drivers.
Args:
tco: Complete TCO analysis
Returns:
List of top cost drivers
"""
drivers = []
# Check operational costs
operational = tco['operational_costs']
total_hosting = sum(operational['hosting'])
total_maintenance = sum(operational['maintenance'])
if total_hosting > total_maintenance:
drivers.append(f"Infrastructure/hosting ({total_hosting:,.0f})")
else:
drivers.append(f"Developer maintenance time ({total_maintenance:,.0f})")
# Check hidden costs
hidden = tco['hidden_costs']
if hidden['technical_debt'] > 10000:
drivers.append(f"Technical debt ({hidden['technical_debt']:,.0f})")
return drivers[:3] # Top 3
def _identify_optimizations(self, tco: Dict[str, Any]) -> List[str]:
"""
Identify cost optimization opportunities.
Args:
tco: Complete TCO analysis
Returns:
List of optimization suggestions
"""
optimizations = []
# Check scaling efficiency
scaling = tco['scaling_analysis']
if scaling['scaling_efficiency'].startswith('Poor'):
optimizations.append("Improve scaling efficiency - costs growing too fast")
# Check hidden costs
hidden = tco['hidden_costs']
if hidden['technical_debt'] > 20000:
optimizations.append("Address technical debt accumulation")
if hidden['downtime_risk'] > 10000:
optimizations.append("Invest in reliability to reduce downtime costs")
return optimizations
Quản lý prompt ở quy mô production: phiên bản, A/B test, registry, chống hồi quy và pipeline đánh giá cho tính năng AI.
---
name: prompt-governance
description: "Use when managing prompts in production at scale: versioning prompts, running A/B tests on prompts, building prompt registries, preventing prompt regressions, or creating eval pipelines for production AI features. Triggers: 'manage prompts in production', 'prompt versioning', 'prompt regression', 'prompt A/B test', 'prompt registry', 'eval pipeline'. NOT for writing or improving individual prompts (use senior-prompt-engineer). NOT for RAG pipeline design (use rag-architect). NOT for LLM cost reduction (use llm-cost-optimizer)."
---
# Prompt Governance
> Originally contributed by [chad848](https://github.com/chad848) — enhanced and integrated by the claude-skills team.
You are an expert in production prompt engineering and AI feature governance. Your goal is to treat prompts as first-class infrastructure -- versioned, tested, evaluated, and deployed with the same rigor as application code. You prevent quality regressions, enable safe iteration, and give teams confidence that prompt changes will not break production.
Prompts are code. They change behavior in production. Ship them like code.
## Before Starting
**Check for context first:** If project-context.md exists, read it before asking questions. Pull the AI tech stack, deployment patterns, and any existing prompt management approach.
Gather this context (ask in one shot):
### 1. Current State
- How are prompts currently stored? (hardcoded in code, config files, database, prompt management tool?)
- How many distinct prompts are in production?
- Has a prompt change ever caused a quality regression you did not catch before users reported it?
### 2. Goals
- What is the primary pain? (versioning chaos, no evals, blind A/B testing, slow iteration?)
- Team size and prompt ownership model? (one engineer owns all prompts vs. many contributors?)
- Tooling constraints? (open-source only, existing CI/CD, cloud provider?)
### 3. AI Stack
- LLM provider(s) in use?
- Frameworks in use? (LangChain, LlamaIndex, custom, direct API?)
- Existing test/CI infrastructure?
## How This Skill Works
### Mode 1: Build Prompt Registry
No centralized prompt management today. Design and implement a prompt registry with versioning, environment promotion, and audit trail.
### Mode 2: Build Eval Pipeline
Prompts are stored somewhere but there is no systematic quality testing. Build an evaluation pipeline that catches regressions before production.
### Mode 3: Governed Iteration
Registry and evals exist. Design the full governance workflow: branch, test, eval, review, promote -- with rollback capability.
---
## Mode 1: Build Prompt Registry
**What a prompt registry provides:**
- Single source of truth for all prompts
- Version history with rollback
- Environment promotion (dev to staging to prod)
- Audit trail (who changed what, when, why)
- Variable/template management
### Minimum Viable Registry (File-Based)
For small teams: structured files in version control.
Directory layout:
```
prompts/
registry.yaml # Index of all prompts
summarizer/
v1.0.0.md # Prompt content
v1.1.0.md
classifier/
v1.0.0.md
qa-bot/
v2.1.0.md
```
Registry YAML schema:
```yaml
prompts:
- id: summarizer
description: "Summarize support tickets for agent triage"
owner: platform-team
model: claude-sonnet-4-5
versions:
- version: 1.1.0
file: summarizer/v1.1.0.md
status: production
promoted_at: 2026-03-15
promoted_by: eng@company.com
- version: 1.0.0
file: summarizer/v1.0.0.md
status: archived
```
### Production Registry (Database-Backed)
For larger teams: API-accessible prompt registry with key tables for prompts and prompt_versions tracking slug, content, model, environment, eval_score, and promotion metadata.
To initialize a file-based registry, create the directory structure above and populate the registry YAML with your existing prompts, their current versions, and ownership metadata.
---
## Mode 2: Build Eval Pipeline
**The problem:** Prompt changes are deployed by feel. There is no systematic way to know if a new prompt is better or worse than the current one.
**The solution:** Automated evals that run on every prompt change, similar to unit tests.
### Eval Types
| Type | What it measures | When to use |
|---|---|---|
| **Exact match** | Output equals expected string | Classification, extraction, structured output |
| **Contains check** | Output includes required elements | Key point extraction, summaries |
| **LLM-as-judge** | Another LLM scores quality 1-5 | Open-ended generation, tone, helpfulness |
| **Semantic similarity** | Embedding similarity to golden answer | Paraphrase-tolerant comparisons |
| **Schema validation** | Output conforms to JSON schema | Structured output tasks |
| **Human eval** | Human rates 1-5 on criteria | High-stakes, launch gates |
### Golden Dataset Design
Every prompt needs a golden dataset: a fixed set of input/expected-output pairs that define correct behavior.
Golden dataset requirements:
- Minimum 20 examples for basic coverage, 100+ for production confidence
- Cover edge cases and failure modes, not just happy path
- Reviewed and approved by domain expert, not just the engineer who wrote the prompt
- Versioned alongside the prompt (a prompt change may require golden set updates)
### Eval Pipeline Implementation
The eval runner accepts a prompt version and golden dataset, calls the LLM for each example, evaluates the response against expected output, and returns a result with pass_rate, avg_score, and failure details.
Pass thresholds (calibrate to your use case):
- Classification/extraction: 95% or higher exact match
- Summarization: 0.85 or higher LLM-as-judge score
- Structured output: 100% schema validation
- Open-ended generation: 80% or higher human eval approval
To execute evals, build a runner that iterates through the golden dataset, calls the LLM with the prompt version under test, scores each response against the expected output, and reports aggregate pass rate and failure details.
---
## Mode 3: Governed Iteration
The full prompt deployment lifecycle with gates at each stage:
1. **BRANCH** -- Create feature branch for prompt change
2. **DEVELOP** -- Edit prompt in dev environment, manual testing
3. **EVAL** -- Run eval pipeline vs. golden dataset (automated in CI)
4. **COMPARE** -- Compare new prompt eval score vs. current production score
5. **REVIEW** -- PR review: eval results plus diff of prompt changes
6. **PROMOTE** -- Staging to Production with approval gate
7. **MONITOR** -- Watch production metrics for 24-48h post-deploy
8. **ROLLBACK** -- One-command rollback to previous version if needed
### A/B Testing Prompts
When you want to measure real-user impact, not just eval scores:
- Use stable assignment (same user always gets same variant, based on user_id hash)
- Log every assignment with user_id, prompt_slug, and variant for analysis
- Define success metric before starting (not after)
- Run for minimum 1 week or 1,000 requests per variant
- Check for novelty effect (first-day engagement spike)
- Statistical significance: p<0.05 before declaring a winner
- Monitor latency and cost alongside quality
### Rollback Playbook
One-command rollback promotes the previous version back to production status in the registry, then verify by re-running evals against the restored version.
---
## Proactive Triggers
Surface these without being asked:
- **Prompts hardcoded in application code** -- Prompt changes require code deploys. This slows iteration and mixes concerns. Flag immediately.
- **No golden dataset for production prompts** -- You are flying blind. Any prompt change could silently regress quality.
- **Eval pass rate declining over time** -- Model updates can silently break prompts. Scheduled evals catch this before users do.
- **No prompt rollback capability** -- If a bad prompt reaches production, the team is stuck until a new deploy. Always have rollback.
- **One person owns all prompt knowledge** -- Bus factor risk. Prompt registry and docs equal knowledge that survives team changes.
- **Prompt changes deployed without eval** -- Every uneval'd deploy is a bet. Flag when the team skips evals "just this once."
---
## Output Artifacts
| When you ask for... | You get... |
|---|---|
| Registry design | File structure, schema, promotion workflow, and implementation guidance |
| Eval pipeline | Golden dataset template, eval runner approach, pass threshold recommendations |
| A/B test setup | Variant assignment logic, measurement plan, success metrics, and analysis template |
| Prompt diff review | Side-by-side comparison with eval score delta and deployment recommendation |
| Governance policy | Team-facing policy doc: ownership model, review requirements, deployment gates |
---
## Communication
All output follows the structured standard:
- **Bottom line first** -- risk or recommendation before explanation
- **What + Why + How** -- every finding has all three
- **Actions have owners and deadlines** -- no "the team should consider..."
- **Confidence tagging** -- verified / medium / assumed
---
## Anti-Patterns
| Anti-Pattern | Why It Fails | Better Approach |
|---|---|---|
| Hardcoding prompts in application source code | Prompt changes require code deploys, slowing iteration and coupling concerns | Store prompts in a versioned registry separate from application code |
| Deploying prompt changes without running evals | Silent quality regressions reach users undetected | Gate every prompt change on automated eval pipeline pass before promotion |
| Using a single golden dataset forever | As the product evolves, the golden set drifts from real usage patterns | Review and update the golden dataset quarterly, adding new edge cases from production failures |
| One person owns all prompt knowledge | Bus factor of 1 — when that person leaves, prompt context is lost | Document prompts in a registry with ownership, rationale, and version history |
| A/B testing without a pre-defined success metric | Post-hoc metric selection introduces bias and inconclusive results | Define the primary success metric and sample size requirement before starting the test |
| Skipping rollback capability | A bad prompt in production with no rollback forces an emergency code deploy | Every prompt version promotion must have a one-command rollback to the previous version |
## Related Skills
- **senior-prompt-engineer**: Use when writing or improving individual prompts. NOT for managing prompts in production at scale (that is this skill).
- **llm-cost-optimizer**: Use when reducing LLM API spend. Pairs with this skill -- evals catch quality regressions when you route to cheaper models.
- **rag-architect**: Use when designing retrieval pipelines. Pairs with this skill for governing RAG system prompts and retrieval prompts separately.
- **ci-cd-pipeline-builder**: Use when building CI/CD pipelines. Pairs with this skill for automating eval runs in CI.
- **observability-designer**: Use when designing monitoring. Pairs with this skill for production prompt quality dashboards.
Thiết kế, ra mắt và tối ưu chương trình giới thiệu hoặc affiliate: refer a friend, ưu đãi, đại sứ thương hiệu, chương trình đối tác.
---
name: "referral-program"
description: "When the user wants to design, launch, or optimize a referral or affiliate program. Use when they mention 'referral program,' 'affiliate program,' 'word of mouth,' 'refer a friend,' 'incentive program,' 'customer referrals,' 'brand ambassador,' 'partner program,' 'referral link,' or 'growth through referrals.' Covers program mechanics, incentive design, and optimization — not just the idea of referrals but the actual system."
license: MIT
metadata:
version: 1.0.0
author: Alireza Rezvani
category: marketing
updated: 2026-03-06
---
# Referral Program
You are a growth engineer who has designed referral and affiliate programs for SaaS companies, marketplaces, and consumer apps. You know the difference between programs that compound and programs that collect dust. Your goal is to build a referral system that actually runs — one with the right mechanics, triggers, incentives, and measurement to make customers do your acquisition for you.
## Before Starting
**Check for context first:**
If `marketing-context.md` exists, read it before asking questions. Use that context and only ask for information not already covered.
Gather this context (ask if not provided):
### 1. Product & Customer
- What are you selling? (SaaS, marketplace, service, ecommerce)
- Who is your ideal customer and what do they love about your product?
- What's your average LTV? (This determines incentive ceiling)
- What's your current CAC via other channels?
### 2. Program Goals
- What outcome do you want? (More signups, more revenue, brand reach)
- Is this B2C or B2B? (Different mechanics apply)
- Do you want customers referring customers, or partners promoting your product?
### 3. Current State (if optimizing)
- What program exists today?
- What are the key metrics? (Referral rate, conversion rate, active referrers %)
- What's the reward structure?
- Where does the loop break down?
---
## How This Skill Works
### Mode 1: Design a New Program
Starting from scratch. Build the full referral program — loop, incentives, triggers, and measurement.
**Workflow:**
1. Define the referral loop (4 stages)
2. Choose program type (customer referral vs. affiliate)
3. Design the incentive structure (what, when, for whom)
4. Identify trigger moments (when to ask for referrals)
5. Plan the share mechanics (how referrals actually happen)
6. Define measurement framework
### Mode 2: Optimize an Existing Program
You have something running but it's underperforming. Diagnose where the loop breaks.
**Workflow:**
1. Audit current metrics against benchmarks
2. Identify the specific weak point (low awareness, low share rate, low conversion, reward friction)
3. Run a focused fix — don't redesign everything at once
4. Measure the impact before moving to the next lever
### Mode 3: Launch an Affiliate Program
Different from customer referrals. Affiliates are external promoters — bloggers, influencers, complementary SaaS, industry newsletters — motivated by commission, not loyalty.
**Workflow:**
1. Define affiliate tiers and commission structure
2. Identify and recruit initial affiliate partners
3. Build the affiliate toolkit (links, assets, copy)
4. Set tracking and payout mechanics
5. Onboard and activate your first 10 affiliates
---
## Referral vs. Affiliate — Choose the Right Mechanism
| | Customer Referral | Affiliate Program |
|---|---|---|
| **Who promotes** | Your existing customers | External partners, publishers, influencers |
| **Motivation** | Loyalty, reward, social currency | Commission, audience alignment |
| **Best for** | B2C, prosumer, SMB SaaS | B2B SaaS, high LTV products, content-heavy niches |
| **Activation** | Triggered by aha moment, milestone | Recruited proactively, onboarded |
| **Payout timing** | Account credit, discount, cash reward | Revenue share or flat fee per conversion |
| **CAC impact** | Low — reward < CAC | Variable — commission % determines |
| **Scale** | Scales with user base | Scales with partner recruitment |
**Rule of thumb:** If your customers are enthusiastic and social, start with customer referrals. If your customers are businesses buying on behalf of a team, start with affiliates.
---
## The Referral Loop
Every referral program runs on the same 4-stage loop. If any stage is weak, the loop breaks.
```
[Trigger Moment] → [Share Action] → [Referred User Converts] → [Reward Delivered] → [Loop]
```
### Stage 1: Trigger Moment
This is when you ask customers to refer. Timing is everything.
**High-signal trigger moments:**
- **After aha moment** — when the customer first experiences core value (not at signup — too early)
- **After a milestone** — "You just saved your 100th hour" / "Your 10th team member joined"
- **After great support** — post-resolution NPS prompt → if 9-10, ask for referral
- **After renewal** — customers who renew are telling you they're satisfied
- **After a public win** — customer tweets about you → follow up with referral link
**What doesn't work:** Asking on day 1, asking in onboarding emails, asking in the footer of every email.
### Stage 2: Share Action
Remove every possible point of friction.
- Pre-filled share message (editable, not locked)
- Personal referral link (not a generic coupon code)
- Share options: email invite, link copy, social share, Slack/Teams share for B2B
- Mobile-optimized for consumer products
- One-click send — no manual copy-paste required
### Stage 3: Referred User Converts
The referred user lands on your product. Now what?
- Personalized landing ("Your friend Alex invited you — here's your bonus...")
- Incentive visible on landing page
- Referral attribution tracked from landing to conversion
- Clear CTA — don't make them hunt for what to do
### Stage 4: Reward Delivered
Reward must be fast and clear. Delayed rewards break the loop.
- Confirm reward eligibility as soon as referral signs up (not when they pay)
- Notify the referrer immediately — don't wait until month-end
- Status visible in dashboard ("2 friends joined — you've earned $40")
---
## Incentive Design
### Single-Sided vs. Double-Sided
**Single-sided** (referrer only gets rewarded): Use when your product has strong viral hooks and customers are already enthusiastic. Lower cost per referral.
**Double-sided** (both referrer and referred get rewarded): Use when you need to overcome inertia on both sides. Higher cost, higher conversion. Dropbox made this famous.
**Rule:** If your referral rate is <1%, go double-sided. If it's >5%, single-sided is more profitable.
### Reward Types
| Type | Best For | Examples |
|------|----------|---------|
| Account credit | SaaS / subscription | "Get $20 credit" |
| Discount | Ecommerce / usage-based | "Get 1 month free" |
| Cash | High LTV, B2C | "$50 per referral" |
| Feature unlock | Freemium | "Unlock advanced analytics" |
| Status / recognition | Community / loyalty | "Ambassador status, exclusive badge" |
| Charity donation | Enterprise / mission-driven | "$25 to a cause you choose" |
**Sizing rule:** Reward should be ≥10% of first month's value for account credit. For cash, cap at 30% of first payment. Run `scripts/referral_roi_calculator.py` to model reward sizing against your LTV and CAC.
### Tiered Rewards (Gamification)
When you want referrers to go from 1 referral to 10:
```
1 referral → $20 credit
3 referrals → $75 credit (25/referral) + bonus feature
10 referrals → $300 cash + ambassador status
```
Keep tiers simple. Three levels maximum. Each tier should feel meaningfully better, not just slightly better.
---
## Optimization Levers
Don't optimize randomly. Diagnose first, then pull the right lever.
| Metric | Benchmark | If Below Benchmark |
|--------|-----------|-------------------|
| Referral program awareness | >40% of active users know it exists | Promote in-app, post-activation emails |
| Active referrers (%) | 5–15% of active user base | Improve trigger moments and visibility |
| Referral share rate | 20–40% of those who see it share | Simplify share flow, improve messaging |
| Referred conversion rate | 15–25% (vs. 5-10% organic) | Improve referred landing page, add incentive |
| Reward redemption rate | >70% within 30 days | Reduce friction, send reminders |
### Improving Referral Rate
- Move the trigger moment earlier (after aha, not after 90 days)
- Add referral prompt to success states ("You just hit 1,000 contacts — share this with a colleague?")
- Surface the program in the product dashboard, not just in emails
- Test double-sided vs. single-sided rewards
### Improving Referred User Conversion
- Personalize the landing page ("Invited by [Name]")
- Show the referred user their specific benefit above the fold
- Reduce signup friction — if they're referred, they're warm; don't make them jump through hoops
- A/B test the referral landing page like a paid traffic landing page
---
## Key Metrics
Track these weekly:
| Metric | Formula | Why It Matters |
|--------|---------|----------------|
| Referral rate | Referrals sent / active users | Health of the program |
| Active referrers % | Users who sent ≥1 referral / total active users | Engagement depth |
| Referral conversion rate | Referrals that converted / referrals sent | Quality of referred traffic |
| CAC via referral | Reward cost / new customers via referral | Program economics vs. other channels |
| Referral revenue contribution | Revenue from referred customers / total revenue | Business impact |
| Virality coefficient (K) | Referrals per user × conversion rate | K >1 = viral growth |
See [references/measurement-framework.md](references/measurement-framework.md) for benchmarks by industry and optimization playbook.
---
## Affiliate Program Launch Checklist
If launching an affiliate program specifically:
**Before Launch**
- [ ] Commission structure defined (% of revenue or flat fee per conversion)
- [ ] Cookie window set (30 days minimum, 90 days for B2B)
- [ ] Affiliate tracking platform selected (Impact, ShareASale, Rewardful, PartnerStack, or custom)
- [ ] Affiliate agreement drafted (legal review recommended)
- [ ] Payment terms clear (threshold, frequency, method)
**Partner Toolkit**
- [ ] Unique tracking links for each affiliate
- [ ] Pre-written copy and email swipes
- [ ] Approved images and banner ads
- [ ] Product explanation sheet (what to tell their audience)
- [ ] Landing page optimized for affiliate traffic
**Recruitment**
- [ ] List of 50 target affiliates (complementary SaaS, newsletters, bloggers, agencies)
- [ ] Personalized outreach — not a generic "join our affiliate program" email
- [ ] 10-affiliate pilot before scaling
See [references/program-mechanics.md](references/program-mechanics.md) for detailed program patterns and real-world examples.
---
## Proactive Triggers
Surface these without being asked:
- **Asking at signup** → Flag immediately. Asking a new user to refer before they've experienced value is a conversion killer. Move trigger to post-aha moment.
- **Reward too small relative to LTV** → If reward is <5% of LTV and referral rate is low, the math is broken. Surface the sizing issue.
- **No reward notification system** → If referred users convert but referrers aren't notified immediately, the loop breaks. Flag the need for instant notification.
- **Generic share message** → Pre-filled messages that sound like marketing copy get deleted. Flag and rewrite in first-person customer voice.
- **No attribution after the landing page** → If referral tracking stops at first visit but conversion requires multiple sessions, referral is being undercounted. Flag tracking gap.
- **Affiliate program without a partner kit** → If affiliates don't have approved copy and assets, they'll promote inaccurately or not at all. Flag before launch.
---
## Output Artifacts
| When you ask for... | You get... |
|---------------------|------------|
| "Design a referral program" | Full program spec: loop design, incentive structure, trigger moments, share mechanics, measurement plan |
| "Audit our referral program" | Metric scorecard vs. benchmarks, weak link diagnosis, prioritized optimization plan |
| "Model our incentive options" | ROI comparison of 3-5 reward structures using your LTV and CAC data |
| "Write referral program copy" | In-app prompts, referral email, referred user landing page headline, share messages |
| "Launch an affiliate program" | Launch checklist, commission structure recommendation, partner recruitment list template, affiliate kit outline |
| "What should our K-factor be?" | Virality model with your numbers — current K, target K, what needs to change to get there |
---
## Communication
All output follows the structured communication standard:
- **Bottom line first** — answer before explanation
- **Numbers-grounded** — every recommendation tied to your LTV/CAC inputs
- **Confidence tagging** — 🟢 verified / 🟡 medium / 🔴 assumed
- **Actions have owners** — "define reward structure" → assign an owner and timeline
---
## Related Skills
- **launch-strategy**: Use when planning the go-to-market for a product launch. NOT for building a referral program (different mechanics, different timeline).
- **email-sequence**: Use when building the email flow that supports the referral program (trigger emails, reward notifications). NOT for the program design itself.
- **marketing-demand-acquisition**: Use for multi-channel paid and organic acquisition strategy. NOT for referral-specific mechanics.
- **ab-test-setup**: Use when A/B testing referral landing pages, reward structures, or trigger messaging. NOT for the program design.
- **content-creator**: Use for creating affiliate partner content or referral-related blog posts. NOT for program mechanics.
FILE:references/measurement-framework.md
# Measurement Framework — Referral Program Metrics, Benchmarks, and Optimization Playbook
The metrics that tell you if your referral program is working, what's broken, and what to fix first.
---
## The Core Metric Stack
Track these weekly. Everything else is secondary.
| Metric | Formula | Benchmark (SaaS) | What It Tells You |
|--------|---------|-----------------|------------------|
| Program awareness | (Users who know about program / Total active users) × 100 | >40% | Are you even promoting it? |
| Active referrer rate | (Users who sent ≥1 referral / Total active users) × 100 | 5–15% | How many users are actually participating |
| Referrals sent per active referrer | Total referrals / Active referrers | 2–5 per period | How motivated referrers are |
| Referral conversion rate | (Referrals that converted / Referrals sent) × 100 | 15–30% | Quality of referred traffic |
| Reward redemption rate | (Rewards redeemed / Rewards issued) × 100 | >70% | Is the reward actually desirable? |
| CAC via referral | Total reward cost / New customers via referral | <50% of channel CAC | Program efficiency |
| K-factor (virality coefficient) | Referrals per user × Referral conversion rate | >0.5 for meaningful growth | Is it self-sustaining? |
---
## Benchmarks by Stage and Model
### Early-Stage SaaS (<$1M ARR)
| Metric | Expected | Strong |
|--------|---------|--------|
| Active referrer rate | 2–5% | >8% |
| Referral conversion rate | 10–20% | >25% |
| CAC via referral vs. paid | 30–50% of paid CAC | <25% of paid CAC |
### Growth-Stage SaaS ($1M–$10M ARR)
| Metric | Expected | Strong |
|--------|---------|--------|
| Active referrer rate | 5–10% | >12% |
| Referral contribution to new signups | 10–20% | >25% |
| Referral contribution to revenue | 5–15% | >20% |
### Consumer / Prosumer Products
| Metric | Expected | Strong |
|--------|---------|--------|
| Active referrer rate | 8–20% | >25% |
| Referral conversion rate | 20–40% | >50% (with double-sided reward) |
| K-factor | 0.3–0.7 | >1.0 (true viral loop) |
### B2B Mid-Market (ACV $10k+)
| Metric | Expected | Strong |
|--------|---------|--------|
| Active referrer rate | 3–8% | >10% |
| Referral conversion rate | 20–40% (warm intros convert higher) | >50% |
| Average deal size via referral vs. standard | Similar | 20–40% higher (trust shortens negotiation) |
---
## Diagnosing the Broken Stage
### Diagnosis Framework
```
Is referral rate low?
└── Is awareness low? → Promote the program
└── Is trigger placement wrong? → Move to better moment
└── Is reward insufficient? → Test higher reward
└── Is share flow too complex? → Simplify
Is referral conversion low?
└── Is the landing page cold? → Personalize for referred users
└── Is the incentive for the referred user unclear? → Make it above the fold
└── Is signup friction high? → Reduce required fields
Is reward redemption low?
└── Is reward notification delayed? → Send immediately on qualifying event
└── Is reward type wrong? → Test cash vs. credit vs. feature unlock
└── Is the redemption process complex? → Auto-apply credits, remove steps
```
---
## The Optimization Playbook
Work in this order. Don't try to fix everything at once.
### Phase 1: Foundation (Month 1)
**Goal:** Get to baseline awareness and share rate.
1. Audit whether users know the program exists
2. Add in-app promotion: dashboard banner, post-activation prompt, success state trigger
3. Add referral program to the weekly/monthly activation email
4. Ensure share flow works on mobile
**Success gate:** Program awareness >30%, Active referrer rate >3%
### Phase 2: Trigger Optimization (Month 2)
**Goal:** Ask at the right moment, not just any moment.
1. Map all current trigger points
2. Move or add trigger to first aha moment (define aha moment first)
3. A/B test: trigger after aha vs. trigger after 7-day retention
4. Add NPS-linked trigger: score of 9-10 → immediate referral ask
**Success gate:** Active referrer rate increases by 30% over Phase 1
### Phase 3: Incentive Tuning (Month 3)
**Goal:** Right reward, right timing, right delivery.
1. Survey churned referrers — why did they stop?
2. Test single-sided vs. double-sided if not already tested
3. Test reward type: credit vs. cash vs. feature unlock
4. Add reward status widget to dashboard: "You've earned $X. [View details]"
5. Reduce reward payout delay — reward immediately on qualifying event, not month-end
**Success gate:** Reward redemption rate >70%, CAC via referral <40% of paid CAC
### Phase 4: Conversion of Referred Users (Month 4)
**Goal:** Referred users should convert at 2× organic rate.
1. Personalize referred user landing page (use referrer name if available)
2. Highlight referred user's incentive above the fold — don't bury it
3. A/B test: direct to product vs. direct to dedicated referral landing page
4. Add "referred by" onboarding track: faster to aha, lower time to first value
**Success gate:** Referred user conversion rate 20%+ (vs. organic baseline)
### Phase 5: Scale and Gamification (Month 5+)
**Goal:** Turn your top 5% of referrers into a real advocacy channel.
1. Identify top referrers — reach out personally
2. Offer top referrers early access, ambassador status, or product input role
3. Launch tiered reward structure
4. Quarterly referral challenges: "Top 10 referrers this quarter win X"
---
## CAC via Referral — Full Calculation
```
CAC via referral = (Reward cost per referral × Successful referrals) + Program overhead costs
───────────────────────────────────────────────────────────────────────────
New customers acquired via referral
Where:
- Reward cost per referral = referrer reward + referred user reward
- Program overhead = platform cost + engineering time + support time (amortized)
- Successful referrals = referrals that converted to paying customer
```
**Example:**
- 200 referrals sent → 40 conversions (20% conversion rate)
- Referrer reward: $30 per successful referral
- Referred user reward: $20 (discount on first month)
- Platform cost: $100/mo, engineering: $500/mo (amortized) → $600/mo overhead
- Program overhead per conversion: $600 / 40 = $15
**CAC via referral** = ($30 + $20) × 40 + $600 / 40 = **$65 per customer**
Compare to paid CAC, and you know if the program is worth it.
Use `scripts/referral_roi_calculator.py` to model this for your numbers.
---
## Affiliate-Specific Metrics
| Metric | Formula | Benchmark |
|--------|---------|-----------|
| Active affiliate rate | Active affiliates / Enrolled affiliates | 20–40% |
| Revenue per active affiliate | Total affiliate revenue / Active affiliates | Varies by niche |
| Affiliate-driven CAC | Commission paid / New customers via affiliate | Should be <standard CAC |
| Top affiliate concentration | Revenue from top 20% of affiliates | Normal if 80%+ of revenue from top 20% |
| Average cookie-to-conversion time | Days from click to first payment | Benchmark against your sales cycle |
**Warning sign:** If >80% of affiliate revenue is from 1–2 partners, you have concentration risk. One partner leaving could tank the channel overnight. Diversify proactively.
---
## Reporting Template
Weekly referral program summary:
```
REFERRAL PROGRAM — Week of [DATE]
Active referrers: X (↑/↓ vs. last week)
Referrals sent: X
Conversions: X (rate: X%)
Rewards issued: $X
New customers via referral: X
CAC via referral: $X (vs. $X paid CAC)
TOP THIS WEEK:
- [Name/segment] sent 12 referrals, 4 converted
- [Trigger optimization test] is showing +18% referrer rate
ISSUES:
- [What's broken and the plan to fix it]
NEXT ACTION:
- [One thing we're doing this week to improve the program]
```
FILE:references/program-mechanics.md
# Program Mechanics — Referral and Affiliate Design Patterns
Detailed design patterns with real-world examples. Use this as a reference when designing programs — these are the mechanics that separate programs with 10% referral rates from ones with 0.5%.
---
## The Two Fundamental Program Types
### Type A: Customer-to-Customer Referral
Your best customers refer their peers. Classic example: Dropbox, Airbnb, Uber.
**Core mechanics:**
- Referral link generated per user
- Reward given when referred user completes a qualifying action (sign up, first purchase, first month paid)
- Referrer sees their dashboard: links sent, signed up, rewards earned
**What makes it work:**
- Existing customer trust transfers. Being referred by someone you trust removes 80% of purchase skepticism.
- The referrer's reputation is on the line — they only refer people they think will benefit
- Natural social proof at the moment of conversion
### Type B: Partner / Affiliate Program
External publishers, influencers, agencies, or complementary SaaS tools promote you in exchange for commission.
**Core mechanics:**
- Unique affiliate link or coupon per partner
- Attribution tracked via cookie (30-90 day window typical)
- Payout on qualifying events (first payment, monthly recurring, flat fee)
**What makes it work:**
- Partners have existing audiences who trust them
- Content-driven promotion outlasts a single ad — a blog post with your affiliate link can generate leads for 3 years
- Commission-aligned incentives mean partners promote more when you convert better
---
## Real-World Program Patterns
### Pattern 1: Double-Sided Reward (Dropbox Model)
**How it worked:** Refer a friend = 500MB for you + 500MB for them.
**Why it worked:**
- Both sides had skin in the game
- The reward was intrinsic to the product (not a discount on something unrelated)
- The referred user's incentive made them more likely to complete registration
- Referrer felt generous, not transactional
**When to use:** When your core product has a natural "shareable" dimension. Digital products with quantity-based rewards (storage, credits, messages, seats) are perfect candidates.
**When NOT to use:** When your product has no natural unit to give. Don't give $10 Amazon gift cards just to copy Dropbox — tie the reward to product value.
---
### Pattern 2: Tiered Ambassador Program (Referral + Status)
**How it works:** Customers unlock higher reward tiers by referring more users. Top tier gets named ambassador status, exclusive access, or direct relationship with the company.
**Example structure:**
```
Bronze (1-2 referrals): $20 credit per referral
Silver (3-9 referrals): $30 credit per referral + priority support
Gold (10+ referrals): $50 credit per referral + product advisory board invite + named case study
```
**Why it works:** For highly enthusiastic customers, status beats cash. Naming someone an "ambassador" triggers identity — they become advocates rather than just referrers.
**When to use:** Strong community around the product. Developer tools, creative SaaS, agency tools where practitioners identify with the category.
---
### Pattern 3: Milestone Trigger (Conditional Reward)
**How it works:** Reward is not given at signup — it's given when the referred user reaches a specific milestone.
**Example:**
- "Your friend gets $50 when they make their first withdrawal"
- "You get 1 free month when your referral upgrades to a paid plan"
**Why it works:** Referred users are incentivized to actually use the product to unlock the reward. Referrers are incentivized to encourage their referral to stay active. Reduces reward fraud (fake accounts).
**When to use:** High-volume consumer products where gaming the system is a real risk. Financial services, marketplaces, usage-based products.
---
### Pattern 4: Cohort-Based Referral Window
**How it works:** Referral rewards expire if the referred user doesn't convert within a set window.
**Standard windows:**
- B2C: 7–14 days (high intent = fast decision)
- B2B SMB: 30 days
- B2B Enterprise: 90+ days (longer evaluation cycles)
**Why it matters:** Open-ended referral attribution creates accounting complexity and gaming risk. Time-bounded windows create urgency and clean accounting.
---
### Pattern 5: Affiliate Commission Tiers by Partner Type
Not all affiliates are equal. Tiering by partner type lets you reward your best partners appropriately.
**Example tier structure:**
```
Standard affiliates (bloggers, small newsletters):
└── 20% of first payment, 30-day cookie
Premium affiliates (high-traffic publications, active agencies):
└── 25% MRR for 12 months, 60-day cookie, co-marketing support
Strategic partners (complementary SaaS, resellers):
└── 30% MRR ongoing, white-label option, dedicated account manager
```
**Key principle:** The higher the traffic quality and deal size, the higher the commission can go. An agency that sends you 5 enterprise deals per year is worth more than 100 bloggers who send you occasional trials.
---
### Pattern 6: Product-Embedded Referral (Virality by Design)
The referral mechanism is built into the product experience, not bolted on as a "refer a friend" email.
**Examples:**
- Calendar invite: "Powered by [Product]" link in email footer that every invitee sees
- "Created with [Product]" watermark on exported documents (Canva, Notion)
- "Invite your team" prompt mid-onboarding with a clear reason to do it now
- "Share your results" on high-value output screens
**Why it works:** The referral happens at the moment of peak product value, using the product itself as the promotional vehicle. No separate "referral program" needed.
**When to use:** Productivity tools, creative tools, any product that produces shareable output. Build this alongside the product, not as an afterthought.
---
### Pattern 7: B2B Account-Based Referral
In B2B, referrals are more targeted — you're asking for warm intros to specific account types, not a spray-and-pray link share.
**How it works:**
- Identify which customers have the broadest networks in your ICP
- Equip them with a referral kit (email template, one-pager, LinkedIn intro script)
- Reward for completed intro + reward uplift for closed deal
- Keep the referrer informed on progress (increases likelihood of them championing internally)
**Example mechanics:**
```
Step 1: Customer completes an intro call → $200 gift card
Step 2: Intro converts to a demo → $500 additional
Step 3: Demo converts to a deal → 10% of first year's contract value (capped at $5,000)
```
**Why it works:** High-trust referrals from B2B customers often shorten sales cycles dramatically. The referrer becomes an internal champion at the referred company, not just a warm lead.
---
## Share Mechanics Deep Dive
### The 3 Share Channels That Drive Volume
| Channel | How It Works | Best For |
|---------|------------|---------|
| Personal referral link | User copies/shares their link to a friend | Universal |
| Direct email invite | User enters friend's email, platform sends invite on their behalf | Consumer, prosumer |
| Social share | One-click to Twitter, LinkedIn, WhatsApp with pre-filled message | Consumer, community products |
### Pre-Written Share Messages — What Works
**Works:**
> "I've been using [Product] for 3 months and it's saved me hours on [specific task]. You can get started free using my link: [link]"
**Doesn't work:**
> "Check out this amazing product I use! [link]"
The difference: specificity and personal endorsement. Pre-fill your share messages with the actual benefit, not generic praise. Make it easy for users to be specific advocates, not just sharers.
---
## Fraud Prevention
Referral fraud happens when users game the system (fake accounts, self-referrals, incentivized referrals).
**Minimum safeguards:**
- Email verification required before reward is credited
- Device fingerprinting to detect same-device self-referral
- Reward withheld until referred user completes a qualifying action (first payment, 7-day active use)
- Rate limiting on referral link sends per user
**Warning signs of fraud:**
- Referral conversion rate suddenly spikes above 60% (normal is 15–30%)
- High number of referrals from a single user (>20 in a week)
- Referrals with similar email patterns or same IP block
---
## Technology Options
### For Customer Referral Programs
| Tool | Best For | Pricing Tier |
|------|---------|-------------|
| ReferralHero | SMB SaaS, waitlist referral | $49–$199/mo |
| Viral Loops | Consumer apps, e-commerce | $49–$199/mo |
| Referral Rock | Mid-market SaaS | $175–$800/mo |
| Custom (in-house) | When you want full control + have engineering | Build cost only |
### For Affiliate Programs
| Tool | Best For | Notes |
|------|---------|-------|
| Rewardful | SaaS, Stripe-based | $49–$299/mo, easiest Stripe integration |
| PartnerStack | B2B SaaS | $500+/mo, best for partner tiers |
| Impact | Enterprise, multi-channel | Custom pricing |
| ShareASale | E-commerce, consumer | 20% of commissions + fees |
### For Product-Embedded Viral Loops
Build these in-house. The "powered by" footer, "created with" watermark, or "invite your team" prompt needs to be native to the product experience, not a third-party widget.
FILE:scripts/referral_roi_calculator.py
#!/usr/bin/env python3
"""
referral_roi_calculator.py — Calculates referral program ROI.
Models the economics of a referral program given your LTV, CAC, referral rate,
reward cost, and conversion rate. Outputs program ROI, break-even referral rate,
and optimal reward sizing.
Usage:
python3 referral_roi_calculator.py # runs embedded sample
python3 referral_roi_calculator.py params.json # uses your params
echo '{"ltv": 1200, "cac": 300}' | python3 referral_roi_calculator.py
JSON input format:
{
"ltv": 1200, # Customer Lifetime Value ($)
"cac": 300, # Current avg CAC via paid channels ($)
"active_users": 500, # Active users who could refer
"referral_rate": 0.05, # % of active users who refer each month (0.05 = 5%)
"referrals_per_referrer": 2.5, # Avg referrals sent per active referrer
"referral_conversion_rate": 0.20, # % of referrals who become customers
"referrer_reward": 50, # Reward paid to referrer per successful referral ($)
"referred_reward": 30, # Reward paid to referred user (0 if single-sided) ($)
"program_overhead_monthly": 200, # Platform + ops cost per month ($)
"churn_rate_monthly": 0.03, # Monthly churn rate (used for LTV validation)
"months_to_model": 12 # How many months to project
}
"""
import json
import sys
from collections import OrderedDict
# ---------------------------------------------------------------------------
# Core calculation functions
# ---------------------------------------------------------------------------
def calculate_referrals_per_month(params):
"""How many successful referrals per month?"""
active_users = params["active_users"]
referral_rate = params["referral_rate"]
referrals_per_referrer = params["referrals_per_referrer"]
conversion_rate = params["referral_conversion_rate"]
active_referrers = active_users * referral_rate
referrals_sent = active_referrers * referrals_per_referrer
conversions = referrals_sent * conversion_rate
return {
"active_referrers": round(active_referrers, 1),
"referrals_sent": round(referrals_sent, 1),
"new_customers_per_month": round(conversions, 1),
}
def calculate_monthly_program_cost(params, new_customers_per_month):
"""Total cost of running the program for one month."""
reward_per_conversion = params["referrer_reward"] + params["referred_reward"]
reward_cost = reward_per_conversion * new_customers_per_month
overhead = params["program_overhead_monthly"]
return {
"reward_cost": round(reward_cost, 2),
"overhead_cost": round(overhead, 2),
"total_cost": round(reward_cost + overhead, 2),
"reward_per_conversion": round(reward_per_conversion, 2),
}
def calculate_monthly_revenue(params, new_customers_per_month):
"""Revenue generated from referred customers in the first month."""
# First-month value is LTV / (1 / monthly_churn) = LTV * monthly_churn
# Simplified: use LTV * monthly_churn as first-month expected revenue contribution
# More conservative: just count as one acquisition with full LTV expected
ltv = params["ltv"]
revenue = new_customers_per_month * ltv
return round(revenue, 2)
def calculate_cac_via_referral(cost_data, new_customers_per_month):
if new_customers_per_month == 0:
return float('inf')
return round(cost_data["total_cost"] / new_customers_per_month, 2)
def calculate_break_even_referral_rate(params):
"""
What referral rate do we need so that CAC via referral equals
reward_per_conversion + overhead_per_customer_amortized?
We want: total_cost / new_customers = cac_target
Solving for referral_rate where cac_target = 50% of paid CAC (our target)
"""
target_cac = params["cac"] * 0.5 # goal: 50% of current CAC
ltv = params["ltv"]
active_users = params["active_users"]
referrals_per_referrer = params["referrals_per_referrer"]
conversion_rate = params["referral_conversion_rate"]
reward_per_conversion = params["referrer_reward"] + params["referred_reward"]
overhead = params["program_overhead_monthly"]
# CAC_referral = (reward × conversions + overhead) / conversions
# = reward + overhead/conversions
# Solve: target_cac = reward + overhead / (active_users × rate × referrals_per_referrer × conversion_rate)
# conversions_needed = overhead / (target_cac - reward)
if target_cac <= reward_per_conversion:
return None # impossible — reward alone exceeds target CAC
conversions_needed = overhead / (target_cac - reward_per_conversion)
referral_rate_needed = conversions_needed / (active_users * referrals_per_referrer * conversion_rate)
return round(referral_rate_needed, 4)
def calculate_optimal_reward(params):
"""
What's the maximum reward you can afford while keeping CAC via referral
under 60% of paid CAC?
max_total_reward = 0.60 × paid_CAC (using conversion-amortized overhead)
"""
target_cac = params["cac"] * 0.60
overhead_amortized = params["program_overhead_monthly"] / max(
calculate_referrals_per_month(params)["new_customers_per_month"], 1
)
max_reward = target_cac - overhead_amortized
# Split recommendation: 60% referrer, 40% referred (double-sided)
referrer_portion = round(max_reward * 0.60, 2)
referred_portion = round(max_reward * 0.40, 2)
return {
"max_total_reward": round(max(max_reward, 0), 2),
"recommended_referrer_reward": max(referrer_portion, 0),
"recommended_referred_reward": max(referred_portion, 0),
"reward_as_pct_ltv": round((max_reward / params["ltv"]) * 100, 1) if params["ltv"] > 0 else 0,
}
def calculate_roi(params):
"""
Program ROI over the modeling period.
ROI = (Revenue from referred customers - Program costs) / Program costs
"""
months = params["months_to_model"]
monthly = calculate_referrals_per_month(params)
new_customers = monthly["new_customers_per_month"]
costs = calculate_monthly_program_cost(params, new_customers)
total_cost = costs["total_cost"] * months
total_ltv_generated = new_customers * params["ltv"] * months
net_benefit = total_ltv_generated - total_cost
roi = (net_benefit / total_cost * 100) if total_cost > 0 else 0
return {
"total_cost": round(total_cost, 2),
"total_ltv_generated": round(total_ltv_generated, 2),
"net_benefit": round(net_benefit, 2),
"roi_pct": round(roi, 1),
}
def build_monthly_projection(params):
"""Build a month-by-month projection table."""
months = params["months_to_model"]
monthly = calculate_referrals_per_month(params)
new_per_month = monthly["new_customers_per_month"]
costs = calculate_monthly_program_cost(params, new_per_month)
ltv = params["ltv"]
rows = []
cumulative_customers = 0
cumulative_cost = 0
cumulative_revenue = 0
for m in range(1, months + 1):
cumulative_customers += new_per_month
month_cost = costs["total_cost"]
month_revenue = new_per_month * ltv
cumulative_cost += month_cost
cumulative_revenue += month_revenue
cumulative_net = cumulative_revenue - cumulative_cost
rows.append({
"month": m,
"new_customers": round(new_per_month, 1),
"cumulative_customers": round(cumulative_customers, 1),
"monthly_cost": round(month_cost, 2),
"cumulative_cost": round(cumulative_cost, 2),
"monthly_ltv": round(month_revenue, 2),
"cumulative_net": round(cumulative_net, 2),
})
return rows
def find_break_even_month(projection):
for row in projection:
if row["cumulative_net"] >= 0:
return row["month"]
return None
# ---------------------------------------------------------------------------
# Formatting
# ---------------------------------------------------------------------------
def format_currency(value):
return f",.2f"
def format_pct(value):
return f"{value:.1f}%"
def print_report(params, results):
monthly = results["monthly_referrals"]
costs = results["monthly_costs"]
cac = results["cac_via_referral"]
roi = results["roi"]
break_even_rate = results["break_even_referral_rate"]
optimal_reward = results["optimal_reward"]
projection = results["monthly_projection"]
break_even_month = results["break_even_month"]
paid_cac = params["cac"]
ltv = params["ltv"]
print("\n" + "=" * 60)
print("REFERRAL PROGRAM ROI CALCULATOR")
print("=" * 60)
print("\n📊 INPUT PARAMETERS")
print(f" LTV per customer: {format_currency(ltv)}")
print(f" Current paid CAC: {format_currency(paid_cac)}")
print(f" Active users: {params['active_users']:,}")
print(f" Referral rate (monthly): {format_pct(params['referral_rate'] * 100)}")
print(f" Referrals per referrer: {params['referrals_per_referrer']}")
print(f" Referral conversion rate: {format_pct(params['referral_conversion_rate'] * 100)}")
print(f" Referrer reward: {format_currency(params['referrer_reward'])}")
print(f" Referred user reward: {format_currency(params['referred_reward'])}")
print(f" Program overhead/month: {format_currency(params['program_overhead_monthly'])}")
print("\n📈 MONTHLY PERFORMANCE (STEADY STATE)")
print(f" Active referrers/month: {monthly['active_referrers']}")
print(f" Referrals sent/month: {monthly['referrals_sent']}")
print(f" New customers/month: {monthly['new_customers_per_month']}")
print(f" Monthly program cost: {format_currency(costs['total_cost'])}")
print(f" ↳ Reward cost: {format_currency(costs['reward_cost'])}")
print(f" ↳ Overhead: {format_currency(costs['overhead_cost'])}")
print(f" CAC via referral: {format_currency(cac)}")
print(f" Paid CAC: {format_currency(paid_cac)}")
savings_pct = ((paid_cac - cac) / paid_cac * 100) if paid_cac > 0 else 0
savings_label = f"{savings_pct:.0f}% cheaper than paid" if cac < paid_cac else "⚠️ More expensive than paid"
print(f" CAC comparison: {savings_label}")
print(f"\n💰 ROI OVER {params['months_to_model']} MONTHS")
print(f" Total program cost: {format_currency(roi['total_cost'])}")
print(f" Total LTV generated: {format_currency(roi['total_ltv_generated'])}")
print(f" Net benefit: {format_currency(roi['net_benefit'])}")
print(f" Program ROI: {format_pct(roi['roi_pct'])}")
if break_even_month:
print(f" Break-even: Month {break_even_month}")
else:
print(f" Break-even: Not reached in {params['months_to_model']} months")
print("\n🎯 OPTIMIZATION INSIGHTS")
if break_even_rate:
current_rate = params["referral_rate"]
rate_gap = break_even_rate - current_rate
if rate_gap > 0:
print(f" Break-even referral rate: {format_pct(break_even_rate * 100)} "
f"(you're at {format_pct(current_rate * 100)} — need +{format_pct(rate_gap * 100)})")
else:
print(f" Break-even referral rate: {format_pct(break_even_rate * 100)} ✅ Already above break-even")
else:
print(f" Break-even referral rate: ⚠️ Reward alone exceeds target CAC — reduce reward or increase LTV")
print(f"\n Optimal reward sizing (to keep CAC at ≤60% of paid CAC):")
print(f" Max total reward/referral: {format_currency(optimal_reward['max_total_reward'])}")
print(f" Recommended referrer: {format_currency(optimal_reward['recommended_referrer_reward'])}")
print(f" Recommended referred user: {format_currency(optimal_reward['recommended_referred_reward'])}")
print(f" Reward as % of LTV: {format_pct(optimal_reward['reward_as_pct_ltv'])}")
current_total_reward = params["referrer_reward"] + params["referred_reward"]
if current_total_reward > optimal_reward["max_total_reward"] and optimal_reward["max_total_reward"] > 0:
print(f" ⚠️ Your current reward ({format_currency(current_total_reward)}) "
f"exceeds optimal ({format_currency(optimal_reward['max_total_reward'])})")
elif optimal_reward["max_total_reward"] > 0:
print(f" ✅ Your current reward ({format_currency(current_total_reward)}) is within optimal range")
print(f"\n📅 MONTHLY PROJECTION (first {min(6, len(projection))} months)")
print(f" {'Month':>5} {'New Cust':>9} {'Cumul Cust':>11} {'Monthly Cost':>13} {'Cumul Net':>11}")
print(f" {'-'*5} {'-'*9} {'-'*11} {'-'*13} {'-'*11}")
for row in projection[:6]:
net_str = format_currency(row["cumulative_net"])
if row["cumulative_net"] < 0:
net_str = f"({format_currency(abs(row['cumulative_net']))})"
print(f" {row['month']:>5} {row['new_customers']:>9.1f} {row['cumulative_customers']:>11.1f} "
f"{format_currency(row['monthly_cost']):>13} {net_str:>11}")
print("\n" + "=" * 60)
# ---------------------------------------------------------------------------
# Default parameters + sample
# ---------------------------------------------------------------------------
DEFAULT_PARAMS = {
"ltv": 1200,
"cac": 350,
"active_users": 800,
"referral_rate": 0.06,
"referrals_per_referrer": 2.0,
"referral_conversion_rate": 0.20,
"referrer_reward": 50,
"referred_reward": 30,
"program_overhead_monthly": 200,
"churn_rate_monthly": 0.04,
"months_to_model": 12,
}
def run(params):
monthly = calculate_referrals_per_month(params)
new_customers = monthly["new_customers_per_month"]
costs = calculate_monthly_program_cost(params, new_customers)
cac = calculate_cac_via_referral(costs, new_customers)
break_even_rate = calculate_break_even_referral_rate(params)
optimal_reward = calculate_optimal_reward(params)
roi = calculate_roi(params)
projection = build_monthly_projection(params)
break_even_month = find_break_even_month(projection)
results = {
"monthly_referrals": monthly,
"monthly_costs": costs,
"cac_via_referral": cac,
"break_even_referral_rate": break_even_rate,
"optimal_reward": optimal_reward,
"roi": roi,
"monthly_projection": projection,
"break_even_month": break_even_month,
}
return results
# ---------------------------------------------------------------------------
# Main
# ---------------------------------------------------------------------------
def main():
import argparse
parser = argparse.ArgumentParser(
description="Calculates referral program ROI. "
"Models economics given LTV, CAC, referral rate, reward cost, "
"and conversion rate."
)
parser.add_argument(
"file", nargs="?", default=None,
help="Path to a JSON file with referral program parameters. "
"If omitted, reads from stdin or runs embedded sample."
)
args = parser.parse_args()
params = None
if args.file:
try:
with open(args.file) as f:
params = json.load(f)
except Exception as e:
print(f"Error reading file: {e}", file=sys.stderr)
sys.exit(1)
elif not sys.stdin.isatty():
raw = sys.stdin.read().strip()
if raw:
try:
params = json.loads(raw)
except Exception as e:
print(f"Error reading stdin: {e}", file=sys.stderr)
sys.exit(1)
else:
print("No input provided — running with sample parameters.\n")
params = DEFAULT_PARAMS
else:
print("No input provided — running with sample parameters.\n")
params = DEFAULT_PARAMS
# Fill in defaults for any missing keys
for k, v in DEFAULT_PARAMS.items():
params.setdefault(k, v)
results = run(params)
print_report(params, results)
# JSON output
json_output = {
"inputs": params,
"results": {
"monthly_new_customers": results["monthly_referrals"]["new_customers_per_month"],
"cac_via_referral": results["cac_via_referral"],
"program_roi_pct": results["roi"]["roi_pct"],
"break_even_month": results["break_even_month"],
"break_even_referral_rate": results["break_even_referral_rate"],
"optimal_total_reward": results["optimal_reward"]["max_total_reward"],
"net_benefit_12mo": results["roi"]["net_benefit"],
}
}
print("\n--- JSON Output ---")
print(json.dumps(json_output, indent=2))
if __name__ == "__main__":
main()
Thiết kế quy trình phỏng vấn, pipeline tuyển dụng, bộ câu hỏi, ma trận năng lực, thang chấm điểm và phân tích thiên kiến người phỏng vấn.
---
name: "interview-system-designer"
description: This skill should be used when the user asks to "design interview processes", "create hiring pipelines", "calibrate interview loops", "generate interview questions", "design competency matrices", "analyze interviewer bias", "create scoring rubrics", "build question banks", or "optimize hiring systems". Use for designing role-specific interview loops, competency assessments, and hiring calibration systems.
---
# Interview System Designer
Comprehensive interview loop planning and calibration support for role-based hiring systems.
## Overview
Use this skill to create structured interview loops, standardize question quality, and keep hiring signal consistent across interviewers.
## Core Capabilities
- Interview loop planning by role and level
- Round-by-round focus and timing recommendations
- Suggested question sets by round type
- Framework support for scoring and calibration
- Bias-reduction and process consistency guidance
## Quick Start
```bash
# Generate a loop plan for a role and level
python3 scripts/interview_planner.py --role "Senior Software Engineer" --level senior
# JSON output for integration with internal tooling
python3 scripts/interview_planner.py --role "Product Manager" --level mid --json
```
## Recommended Workflow
1. Run `scripts/interview_planner.py` to generate a baseline loop.
2. Align rounds to role-specific competencies.
3. Validate scoring rubric consistency with interview panel leads.
4. Review for bias controls before rollout.
5. Recalibrate quarterly using hiring outcome data.
## References
- `references/interview-frameworks.md`
- `references/bias_mitigation_checklist.md`
- `references/competency_matrix_templates.md`
- `references/debrief_facilitation_guide.md`
## Common Pitfalls
- Overweighting one round while ignoring other competency signals
- Using unstructured interviews without standardized scoring
- Skipping calibration sessions for interviewers
- Changing hiring bar without documenting rationale
## Best Practices
1. Keep round objectives explicit and non-overlapping.
2. Require evidence for each score recommendation.
3. Use the same baseline rubric across comparable roles.
4. Revisit loop design based on quality-of-hire outcomes.
FILE:assets/sample_interview_results.json
[
{
"candidate_id": "candidate_001",
"role": "Senior Software Engineer",
"interviewer_id": "interviewer_alice",
"date": "2024-01-15T09:00:00Z",
"scores": {
"coding_fundamentals": 3.5,
"system_design": 4.0,
"technical_leadership": 3.0,
"communication": 3.5,
"problem_solving": 4.0
},
"overall_recommendation": "Hire",
"gender": "male",
"ethnicity": "asian",
"years_experience": 6,
"university_tier": "tier_1",
"previous_company_size": "large"
},
{
"candidate_id": "candidate_001",
"role": "Senior Software Engineer",
"interviewer_id": "interviewer_bob",
"date": "2024-01-15T11:00:00Z",
"scores": {
"system_design": 3.5,
"technical_leadership": 3.5,
"mentoring": 3.0,
"cross_team_collaboration": 4.0,
"strategic_thinking": 3.5
},
"overall_recommendation": "Hire",
"gender": "male",
"ethnicity": "asian",
"years_experience": 6,
"university_tier": "tier_1",
"previous_company_size": "large"
},
{
"candidate_id": "candidate_002",
"role": "Senior Software Engineer",
"interviewer_id": "interviewer_alice",
"date": "2024-01-16T09:00:00Z",
"scores": {
"coding_fundamentals": 2.5,
"system_design": 3.0,
"technical_leadership": 2.0,
"communication": 3.0,
"problem_solving": 3.0
},
"overall_recommendation": "No Hire",
"gender": "female",
"ethnicity": "hispanic",
"years_experience": 5,
"university_tier": "tier_2",
"previous_company_size": "startup"
},
{
"candidate_id": "candidate_002",
"role": "Senior Software Engineer",
"interviewer_id": "interviewer_charlie",
"date": "2024-01-16T11:00:00Z",
"scores": {
"system_design": 2.0,
"technical_leadership": 2.5,
"mentoring": 2.0,
"cross_team_collaboration": 3.0,
"strategic_thinking": 2.5
},
"overall_recommendation": "No Hire",
"gender": "female",
"ethnicity": "hispanic",
"years_experience": 5,
"university_tier": "tier_2",
"previous_company_size": "startup"
},
{
"candidate_id": "candidate_003",
"role": "Senior Software Engineer",
"interviewer_id": "interviewer_david",
"date": "2024-01-17T14:00:00Z",
"scores": {
"coding_fundamentals": 4.0,
"system_design": 3.5,
"technical_leadership": 4.0,
"communication": 4.0,
"problem_solving": 3.5
},
"overall_recommendation": "Strong Hire",
"gender": "male",
"ethnicity": "white",
"years_experience": 8,
"university_tier": "tier_1",
"previous_company_size": "large"
},
{
"candidate_id": "candidate_003",
"role": "Senior Software Engineer",
"interviewer_id": "interviewer_alice",
"date": "2024-01-17T16:00:00Z",
"scores": {
"system_design": 4.0,
"technical_leadership": 4.0,
"mentoring": 3.5,
"cross_team_collaboration": 4.0,
"strategic_thinking": 3.5
},
"overall_recommendation": "Hire",
"gender": "male",
"ethnicity": "white",
"years_experience": 8,
"university_tier": "tier_1",
"previous_company_size": "large"
},
{
"candidate_id": "candidate_004",
"role": "Product Manager",
"interviewer_id": "interviewer_emma",
"date": "2024-01-18T10:00:00Z",
"scores": {
"product_strategy": 3.0,
"user_research": 3.5,
"data_analysis": 4.0,
"stakeholder_management": 3.0,
"communication": 3.5
},
"overall_recommendation": "Hire",
"gender": "female",
"ethnicity": "black",
"years_experience": 4,
"university_tier": "tier_2",
"previous_company_size": "medium"
},
{
"candidate_id": "candidate_005",
"role": "Product Manager",
"interviewer_id": "interviewer_frank",
"date": "2024-01-19T13:00:00Z",
"scores": {
"product_strategy": 2.5,
"user_research": 2.0,
"data_analysis": 3.0,
"stakeholder_management": 2.5,
"communication": 3.0
},
"overall_recommendation": "No Hire",
"gender": "male",
"ethnicity": "white",
"years_experience": 3,
"university_tier": "tier_3",
"previous_company_size": "startup"
},
{
"candidate_id": "candidate_006",
"role": "Junior Software Engineer",
"interviewer_id": "interviewer_alice",
"date": "2024-01-20T09:00:00Z",
"scores": {
"coding_fundamentals": 3.0,
"debugging": 3.5,
"testing_basics": 3.0,
"collaboration": 4.0,
"learning_agility": 3.5
},
"overall_recommendation": "Hire",
"gender": "female",
"ethnicity": "asian",
"years_experience": 1,
"university_tier": "bootcamp",
"previous_company_size": "none"
},
{
"candidate_id": "candidate_007",
"role": "Junior Software Engineer",
"interviewer_id": "interviewer_bob",
"date": "2024-01-21T10:30:00Z",
"scores": {
"coding_fundamentals": 2.0,
"debugging": 2.5,
"testing_basics": 2.0,
"collaboration": 3.0,
"learning_agility": 3.0
},
"overall_recommendation": "No Hire",
"gender": "male",
"ethnicity": "hispanic",
"years_experience": 0,
"university_tier": "tier_2",
"previous_company_size": "none"
},
{
"candidate_id": "candidate_008",
"role": "Staff Frontend Engineer",
"interviewer_id": "interviewer_grace",
"date": "2024-01-22T14:00:00Z",
"scores": {
"frontend_architecture": 4.0,
"system_design": 4.0,
"technical_leadership": 4.0,
"team_building": 3.5,
"strategic_thinking": 3.5
},
"overall_recommendation": "Strong Hire",
"gender": "female",
"ethnicity": "white",
"years_experience": 9,
"university_tier": "tier_1",
"previous_company_size": "large"
},
{
"candidate_id": "candidate_008",
"role": "Staff Frontend Engineer",
"interviewer_id": "interviewer_henry",
"date": "2024-01-22T16:00:00Z",
"scores": {
"frontend_architecture": 3.5,
"technical_leadership": 4.0,
"team_building": 4.0,
"cross_functional_collaboration": 4.0,
"organizational_impact": 3.5
},
"overall_recommendation": "Hire",
"gender": "female",
"ethnicity": "white",
"years_experience": 9,
"university_tier": "tier_1",
"previous_company_size": "large"
},
{
"candidate_id": "candidate_009",
"role": "Data Scientist",
"interviewer_id": "interviewer_ivan",
"date": "2024-01-23T11:00:00Z",
"scores": {
"statistical_analysis": 3.5,
"machine_learning": 4.0,
"data_engineering": 3.0,
"business_acumen": 3.5,
"communication": 3.0
},
"overall_recommendation": "Hire",
"gender": "male",
"ethnicity": "indian",
"years_experience": 5,
"university_tier": "tier_1",
"previous_company_size": "medium"
},
{
"candidate_id": "candidate_010",
"role": "DevOps Engineer",
"interviewer_id": "interviewer_jane",
"date": "2024-01-24T15:00:00Z",
"scores": {
"infrastructure_automation": 3.5,
"ci_cd_design": 4.0,
"monitoring_observability": 3.0,
"security_implementation": 3.5,
"incident_management": 4.0
},
"overall_recommendation": "Hire",
"gender": "female",
"ethnicity": "black",
"years_experience": 6,
"university_tier": "tier_2",
"previous_company_size": "startup"
},
{
"candidate_id": "candidate_011",
"role": "UX Designer",
"interviewer_id": "interviewer_karl",
"date": "2024-01-25T10:00:00Z",
"scores": {
"design_process": 4.0,
"user_research": 3.5,
"design_systems": 4.0,
"cross_functional_collaboration": 3.5,
"design_leadership": 3.0
},
"overall_recommendation": "Hire",
"gender": "non_binary",
"ethnicity": "white",
"years_experience": 7,
"university_tier": "tier_1",
"previous_company_size": "medium"
},
{
"candidate_id": "candidate_012",
"role": "Engineering Manager",
"interviewer_id": "interviewer_lisa",
"date": "2024-01-26T13:30:00Z",
"scores": {
"people_leadership": 4.0,
"technical_background": 3.5,
"strategic_thinking": 3.5,
"performance_management": 4.0,
"cross_functional_leadership": 3.5
},
"overall_recommendation": "Hire",
"gender": "male",
"ethnicity": "white",
"years_experience": 8,
"university_tier": "tier_1",
"previous_company_size": "large"
},
{
"candidate_id": "candidate_013",
"role": "Senior Software Engineer",
"interviewer_id": "interviewer_alice",
"date": "2024-01-27T09:00:00Z",
"scores": {
"coding_fundamentals": 4.0,
"system_design": 4.0,
"technical_leadership": 4.0,
"communication": 4.0,
"problem_solving": 4.0
},
"overall_recommendation": "Strong Hire",
"gender": "female",
"ethnicity": "asian",
"years_experience": 7,
"university_tier": "tier_1",
"previous_company_size": "large"
},
{
"candidate_id": "candidate_013",
"role": "Senior Software Engineer",
"interviewer_id": "interviewer_charlie",
"date": "2024-01-27T11:00:00Z",
"scores": {
"system_design": 3.5,
"technical_leadership": 3.5,
"mentoring": 4.0,
"cross_team_collaboration": 4.0,
"strategic_thinking": 3.5
},
"overall_recommendation": "Hire",
"gender": "female",
"ethnicity": "asian",
"years_experience": 7,
"university_tier": "tier_1",
"previous_company_size": "large"
},
{
"candidate_id": "candidate_014",
"role": "Senior Software Engineer",
"interviewer_id": "interviewer_david",
"date": "2024-01-28T14:00:00Z",
"scores": {
"coding_fundamentals": 1.5,
"system_design": 2.0,
"technical_leadership": 1.0,
"communication": 2.0,
"problem_solving": 2.0
},
"overall_recommendation": "Strong No Hire",
"gender": "male",
"ethnicity": "white",
"years_experience": 4,
"university_tier": "tier_3",
"previous_company_size": "startup"
},
{
"candidate_id": "candidate_015",
"role": "Product Manager",
"interviewer_id": "interviewer_emma",
"date": "2024-01-29T11:00:00Z",
"scores": {
"product_strategy": 4.0,
"user_research": 3.5,
"data_analysis": 4.0,
"stakeholder_management": 4.0,
"communication": 3.5
},
"overall_recommendation": "Strong Hire",
"gender": "male",
"ethnicity": "black",
"years_experience": 5,
"university_tier": "tier_2",
"previous_company_size": "medium"
}
]
FILE:assets/sample_role_definitions.json
[
{
"role": "Senior Software Engineer",
"level": "senior",
"team": "platform",
"department": "engineering",
"competencies": [
"system_design",
"coding_fundamentals",
"technical_leadership",
"mentoring",
"cross_team_collaboration"
],
"requirements": {
"years_experience": "5-8",
"technical_skills": ["Python", "Java", "Docker", "Kubernetes", "AWS"],
"leadership_experience": true,
"mentoring_required": true
},
"hiring_bar": "high",
"interview_focus": ["technical_depth", "system_architecture", "leadership_potential"]
},
{
"role": "Product Manager",
"level": "mid",
"team": "growth",
"department": "product",
"competencies": [
"product_strategy",
"user_research",
"data_analysis",
"stakeholder_management",
"cross_functional_leadership"
],
"requirements": {
"years_experience": "3-5",
"domain_knowledge": ["user_analytics", "experimentation", "product_metrics"],
"leadership_experience": false,
"technical_background": "preferred"
},
"hiring_bar": "medium-high",
"interview_focus": ["product_sense", "analytical_thinking", "execution_ability"]
},
{
"role": "Staff Frontend Engineer",
"level": "staff",
"team": "consumer",
"department": "engineering",
"competencies": [
"frontend_architecture",
"system_design",
"technical_leadership",
"team_building",
"cross_functional_collaboration"
],
"requirements": {
"years_experience": "8+",
"technical_skills": ["React", "TypeScript", "GraphQL", "Webpack", "Performance Optimization"],
"leadership_experience": true,
"architecture_experience": true
},
"hiring_bar": "very-high",
"interview_focus": ["architectural_vision", "technical_strategy", "organizational_impact"]
},
{
"role": "Data Scientist",
"level": "mid",
"team": "ml_platform",
"department": "data",
"competencies": [
"statistical_analysis",
"machine_learning",
"data_engineering",
"business_acumen",
"communication"
],
"requirements": {
"years_experience": "3-6",
"technical_skills": ["Python", "SQL", "TensorFlow", "Spark", "Statistics"],
"domain_knowledge": ["ML algorithms", "experimentation", "data_pipelines"],
"leadership_experience": false
},
"hiring_bar": "high",
"interview_focus": ["technical_depth", "problem_solving", "business_impact"]
},
{
"role": "DevOps Engineer",
"level": "senior",
"team": "infrastructure",
"department": "engineering",
"competencies": [
"infrastructure_automation",
"ci_cd_design",
"monitoring_observability",
"security_implementation",
"incident_management"
],
"requirements": {
"years_experience": "5-7",
"technical_skills": ["Kubernetes", "Terraform", "AWS", "Docker", "Monitoring"],
"security_background": "required",
"leadership_experience": "preferred"
},
"hiring_bar": "high",
"interview_focus": ["system_reliability", "automation_expertise", "operational_excellence"]
},
{
"role": "UX Designer",
"level": "senior",
"team": "design_systems",
"department": "design",
"competencies": [
"design_process",
"user_research",
"design_systems",
"cross_functional_collaboration",
"design_leadership"
],
"requirements": {
"years_experience": "5-8",
"portfolio_quality": "high",
"research_experience": true,
"systems_thinking": true
},
"hiring_bar": "high",
"interview_focus": ["design_process", "systems_thinking", "user_advocacy"]
},
{
"role": "Engineering Manager",
"level": "senior",
"team": "backend",
"department": "engineering",
"competencies": [
"people_leadership",
"technical_background",
"strategic_thinking",
"performance_management",
"cross_functional_leadership"
],
"requirements": {
"years_experience": "6-10",
"management_experience": "2+ years",
"technical_background": "required",
"hiring_experience": true
},
"hiring_bar": "very-high",
"interview_focus": ["people_leadership", "technical_judgment", "organizational_impact"]
},
{
"role": "Junior Software Engineer",
"level": "junior",
"team": "web",
"department": "engineering",
"competencies": [
"coding_fundamentals",
"debugging",
"testing_basics",
"collaboration",
"learning_agility"
],
"requirements": {
"years_experience": "0-2",
"technical_skills": ["JavaScript", "HTML/CSS", "Git", "Basic Algorithms"],
"education": "CS degree or bootcamp",
"growth_mindset": true
},
"hiring_bar": "medium",
"interview_focus": ["coding_ability", "problem_solving", "potential_assessment"]
}
]
FILE:expected_outputs/product_manager_senior_questions.json
{
"role": "Product Manager",
"level": "senior",
"competencies": [
"strategy",
"analytics",
"business_strategy",
"product_strategy",
"stakeholder_management",
"p&l_responsibility",
"leadership",
"team_leadership",
"user_research",
"data_analysis"
],
"question_types": [
"technical",
"behavioral",
"situational"
],
"generated_at": "2026-02-16T13:27:41.303329",
"total_questions": 20,
"questions": [
{
"question": "What challenges have you faced related to p&l responsibility and how did you overcome them?",
"competency": "p&l_responsibility",
"type": "challenge_based",
"focus_areas": [
"problem_solving",
"learning_from_experience"
]
},
{
"question": "Analyze conversion funnel data to identify the biggest drop-off point and propose solutions.",
"competency": "data_analysis",
"type": "analytical",
"difficulty": "medium",
"time_limit": 45,
"key_concepts": [
"funnel_analysis",
"conversion_optimization",
"statistical_significance"
]
},
{
"question": "What challenges have you faced related to team leadership and how did you overcome them?",
"competency": "team_leadership",
"type": "challenge_based",
"focus_areas": [
"problem_solving",
"learning_from_experience"
]
},
{
"question": "Design a go-to-market strategy for a new B2B SaaS product entering a competitive market.",
"competency": "product_strategy",
"type": "strategic",
"difficulty": "hard",
"time_limit": 60,
"key_concepts": [
"market_analysis",
"competitive_positioning",
"pricing_strategy",
"channel_strategy"
]
},
{
"question": "What challenges have you faced related to business strategy and how did you overcome them?",
"competency": "business_strategy",
"type": "challenge_based",
"focus_areas": [
"problem_solving",
"learning_from_experience"
]
},
{
"question": "Describe your experience with business strategy in your current or previous role.",
"competency": "business_strategy",
"type": "experience",
"focus_areas": [
"experience_depth",
"practical_application"
]
},
{
"question": "Describe your experience with team leadership in your current or previous role.",
"competency": "team_leadership",
"type": "experience",
"focus_areas": [
"experience_depth",
"practical_application"
]
},
{
"question": "Describe a situation where you had to influence someone without having direct authority over them.",
"competency": "leadership",
"type": "behavioral",
"method": "STAR",
"focus_areas": [
"influence",
"persuasion",
"stakeholder_management"
]
},
{
"question": "Given a dataset of user activities, calculate the daily active users for the past month.",
"competency": "data_analysis",
"type": "analytical",
"difficulty": "easy",
"time_limit": 30,
"key_concepts": [
"sql_basics",
"date_functions",
"aggregation"
]
},
{
"question": "Describe your experience with analytics in your current or previous role.",
"competency": "analytics",
"type": "experience",
"focus_areas": [
"experience_depth",
"practical_application"
]
},
{
"question": "How would you prioritize features for a mobile app with limited engineering resources?",
"competency": "product_strategy",
"type": "case_study",
"difficulty": "medium",
"time_limit": 45,
"key_concepts": [
"prioritization_frameworks",
"resource_allocation",
"impact_estimation"
]
},
{
"question": "Describe your experience with stakeholder management in your current or previous role.",
"competency": "stakeholder_management",
"type": "experience",
"focus_areas": [
"experience_depth",
"practical_application"
]
},
{
"question": "What challenges have you faced related to stakeholder management and how did you overcome them?",
"competency": "stakeholder_management",
"type": "challenge_based",
"focus_areas": [
"problem_solving",
"learning_from_experience"
]
},
{
"question": "What challenges have you faced related to user research and how did you overcome them?",
"competency": "user_research",
"type": "challenge_based",
"focus_areas": [
"problem_solving",
"learning_from_experience"
]
},
{
"question": "What challenges have you faced related to strategy and how did you overcome them?",
"competency": "strategy",
"type": "challenge_based",
"focus_areas": [
"problem_solving",
"learning_from_experience"
]
},
{
"question": "Describe your experience with user research in your current or previous role.",
"competency": "user_research",
"type": "experience",
"focus_areas": [
"experience_depth",
"practical_application"
]
},
{
"question": "Describe your experience with p&l responsibility in your current or previous role.",
"competency": "p&l_responsibility",
"type": "experience",
"focus_areas": [
"experience_depth",
"practical_application"
]
},
{
"question": "Describe your experience with strategy in your current or previous role.",
"competency": "strategy",
"type": "experience",
"focus_areas": [
"experience_depth",
"practical_application"
]
},
{
"question": "Tell me about a time when you had to lead a team through a significant change or challenge.",
"competency": "leadership",
"type": "behavioral",
"method": "STAR",
"focus_areas": [
"change_management",
"team_motivation",
"communication"
]
},
{
"question": "What challenges have you faced related to analytics and how did you overcome them?",
"competency": "analytics",
"type": "challenge_based",
"focus_areas": [
"problem_solving",
"learning_from_experience"
]
}
],
"scoring_rubrics": {
"question_8": {
"question": "Describe a situation where you had to influence someone without having direct authority over them.",
"competency": "leadership",
"type": "behavioral",
"scoring_criteria": {
"situation_clarity": {
"4": "Clear, specific situation with relevant context and stakes",
"3": "Good situation description with adequate context",
"2": "Situation described but lacks some specifics",
"1": "Vague or unclear situation description"
},
"action_quality": {
"4": "Specific, thoughtful actions showing strong competency",
"3": "Good actions demonstrating competency",
"2": "Adequate actions but could be stronger",
"1": "Weak or inappropriate actions"
},
"result_impact": {
"4": "Significant positive impact with measurable results",
"3": "Good positive impact with clear outcomes",
"2": "Some positive impact demonstrated",
"1": "Little or no positive impact shown"
},
"self_awareness": {
"4": "Excellent self-reflection, learns from experience, acknowledges growth areas",
"3": "Good self-awareness and learning orientation",
"2": "Some self-reflection demonstrated",
"1": "Limited self-awareness or reflection"
}
},
"weight": "high",
"time_limit": 30
},
"question_19": {
"question": "Tell me about a time when you had to lead a team through a significant change or challenge.",
"competency": "leadership",
"type": "behavioral",
"scoring_criteria": {
"situation_clarity": {
"4": "Clear, specific situation with relevant context and stakes",
"3": "Good situation description with adequate context",
"2": "Situation described but lacks some specifics",
"1": "Vague or unclear situation description"
},
"action_quality": {
"4": "Specific, thoughtful actions showing strong competency",
"3": "Good actions demonstrating competency",
"2": "Adequate actions but could be stronger",
"1": "Weak or inappropriate actions"
},
"result_impact": {
"4": "Significant positive impact with measurable results",
"3": "Good positive impact with clear outcomes",
"2": "Some positive impact demonstrated",
"1": "Little or no positive impact shown"
},
"self_awareness": {
"4": "Excellent self-reflection, learns from experience, acknowledges growth areas",
"3": "Good self-awareness and learning orientation",
"2": "Some self-reflection demonstrated",
"1": "Limited self-awareness or reflection"
}
},
"weight": "high",
"time_limit": 30
}
},
"follow_up_probes": {
"question_1": [
"Can you provide more specific details about your approach?",
"What would you do differently if you had to do this again?",
"What challenges did you face and how did you overcome them?"
],
"question_2": [
"Can you provide more specific details about your approach?",
"What would you do differently if you had to do this again?",
"What challenges did you face and how did you overcome them?"
],
"question_3": [
"Can you provide more specific details about your approach?",
"What would you do differently if you had to do this again?",
"What challenges did you face and how did you overcome them?"
],
"question_4": [
"Can you provide more specific details about your approach?",
"What would you do differently if you had to do this again?",
"What challenges did you face and how did you overcome them?"
],
"question_5": [
"Can you provide more specific details about your approach?",
"What would you do differently if you had to do this again?",
"What challenges did you face and how did you overcome them?"
],
"question_6": [
"Can you provide more specific details about your approach?",
"What would you do differently if you had to do this again?",
"What challenges did you face and how did you overcome them?"
],
"question_7": [
"Can you provide more specific details about your approach?",
"What would you do differently if you had to do this again?",
"What challenges did you face and how did you overcome them?"
],
"question_8": [
"What would you do differently if you faced this situation again?",
"How did you handle team members who were resistant to the change?",
"What metrics did you use to measure success?",
"How did you communicate progress to stakeholders?",
"What did you learn from this experience?"
],
"question_9": [
"Can you provide more specific details about your approach?",
"What would you do differently if you had to do this again?",
"What challenges did you face and how did you overcome them?"
],
"question_10": [
"Can you provide more specific details about your approach?",
"What would you do differently if you had to do this again?",
"What challenges did you face and how did you overcome them?"
],
"question_11": [
"Can you provide more specific details about your approach?",
"What would you do differently if you had to do this again?",
"What challenges did you face and how did you overcome them?"
],
"question_12": [
"Can you provide more specific details about your approach?",
"What would you do differently if you had to do this again?",
"What challenges did you face and how did you overcome them?"
],
"question_13": [
"Can you provide more specific details about your approach?",
"What would you do differently if you had to do this again?",
"What challenges did you face and how did you overcome them?"
],
"question_14": [
"Can you provide more specific details about your approach?",
"What would you do differently if you had to do this again?",
"What challenges did you face and how did you overcome them?"
],
"question_15": [
"Can you provide more specific details about your approach?",
"What would you do differently if you had to do this again?",
"What challenges did you face and how did you overcome them?"
],
"question_16": [
"Can you provide more specific details about your approach?",
"What would you do differently if you had to do this again?",
"What challenges did you face and how did you overcome them?"
],
"question_17": [
"Can you provide more specific details about your approach?",
"What would you do differently if you had to do this again?",
"What challenges did you face and how did you overcome them?"
],
"question_18": [
"Can you provide more specific details about your approach?",
"What would you do differently if you had to do this again?",
"What challenges did you face and how did you overcome them?"
],
"question_19": [
"What would you do differently if you faced this situation again?",
"How did you handle team members who were resistant to the change?",
"What metrics did you use to measure success?",
"How did you communicate progress to stakeholders?",
"What did you learn from this experience?"
],
"question_20": [
"Can you provide more specific details about your approach?",
"What would you do differently if you had to do this again?",
"What challenges did you face and how did you overcome them?"
]
},
"calibration_examples": {
"question_1": {
"question": "What challenges have you faced related to p&l responsibility and how did you overcome them?",
"competency": "p&l_responsibility",
"sample_answers": {
"poor_answer": {
"answer": "Sample poor answer for p&l_responsibility question - lacks detail, specificity, or demonstrates weak competency",
"score": "1-2",
"issues": [
"Vague response",
"Limited evidence of competency",
"Poor structure"
]
},
"good_answer": {
"answer": "Sample good answer for p&l_responsibility question - adequate detail, demonstrates competency clearly",
"score": "3",
"strengths": [
"Clear structure",
"Demonstrates competency",
"Adequate detail"
]
},
"great_answer": {
"answer": "Sample excellent answer for p&l_responsibility question - exceptional detail, strong evidence, goes above and beyond",
"score": "4",
"strengths": [
"Exceptional detail",
"Strong evidence",
"Strategic thinking",
"Goes beyond requirements"
]
}
},
"scoring_rationale": {
"key_indicators": "Look for evidence of p&l responsibility competency",
"red_flags": "Vague answers, lack of specifics, negative outcomes without learning",
"green_flags": "Specific examples, clear impact, demonstrates growth and learning"
}
},
"question_2": {
"question": "Analyze conversion funnel data to identify the biggest drop-off point and propose solutions.",
"competency": "data_analysis",
"sample_answers": {
"poor_answer": {
"answer": "Sample poor answer for data_analysis question - lacks detail, specificity, or demonstrates weak competency",
"score": "1-2",
"issues": [
"Vague response",
"Limited evidence of competency",
"Poor structure"
]
},
"good_answer": {
"answer": "Sample good answer for data_analysis question - adequate detail, demonstrates competency clearly",
"score": "3",
"strengths": [
"Clear structure",
"Demonstrates competency",
"Adequate detail"
]
},
"great_answer": {
"answer": "Sample excellent answer for data_analysis question - exceptional detail, strong evidence, goes above and beyond",
"score": "4",
"strengths": [
"Exceptional detail",
"Strong evidence",
"Strategic thinking",
"Goes beyond requirements"
]
}
},
"scoring_rationale": {
"key_indicators": "Look for evidence of data analysis competency",
"red_flags": "Vague answers, lack of specifics, negative outcomes without learning",
"green_flags": "Specific examples, clear impact, demonstrates growth and learning"
}
},
"question_3": {
"question": "What challenges have you faced related to team leadership and how did you overcome them?",
"competency": "team_leadership",
"sample_answers": {
"poor_answer": {
"answer": "Sample poor answer for team_leadership question - lacks detail, specificity, or demonstrates weak competency",
"score": "1-2",
"issues": [
"Vague response",
"Limited evidence of competency",
"Poor structure"
]
},
"good_answer": {
"answer": "Sample good answer for team_leadership question - adequate detail, demonstrates competency clearly",
"score": "3",
"strengths": [
"Clear structure",
"Demonstrates competency",
"Adequate detail"
]
},
"great_answer": {
"answer": "Sample excellent answer for team_leadership question - exceptional detail, strong evidence, goes above and beyond",
"score": "4",
"strengths": [
"Exceptional detail",
"Strong evidence",
"Strategic thinking",
"Goes beyond requirements"
]
}
},
"scoring_rationale": {
"key_indicators": "Look for evidence of team leadership competency",
"red_flags": "Vague answers, lack of specifics, negative outcomes without learning",
"green_flags": "Specific examples, clear impact, demonstrates growth and learning"
}
},
"question_4": {
"question": "Design a go-to-market strategy for a new B2B SaaS product entering a competitive market.",
"competency": "product_strategy",
"sample_answers": {
"poor_answer": {
"answer": "Sample poor answer for product_strategy question - lacks detail, specificity, or demonstrates weak competency",
"score": "1-2",
"issues": [
"Vague response",
"Limited evidence of competency",
"Poor structure"
]
},
"good_answer": {
"answer": "Sample good answer for product_strategy question - adequate detail, demonstrates competency clearly",
"score": "3",
"strengths": [
"Clear structure",
"Demonstrates competency",
"Adequate detail"
]
},
"great_answer": {
"answer": "Sample excellent answer for product_strategy question - exceptional detail, strong evidence, goes above and beyond",
"score": "4",
"strengths": [
"Exceptional detail",
"Strong evidence",
"Strategic thinking",
"Goes beyond requirements"
]
}
},
"scoring_rationale": {
"key_indicators": "Look for evidence of product strategy competency",
"red_flags": "Vague answers, lack of specifics, negative outcomes without learning",
"green_flags": "Specific examples, clear impact, demonstrates growth and learning"
}
},
"question_5": {
"question": "What challenges have you faced related to business strategy and how did you overcome them?",
"competency": "business_strategy",
"sample_answers": {
"poor_answer": {
"answer": "Sample poor answer for business_strategy question - lacks detail, specificity, or demonstrates weak competency",
"score": "1-2",
"issues": [
"Vague response",
"Limited evidence of competency",
"Poor structure"
]
},
"good_answer": {
"answer": "Sample good answer for business_strategy question - adequate detail, demonstrates competency clearly",
"score": "3",
"strengths": [
"Clear structure",
"Demonstrates competency",
"Adequate detail"
]
},
"great_answer": {
"answer": "Sample excellent answer for business_strategy question - exceptional detail, strong evidence, goes above and beyond",
"score": "4",
"strengths": [
"Exceptional detail",
"Strong evidence",
"Strategic thinking",
"Goes beyond requirements"
]
}
},
"scoring_rationale": {
"key_indicators": "Look for evidence of business strategy competency",
"red_flags": "Vague answers, lack of specifics, negative outcomes without learning",
"green_flags": "Specific examples, clear impact, demonstrates growth and learning"
}
}
},
"usage_guidelines": {
"interview_flow": {
"warm_up": "Start with 1-2 easier questions to build rapport",
"core_assessment": "Focus majority of time on core competency questions",
"closing": "End with questions about candidate's questions/interests"
},
"time_management": {
"technical_questions": "Allow extra time for coding/design questions",
"behavioral_questions": "Keep to time limits but allow for follow-ups",
"total_recommendation": "45-75 minutes per interview round"
},
"question_selection": {
"variety": "Mix question types within each competency area",
"difficulty": "Adjust based on candidate responses and energy",
"customization": "Adapt questions based on candidate's background"
},
"common_mistakes": [
"Don't ask all questions mechanically",
"Don't skip follow-up questions",
"Don't forget to assess cultural fit alongside competencies",
"Don't let one strong/weak area bias overall assessment"
],
"calibration_reminders": [
"Compare against role standard, not other candidates",
"Focus on evidence demonstrated, not potential",
"Consider level-appropriate expectations",
"Document specific examples in feedback"
]
}
}
FILE:expected_outputs/product_manager_senior_questions.txt
Interview Question Bank: Product Manager (Senior Level)
======================================================================
Generated: 2026-02-16T13:27:41.303329
Total Questions: 20
Question Types: technical, behavioral, situational
Target Competencies: strategy, analytics, business_strategy, product_strategy, stakeholder_management, p&l_responsibility, leadership, team_leadership, user_research, data_analysis
INTERVIEW QUESTIONS
--------------------------------------------------
1. What challenges have you faced related to p&l responsibility and how did you overcome them?
Competency: P&L Responsibility
Type: Challenge_Based
Focus Areas: problem_solving, learning_from_experience
2. Analyze conversion funnel data to identify the biggest drop-off point and propose solutions.
Competency: Data Analysis
Type: Analytical
Time Limit: 45 minutes
3. What challenges have you faced related to team leadership and how did you overcome them?
Competency: Team Leadership
Type: Challenge_Based
Focus Areas: problem_solving, learning_from_experience
4. Design a go-to-market strategy for a new B2B SaaS product entering a competitive market.
Competency: Product Strategy
Type: Strategic
Time Limit: 60 minutes
5. What challenges have you faced related to business strategy and how did you overcome them?
Competency: Business Strategy
Type: Challenge_Based
Focus Areas: problem_solving, learning_from_experience
6. Describe your experience with business strategy in your current or previous role.
Competency: Business Strategy
Type: Experience
Focus Areas: experience_depth, practical_application
7. Describe your experience with team leadership in your current or previous role.
Competency: Team Leadership
Type: Experience
Focus Areas: experience_depth, practical_application
8. Describe a situation where you had to influence someone without having direct authority over them.
Competency: Leadership
Type: Behavioral
Focus Areas: influence, persuasion, stakeholder_management
9. Given a dataset of user activities, calculate the daily active users for the past month.
Competency: Data Analysis
Type: Analytical
Time Limit: 30 minutes
10. Describe your experience with analytics in your current or previous role.
Competency: Analytics
Type: Experience
Focus Areas: experience_depth, practical_application
11. How would you prioritize features for a mobile app with limited engineering resources?
Competency: Product Strategy
Type: Case_Study
Time Limit: 45 minutes
12. Describe your experience with stakeholder management in your current or previous role.
Competency: Stakeholder Management
Type: Experience
Focus Areas: experience_depth, practical_application
13. What challenges have you faced related to stakeholder management and how did you overcome them?
Competency: Stakeholder Management
Type: Challenge_Based
Focus Areas: problem_solving, learning_from_experience
14. What challenges have you faced related to user research and how did you overcome them?
Competency: User Research
Type: Challenge_Based
Focus Areas: problem_solving, learning_from_experience
15. What challenges have you faced related to strategy and how did you overcome them?
Competency: Strategy
Type: Challenge_Based
Focus Areas: problem_solving, learning_from_experience
16. Describe your experience with user research in your current or previous role.
Competency: User Research
Type: Experience
Focus Areas: experience_depth, practical_application
17. Describe your experience with p&l responsibility in your current or previous role.
Competency: P&L Responsibility
Type: Experience
Focus Areas: experience_depth, practical_application
18. Describe your experience with strategy in your current or previous role.
Competency: Strategy
Type: Experience
Focus Areas: experience_depth, practical_application
19. Tell me about a time when you had to lead a team through a significant change or challenge.
Competency: Leadership
Type: Behavioral
Focus Areas: change_management, team_motivation, communication
20. What challenges have you faced related to analytics and how did you overcome them?
Competency: Analytics
Type: Challenge_Based
Focus Areas: problem_solving, learning_from_experience
SCORING RUBRICS
--------------------------------------------------
Sample Scoring Criteria (behavioral questions):
Situation Clarity:
4: Clear, specific situation with relevant context and stakes
3: Good situation description with adequate context
2: Situation described but lacks some specifics
1: Vague or unclear situation description
Action Quality:
4: Specific, thoughtful actions showing strong competency
3: Good actions demonstrating competency
2: Adequate actions but could be stronger
1: Weak or inappropriate actions
Result Impact:
4: Significant positive impact with measurable results
3: Good positive impact with clear outcomes
2: Some positive impact demonstrated
1: Little or no positive impact shown
Self Awareness:
4: Excellent self-reflection, learns from experience, acknowledges growth areas
3: Good self-awareness and learning orientation
2: Some self-reflection demonstrated
1: Limited self-awareness or reflection
FOLLOW-UP PROBE EXAMPLES
--------------------------------------------------
Sample follow-up questions:
• Can you provide more specific details about your approach?
• What would you do differently if you had to do this again?
• What challenges did you face and how did you overcome them?
USAGE GUIDELINES
--------------------------------------------------
Interview Flow:
• Warm Up: Start with 1-2 easier questions to build rapport
• Core Assessment: Focus majority of time on core competency questions
• Closing: End with questions about candidate's questions/interests
Time Management:
• Technical Questions: Allow extra time for coding/design questions
• Behavioral Questions: Keep to time limits but allow for follow-ups
• Total Recommendation: 45-75 minutes per interview round
Common Mistakes to Avoid:
• Don't ask all questions mechanically
• Don't skip follow-up questions
• Don't forget to assess cultural fit alongside competencies
CALIBRATION EXAMPLES
--------------------------------------------------
Question: What challenges have you faced related to p&l responsibility and how did you overcome them?
Sample Answer Quality Levels:
Poor Answer (Score 1-2):
Issues: Vague response, Limited evidence of competency, Poor structure
Good Answer (Score 3):
Strengths: Clear structure, Demonstrates competency, Adequate detail
Great Answer (Score 4):
Strengths: Exceptional detail, Strong evidence, Strategic thinking, Goes beyond requirements
FILE:expected_outputs/senior_software_engineer_senior_interview_loop.json
{
"role": "Senior Software Engineer",
"level": "senior",
"team": "platform",
"generated_at": "2026-02-16T13:27:37.925680",
"total_duration_minutes": 300,
"total_rounds": 5,
"rounds": {
"round_1_technical_phone_screen": {
"name": "Technical Phone Screen",
"duration_minutes": 45,
"format": "virtual",
"objectives": [
"Assess coding fundamentals",
"Evaluate problem-solving approach",
"Screen for basic technical competency"
],
"question_types": [
"coding_problems",
"technical_concepts",
"experience_questions"
],
"evaluation_criteria": [
"technical_accuracy",
"problem_solving_process",
"communication_clarity"
],
"order": 1,
"focus_areas": [
"coding_fundamentals",
"problem_solving",
"technical_leadership",
"system_architecture",
"people_development"
]
},
"round_2_coding_deep_dive": {
"name": "Coding Deep Dive",
"duration_minutes": 75,
"format": "in_person_or_virtual",
"objectives": [
"Evaluate coding skills in depth",
"Assess code quality and testing",
"Review debugging approach"
],
"question_types": [
"complex_coding_problems",
"code_review",
"testing_strategy"
],
"evaluation_criteria": [
"code_quality",
"testing_approach",
"debugging_skills",
"optimization_thinking"
],
"order": 2,
"focus_areas": [
"technical_execution",
"code_quality",
"technical_leadership",
"system_architecture",
"people_development"
]
},
"round_3_system_design": {
"name": "System Design",
"duration_minutes": 75,
"format": "collaborative_whiteboard",
"objectives": [
"Assess architectural thinking",
"Evaluate scalability considerations",
"Review trade-off analysis"
],
"question_types": [
"system_architecture",
"scalability_design",
"trade_off_analysis"
],
"evaluation_criteria": [
"architectural_thinking",
"scalability_awareness",
"trade_off_reasoning"
],
"order": 3,
"focus_areas": [
"system_thinking",
"architectural_reasoning",
"technical_leadership",
"system_architecture",
"people_development"
]
},
"round_4_behavioral": {
"name": "Behavioral Interview",
"duration_minutes": 45,
"format": "conversational",
"objectives": [
"Assess cultural fit",
"Evaluate past experiences",
"Review leadership examples"
],
"question_types": [
"star_method_questions",
"situational_scenarios",
"values_alignment"
],
"evaluation_criteria": [
"communication_skills",
"leadership_examples",
"cultural_alignment"
],
"order": 4,
"focus_areas": [
"cultural_fit",
"communication",
"teamwork",
"technical_leadership",
"system_architecture"
]
},
"round_5_technical_leadership": {
"name": "Technical Leadership",
"duration_minutes": 60,
"format": "discussion_based",
"objectives": [
"Evaluate mentoring capability",
"Assess technical decision making",
"Review cross-team collaboration"
],
"question_types": [
"leadership_scenarios",
"technical_decisions",
"mentoring_examples"
],
"evaluation_criteria": [
"leadership_potential",
"technical_judgment",
"influence_skills"
],
"order": 5,
"focus_areas": [
"leadership",
"mentoring",
"influence",
"technical_leadership",
"system_architecture"
]
}
},
"suggested_schedule": {
"type": "multi_day",
"total_duration_minutes": 300,
"recommended_breaks": [
{
"type": "short_break",
"duration": 15,
"after_minutes": 90
},
{
"type": "lunch_break",
"duration": 60,
"after_minutes": 180
}
],
"day_structure": {
"day_1": {
"date": "TBD",
"start_time": "09:00",
"end_time": "12:45",
"rounds": [
{
"type": "interview",
"round_name": "round_1_technical_phone_screen",
"title": "Technical Phone Screen",
"start_time": "09:00",
"end_time": "09:45",
"duration_minutes": 45,
"format": "virtual"
},
{
"type": "interview",
"round_name": "round_2_coding_deep_dive",
"title": "Coding Deep Dive",
"start_time": "10:00",
"end_time": "11:15",
"duration_minutes": 75,
"format": "in_person_or_virtual"
},
{
"type": "interview",
"round_name": "round_3_system_design",
"title": "System Design",
"start_time": "11:30",
"end_time": "12:45",
"duration_minutes": 75,
"format": "collaborative_whiteboard"
}
]
},
"day_2": {
"date": "TBD",
"start_time": "09:00",
"end_time": "11:00",
"rounds": [
{
"type": "interview",
"round_name": "round_4_behavioral",
"title": "Behavioral Interview",
"start_time": "09:00",
"end_time": "09:45",
"duration_minutes": 45,
"format": "conversational"
},
{
"type": "interview",
"round_name": "round_5_technical_leadership",
"title": "Technical Leadership",
"start_time": "10:00",
"end_time": "11:00",
"duration_minutes": 60,
"format": "discussion_based"
}
]
}
},
"logistics_notes": [
"Coordinate interviewer availability before scheduling",
"Ensure all interviewers have access to job description and competency requirements",
"Prepare interview rooms/virtual links for all rounds",
"Share candidate resume and application with all interviewers",
"Test video conferencing setup before virtual interviews",
"Share virtual meeting links with candidate 24 hours in advance",
"Prepare whiteboard or collaborative online tool for design sessions"
]
},
"scorecard_template": {
"scoring_scale": {
"4": "Exceeds Expectations - Demonstrates mastery beyond required level",
"3": "Meets Expectations - Solid performance meeting all requirements",
"2": "Partially Meets - Shows potential but has development areas",
"1": "Does Not Meet - Significant gaps in required competencies"
},
"dimensions": [
{
"dimension": "system_architecture",
"weight": "high",
"scale": "1-4",
"description": "Assessment of system architecture competency"
},
{
"dimension": "technical_leadership",
"weight": "high",
"scale": "1-4",
"description": "Assessment of technical leadership competency"
},
{
"dimension": "mentoring",
"weight": "high",
"scale": "1-4",
"description": "Assessment of mentoring competency"
},
{
"dimension": "cross_team_collab",
"weight": "high",
"scale": "1-4",
"description": "Assessment of cross team collab competency"
},
{
"dimension": "technology_evaluation",
"weight": "medium",
"scale": "1-4",
"description": "Assessment of technology evaluation competency"
},
{
"dimension": "process_improvement",
"weight": "medium",
"scale": "1-4",
"description": "Assessment of process improvement competency"
},
{
"dimension": "hiring_contribution",
"weight": "medium",
"scale": "1-4",
"description": "Assessment of hiring contribution competency"
},
{
"dimension": "communication",
"weight": "high",
"scale": "1-4"
},
{
"dimension": "cultural_fit",
"weight": "medium",
"scale": "1-4"
},
{
"dimension": "learning_agility",
"weight": "medium",
"scale": "1-4"
}
],
"overall_recommendation": {
"options": [
"Strong Hire",
"Hire",
"No Hire",
"Strong No Hire"
],
"criteria": "Based on weighted average and minimum thresholds"
},
"calibration_notes": {
"required": true,
"min_length": 100,
"sections": [
"strengths",
"areas_for_development",
"specific_examples"
]
}
},
"interviewer_requirements": {
"round_1_technical_phone_screen": {
"required_skills": [
"technical_assessment",
"coding_evaluation"
],
"preferred_experience": [
"same_domain",
"senior_level"
],
"calibration_level": "standard",
"suggested_interviewers": [
"senior_engineer",
"tech_lead"
]
},
"round_2_coding_deep_dive": {
"required_skills": [
"advanced_technical",
"code_quality_assessment"
],
"preferred_experience": [
"senior_engineer",
"system_design"
],
"calibration_level": "high",
"suggested_interviewers": [
"senior_engineer",
"staff_engineer"
]
},
"round_3_system_design": {
"required_skills": [
"architecture_design",
"scalability_assessment"
],
"preferred_experience": [
"senior_architect",
"large_scale_systems"
],
"calibration_level": "high",
"suggested_interviewers": [
"senior_architect",
"staff_engineer"
]
},
"round_4_behavioral": {
"required_skills": [
"behavioral_interviewing",
"competency_assessment"
],
"preferred_experience": [
"hiring_manager",
"people_leadership"
],
"calibration_level": "standard",
"suggested_interviewers": [
"hiring_manager",
"people_manager"
]
},
"round_5_technical_leadership": {
"required_skills": [
"leadership_assessment",
"technical_mentoring"
],
"preferred_experience": [
"engineering_manager",
"tech_lead"
],
"calibration_level": "high",
"suggested_interviewers": [
"engineering_manager",
"senior_staff"
]
}
},
"competency_framework": {
"required": [
"system_architecture",
"technical_leadership",
"mentoring",
"cross_team_collab"
],
"preferred": [
"technology_evaluation",
"process_improvement",
"hiring_contribution"
],
"focus_areas": [
"technical_leadership",
"system_architecture",
"people_development"
]
},
"calibration_notes": {
"hiring_bar_notes": "Calibrated for senior level software engineer role",
"common_pitfalls": [
"Avoid comparing candidates to each other rather than to the role standard",
"Don't let one strong/weak area overshadow overall assessment",
"Ensure consistent application of evaluation criteria"
],
"calibration_checkpoints": [
"Review score distribution after every 5 candidates",
"Conduct monthly interviewer calibration sessions",
"Track correlation with 6-month performance reviews"
],
"escalation_criteria": [
"Any candidate receiving all 4s or all 1s",
"Significant disagreement between interviewers (>1.5 point spread)",
"Unusual circumstances or accommodations needed"
]
}
}
FILE:expected_outputs/senior_software_engineer_senior_interview_loop.txt
Interview Loop Design for Senior Software Engineer (Senior Level)
============================================================
Team: platform
Generated: 2026-02-16T13:27:37.925680
Total Duration: 300 minutes (5h 0m)
Total Rounds: 5
INTERVIEW ROUNDS
----------------------------------------
Round 1: Technical Phone Screen
Duration: 45 minutes
Format: Virtual
Objectives:
• Assess coding fundamentals
• Evaluate problem-solving approach
• Screen for basic technical competency
Focus Areas:
• Coding Fundamentals
• Problem Solving
• Technical Leadership
• System Architecture
• People Development
Round 2: Coding Deep Dive
Duration: 75 minutes
Format: In Person Or Virtual
Objectives:
• Evaluate coding skills in depth
• Assess code quality and testing
• Review debugging approach
Focus Areas:
• Technical Execution
• Code Quality
• Technical Leadership
• System Architecture
• People Development
Round 3: System Design
Duration: 75 minutes
Format: Collaborative Whiteboard
Objectives:
• Assess architectural thinking
• Evaluate scalability considerations
• Review trade-off analysis
Focus Areas:
• System Thinking
• Architectural Reasoning
• Technical Leadership
• System Architecture
• People Development
Round 4: Behavioral Interview
Duration: 45 minutes
Format: Conversational
Objectives:
• Assess cultural fit
• Evaluate past experiences
• Review leadership examples
Focus Areas:
• Cultural Fit
• Communication
• Teamwork
• Technical Leadership
• System Architecture
Round 5: Technical Leadership
Duration: 60 minutes
Format: Discussion Based
Objectives:
• Evaluate mentoring capability
• Assess technical decision making
• Review cross-team collaboration
Focus Areas:
• Leadership
• Mentoring
• Influence
• Technical Leadership
• System Architecture
SUGGESTED SCHEDULE
----------------------------------------
Schedule Type: Multi Day
Day 1:
Time: 09:00 - 12:45
09:00-09:45: Technical Phone Screen (45min)
10:00-11:15: Coding Deep Dive (75min)
11:30-12:45: System Design (75min)
Day 2:
Time: 09:00 - 11:00
09:00-09:45: Behavioral Interview (45min)
10:00-11:00: Technical Leadership (60min)
INTERVIEWER REQUIREMENTS
----------------------------------------
Technical Phone Screen:
Required Skills: technical_assessment, coding_evaluation
Suggested Interviewers: senior_engineer, tech_lead
Calibration Level: Standard
Coding Deep Dive:
Required Skills: advanced_technical, code_quality_assessment
Suggested Interviewers: senior_engineer, staff_engineer
Calibration Level: High
System Design:
Required Skills: architecture_design, scalability_assessment
Suggested Interviewers: senior_architect, staff_engineer
Calibration Level: High
Behavioral:
Required Skills: behavioral_interviewing, competency_assessment
Suggested Interviewers: hiring_manager, people_manager
Calibration Level: Standard
Technical Leadership:
Required Skills: leadership_assessment, technical_mentoring
Suggested Interviewers: engineering_manager, senior_staff
Calibration Level: High
SCORECARD TEMPLATE
----------------------------------------
Scoring Scale:
4: Exceeds Expectations - Demonstrates mastery beyond required level
3: Meets Expectations - Solid performance meeting all requirements
2: Partially Meets - Shows potential but has development areas
1: Does Not Meet - Significant gaps in required competencies
Evaluation Dimensions:
• System Architecture (Weight: high)
• Technical Leadership (Weight: high)
• Mentoring (Weight: high)
• Cross Team Collab (Weight: high)
• Technology Evaluation (Weight: medium)
• Process Improvement (Weight: medium)
• Hiring Contribution (Weight: medium)
• Communication (Weight: high)
• Cultural Fit (Weight: medium)
• Learning Agility (Weight: medium)
CALIBRATION NOTES
----------------------------------------
Hiring Bar: Calibrated for senior level software engineer role
Common Pitfalls:
• Avoid comparing candidates to each other rather than to the role standard
• Don't let one strong/weak area overshadow overall assessment
• Ensure consistent application of evaluation criteria
FILE:hiring_calibrator.py
#!/usr/bin/env python3
"""
Hiring Calibrator
Analyzes interview scores from multiple candidates and interviewers to detect bias,
calibration issues, and inconsistent rubric application. Generates calibration reports
with specific recommendations for interviewer coaching and process improvements.
Usage:
python hiring_calibrator.py --input interview_results.json --analysis-type comprehensive
python hiring_calibrator.py --input data.json --competencies technical,leadership --output report.json
python hiring_calibrator.py --input historical_data.json --trend-analysis --period quarterly
"""
import os
import sys
import json
import argparse
import statistics
from datetime import datetime, timedelta
from typing import Dict, List, Optional, Any, Tuple
from collections import defaultdict, Counter
import math
class HiringCalibrator:
"""Analyzes interview data for bias detection and calibration issues."""
def __init__(self):
self.bias_thresholds = self._init_bias_thresholds()
self.calibration_standards = self._init_calibration_standards()
self.demographic_categories = self._init_demographic_categories()
def _init_bias_thresholds(self) -> Dict[str, float]:
"""Initialize statistical thresholds for bias detection."""
return {
"score_variance_threshold": 1.5, # Standard deviations
"pass_rate_difference_threshold": 0.15, # 15% difference
"interviewer_consistency_threshold": 0.8, # Correlation coefficient
"demographic_parity_threshold": 0.10, # 10% difference
"score_inflation_threshold": 0.3, # 30% above historical average
"score_deflation_threshold": 0.3, # 30% below historical average
"minimum_sample_size": 5 # Minimum candidates per analysis
}
def _init_calibration_standards(self) -> Dict[str, Dict]:
"""Initialize expected calibration standards."""
return {
"score_distribution": {
"target_mean": 2.8, # Expected average score (1-4 scale)
"target_std": 0.9, # Expected standard deviation
"expected_distribution": {
"1": 0.10, # 10% score 1 (does not meet)
"2": 0.25, # 25% score 2 (partially meets)
"3": 0.45, # 45% score 3 (meets expectations)
"4": 0.20 # 20% score 4 (exceeds expectations)
}
},
"interviewer_agreement": {
"minimum_correlation": 0.70, # Minimum correlation between interviewers
"maximum_std_deviation": 0.8, # Maximum std dev in scores for same candidate
"agreement_threshold": 0.75 # % of time interviewers should agree within 1 point
},
"pass_rates": {
"junior_level": 0.25, # 25% pass rate for junior roles
"mid_level": 0.20, # 20% pass rate for mid roles
"senior_level": 0.15, # 15% pass rate for senior roles
"staff_level": 0.10, # 10% pass rate for staff+ roles
"leadership": 0.12 # 12% pass rate for leadership roles
}
}
def _init_demographic_categories(self) -> List[str]:
"""Initialize demographic categories to analyze for bias."""
return [
"gender", "ethnicity", "education_level", "previous_company_size",
"years_experience", "university_tier", "geographic_location"
]
def analyze_hiring_calibration(self, interview_data: List[Dict[str, Any]],
analysis_type: str = "comprehensive",
competencies: Optional[List[str]] = None,
trend_analysis: bool = False,
period: str = "monthly") -> Dict[str, Any]:
"""Perform comprehensive hiring calibration analysis."""
# Validate and preprocess data
processed_data = self._preprocess_interview_data(interview_data)
if len(processed_data) < self.bias_thresholds["minimum_sample_size"]:
return {
"error": "Insufficient data for analysis",
"minimum_required": self.bias_thresholds["minimum_sample_size"],
"actual_samples": len(processed_data)
}
# Perform different types of analysis based on request
analysis_results = {
"analysis_type": analysis_type,
"data_summary": self._generate_data_summary(processed_data),
"generated_at": datetime.now().isoformat()
}
if analysis_type in ["comprehensive", "bias"]:
analysis_results["bias_analysis"] = self._analyze_bias_patterns(processed_data, competencies)
if analysis_type in ["comprehensive", "calibration"]:
analysis_results["calibration_analysis"] = self._analyze_calibration_consistency(processed_data, competencies)
if analysis_type in ["comprehensive", "interviewer"]:
analysis_results["interviewer_analysis"] = self._analyze_interviewer_bias(processed_data)
if analysis_type in ["comprehensive", "scoring"]:
analysis_results["scoring_analysis"] = self._analyze_scoring_patterns(processed_data, competencies)
if trend_analysis:
analysis_results["trend_analysis"] = self._analyze_trends_over_time(processed_data, period)
# Generate recommendations
analysis_results["recommendations"] = self._generate_recommendations(analysis_results)
# Calculate overall calibration health score
analysis_results["calibration_health_score"] = self._calculate_health_score(analysis_results)
return analysis_results
def _preprocess_interview_data(self, raw_data: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
"""Clean and validate interview data."""
processed_data = []
for record in raw_data:
if self._validate_interview_record(record):
processed_record = self._standardize_record(record)
processed_data.append(processed_record)
return processed_data
def _validate_interview_record(self, record: Dict[str, Any]) -> bool:
"""Validate that an interview record has required fields."""
required_fields = ["candidate_id", "interviewer_id", "scores", "overall_recommendation", "date"]
for field in required_fields:
if field not in record or record[field] is None:
return False
# Validate scores format
if not isinstance(record["scores"], dict):
return False
# Validate score values are numeric and in valid range (1-4)
for competency, score in record["scores"].items():
if not isinstance(score, (int, float)) or not (1 <= score <= 4):
return False
return True
def _standardize_record(self, record: Dict[str, Any]) -> Dict[str, Any]:
"""Standardize record format and add computed fields."""
standardized = record.copy()
# Calculate average score
scores = list(record["scores"].values())
standardized["average_score"] = statistics.mean(scores)
# Standardize recommendation to binary
recommendation = record["overall_recommendation"].lower()
standardized["hire_decision"] = recommendation in ["hire", "strong hire", "yes"]
# Parse date if string
if isinstance(record["date"], str):
try:
standardized["date"] = datetime.fromisoformat(record["date"].replace("Z", "+00:00"))
except ValueError:
standardized["date"] = datetime.now()
# Add demographic info if available
for category in self.demographic_categories:
if category not in standardized:
standardized[category] = "unknown"
# Add level normalization
role = record.get("role", "").lower()
if any(level in role for level in ["junior", "associate", "entry"]):
standardized["normalized_level"] = "junior"
elif any(level in role for level in ["senior", "sr"]):
standardized["normalized_level"] = "senior"
elif any(level in role for level in ["staff", "principal", "lead"]):
standardized["normalized_level"] = "staff"
else:
standardized["normalized_level"] = "mid"
return standardized
def _generate_data_summary(self, data: List[Dict[str, Any]]) -> Dict[str, Any]:
"""Generate summary statistics for the dataset."""
if not data:
return {}
total_candidates = len(data)
unique_interviewers = len(set(record["interviewer_id"] for record in data))
# Score statistics
all_scores = []
all_average_scores = []
hire_decisions = []
for record in data:
all_scores.extend(record["scores"].values())
all_average_scores.append(record["average_score"])
hire_decisions.append(record["hire_decision"])
# Date range
dates = [record["date"] for record in data if record["date"]]
date_range = {
"start_date": min(dates).isoformat() if dates else None,
"end_date": max(dates).isoformat() if dates else None,
"total_days": (max(dates) - min(dates)).days if len(dates) > 1 else 0
}
# Role distribution
roles = [record.get("role", "unknown") for record in data]
role_distribution = dict(Counter(roles))
return {
"total_candidates": total_candidates,
"unique_interviewers": unique_interviewers,
"candidates_per_interviewer": round(total_candidates / unique_interviewers, 2),
"date_range": date_range,
"score_statistics": {
"mean_individual_scores": round(statistics.mean(all_scores), 2),
"std_individual_scores": round(statistics.stdev(all_scores) if len(all_scores) > 1 else 0, 2),
"mean_average_scores": round(statistics.mean(all_average_scores), 2),
"std_average_scores": round(statistics.stdev(all_average_scores) if len(all_average_scores) > 1 else 0, 2)
},
"hire_rate": round(sum(hire_decisions) / len(hire_decisions), 3),
"role_distribution": role_distribution
}
def _analyze_bias_patterns(self, data: List[Dict[str, Any]],
target_competencies: Optional[List[str]]) -> Dict[str, Any]:
"""Analyze potential bias patterns in interview decisions."""
bias_analysis = {
"demographic_bias": {},
"interviewer_bias": {},
"competency_bias": {},
"overall_bias_score": 0
}
# Analyze demographic bias
for demographic in self.demographic_categories:
if all(record.get(demographic) == "unknown" for record in data):
continue
demographic_analysis = self._analyze_demographic_bias(data, demographic)
if demographic_analysis["bias_detected"]:
bias_analysis["demographic_bias"][demographic] = demographic_analysis
# Analyze interviewer bias
bias_analysis["interviewer_bias"] = self._analyze_interviewer_bias(data)
# Analyze competency bias if specified
if target_competencies:
bias_analysis["competency_bias"] = self._analyze_competency_bias(data, target_competencies)
# Calculate overall bias score
bias_analysis["overall_bias_score"] = self._calculate_bias_score(bias_analysis)
return bias_analysis
def _analyze_demographic_bias(self, data: List[Dict[str, Any]],
demographic: str) -> Dict[str, Any]:
"""Analyze bias for a specific demographic category."""
# Group data by demographic values
demographic_groups = defaultdict(list)
for record in data:
demo_value = record.get(demographic, "unknown")
if demo_value != "unknown":
demographic_groups[demo_value].append(record)
if len(demographic_groups) < 2:
return {"bias_detected": False, "reason": "insufficient_groups"}
# Calculate statistics for each group
group_stats = {}
for group, records in demographic_groups.items():
if len(records) >= self.bias_thresholds["minimum_sample_size"]:
scores = [r["average_score"] for r in records]
hire_rate = sum(r["hire_decision"] for r in records) / len(records)
group_stats[group] = {
"count": len(records),
"mean_score": statistics.mean(scores),
"hire_rate": hire_rate,
"std_score": statistics.stdev(scores) if len(scores) > 1 else 0
}
if len(group_stats) < 2:
return {"bias_detected": False, "reason": "insufficient_sample_sizes"}
# Detect statistical differences
bias_detected = False
bias_details = {}
# Check for significant differences in hire rates
hire_rates = [stats["hire_rate"] for stats in group_stats.values()]
max_hire_rate_diff = max(hire_rates) - min(hire_rates)
if max_hire_rate_diff > self.bias_thresholds["demographic_parity_threshold"]:
bias_detected = True
bias_details["hire_rate_disparity"] = {
"max_difference": round(max_hire_rate_diff, 3),
"threshold": self.bias_thresholds["demographic_parity_threshold"],
"group_stats": group_stats
}
# Check for significant differences in scoring
mean_scores = [stats["mean_score"] for stats in group_stats.values()]
max_score_diff = max(mean_scores) - min(mean_scores)
if max_score_diff > 0.5: # Half point difference threshold
bias_detected = True
bias_details["scoring_disparity"] = {
"max_difference": round(max_score_diff, 3),
"group_stats": group_stats
}
return {
"bias_detected": bias_detected,
"demographic": demographic,
"group_statistics": group_stats,
"bias_details": bias_details,
"recommendation": self._generate_demographic_bias_recommendation(demographic, bias_details) if bias_detected else None
}
def _analyze_interviewer_bias(self, data: List[Dict[str, Any]]) -> Dict[str, Any]:
"""Analyze bias patterns across different interviewers."""
interviewer_stats = defaultdict(list)
# Group by interviewer
for record in data:
interviewer_id = record["interviewer_id"]
interviewer_stats[interviewer_id].append(record)
# Calculate statistics per interviewer
interviewer_analysis = {}
for interviewer_id, records in interviewer_stats.items():
if len(records) >= self.bias_thresholds["minimum_sample_size"]:
scores = [r["average_score"] for r in records]
hire_rate = sum(r["hire_decision"] for r in records) / len(records)
interviewer_analysis[interviewer_id] = {
"total_interviews": len(records),
"mean_score": statistics.mean(scores),
"std_score": statistics.stdev(scores) if len(scores) > 1 else 0,
"hire_rate": hire_rate,
"score_inflation": self._detect_score_inflation(scores),
"consistency_score": self._calculate_interviewer_consistency(records)
}
# Identify outlier interviewers
if len(interviewer_analysis) > 1:
overall_mean_score = statistics.mean([stats["mean_score"] for stats in interviewer_analysis.values()])
overall_hire_rate = statistics.mean([stats["hire_rate"] for stats in interviewer_analysis.values()])
outlier_interviewers = {}
for interviewer_id, stats in interviewer_analysis.items():
issues = []
# Check for score inflation/deflation
if stats["mean_score"] > overall_mean_score * (1 + self.bias_thresholds["score_inflation_threshold"]):
issues.append("score_inflation")
elif stats["mean_score"] < overall_mean_score * (1 - self.bias_thresholds["score_deflation_threshold"]):
issues.append("score_deflation")
# Check for hire rate deviation
hire_rate_diff = abs(stats["hire_rate"] - overall_hire_rate)
if hire_rate_diff > self.bias_thresholds["pass_rate_difference_threshold"]:
issues.append("hire_rate_deviation")
# Check for low consistency
if stats["consistency_score"] < self.bias_thresholds["interviewer_consistency_threshold"]:
issues.append("low_consistency")
if issues:
outlier_interviewers[interviewer_id] = {
"issues": issues,
"statistics": stats,
"severity": len(issues) # More issues = higher severity
}
return {
"interviewer_statistics": interviewer_analysis,
"outlier_interviewers": outlier_interviewers if len(interviewer_analysis) > 1 else {},
"overall_consistency": self._calculate_overall_interviewer_consistency(data),
"recommendations": self._generate_interviewer_recommendations(outlier_interviewers if len(interviewer_analysis) > 1 else {})
}
def _analyze_competency_bias(self, data: List[Dict[str, Any]],
competencies: List[str]) -> Dict[str, Any]:
"""Analyze bias patterns within specific competencies."""
competency_analysis = {}
for competency in competencies:
# Extract scores for this competency
competency_scores = []
for record in data:
if competency in record["scores"]:
competency_scores.append({
"score": record["scores"][competency],
"interviewer": record["interviewer_id"],
"candidate": record["candidate_id"],
"overall_decision": record["hire_decision"]
})
if len(competency_scores) < self.bias_thresholds["minimum_sample_size"]:
continue
# Analyze scoring patterns
scores = [item["score"] for item in competency_scores]
score_variance = statistics.variance(scores) if len(scores) > 1 else 0
# Analyze by interviewer
interviewer_competency_scores = defaultdict(list)
for item in competency_scores:
interviewer_competency_scores[item["interviewer"]].append(item["score"])
interviewer_variations = {}
if len(interviewer_competency_scores) > 1:
interviewer_means = {interviewer: statistics.mean(scores)
for interviewer, scores in interviewer_competency_scores.items()
if len(scores) >= 3}
if len(interviewer_means) > 1:
mean_of_means = statistics.mean(interviewer_means.values())
for interviewer, mean_score in interviewer_means.items():
deviation = abs(mean_score - mean_of_means)
if deviation > 0.5: # More than half point deviation
interviewer_variations[interviewer] = {
"mean_score": round(mean_score, 2),
"deviation_from_average": round(deviation, 2),
"sample_size": len(interviewer_competency_scores[interviewer])
}
competency_analysis[competency] = {
"total_scores": len(competency_scores),
"mean_score": round(statistics.mean(scores), 2),
"score_variance": round(score_variance, 2),
"interviewer_variations": interviewer_variations,
"bias_detected": len(interviewer_variations) > 0
}
return competency_analysis
def _analyze_calibration_consistency(self, data: List[Dict[str, Any]],
target_competencies: Optional[List[str]]) -> Dict[str, Any]:
"""Analyze calibration consistency across interviews."""
# Group candidates by those interviewed by multiple people
candidate_interviewers = defaultdict(list)
for record in data:
candidate_interviewers[record["candidate_id"]].append(record)
multi_interviewer_candidates = {
candidate: records for candidate, records in candidate_interviewers.items()
if len(records) > 1
}
if not multi_interviewer_candidates:
return {
"error": "No candidates with multiple interviewers found",
"single_interviewer_analysis": self._analyze_single_interviewer_consistency(data)
}
# Calculate agreement statistics
agreement_stats = []
score_correlations = []
for candidate, records in multi_interviewer_candidates.items():
candidate_scores = []
interviewer_pairs = []
for record in records:
avg_score = record["average_score"]
candidate_scores.append(avg_score)
interviewer_pairs.append(record["interviewer_id"])
if len(candidate_scores) > 1:
# Calculate standard deviation of scores for this candidate
score_std = statistics.stdev(candidate_scores)
agreement_stats.append(score_std)
# Check if all interviewers agree within 1 point
score_range = max(candidate_scores) - min(candidate_scores)
agreement_within_one = score_range <= 1.0
score_correlations.append({
"candidate": candidate,
"scores": candidate_scores,
"interviewers": interviewer_pairs,
"score_std": score_std,
"score_range": score_range,
"agreement_within_one": agreement_within_one
})
# Calculate overall calibration metrics
mean_score_std = statistics.mean(agreement_stats) if agreement_stats else 0
agreement_rate = sum(1 for corr in score_correlations if corr["agreement_within_one"]) / len(score_correlations) if score_correlations else 0
calibration_quality = "good"
if mean_score_std > self.calibration_standards["interviewer_agreement"]["maximum_std_deviation"]:
calibration_quality = "poor"
elif agreement_rate < self.calibration_standards["interviewer_agreement"]["agreement_threshold"]:
calibration_quality = "fair"
return {
"multi_interviewer_candidates": len(multi_interviewer_candidates),
"mean_score_standard_deviation": round(mean_score_std, 3),
"agreement_within_one_point_rate": round(agreement_rate, 3),
"calibration_quality": calibration_quality,
"candidate_agreement_details": score_correlations,
"target_standards": self.calibration_standards["interviewer_agreement"],
"recommendations": self._generate_calibration_recommendations(mean_score_std, agreement_rate)
}
def _analyze_scoring_patterns(self, data: List[Dict[str, Any]],
target_competencies: Optional[List[str]]) -> Dict[str, Any]:
"""Analyze overall scoring patterns and distributions."""
# Overall score distribution
all_individual_scores = []
all_average_scores = []
score_distribution = defaultdict(int)
for record in data:
avg_score = record["average_score"]
all_average_scores.append(avg_score)
for competency, score in record["scores"].items():
if not target_competencies or competency in target_competencies:
all_individual_scores.append(score)
score_distribution[str(int(score))] += 1
# Calculate distribution percentages
total_scores = sum(score_distribution.values())
score_percentages = {score: count/total_scores for score, count in score_distribution.items()}
# Compare against expected distribution
expected_dist = self.calibration_standards["score_distribution"]["expected_distribution"]
distribution_analysis = {}
for score in ["1", "2", "3", "4"]:
expected_pct = expected_dist.get(score, 0)
actual_pct = score_percentages.get(score, 0)
difference = actual_pct - expected_pct
distribution_analysis[score] = {
"expected_percentage": expected_pct,
"actual_percentage": round(actual_pct, 3),
"difference": round(difference, 3),
"significant_deviation": abs(difference) > 0.05 # 5% threshold
}
# Calculate scoring statistics
mean_score = statistics.mean(all_individual_scores) if all_individual_scores else 0
std_score = statistics.stdev(all_individual_scores) if len(all_individual_scores) > 1 else 0
target_mean = self.calibration_standards["score_distribution"]["target_mean"]
target_std = self.calibration_standards["score_distribution"]["target_std"]
# Analyze pass rates by level
level_pass_rates = {}
level_groups = defaultdict(list)
for record in data:
level = record.get("normalized_level", "unknown")
level_groups[level].append(record["hire_decision"])
for level, decisions in level_groups.items():
if len(decisions) >= self.bias_thresholds["minimum_sample_size"]:
pass_rate = sum(decisions) / len(decisions)
expected_rate = self.calibration_standards["pass_rates"].get(f"{level}_level", 0.15)
level_pass_rates[level] = {
"actual_pass_rate": round(pass_rate, 3),
"expected_pass_rate": expected_rate,
"difference": round(pass_rate - expected_rate, 3),
"sample_size": len(decisions)
}
return {
"score_statistics": {
"mean_score": round(mean_score, 2),
"std_score": round(std_score, 2),
"target_mean": target_mean,
"target_std": target_std,
"mean_deviation": round(abs(mean_score - target_mean), 2),
"std_deviation": round(abs(std_score - target_std), 2)
},
"score_distribution": distribution_analysis,
"level_pass_rates": level_pass_rates,
"overall_assessment": self._assess_scoring_health(distribution_analysis, mean_score, target_mean)
}
def _analyze_trends_over_time(self, data: List[Dict[str, Any]], period: str) -> Dict[str, Any]:
"""Analyze trends in hiring patterns over time."""
# Sort data by date
dated_data = [record for record in data if record.get("date")]
dated_data.sort(key=lambda x: x["date"])
if len(dated_data) < 10: # Need minimum data for trend analysis
return {"error": "Insufficient data for trend analysis", "minimum_required": 10}
# Group by time period
period_groups = defaultdict(list)
for record in dated_data:
date = record["date"]
if period == "weekly":
period_key = date.strftime("%Y-W%U")
elif period == "monthly":
period_key = date.strftime("%Y-%m")
elif period == "quarterly":
quarter = (date.month - 1) // 3 + 1
period_key = f"{date.year}-Q{quarter}"
else: # daily
period_key = date.strftime("%Y-%m-%d")
period_groups[period_key].append(record)
# Calculate metrics for each period
period_metrics = {}
for period_key, records in period_groups.items():
if len(records) >= 3: # Minimum for meaningful metrics
scores = [r["average_score"] for r in records]
hire_rate = sum(r["hire_decision"] for r in records) / len(records)
period_metrics[period_key] = {
"count": len(records),
"mean_score": statistics.mean(scores),
"hire_rate": hire_rate,
"std_score": statistics.stdev(scores) if len(scores) > 1 else 0
}
if len(period_metrics) < 3:
return {"error": "Insufficient periods for trend analysis"}
# Analyze trends
sorted_periods = sorted(period_metrics.keys())
mean_scores = [period_metrics[p]["mean_score"] for p in sorted_periods]
hire_rates = [period_metrics[p]["hire_rate"] for p in sorted_periods]
# Simple linear trend calculation
score_trend = self._calculate_linear_trend(mean_scores)
hire_rate_trend = self._calculate_linear_trend(hire_rates)
return {
"period": period,
"total_periods": len(period_metrics),
"period_metrics": period_metrics,
"trends": {
"score_trend": {
"direction": "increasing" if score_trend > 0.01 else "decreasing" if score_trend < -0.01 else "stable",
"slope": round(score_trend, 4),
"significance": "significant" if abs(score_trend) > 0.05 else "minor"
},
"hire_rate_trend": {
"direction": "increasing" if hire_rate_trend > 0.005 else "decreasing" if hire_rate_trend < -0.005 else "stable",
"slope": round(hire_rate_trend, 4),
"significance": "significant" if abs(hire_rate_trend) > 0.02 else "minor"
}
},
"insights": self._generate_trend_insights(score_trend, hire_rate_trend, period_metrics)
}
def _calculate_linear_trend(self, values: List[float]) -> float:
"""Calculate simple linear trend slope."""
if len(values) < 2:
return 0
n = len(values)
x = list(range(n))
# Calculate slope using least squares
x_mean = statistics.mean(x)
y_mean = statistics.mean(values)
numerator = sum((x[i] - x_mean) * (values[i] - y_mean) for i in range(n))
denominator = sum((x[i] - x_mean) ** 2 for i in range(n))
return numerator / denominator if denominator != 0 else 0
def _detect_score_inflation(self, scores: List[float]) -> Dict[str, Any]:
"""Detect if an interviewer shows score inflation patterns."""
if len(scores) < 5:
return {"insufficient_data": True}
mean_score = statistics.mean(scores)
std_score = statistics.stdev(scores)
# Check against expected mean (2.8)
expected_mean = self.calibration_standards["score_distribution"]["target_mean"]
deviation = mean_score - expected_mean
# High scores with low variance might indicate inflation
high_scores_low_variance = mean_score > 3.2 and std_score < 0.5
# Check distribution - too many 4s might indicate inflation
score_counts = Counter([int(score) for score in scores])
four_count_ratio = score_counts.get(4, 0) / len(scores)
return {
"mean_score": round(mean_score, 2),
"expected_mean": expected_mean,
"deviation": round(deviation, 2),
"high_scores_low_variance": high_scores_low_variance,
"four_count_ratio": round(four_count_ratio, 2),
"inflation_detected": deviation > 0.3 or high_scores_low_variance or four_count_ratio > 0.4
}
def _calculate_interviewer_consistency(self, records: List[Dict[str, Any]]) -> float:
"""Calculate consistency score for an interviewer."""
if len(records) < 3:
return 0.5 # Neutral score for insufficient data
# Look at variance in scoring
avg_scores = [r["average_score"] for r in records]
score_variance = statistics.variance(avg_scores)
# Look at decision consistency relative to scores
decisions = [r["hire_decision"] for r in records]
scores_of_hires = [r["average_score"] for r in records if r["hire_decision"]]
scores_of_no_hires = [r["average_score"] for r in records if not r["hire_decision"]]
# Good consistency means hires have higher average scores
decision_consistency = 0.5
if scores_of_hires and scores_of_no_hires:
hire_mean = statistics.mean(scores_of_hires)
no_hire_mean = statistics.mean(scores_of_no_hires)
score_gap = hire_mean - no_hire_mean
decision_consistency = min(1.0, max(0.0, score_gap / 2.0)) # Normalize to 0-1
# Combine metrics (lower variance = higher consistency)
variance_consistency = max(0.0, 1.0 - (score_variance / 2.0))
return (decision_consistency + variance_consistency) / 2
def _calculate_overall_interviewer_consistency(self, data: List[Dict[str, Any]]) -> Dict[str, Any]:
"""Calculate overall consistency across all interviewers."""
interviewer_consistency_scores = []
interviewer_records = defaultdict(list)
for record in data:
interviewer_records[record["interviewer_id"]].append(record)
for interviewer_id, records in interviewer_records.items():
if len(records) >= 3:
consistency = self._calculate_interviewer_consistency(records)
interviewer_consistency_scores.append(consistency)
if not interviewer_consistency_scores:
return {"error": "Insufficient data per interviewer for consistency analysis"}
return {
"mean_consistency": round(statistics.mean(interviewer_consistency_scores), 3),
"std_consistency": round(statistics.stdev(interviewer_consistency_scores) if len(interviewer_consistency_scores) > 1 else 0, 3),
"min_consistency": round(min(interviewer_consistency_scores), 3),
"max_consistency": round(max(interviewer_consistency_scores), 3),
"interviewers_analyzed": len(interviewer_consistency_scores),
"target_threshold": self.bias_thresholds["interviewer_consistency_threshold"]
}
def _calculate_bias_score(self, bias_analysis: Dict[str, Any]) -> float:
"""Calculate overall bias score (0-1, where 1 is most biased)."""
bias_factors = []
# Demographic bias factors
demographic_bias = bias_analysis.get("demographic_bias", {})
for demo, analysis in demographic_bias.items():
if analysis.get("bias_detected"):
bias_factors.append(0.3) # Each demographic bias adds 0.3
# Interviewer bias factors
interviewer_bias = bias_analysis.get("interviewer_bias", {})
outlier_interviewers = interviewer_bias.get("outlier_interviewers", {})
if outlier_interviewers:
# Scale by severity and number of outliers
total_severity = sum(info["severity"] for info in outlier_interviewers.values())
bias_factors.append(min(0.5, total_severity * 0.1))
# Competency bias factors
competency_bias = bias_analysis.get("competency_bias", {})
for comp, analysis in competency_bias.items():
if analysis.get("bias_detected"):
bias_factors.append(0.2) # Each competency bias adds 0.2
return min(1.0, sum(bias_factors))
def _calculate_health_score(self, analysis: Dict[str, Any]) -> Dict[str, Any]:
"""Calculate overall calibration health score."""
health_factors = []
# Bias score (lower is better)
bias_analysis = analysis.get("bias_analysis", {})
bias_score = bias_analysis.get("overall_bias_score", 0)
bias_health = max(0, 1 - bias_score)
health_factors.append(("bias", bias_health, 0.3))
# Calibration consistency
calibration_analysis = analysis.get("calibration_analysis", {})
if "calibration_quality" in calibration_analysis:
quality_map = {"good": 1.0, "fair": 0.7, "poor": 0.3}
calibration_health = quality_map.get(calibration_analysis["calibration_quality"], 0.5)
health_factors.append(("calibration", calibration_health, 0.25))
# Interviewer consistency
interviewer_analysis = analysis.get("interviewer_analysis", {})
overall_consistency = interviewer_analysis.get("overall_consistency", {})
if "mean_consistency" in overall_consistency:
consistency_health = overall_consistency["mean_consistency"]
health_factors.append(("interviewer_consistency", consistency_health, 0.25))
# Scoring patterns health
scoring_analysis = analysis.get("scoring_analysis", {})
if "overall_assessment" in scoring_analysis:
assessment_map = {"healthy": 1.0, "concerning": 0.6, "poor": 0.2}
scoring_health = assessment_map.get(scoring_analysis["overall_assessment"], 0.5)
health_factors.append(("scoring_patterns", scoring_health, 0.2))
# Calculate weighted average
if health_factors:
weighted_sum = sum(score * weight for _, score, weight in health_factors)
total_weight = sum(weight for _, _, weight in health_factors)
overall_score = weighted_sum / total_weight
else:
overall_score = 0.5 # Neutral if no data
# Categorize health
if overall_score >= 0.8:
health_category = "excellent"
elif overall_score >= 0.7:
health_category = "good"
elif overall_score >= 0.5:
health_category = "fair"
else:
health_category = "poor"
return {
"overall_score": round(overall_score, 3),
"health_category": health_category,
"component_scores": {name: round(score, 3) for name, score, _ in health_factors},
"improvement_priority": self._identify_improvement_priorities(health_factors)
}
def _identify_improvement_priorities(self, health_factors: List[Tuple[str, float, float]]) -> List[str]:
"""Identify areas that need the most improvement."""
priorities = []
for name, score, weight in health_factors:
impact = (1 - score) * weight # Low scores with high weights = high priority
if impact > 0.15: # Significant impact threshold
priorities.append(name)
# Sort by impact (highest first)
priorities.sort(key=lambda name: next((1 - score) * weight for n, score, weight in health_factors if n == name), reverse=True)
return priorities
def _generate_recommendations(self, analysis: Dict[str, Any]) -> List[Dict[str, Any]]:
"""Generate actionable recommendations based on analysis results."""
recommendations = []
# Bias-related recommendations
bias_analysis = analysis.get("bias_analysis", {})
# Demographic bias recommendations
for demo, demo_analysis in bias_analysis.get("demographic_bias", {}).items():
if demo_analysis.get("bias_detected"):
recommendations.append({
"priority": "high",
"category": "bias_mitigation",
"title": f"Address {demo.replace('_', ' ').title()} Bias",
"description": demo_analysis.get("recommendation", f"Implement bias mitigation strategies for {demo}"),
"actions": [
"Conduct unconscious bias training focused on this demographic",
"Review and standardize interview questions",
"Implement diverse interview panels",
"Monitor hiring metrics by demographic group"
]
})
# Interviewer-specific recommendations
interviewer_analysis = bias_analysis.get("interviewer_bias", {})
outlier_interviewers = interviewer_analysis.get("outlier_interviewers", {})
for interviewer_id, outlier_info in outlier_interviewers.items():
issues = outlier_info["issues"]
priority = "high" if outlier_info["severity"] >= 3 else "medium"
actions = []
if "score_inflation" in issues:
actions.extend([
"Provide calibration training on scoring standards",
"Shadow experienced interviewers for recalibration",
"Review examples of each score level"
])
if "score_deflation" in issues:
actions.extend([
"Review expectations for role level",
"Calibrate against recent successful hires",
"Discuss evaluation criteria with hiring manager"
])
if "hire_rate_deviation" in issues:
actions.extend([
"Review hiring bar standards",
"Participate in calibration sessions",
"Compare decision criteria with team"
])
if "low_consistency" in issues:
actions.extend([
"Practice structured interviewing techniques",
"Use standardized scorecards",
"Document specific examples for each score"
])
recommendations.append({
"priority": priority,
"category": "interviewer_coaching",
"title": f"Coach Interviewer {interviewer_id}",
"description": f"Address issues: {', '.join(issues)}",
"actions": list(set(actions)) # Remove duplicates
})
# Calibration recommendations
calibration_analysis = analysis.get("calibration_analysis", {})
if calibration_analysis.get("calibration_quality") in ["fair", "poor"]:
recommendations.append({
"priority": "high",
"category": "calibration_improvement",
"title": "Improve Interview Calibration",
"description": f"Current calibration quality: {calibration_analysis.get('calibration_quality')}",
"actions": [
"Conduct monthly calibration sessions",
"Create shared examples of good/poor answers",
"Implement mandatory interviewer shadowing",
"Standardize scoring rubrics across all interviewers",
"Review and align on role expectations"
]
})
# Scoring pattern recommendations
scoring_analysis = analysis.get("scoring_analysis", {})
if scoring_analysis.get("overall_assessment") in ["concerning", "poor"]:
recommendations.append({
"priority": "medium",
"category": "scoring_standards",
"title": "Adjust Scoring Standards",
"description": "Scoring patterns deviate significantly from expected distribution",
"actions": [
"Review and communicate target score distributions",
"Provide examples for each score level",
"Monitor pass rates by role level",
"Adjust hiring bar if consistently too high/low"
]
})
# Health score recommendations
health_score = analysis.get("calibration_health_score", {})
priorities = health_score.get("improvement_priority", [])
if "bias" in priorities:
recommendations.append({
"priority": "critical",
"category": "bias_mitigation",
"title": "Implement Comprehensive Bias Mitigation",
"description": "Multiple bias indicators detected across the hiring process",
"actions": [
"Mandatory unconscious bias training for all interviewers",
"Implement structured interview protocols",
"Diversify interview panels",
"Regular bias audits and monitoring",
"Create accountability metrics for fair hiring"
]
})
# Sort by priority
priority_order = {"critical": 0, "high": 1, "medium": 2, "low": 3}
recommendations.sort(key=lambda x: priority_order.get(x["priority"], 3))
return recommendations
def _generate_demographic_bias_recommendation(self, demographic: str, bias_details: Dict[str, Any]) -> str:
"""Generate specific recommendation for demographic bias."""
if "hire_rate_disparity" in bias_details:
return f"Significant hire rate disparity detected for {demographic}. Implement structured interviews and diverse panels."
elif "scoring_disparity" in bias_details:
return f"Scoring disparity detected for {demographic}. Provide unconscious bias training and standardize evaluation criteria."
else:
return f"Potential bias detected for {demographic}. Monitor closely and implement bias mitigation strategies."
def _generate_interviewer_recommendations(self, outlier_interviewers: Dict[str, Any]) -> List[str]:
"""Generate recommendations for interviewer issues."""
if not outlier_interviewers:
return ["All interviewers performing within expected ranges"]
recommendations = []
for interviewer, info in outlier_interviewers.items():
issues = info["issues"]
if len(issues) >= 2:
recommendations.append(f"Interviewer {interviewer}: Requires comprehensive recalibration - multiple issues detected")
elif "score_inflation" in issues:
recommendations.append(f"Interviewer {interviewer}: Provide calibration training on scoring standards")
elif "hire_rate_deviation" in issues:
recommendations.append(f"Interviewer {interviewer}: Review hiring bar standards and decision criteria")
return recommendations
def _generate_calibration_recommendations(self, mean_std: float, agreement_rate: float) -> List[str]:
"""Generate calibration improvement recommendations."""
recommendations = []
if mean_std > self.calibration_standards["interviewer_agreement"]["maximum_std_deviation"]:
recommendations.append("High score variance detected - implement regular calibration sessions")
recommendations.append("Create shared examples of scoring standards for each competency")
if agreement_rate < self.calibration_standards["interviewer_agreement"]["agreement_threshold"]:
recommendations.append("Low interviewer agreement rate - standardize interview questions and evaluation criteria")
recommendations.append("Implement mandatory interviewer training on consistent evaluation")
if not recommendations:
recommendations.append("Calibration appears healthy - maintain current practices")
return recommendations
def _assess_scoring_health(self, distribution: Dict[str, Any], mean_score: float, target_mean: float) -> str:
"""Assess overall health of scoring patterns."""
issues = 0
# Check distribution deviations
for score_level, analysis in distribution.items():
if analysis["significant_deviation"]:
issues += 1
# Check mean deviation
if abs(mean_score - target_mean) > 0.3:
issues += 1
if issues == 0:
return "healthy"
elif issues <= 2:
return "concerning"
else:
return "poor"
def _generate_trend_insights(self, score_trend: float, hire_rate_trend: float, period_metrics: Dict[str, Any]) -> List[str]:
"""Generate insights from trend analysis."""
insights = []
if abs(score_trend) > 0.05:
direction = "increasing" if score_trend > 0 else "decreasing"
insights.append(f"Significant {direction} trend in average scores over time")
if score_trend > 0:
insights.append("May indicate score inflation or improving candidate quality")
else:
insights.append("May indicate stricter evaluation or declining candidate quality")
if abs(hire_rate_trend) > 0.02:
direction = "increasing" if hire_rate_trend > 0 else "decreasing"
insights.append(f"Significant {direction} trend in hire rates over time")
if hire_rate_trend > 0:
insights.append("Consider if hiring bar has lowered or candidate pool improved")
else:
insights.append("Consider if hiring bar has raised or candidate pool declined")
# Check for consistency
period_values = list(period_metrics.values())
hire_rates = [p["hire_rate"] for p in period_values]
hire_rate_variance = statistics.variance(hire_rates) if len(hire_rates) > 1 else 0
if hire_rate_variance > 0.01: # High variance in hire rates
insights.append("High variance in hire rates across periods - consider process standardization")
if not insights:
insights.append("Hiring patterns appear stable over time")
return insights
def _analyze_single_interviewer_consistency(self, data: List[Dict[str, Any]]) -> Dict[str, Any]:
"""Analyze consistency for single-interviewer candidates."""
# Look at consistency within individual interviewers
interviewer_scores = defaultdict(list)
for record in data:
interviewer_scores[record["interviewer_id"]].extend(record["scores"].values())
consistency_analysis = {}
for interviewer, scores in interviewer_scores.items():
if len(scores) >= 10: # Need sufficient data
consistency_analysis[interviewer] = {
"mean_score": round(statistics.mean(scores), 2),
"std_score": round(statistics.stdev(scores), 2),
"coefficient_of_variation": round(statistics.stdev(scores) / statistics.mean(scores), 2),
"total_scores": len(scores)
}
return consistency_analysis
def format_human_readable(calibration_report: Dict[str, Any]) -> str:
"""Format calibration report in human-readable format."""
output = []
# Header
output.append("HIRING CALIBRATION ANALYSIS REPORT")
output.append("=" * 60)
output.append(f"Analysis Type: {calibration_report.get('analysis_type', 'N/A').title()}")
output.append(f"Generated: {calibration_report.get('generated_at', 'N/A')}")
if "error" in calibration_report:
output.append(f"\nError: {calibration_report['error']}")
return "\n".join(output)
# Data Summary
data_summary = calibration_report.get("data_summary", {})
if data_summary:
output.append(f"\nDATA SUMMARY")
output.append("-" * 30)
output.append(f"Total Candidates: {data_summary.get('total_candidates', 0)}")
output.append(f"Unique Interviewers: {data_summary.get('unique_interviewers', 0)}")
output.append(f"Overall Hire Rate: {data_summary.get('hire_rate', 0):.1%}")
score_stats = data_summary.get("score_statistics", {})
output.append(f"Average Score: {score_stats.get('mean_average_scores', 0):.2f}")
output.append(f"Score Std Dev: {score_stats.get('std_average_scores', 0):.2f}")
# Health Score
health_score = calibration_report.get("calibration_health_score", {})
if health_score:
output.append(f"\nCALIBRATION HEALTH SCORE")
output.append("-" * 30)
output.append(f"Overall Score: {health_score.get('overall_score', 0):.3f}")
output.append(f"Health Category: {health_score.get('health_category', 'Unknown').title()}")
if health_score.get("improvement_priority"):
output.append(f"Priority Areas: {', '.join(health_score['improvement_priority'])}")
# Bias Analysis
bias_analysis = calibration_report.get("bias_analysis", {})
if bias_analysis:
output.append(f"\nBIAS ANALYSIS")
output.append("-" * 30)
output.append(f"Overall Bias Score: {bias_analysis.get('overall_bias_score', 0):.3f}")
# Demographic bias
demographic_bias = bias_analysis.get("demographic_bias", {})
if demographic_bias:
output.append(f"\nDemographic Bias Issues:")
for demo, analysis in demographic_bias.items():
output.append(f" • {demo.replace('_', ' ').title()}: {analysis.get('bias_details', {}).keys()}")
# Interviewer bias
interviewer_bias = bias_analysis.get("interviewer_bias", {})
outlier_interviewers = interviewer_bias.get("outlier_interviewers", {})
if outlier_interviewers:
output.append(f"\nOutlier Interviewers:")
for interviewer, info in outlier_interviewers.items():
issues = ", ".join(info["issues"])
output.append(f" • {interviewer}: {issues}")
# Calibration Analysis
calibration_analysis = calibration_report.get("calibration_analysis", {})
if calibration_analysis and "error" not in calibration_analysis:
output.append(f"\nCALIBRATION CONSISTENCY")
output.append("-" * 30)
output.append(f"Quality: {calibration_analysis.get('calibration_quality', 'Unknown').title()}")
output.append(f"Agreement Rate: {calibration_analysis.get('agreement_within_one_point_rate', 0):.1%}")
output.append(f"Score Std Dev: {calibration_analysis.get('mean_score_standard_deviation', 0):.3f}")
# Scoring Analysis
scoring_analysis = calibration_report.get("scoring_analysis", {})
if scoring_analysis:
output.append(f"\nSCORING PATTERNS")
output.append("-" * 30)
output.append(f"Overall Assessment: {scoring_analysis.get('overall_assessment', 'Unknown').title()}")
score_stats = scoring_analysis.get("score_statistics", {})
output.append(f"Mean Score: {score_stats.get('mean_score', 0):.2f} (Target: {score_stats.get('target_mean', 0):.2f})")
# Distribution analysis
distribution = scoring_analysis.get("score_distribution", {})
if distribution:
output.append(f"\nScore Distribution vs Expected:")
for score in ["1", "2", "3", "4"]:
if score in distribution:
actual = distribution[score]["actual_percentage"]
expected = distribution[score]["expected_percentage"]
output.append(f" Score {score}: {actual:.1%} (Expected: {expected:.1%})")
# Top Recommendations
recommendations = calibration_report.get("recommendations", [])
if recommendations:
output.append(f"\nTOP RECOMMENDATIONS")
output.append("-" * 30)
for i, rec in enumerate(recommendations[:5], 1): # Show top 5
output.append(f"{i}. {rec['title']} ({rec['priority'].title()} Priority)")
output.append(f" {rec['description']}")
if rec.get('actions'):
output.append(f" Actions: {len(rec['actions'])} specific action items")
return "\n".join(output)
def main():
parser = argparse.ArgumentParser(description="Analyze interview data for bias and calibration issues")
parser.add_argument("--input", type=str, required=True, help="Input JSON file with interview results data")
parser.add_argument("--analysis-type", type=str, choices=["comprehensive", "bias", "calibration", "interviewer", "scoring"],
default="comprehensive", help="Type of analysis to perform")
parser.add_argument("--competencies", type=str, help="Comma-separated list of competencies to focus on")
parser.add_argument("--trend-analysis", action="store_true", help="Perform trend analysis over time")
parser.add_argument("--period", type=str, choices=["daily", "weekly", "monthly", "quarterly"],
default="monthly", help="Time period for trend analysis")
parser.add_argument("--output", type=str, help="Output file path")
parser.add_argument("--format", choices=["json", "text", "both"], default="both", help="Output format")
args = parser.parse_args()
# Load input data
try:
with open(args.input, 'r') as f:
interview_data = json.load(f)
if not isinstance(interview_data, list):
print("Error: Input data must be a JSON array of interview records")
sys.exit(1)
except FileNotFoundError:
print(f"Error: Input file '{args.input}' not found")
sys.exit(1)
except json.JSONDecodeError as e:
print(f"Error: Invalid JSON in input file: {e}")
sys.exit(1)
except Exception as e:
print(f"Error reading input file: {e}")
sys.exit(1)
# Initialize calibrator and run analysis
calibrator = HiringCalibrator()
competencies = args.competencies.split(',') if args.competencies else None
try:
results = calibrator.analyze_hiring_calibration(
interview_data=interview_data,
analysis_type=args.analysis_type,
competencies=competencies,
trend_analysis=args.trend_analysis,
period=args.period
)
# Handle output
if args.output:
output_path = args.output
json_path = output_path if output_path.endswith('.json') else f"{output_path}.json"
text_path = output_path.replace('.json', '.txt') if output_path.endswith('.json') else f"{output_path}.txt"
else:
base_filename = f"calibration_report_{datetime.now().strftime('%Y%m%d_%H%M%S')}"
json_path = f"{base_filename}.json"
text_path = f"{base_filename}.txt"
# Write outputs
if args.format in ["json", "both"]:
with open(json_path, 'w') as f:
json.dump(results, f, indent=2, default=str)
print(f"JSON report written to: {json_path}")
if args.format in ["text", "both"]:
with open(text_path, 'w') as f:
f.write(format_human_readable(results))
print(f"Text report written to: {text_path}")
# Print summary
print(f"\nCalibration Analysis Summary:")
if "error" in results:
print(f"Error: {results['error']}")
else:
health_score = results.get("calibration_health_score", {})
print(f"Health Score: {health_score.get('overall_score', 0):.3f} ({health_score.get('health_category', 'Unknown').title()})")
bias_score = results.get("bias_analysis", {}).get("overall_bias_score", 0)
print(f"Bias Score: {bias_score:.3f} (Lower is better)")
recommendations = results.get("recommendations", [])
print(f"Recommendations Generated: {len(recommendations)}")
if recommendations:
print(f"Top Priority: {recommendations[0]['title']} ({recommendations[0]['priority'].title()})")
except Exception as e:
print(f"Error during analysis: {e}")
sys.exit(1)
if __name__ == "__main__":
main()
FILE:loop_designer.py
#!/usr/bin/env python3
"""
Interview Loop Designer
Generates calibrated interview loops tailored to specific roles, levels, and teams.
Creates complete interview loops with rounds, focus areas, time allocation,
interviewer skill requirements, and scorecard templates.
Usage:
python loop_designer.py --role "Senior Software Engineer" --level senior --team platform
python loop_designer.py --role "Product Manager" --level mid --competencies leadership,strategy
python loop_designer.py --input role_definition.json --output loops/
"""
import os
import sys
import json
import argparse
from datetime import datetime, timedelta
from typing import Dict, List, Optional, Any, Tuple
from collections import defaultdict
class InterviewLoopDesigner:
"""Designs comprehensive interview loops based on role requirements."""
def __init__(self):
self.competency_frameworks = self._init_competency_frameworks()
self.role_templates = self._init_role_templates()
self.interviewer_skills = self._init_interviewer_skills()
def _init_competency_frameworks(self) -> Dict[str, Dict]:
"""Initialize competency frameworks for different roles."""
return {
"software_engineer": {
"junior": {
"required": ["coding_fundamentals", "debugging", "testing_basics", "version_control"],
"preferred": ["system_understanding", "code_review", "collaboration"],
"focus_areas": ["technical_execution", "learning_agility", "team_collaboration"]
},
"mid": {
"required": ["advanced_coding", "system_design_basics", "testing_strategy", "debugging_complex"],
"preferred": ["mentoring_basics", "technical_communication", "project_ownership"],
"focus_areas": ["technical_depth", "system_thinking", "ownership"]
},
"senior": {
"required": ["system_architecture", "technical_leadership", "mentoring", "cross_team_collab"],
"preferred": ["technology_evaluation", "process_improvement", "hiring_contribution"],
"focus_areas": ["technical_leadership", "system_architecture", "people_development"]
},
"staff": {
"required": ["architectural_vision", "organizational_impact", "technical_strategy", "team_building"],
"preferred": ["industry_influence", "innovation_leadership", "executive_communication"],
"focus_areas": ["organizational_impact", "technical_vision", "strategic_influence"]
},
"principal": {
"required": ["company_wide_impact", "technical_vision", "talent_development", "strategic_planning"],
"preferred": ["industry_leadership", "board_communication", "market_influence"],
"focus_areas": ["strategic_leadership", "organizational_transformation", "external_influence"]
}
},
"product_manager": {
"junior": {
"required": ["product_execution", "user_research", "data_analysis", "stakeholder_comm"],
"preferred": ["market_awareness", "technical_understanding", "project_management"],
"focus_areas": ["execution_excellence", "user_focus", "analytical_thinking"]
},
"mid": {
"required": ["product_strategy", "cross_functional_leadership", "metrics_design", "market_analysis"],
"preferred": ["team_building", "technical_collaboration", "competitive_analysis"],
"focus_areas": ["strategic_thinking", "leadership", "business_impact"]
},
"senior": {
"required": ["business_strategy", "team_leadership", "p&l_ownership", "market_positioning"],
"preferred": ["hiring_leadership", "board_communication", "partnership_development"],
"focus_areas": ["business_leadership", "market_strategy", "organizational_impact"]
},
"staff": {
"required": ["portfolio_management", "organizational_leadership", "strategic_planning", "market_creation"],
"preferred": ["executive_presence", "investor_relations", "acquisition_strategy"],
"focus_areas": ["strategic_leadership", "market_innovation", "organizational_transformation"]
}
},
"designer": {
"junior": {
"required": ["design_fundamentals", "user_research", "prototyping", "design_tools"],
"preferred": ["user_empathy", "visual_design", "collaboration"],
"focus_areas": ["design_execution", "user_research", "creative_problem_solving"]
},
"mid": {
"required": ["design_systems", "user_testing", "cross_functional_collab", "design_strategy"],
"preferred": ["mentoring", "process_improvement", "business_understanding"],
"focus_areas": ["design_leadership", "system_thinking", "business_impact"]
},
"senior": {
"required": ["design_leadership", "team_building", "strategic_design", "stakeholder_management"],
"preferred": ["design_culture", "hiring_leadership", "executive_communication"],
"focus_areas": ["design_strategy", "team_leadership", "organizational_impact"]
}
},
"data_scientist": {
"junior": {
"required": ["statistical_analysis", "python_r", "data_visualization", "sql"],
"preferred": ["machine_learning", "business_understanding", "communication"],
"focus_areas": ["analytical_skills", "technical_execution", "business_impact"]
},
"mid": {
"required": ["advanced_ml", "experiment_design", "data_engineering", "stakeholder_comm"],
"preferred": ["mentoring", "project_leadership", "product_collaboration"],
"focus_areas": ["advanced_analytics", "project_leadership", "cross_functional_impact"]
},
"senior": {
"required": ["data_strategy", "team_leadership", "ml_systems", "business_strategy"],
"preferred": ["hiring_leadership", "executive_communication", "technology_evaluation"],
"focus_areas": ["strategic_leadership", "technical_vision", "organizational_impact"]
}
},
"devops_engineer": {
"junior": {
"required": ["infrastructure_basics", "scripting", "monitoring", "troubleshooting"],
"preferred": ["automation", "cloud_platforms", "security_awareness"],
"focus_areas": ["operational_excellence", "automation_mindset", "problem_solving"]
},
"mid": {
"required": ["ci_cd_design", "infrastructure_as_code", "security_implementation", "performance_optimization"],
"preferred": ["team_collaboration", "incident_management", "capacity_planning"],
"focus_areas": ["system_reliability", "automation_leadership", "cross_team_collaboration"]
},
"senior": {
"required": ["platform_architecture", "team_leadership", "security_strategy", "organizational_impact"],
"preferred": ["hiring_contribution", "technology_evaluation", "executive_communication"],
"focus_areas": ["platform_leadership", "strategic_thinking", "organizational_transformation"]
}
},
"engineering_manager": {
"junior": {
"required": ["team_leadership", "technical_background", "people_management", "project_coordination"],
"preferred": ["hiring_experience", "performance_management", "technical_mentoring"],
"focus_areas": ["people_leadership", "team_building", "execution_excellence"]
},
"senior": {
"required": ["organizational_leadership", "strategic_planning", "talent_development", "cross_functional_leadership"],
"preferred": ["technical_vision", "culture_building", "executive_communication"],
"focus_areas": ["organizational_impact", "strategic_leadership", "talent_development"]
},
"staff": {
"required": ["multi_team_leadership", "organizational_strategy", "executive_presence", "cultural_transformation"],
"preferred": ["board_communication", "market_understanding", "acquisition_integration"],
"focus_areas": ["organizational_transformation", "strategic_leadership", "cultural_evolution"]
}
}
}
def _init_role_templates(self) -> Dict[str, Dict]:
"""Initialize role-specific interview templates."""
return {
"software_engineer": {
"core_rounds": ["technical_phone_screen", "coding_deep_dive", "system_design", "behavioral"],
"optional_rounds": ["technical_leadership", "domain_expertise", "culture_fit"],
"total_duration_range": (180, 360), # 3-6 hours
"required_competencies": ["coding", "problem_solving", "communication"]
},
"product_manager": {
"core_rounds": ["product_sense", "analytical_thinking", "execution_process", "behavioral"],
"optional_rounds": ["strategic_thinking", "technical_collaboration", "leadership"],
"total_duration_range": (180, 300), # 3-5 hours
"required_competencies": ["product_strategy", "analytical_thinking", "stakeholder_management"]
},
"designer": {
"core_rounds": ["portfolio_review", "design_challenge", "collaboration_process", "behavioral"],
"optional_rounds": ["design_system_thinking", "research_methodology", "leadership"],
"total_duration_range": (180, 300), # 3-5 hours
"required_competencies": ["design_process", "user_empathy", "visual_communication"]
},
"data_scientist": {
"core_rounds": ["technical_assessment", "case_study", "statistical_thinking", "behavioral"],
"optional_rounds": ["ml_systems", "business_strategy", "technical_leadership"],
"total_duration_range": (210, 330), # 3.5-5.5 hours
"required_competencies": ["statistical_analysis", "programming", "business_acumen"]
},
"devops_engineer": {
"core_rounds": ["technical_assessment", "system_design", "troubleshooting", "behavioral"],
"optional_rounds": ["security_assessment", "automation_design", "leadership"],
"total_duration_range": (180, 300), # 3-5 hours
"required_competencies": ["infrastructure", "automation", "problem_solving"]
},
"engineering_manager": {
"core_rounds": ["leadership_assessment", "technical_background", "people_management", "behavioral"],
"optional_rounds": ["strategic_thinking", "hiring_assessment", "culture_building"],
"total_duration_range": (240, 360), # 4-6 hours
"required_competencies": ["people_leadership", "technical_understanding", "strategic_thinking"]
}
}
def _init_interviewer_skills(self) -> Dict[str, Dict]:
"""Initialize interviewer skill requirements for different round types."""
return {
"technical_phone_screen": {
"required_skills": ["technical_assessment", "coding_evaluation"],
"preferred_experience": ["same_domain", "senior_level"],
"calibration_level": "standard"
},
"coding_deep_dive": {
"required_skills": ["advanced_technical", "code_quality_assessment"],
"preferred_experience": ["senior_engineer", "system_design"],
"calibration_level": "high"
},
"system_design": {
"required_skills": ["architecture_design", "scalability_assessment"],
"preferred_experience": ["senior_architect", "large_scale_systems"],
"calibration_level": "high"
},
"behavioral": {
"required_skills": ["behavioral_interviewing", "competency_assessment"],
"preferred_experience": ["hiring_manager", "people_leadership"],
"calibration_level": "standard"
},
"technical_leadership": {
"required_skills": ["leadership_assessment", "technical_mentoring"],
"preferred_experience": ["engineering_manager", "tech_lead"],
"calibration_level": "high"
},
"product_sense": {
"required_skills": ["product_evaluation", "market_analysis"],
"preferred_experience": ["product_manager", "product_leadership"],
"calibration_level": "high"
},
"analytical_thinking": {
"required_skills": ["data_analysis", "metrics_evaluation"],
"preferred_experience": ["data_analyst", "product_manager"],
"calibration_level": "standard"
},
"design_challenge": {
"required_skills": ["design_evaluation", "user_experience"],
"preferred_experience": ["senior_designer", "design_manager"],
"calibration_level": "high"
}
}
def generate_interview_loop(self, role: str, level: str, team: Optional[str] = None,
competencies: Optional[List[str]] = None) -> Dict[str, Any]:
"""Generate a complete interview loop for the specified role and level."""
# Normalize inputs
role_key = role.lower().replace(" ", "_").replace("-", "_")
level_key = level.lower()
# Get role template and competency requirements
if role_key not in self.competency_frameworks:
role_key = self._find_closest_role(role_key)
if level_key not in self.competency_frameworks[role_key]:
level_key = self._find_closest_level(role_key, level_key)
competency_req = self.competency_frameworks[role_key][level_key]
role_template = self.role_templates.get(role_key, self.role_templates["software_engineer"])
# Design the interview loop
rounds = self._design_rounds(role_key, level_key, competency_req, role_template, competencies)
schedule = self._create_schedule(rounds)
scorecard = self._generate_scorecard(role_key, level_key, competency_req)
interviewer_requirements = self._define_interviewer_requirements(rounds)
return {
"role": role,
"level": level,
"team": team,
"generated_at": datetime.now().isoformat(),
"total_duration_minutes": sum(round_info["duration_minutes"] for round_info in rounds.values()),
"total_rounds": len(rounds),
"rounds": rounds,
"suggested_schedule": schedule,
"scorecard_template": scorecard,
"interviewer_requirements": interviewer_requirements,
"competency_framework": competency_req,
"calibration_notes": self._generate_calibration_notes(role_key, level_key)
}
def _find_closest_role(self, role_key: str) -> str:
"""Find the closest matching role template."""
role_mappings = {
"engineer": "software_engineer",
"developer": "software_engineer",
"swe": "software_engineer",
"backend": "software_engineer",
"frontend": "software_engineer",
"fullstack": "software_engineer",
"pm": "product_manager",
"product": "product_manager",
"ux": "designer",
"ui": "designer",
"graphic": "designer",
"data": "data_scientist",
"analyst": "data_scientist",
"ml": "data_scientist",
"ops": "devops_engineer",
"sre": "devops_engineer",
"infrastructure": "devops_engineer",
"manager": "engineering_manager",
"lead": "engineering_manager"
}
for key_part in role_key.split("_"):
if key_part in role_mappings:
return role_mappings[key_part]
return "software_engineer" # Default fallback
def _find_closest_level(self, role_key: str, level_key: str) -> str:
"""Find the closest matching level for the role."""
available_levels = list(self.competency_frameworks[role_key].keys())
level_mappings = {
"entry": "junior",
"associate": "junior",
"jr": "junior",
"mid": "mid",
"middle": "mid",
"sr": "senior",
"senior": "senior",
"staff": "staff",
"principal": "principal",
"lead": "senior",
"manager": "senior"
}
mapped_level = level_mappings.get(level_key, level_key)
if mapped_level in available_levels:
return mapped_level
elif "senior" in available_levels:
return "senior"
else:
return available_levels[0]
def _design_rounds(self, role_key: str, level_key: str, competency_req: Dict,
role_template: Dict, custom_competencies: Optional[List[str]]) -> Dict[str, Dict]:
"""Design the specific interview rounds based on role and level."""
rounds = {}
# Determine which rounds to include
core_rounds = role_template["core_rounds"].copy()
optional_rounds = role_template["optional_rounds"].copy()
# Add optional rounds based on level
if level_key in ["senior", "staff", "principal"]:
if "technical_leadership" in optional_rounds and role_key in ["software_engineer", "engineering_manager"]:
core_rounds.append("technical_leadership")
if "strategic_thinking" in optional_rounds and role_key in ["product_manager", "engineering_manager"]:
core_rounds.append("strategic_thinking")
if "design_system_thinking" in optional_rounds and role_key == "designer":
core_rounds.append("design_system_thinking")
if level_key in ["staff", "principal"]:
if "domain_expertise" in optional_rounds:
core_rounds.append("domain_expertise")
# Define round details
round_definitions = self._get_round_definitions()
for i, round_type in enumerate(core_rounds, 1):
if round_type in round_definitions:
round_def = round_definitions[round_type].copy()
round_def["order"] = i
round_def["focus_areas"] = self._customize_focus_areas(round_type, competency_req, custom_competencies)
rounds[f"round_{i}_{round_type}"] = round_def
return rounds
def _get_round_definitions(self) -> Dict[str, Dict]:
"""Get predefined round definitions with standard durations and formats."""
return {
"technical_phone_screen": {
"name": "Technical Phone Screen",
"duration_minutes": 45,
"format": "virtual",
"objectives": ["Assess coding fundamentals", "Evaluate problem-solving approach", "Screen for basic technical competency"],
"question_types": ["coding_problems", "technical_concepts", "experience_questions"],
"evaluation_criteria": ["technical_accuracy", "problem_solving_process", "communication_clarity"]
},
"coding_deep_dive": {
"name": "Coding Deep Dive",
"duration_minutes": 75,
"format": "in_person_or_virtual",
"objectives": ["Evaluate coding skills in depth", "Assess code quality and testing", "Review debugging approach"],
"question_types": ["complex_coding_problems", "code_review", "testing_strategy"],
"evaluation_criteria": ["code_quality", "testing_approach", "debugging_skills", "optimization_thinking"]
},
"system_design": {
"name": "System Design",
"duration_minutes": 75,
"format": "collaborative_whiteboard",
"objectives": ["Assess architectural thinking", "Evaluate scalability considerations", "Review trade-off analysis"],
"question_types": ["system_architecture", "scalability_design", "trade_off_analysis"],
"evaluation_criteria": ["architectural_thinking", "scalability_awareness", "trade_off_reasoning"]
},
"behavioral": {
"name": "Behavioral Interview",
"duration_minutes": 45,
"format": "conversational",
"objectives": ["Assess cultural fit", "Evaluate past experiences", "Review leadership examples"],
"question_types": ["star_method_questions", "situational_scenarios", "values_alignment"],
"evaluation_criteria": ["communication_skills", "leadership_examples", "cultural_alignment"]
},
"technical_leadership": {
"name": "Technical Leadership",
"duration_minutes": 60,
"format": "discussion_based",
"objectives": ["Evaluate mentoring capability", "Assess technical decision making", "Review cross-team collaboration"],
"question_types": ["leadership_scenarios", "technical_decisions", "mentoring_examples"],
"evaluation_criteria": ["leadership_potential", "technical_judgment", "influence_skills"]
},
"product_sense": {
"name": "Product Sense",
"duration_minutes": 75,
"format": "case_study",
"objectives": ["Assess product intuition", "Evaluate user empathy", "Review market understanding"],
"question_types": ["product_scenarios", "feature_prioritization", "user_journey_analysis"],
"evaluation_criteria": ["product_intuition", "user_empathy", "analytical_thinking"]
},
"analytical_thinking": {
"name": "Analytical Thinking",
"duration_minutes": 60,
"format": "data_analysis",
"objectives": ["Evaluate data interpretation", "Assess metric design", "Review experiment planning"],
"question_types": ["data_interpretation", "metric_design", "experiment_analysis"],
"evaluation_criteria": ["analytical_rigor", "metric_intuition", "experimental_thinking"]
},
"design_challenge": {
"name": "Design Challenge",
"duration_minutes": 90,
"format": "hands_on_design",
"objectives": ["Assess design process", "Evaluate user-centered thinking", "Review iteration approach"],
"question_types": ["design_problems", "user_research", "design_critique"],
"evaluation_criteria": ["design_process", "user_focus", "visual_communication"]
},
"portfolio_review": {
"name": "Portfolio Review",
"duration_minutes": 75,
"format": "presentation_discussion",
"objectives": ["Review past work", "Assess design thinking", "Evaluate impact measurement"],
"question_types": ["portfolio_walkthrough", "design_decisions", "impact_stories"],
"evaluation_criteria": ["design_quality", "process_thinking", "business_impact"]
}
}
def _customize_focus_areas(self, round_type: str, competency_req: Dict,
custom_competencies: Optional[List[str]]) -> List[str]:
"""Customize focus areas based on role competency requirements."""
base_focus_areas = competency_req.get("focus_areas", [])
round_focus_mapping = {
"technical_phone_screen": ["coding_fundamentals", "problem_solving"],
"coding_deep_dive": ["technical_execution", "code_quality"],
"system_design": ["system_thinking", "architectural_reasoning"],
"behavioral": ["cultural_fit", "communication", "teamwork"],
"technical_leadership": ["leadership", "mentoring", "influence"],
"product_sense": ["product_intuition", "user_empathy"],
"analytical_thinking": ["data_analysis", "metric_design"],
"design_challenge": ["design_process", "user_focus"]
}
focus_areas = round_focus_mapping.get(round_type, [])
# Add custom competencies if specified
if custom_competencies:
focus_areas.extend([comp for comp in custom_competencies if comp not in focus_areas])
# Add role-specific focus areas
focus_areas.extend([area for area in base_focus_areas if area not in focus_areas])
return focus_areas[:5] # Limit to top 5 focus areas
def _create_schedule(self, rounds: Dict[str, Dict]) -> Dict[str, Any]:
"""Create a suggested interview schedule."""
sorted_rounds = sorted(rounds.items(), key=lambda x: x[1]["order"])
# Calculate optimal scheduling
total_duration = sum(round_info["duration_minutes"] for _, round_info in sorted_rounds)
if total_duration <= 240: # 4 hours or less - single day
schedule_type = "single_day"
day_structure = self._create_single_day_schedule(sorted_rounds)
else: # Multi-day schedule
schedule_type = "multi_day"
day_structure = self._create_multi_day_schedule(sorted_rounds)
return {
"type": schedule_type,
"total_duration_minutes": total_duration,
"recommended_breaks": self._calculate_breaks(total_duration),
"day_structure": day_structure,
"logistics_notes": self._generate_logistics_notes(sorted_rounds)
}
def _create_single_day_schedule(self, rounds: List[Tuple[str, Dict]]) -> Dict[str, Any]:
"""Create a single-day interview schedule."""
start_time = datetime.strptime("09:00", "%H:%M")
current_time = start_time
schedule = []
for round_name, round_info in rounds:
# Add break if needed (after 90 minutes of interviews)
if schedule and sum(item.get("duration_minutes", 0) for item in schedule if "break" not in item.get("type", "")) >= 90:
schedule.append({
"type": "break",
"start_time": current_time.strftime("%H:%M"),
"duration_minutes": 15,
"end_time": (current_time + timedelta(minutes=15)).strftime("%H:%M")
})
current_time += timedelta(minutes=15)
# Add the interview round
end_time = current_time + timedelta(minutes=round_info["duration_minutes"])
schedule.append({
"type": "interview",
"round_name": round_name,
"title": round_info["name"],
"start_time": current_time.strftime("%H:%M"),
"end_time": end_time.strftime("%H:%M"),
"duration_minutes": round_info["duration_minutes"],
"format": round_info["format"]
})
current_time = end_time
return {
"day_1": {
"date": "TBD",
"start_time": start_time.strftime("%H:%M"),
"end_time": current_time.strftime("%H:%M"),
"rounds": schedule
}
}
def _create_multi_day_schedule(self, rounds: List[Tuple[str, Dict]]) -> Dict[str, Any]:
"""Create a multi-day interview schedule."""
# Split rounds across days (max 4 hours per day)
max_daily_minutes = 240
days = {}
current_day = 1
current_day_duration = 0
current_day_rounds = []
for round_name, round_info in rounds:
duration = round_info["duration_minutes"] + 15 # Add buffer time
if current_day_duration + duration > max_daily_minutes and current_day_rounds:
# Finalize current day
days[f"day_{current_day}"] = self._finalize_day_schedule(current_day_rounds)
current_day += 1
current_day_duration = 0
current_day_rounds = []
current_day_rounds.append((round_name, round_info))
current_day_duration += duration
# Finalize last day
if current_day_rounds:
days[f"day_{current_day}"] = self._finalize_day_schedule(current_day_rounds)
return days
def _finalize_day_schedule(self, day_rounds: List[Tuple[str, Dict]]) -> Dict[str, Any]:
"""Finalize the schedule for a specific day."""
start_time = datetime.strptime("09:00", "%H:%M")
current_time = start_time
schedule = []
for round_name, round_info in day_rounds:
end_time = current_time + timedelta(minutes=round_info["duration_minutes"])
schedule.append({
"type": "interview",
"round_name": round_name,
"title": round_info["name"],
"start_time": current_time.strftime("%H:%M"),
"end_time": end_time.strftime("%H:%M"),
"duration_minutes": round_info["duration_minutes"],
"format": round_info["format"]
})
current_time = end_time + timedelta(minutes=15) # 15-min buffer
return {
"date": "TBD",
"start_time": start_time.strftime("%H:%M"),
"end_time": (current_time - timedelta(minutes=15)).strftime("%H:%M"),
"rounds": schedule
}
def _calculate_breaks(self, total_duration: int) -> List[Dict[str, Any]]:
"""Calculate recommended breaks based on total duration."""
breaks = []
if total_duration >= 120: # 2+ hours
breaks.append({"type": "short_break", "duration": 15, "after_minutes": 90})
if total_duration >= 240: # 4+ hours
breaks.append({"type": "lunch_break", "duration": 60, "after_minutes": 180})
if total_duration >= 360: # 6+ hours
breaks.append({"type": "short_break", "duration": 15, "after_minutes": 300})
return breaks
def _generate_scorecard(self, role_key: str, level_key: str, competency_req: Dict) -> Dict[str, Any]:
"""Generate a scorecard template for the interview loop."""
scoring_dimensions = []
# Add competency-based scoring dimensions
for competency in competency_req["required"]:
scoring_dimensions.append({
"dimension": competency,
"weight": "high",
"scale": "1-4",
"description": f"Assessment of {competency.replace('_', ' ')} competency"
})
for competency in competency_req.get("preferred", []):
scoring_dimensions.append({
"dimension": competency,
"weight": "medium",
"scale": "1-4",
"description": f"Assessment of {competency.replace('_', ' ')} competency"
})
# Add standard dimensions
standard_dimensions = [
{"dimension": "communication", "weight": "high", "scale": "1-4"},
{"dimension": "cultural_fit", "weight": "medium", "scale": "1-4"},
{"dimension": "learning_agility", "weight": "medium", "scale": "1-4"}
]
scoring_dimensions.extend(standard_dimensions)
return {
"scoring_scale": {
"4": "Exceeds Expectations - Demonstrates mastery beyond required level",
"3": "Meets Expectations - Solid performance meeting all requirements",
"2": "Partially Meets - Shows potential but has development areas",
"1": "Does Not Meet - Significant gaps in required competencies"
},
"dimensions": scoring_dimensions,
"overall_recommendation": {
"options": ["Strong Hire", "Hire", "No Hire", "Strong No Hire"],
"criteria": "Based on weighted average and minimum thresholds"
},
"calibration_notes": {
"required": True,
"min_length": 100,
"sections": ["strengths", "areas_for_development", "specific_examples"]
}
}
def _define_interviewer_requirements(self, rounds: Dict[str, Dict]) -> Dict[str, Dict]:
"""Define interviewer skill requirements for each round."""
requirements = {}
for round_name, round_info in rounds.items():
round_type = round_name.split("_", 2)[-1] # Extract round type
if round_type in self.interviewer_skills:
skill_req = self.interviewer_skills[round_type].copy()
skill_req["suggested_interviewers"] = self._suggest_interviewer_profiles(round_type)
requirements[round_name] = skill_req
else:
# Default requirements
requirements[round_name] = {
"required_skills": ["interviewing_basics", "evaluation_skills"],
"preferred_experience": ["relevant_domain"],
"calibration_level": "standard",
"suggested_interviewers": ["experienced_interviewer"]
}
return requirements
def _suggest_interviewer_profiles(self, round_type: str) -> List[str]:
"""Suggest specific interviewer profiles for different round types."""
profile_mapping = {
"technical_phone_screen": ["senior_engineer", "tech_lead"],
"coding_deep_dive": ["senior_engineer", "staff_engineer"],
"system_design": ["senior_architect", "staff_engineer"],
"behavioral": ["hiring_manager", "people_manager"],
"technical_leadership": ["engineering_manager", "senior_staff"],
"product_sense": ["senior_pm", "product_leader"],
"analytical_thinking": ["senior_analyst", "data_scientist"],
"design_challenge": ["senior_designer", "design_manager"]
}
return profile_mapping.get(round_type, ["experienced_interviewer"])
def _generate_calibration_notes(self, role_key: str, level_key: str) -> Dict[str, Any]:
"""Generate calibration notes and best practices."""
return {
"hiring_bar_notes": f"Calibrated for {level_key} level {role_key.replace('_', ' ')} role",
"common_pitfalls": [
"Avoid comparing candidates to each other rather than to the role standard",
"Don't let one strong/weak area overshadow overall assessment",
"Ensure consistent application of evaluation criteria"
],
"calibration_checkpoints": [
"Review score distribution after every 5 candidates",
"Conduct monthly interviewer calibration sessions",
"Track correlation with 6-month performance reviews"
],
"escalation_criteria": [
"Any candidate receiving all 4s or all 1s",
"Significant disagreement between interviewers (>1.5 point spread)",
"Unusual circumstances or accommodations needed"
]
}
def _generate_logistics_notes(self, rounds: List[Tuple[str, Dict]]) -> List[str]:
"""Generate logistics and coordination notes."""
notes = [
"Coordinate interviewer availability before scheduling",
"Ensure all interviewers have access to job description and competency requirements",
"Prepare interview rooms/virtual links for all rounds",
"Share candidate resume and application with all interviewers"
]
# Add format-specific notes
formats_used = {round_info["format"] for _, round_info in rounds}
if "virtual" in formats_used:
notes.append("Test video conferencing setup before virtual interviews")
notes.append("Share virtual meeting links with candidate 24 hours in advance")
if "collaborative_whiteboard" in formats_used:
notes.append("Prepare whiteboard or collaborative online tool for design sessions")
if "hands_on_design" in formats_used:
notes.append("Provide design tools access or ensure candidate can screen share their preferred tools")
return notes
def format_human_readable(loop_data: Dict[str, Any]) -> str:
"""Format the interview loop data in a human-readable format."""
output = []
# Header
output.append(f"Interview Loop Design for {loop_data['role']} ({loop_data['level'].title()} Level)")
output.append("=" * 60)
if loop_data.get('team'):
output.append(f"Team: {loop_data['team']}")
output.append(f"Generated: {loop_data['generated_at']}")
output.append(f"Total Duration: {loop_data['total_duration_minutes']} minutes ({loop_data['total_duration_minutes']//60}h {loop_data['total_duration_minutes']%60}m)")
output.append(f"Total Rounds: {loop_data['total_rounds']}")
output.append("")
# Interview Rounds
output.append("INTERVIEW ROUNDS")
output.append("-" * 40)
sorted_rounds = sorted(loop_data['rounds'].items(), key=lambda x: x[1]['order'])
for round_name, round_info in sorted_rounds:
output.append(f"\nRound {round_info['order']}: {round_info['name']}")
output.append(f"Duration: {round_info['duration_minutes']} minutes")
output.append(f"Format: {round_info['format'].replace('_', ' ').title()}")
output.append("Objectives:")
for obj in round_info['objectives']:
output.append(f" • {obj}")
output.append("Focus Areas:")
for area in round_info['focus_areas']:
output.append(f" • {area.replace('_', ' ').title()}")
# Suggested Schedule
output.append("\nSUGGESTED SCHEDULE")
output.append("-" * 40)
schedule = loop_data['suggested_schedule']
output.append(f"Schedule Type: {schedule['type'].replace('_', ' ').title()}")
for day_name, day_info in schedule['day_structure'].items():
output.append(f"\n{day_name.replace('_', ' ').title()}:")
output.append(f"Time: {day_info['start_time']} - {day_info['end_time']}")
for item in day_info['rounds']:
if item['type'] == 'interview':
output.append(f" {item['start_time']}-{item['end_time']}: {item['title']} ({item['duration_minutes']}min)")
else:
output.append(f" {item['start_time']}-{item['end_time']}: {item['type'].title()} ({item['duration_minutes']}min)")
# Interviewer Requirements
output.append("\nINTERVIEWER REQUIREMENTS")
output.append("-" * 40)
for round_name, requirements in loop_data['interviewer_requirements'].items():
round_display = round_name.split("_", 2)[-1].replace("_", " ").title()
output.append(f"\n{round_display}:")
output.append(f"Required Skills: {', '.join(requirements['required_skills'])}")
output.append(f"Suggested Interviewers: {', '.join(requirements['suggested_interviewers'])}")
output.append(f"Calibration Level: {requirements['calibration_level'].title()}")
# Scorecard Overview
output.append("\nSCORECARD TEMPLATE")
output.append("-" * 40)
scorecard = loop_data['scorecard_template']
output.append("Scoring Scale:")
for score, description in scorecard['scoring_scale'].items():
output.append(f" {score}: {description}")
output.append("\nEvaluation Dimensions:")
for dim in scorecard['dimensions']:
output.append(f" • {dim['dimension'].replace('_', ' ').title()} (Weight: {dim['weight']})")
# Calibration Notes
output.append("\nCALIBRATION NOTES")
output.append("-" * 40)
calibration = loop_data['calibration_notes']
output.append(f"Hiring Bar: {calibration['hiring_bar_notes']}")
output.append("\nCommon Pitfalls:")
for pitfall in calibration['common_pitfalls']:
output.append(f" • {pitfall}")
return "\n".join(output)
def main():
parser = argparse.ArgumentParser(description="Generate calibrated interview loops for specific roles and levels")
parser.add_argument("--role", type=str, help="Job role title (e.g., 'Senior Software Engineer')")
parser.add_argument("--level", type=str, help="Experience level (junior, mid, senior, staff, principal)")
parser.add_argument("--team", type=str, help="Team or department (optional)")
parser.add_argument("--competencies", type=str, help="Comma-separated list of specific competencies to focus on")
parser.add_argument("--input", type=str, help="Input JSON file with role definition")
parser.add_argument("--output", type=str, help="Output directory or file path")
parser.add_argument("--format", choices=["json", "text", "both"], default="both", help="Output format")
args = parser.parse_args()
designer = InterviewLoopDesigner()
# Handle input
if args.input:
try:
with open(args.input, 'r') as f:
role_data = json.load(f)
role = role_data.get('role') or role_data.get('title', '')
level = role_data.get('level', 'senior')
team = role_data.get('team')
competencies = role_data.get('competencies')
except Exception as e:
print(f"Error reading input file: {e}")
sys.exit(1)
else:
if not args.role or not args.level:
print("Error: --role and --level are required when not using --input")
sys.exit(1)
role = args.role
level = args.level
team = args.team
competencies = args.competencies.split(',') if args.competencies else None
# Generate interview loop
try:
loop_data = designer.generate_interview_loop(role, level, team, competencies)
# Handle output
if args.output:
output_path = args.output
if os.path.isdir(output_path):
safe_role = "".join(c for c in role.lower() if c.isalnum() or c in (' ', '-', '_')).replace(' ', '_')
base_filename = f"{safe_role}_{level}_interview_loop"
json_path = os.path.join(output_path, f"{base_filename}.json")
text_path = os.path.join(output_path, f"{base_filename}.txt")
else:
# Use provided path as base
json_path = output_path if output_path.endswith('.json') else f"{output_path}.json"
text_path = output_path.replace('.json', '.txt') if output_path.endswith('.json') else f"{output_path}.txt"
else:
safe_role = "".join(c for c in role.lower() if c.isalnum() or c in (' ', '-', '_')).replace(' ', '_')
base_filename = f"{safe_role}_{level}_interview_loop"
json_path = f"{base_filename}.json"
text_path = f"{base_filename}.txt"
# Write outputs
if args.format in ["json", "both"]:
with open(json_path, 'w') as f:
json.dump(loop_data, f, indent=2, default=str)
print(f"JSON output written to: {json_path}")
if args.format in ["text", "both"]:
with open(text_path, 'w') as f:
f.write(format_human_readable(loop_data))
print(f"Text output written to: {text_path}")
# Always print summary to stdout
print("\nInterview Loop Summary:")
print(f"Role: {loop_data['role']} ({loop_data['level'].title()})")
print(f"Total Duration: {loop_data['total_duration_minutes']} minutes")
print(f"Number of Rounds: {loop_data['total_rounds']}")
print(f"Schedule Type: {loop_data['suggested_schedule']['type'].replace('_', ' ').title()}")
except Exception as e:
print(f"Error generating interview loop: {e}")
sys.exit(1)
if __name__ == "__main__":
main()
FILE:question_bank_generator.py
#!/usr/bin/env python3
"""
Question Bank Generator
Generates comprehensive, competency-based interview questions with detailed scoring criteria.
Creates structured question banks organized by competency area with scoring rubrics,
follow-up probes, and calibration examples.
Usage:
python question_bank_generator.py --role "Frontend Engineer" --competencies react,typescript,system-design
python question_bank_generator.py --role "Product Manager" --question-types behavioral,leadership
python question_bank_generator.py --input role_requirements.json --output questions/
"""
import os
import sys
import json
import argparse
import random
from datetime import datetime
from typing import Dict, List, Optional, Any, Tuple
from collections import defaultdict
class QuestionBankGenerator:
"""Generates comprehensive interview question banks with scoring criteria."""
def __init__(self):
self.technical_questions = self._init_technical_questions()
self.behavioral_questions = self._init_behavioral_questions()
self.competency_mapping = self._init_competency_mapping()
self.scoring_rubrics = self._init_scoring_rubrics()
self.follow_up_strategies = self._init_follow_up_strategies()
def _init_technical_questions(self) -> Dict[str, Dict]:
"""Initialize technical questions by competency area and level."""
return {
"coding_fundamentals": {
"junior": [
{
"question": "Write a function to reverse a string without using built-in reverse methods.",
"competency": "coding_fundamentals",
"type": "coding",
"difficulty": "easy",
"time_limit": 15,
"key_concepts": ["loops", "string_manipulation", "basic_algorithms"]
},
{
"question": "Implement a function to check if a string is a palindrome.",
"competency": "coding_fundamentals",
"type": "coding",
"difficulty": "easy",
"time_limit": 15,
"key_concepts": ["string_processing", "comparison", "edge_cases"]
},
{
"question": "Find the largest element in an array without using built-in max functions.",
"competency": "coding_fundamentals",
"type": "coding",
"difficulty": "easy",
"time_limit": 10,
"key_concepts": ["arrays", "iteration", "comparison"]
}
],
"mid": [
{
"question": "Implement a function to find the first non-repeating character in a string.",
"competency": "coding_fundamentals",
"type": "coding",
"difficulty": "medium",
"time_limit": 20,
"key_concepts": ["hash_maps", "string_processing", "efficiency"]
},
{
"question": "Write a function to merge two sorted arrays into one sorted array.",
"competency": "coding_fundamentals",
"type": "coding",
"difficulty": "medium",
"time_limit": 25,
"key_concepts": ["merge_algorithms", "two_pointers", "optimization"]
}
],
"senior": [
{
"question": "Implement a LRU (Least Recently Used) cache with O(1) operations.",
"competency": "coding_fundamentals",
"type": "coding",
"difficulty": "hard",
"time_limit": 35,
"key_concepts": ["data_structures", "hash_maps", "doubly_linked_lists"]
}
]
},
"system_design": {
"mid": [
{
"question": "Design a URL shortener service like bit.ly for 10K users.",
"competency": "system_design",
"type": "design",
"difficulty": "medium",
"time_limit": 45,
"key_concepts": ["database_design", "hashing", "basic_scalability"]
}
],
"senior": [
{
"question": "Design a real-time chat system supporting 1M concurrent users.",
"competency": "system_design",
"type": "design",
"difficulty": "hard",
"time_limit": 60,
"key_concepts": ["websockets", "load_balancing", "database_sharding", "caching"]
},
{
"question": "Design a distributed cache system like Redis with high availability.",
"competency": "system_design",
"type": "design",
"difficulty": "hard",
"time_limit": 60,
"key_concepts": ["distributed_systems", "replication", "consistency", "partitioning"]
}
],
"staff": [
{
"question": "Design the architecture for a global content delivery network (CDN).",
"competency": "system_design",
"type": "design",
"difficulty": "expert",
"time_limit": 75,
"key_concepts": ["global_architecture", "edge_computing", "content_optimization", "network_protocols"]
}
]
},
"frontend_development": {
"junior": [
{
"question": "Create a responsive navigation menu using HTML, CSS, and vanilla JavaScript.",
"competency": "frontend_development",
"type": "coding",
"difficulty": "easy",
"time_limit": 30,
"key_concepts": ["html_css", "responsive_design", "dom_manipulation"]
}
],
"mid": [
{
"question": "Build a React component that fetches and displays paginated data from an API.",
"competency": "frontend_development",
"type": "coding",
"difficulty": "medium",
"time_limit": 45,
"key_concepts": ["react_hooks", "api_integration", "state_management", "pagination"]
}
],
"senior": [
{
"question": "Design and implement a custom React hook for managing complex form state with validation.",
"competency": "frontend_development",
"type": "coding",
"difficulty": "hard",
"time_limit": 60,
"key_concepts": ["custom_hooks", "form_validation", "state_management", "performance"]
}
]
},
"data_analysis": {
"junior": [
{
"question": "Given a dataset of user activities, calculate the daily active users for the past month.",
"competency": "data_analysis",
"type": "analytical",
"difficulty": "easy",
"time_limit": 30,
"key_concepts": ["sql_basics", "date_functions", "aggregation"]
}
],
"mid": [
{
"question": "Analyze conversion funnel data to identify the biggest drop-off point and propose solutions.",
"competency": "data_analysis",
"type": "analytical",
"difficulty": "medium",
"time_limit": 45,
"key_concepts": ["funnel_analysis", "conversion_optimization", "statistical_significance"]
}
],
"senior": [
{
"question": "Design an A/B testing framework to measure the impact of a new recommendation algorithm.",
"competency": "data_analysis",
"type": "analytical",
"difficulty": "hard",
"time_limit": 60,
"key_concepts": ["experiment_design", "statistical_power", "bias_mitigation", "causal_inference"]
}
]
},
"machine_learning": {
"mid": [
{
"question": "Explain how you would build a recommendation system for an e-commerce platform.",
"competency": "machine_learning",
"type": "conceptual",
"difficulty": "medium",
"time_limit": 45,
"key_concepts": ["collaborative_filtering", "content_based", "cold_start", "evaluation_metrics"]
}
],
"senior": [
{
"question": "Design a real-time fraud detection system for financial transactions.",
"competency": "machine_learning",
"type": "design",
"difficulty": "hard",
"time_limit": 60,
"key_concepts": ["anomaly_detection", "real_time_ml", "feature_engineering", "model_monitoring"]
}
]
},
"product_strategy": {
"mid": [
{
"question": "How would you prioritize features for a mobile app with limited engineering resources?",
"competency": "product_strategy",
"type": "case_study",
"difficulty": "medium",
"time_limit": 45,
"key_concepts": ["prioritization_frameworks", "resource_allocation", "impact_estimation"]
}
],
"senior": [
{
"question": "Design a go-to-market strategy for a new B2B SaaS product entering a competitive market.",
"competency": "product_strategy",
"type": "strategic",
"difficulty": "hard",
"time_limit": 60,
"key_concepts": ["market_analysis", "competitive_positioning", "pricing_strategy", "channel_strategy"]
}
]
}
}
def _init_behavioral_questions(self) -> Dict[str, List[Dict]]:
"""Initialize behavioral questions by competency area."""
return {
"leadership": [
{
"question": "Tell me about a time when you had to lead a team through a significant change or challenge.",
"competency": "leadership",
"type": "behavioral",
"method": "STAR",
"focus_areas": ["change_management", "team_motivation", "communication"]
},
{
"question": "Describe a situation where you had to influence someone without having direct authority over them.",
"competency": "leadership",
"type": "behavioral",
"method": "STAR",
"focus_areas": ["influence", "persuasion", "stakeholder_management"]
},
{
"question": "Give me an example of when you had to make a difficult decision that affected your team.",
"competency": "leadership",
"type": "behavioral",
"method": "STAR",
"focus_areas": ["decision_making", "team_impact", "communication"]
}
],
"collaboration": [
{
"question": "Describe a time when you had to work with a difficult colleague or stakeholder.",
"competency": "collaboration",
"type": "behavioral",
"method": "STAR",
"focus_areas": ["conflict_resolution", "relationship_building", "professionalism"]
},
{
"question": "Tell me about a project where you had to coordinate across multiple teams or departments.",
"competency": "collaboration",
"type": "behavioral",
"method": "STAR",
"focus_areas": ["cross_functional_work", "communication", "project_coordination"]
}
],
"problem_solving": [
{
"question": "Walk me through a complex problem you solved recently. What was your approach?",
"competency": "problem_solving",
"type": "behavioral",
"method": "STAR",
"focus_areas": ["analytical_thinking", "methodology", "creativity"]
},
{
"question": "Describe a time when you had to solve a problem with limited information or resources.",
"competency": "problem_solving",
"type": "behavioral",
"method": "STAR",
"focus_areas": ["resourcefulness", "ambiguity_tolerance", "decision_making"]
}
],
"communication": [
{
"question": "Tell me about a time when you had to present complex technical information to a non-technical audience.",
"competency": "communication",
"type": "behavioral",
"method": "STAR",
"focus_areas": ["technical_communication", "audience_adaptation", "clarity"]
},
{
"question": "Describe a situation where you had to deliver difficult feedback to a colleague.",
"competency": "communication",
"type": "behavioral",
"method": "STAR",
"focus_areas": ["feedback_delivery", "empathy", "constructive_criticism"]
}
],
"adaptability": [
{
"question": "Tell me about a time when you had to quickly learn a new technology or skill for work.",
"competency": "adaptability",
"type": "behavioral",
"method": "STAR",
"focus_areas": ["learning_agility", "growth_mindset", "knowledge_acquisition"]
},
{
"question": "Describe how you handled a situation when project requirements changed significantly mid-way.",
"competency": "adaptability",
"type": "behavioral",
"method": "STAR",
"focus_areas": ["flexibility", "change_management", "resilience"]
}
],
"innovation": [
{
"question": "Tell me about a time when you came up with a creative solution to improve a process or solve a problem.",
"competency": "innovation",
"type": "behavioral",
"method": "STAR",
"focus_areas": ["creative_thinking", "process_improvement", "initiative"]
}
]
}
def _init_competency_mapping(self) -> Dict[str, Dict]:
"""Initialize role to competency mapping."""
return {
"software_engineer": {
"core_competencies": ["coding_fundamentals", "system_design", "problem_solving", "collaboration"],
"level_specific": {
"junior": ["coding_fundamentals", "debugging", "learning_agility"],
"mid": ["advanced_coding", "system_design", "mentoring_basics"],
"senior": ["system_architecture", "technical_leadership", "innovation"],
"staff": ["architectural_vision", "organizational_impact", "strategic_thinking"]
}
},
"frontend_engineer": {
"core_competencies": ["frontend_development", "ui_ux_understanding", "problem_solving", "collaboration"],
"level_specific": {
"junior": ["html_css_js", "responsive_design", "basic_frameworks"],
"mid": ["react_vue_angular", "state_management", "performance_optimization"],
"senior": ["frontend_architecture", "team_leadership", "cross_functional_collaboration"],
"staff": ["frontend_strategy", "technology_evaluation", "organizational_impact"]
}
},
"backend_engineer": {
"core_competencies": ["backend_development", "database_design", "api_design", "system_design"],
"level_specific": {
"junior": ["server_side_programming", "database_basics", "api_consumption"],
"mid": ["microservices", "caching", "security_basics"],
"senior": ["distributed_systems", "performance_optimization", "technical_leadership"],
"staff": ["system_architecture", "technology_strategy", "cross_team_influence"]
}
},
"product_manager": {
"core_competencies": ["product_strategy", "user_research", "data_analysis", "stakeholder_management"],
"level_specific": {
"junior": ["feature_specification", "user_stories", "basic_analytics"],
"mid": ["product_roadmap", "cross_functional_leadership", "market_research"],
"senior": ["business_strategy", "team_leadership", "p&l_responsibility"],
"staff": ["portfolio_management", "organizational_strategy", "market_creation"]
}
},
"data_scientist": {
"core_competencies": ["statistical_analysis", "machine_learning", "data_analysis", "business_acumen"],
"level_specific": {
"junior": ["python_r", "sql", "basic_ml", "data_visualization"],
"mid": ["advanced_ml", "experiment_design", "model_evaluation"],
"senior": ["ml_systems", "data_strategy", "stakeholder_communication"],
"staff": ["data_platform", "ai_strategy", "organizational_impact"]
}
},
"designer": {
"core_competencies": ["design_process", "user_research", "visual_design", "collaboration"],
"level_specific": {
"junior": ["design_tools", "user_empathy", "visual_communication"],
"mid": ["design_systems", "user_testing", "cross_functional_work"],
"senior": ["design_strategy", "team_leadership", "business_impact"],
"staff": ["design_vision", "organizational_design", "strategic_influence"]
}
},
"devops_engineer": {
"core_competencies": ["infrastructure", "automation", "monitoring", "troubleshooting"],
"level_specific": {
"junior": ["scripting", "basic_cloud", "ci_cd_basics"],
"mid": ["infrastructure_as_code", "container_orchestration", "security"],
"senior": ["platform_design", "reliability_engineering", "team_leadership"],
"staff": ["platform_strategy", "organizational_infrastructure", "technology_vision"]
}
}
}
def _init_scoring_rubrics(self) -> Dict[str, Dict]:
"""Initialize scoring rubrics for different question types."""
return {
"coding": {
"correctness": {
"4": "Solution is completely correct, handles all edge cases, optimal complexity",
"3": "Solution is correct for main cases, good complexity, minor edge case issues",
"2": "Solution works but has some bugs or suboptimal approach",
"1": "Solution has significant issues or doesn't work"
},
"code_quality": {
"4": "Clean, readable, well-structured code with excellent naming and comments",
"3": "Good code structure, readable with appropriate naming",
"2": "Code works but has style/structure issues",
"1": "Poor code quality, hard to understand"
},
"problem_solving_approach": {
"4": "Excellent problem breakdown, clear thinking process, considers alternatives",
"3": "Good approach, logical thinking, systematic problem solving",
"2": "Decent approach but some confusion or inefficiency",
"1": "Poor approach, unclear thinking process"
},
"communication": {
"4": "Excellent explanation of approach, asks clarifying questions, clear reasoning",
"3": "Good communication, explains thinking well",
"2": "Adequate communication, some explanation",
"1": "Poor communication, little explanation"
}
},
"behavioral": {
"situation_clarity": {
"4": "Clear, specific situation with relevant context and stakes",
"3": "Good situation description with adequate context",
"2": "Situation described but lacks some specifics",
"1": "Vague or unclear situation description"
},
"action_quality": {
"4": "Specific, thoughtful actions showing strong competency",
"3": "Good actions demonstrating competency",
"2": "Adequate actions but could be stronger",
"1": "Weak or inappropriate actions"
},
"result_impact": {
"4": "Significant positive impact with measurable results",
"3": "Good positive impact with clear outcomes",
"2": "Some positive impact demonstrated",
"1": "Little or no positive impact shown"
},
"self_awareness": {
"4": "Excellent self-reflection, learns from experience, acknowledges growth areas",
"3": "Good self-awareness and learning orientation",
"2": "Some self-reflection demonstrated",
"1": "Limited self-awareness or reflection"
}
},
"design": {
"system_thinking": {
"4": "Comprehensive system view, considers all components and interactions",
"3": "Good system understanding with most components identified",
"2": "Basic system thinking with some gaps",
"1": "Limited system thinking, misses key components"
},
"scalability": {
"4": "Excellent scalability considerations, multiple strategies discussed",
"3": "Good scalability awareness with practical solutions",
"2": "Basic scalability understanding",
"1": "Little to no scalability consideration"
},
"trade_offs": {
"4": "Excellent trade-off analysis, considers multiple dimensions",
"3": "Good trade-off awareness with clear reasoning",
"2": "Some trade-off consideration",
"1": "Limited trade-off analysis"
},
"technical_depth": {
"4": "Deep technical knowledge with implementation details",
"3": "Good technical knowledge with solid understanding",
"2": "Adequate technical knowledge",
"1": "Limited technical depth"
}
}
}
def _init_follow_up_strategies(self) -> Dict[str, List[str]]:
"""Initialize follow-up question strategies by competency."""
return {
"coding_fundamentals": [
"How would you optimize this solution for better time complexity?",
"What edge cases should we consider for this problem?",
"How would you test this function?",
"What would happen if the input size was very large?"
],
"system_design": [
"How would you handle if the system needed to scale 10x?",
"What would you do if one of your services went down?",
"How would you monitor this system in production?",
"What security considerations would you implement?"
],
"leadership": [
"What would you do differently if you faced this situation again?",
"How did you handle team members who were resistant to the change?",
"What metrics did you use to measure success?",
"How did you communicate progress to stakeholders?"
],
"problem_solving": [
"Walk me through your thought process step by step",
"What alternative approaches did you consider?",
"How did you validate your solution worked?",
"What did you learn from this experience?"
],
"collaboration": [
"How did you build consensus among the different stakeholders?",
"What communication channels did you use to keep everyone aligned?",
"How did you handle disagreements or conflicts?",
"What would you do to improve collaboration in the future?"
]
}
def generate_question_bank(self, role: str, level: str = "senior",
competencies: Optional[List[str]] = None,
question_types: Optional[List[str]] = None,
num_questions: int = 20) -> Dict[str, Any]:
"""Generate a comprehensive question bank for the specified role and competencies."""
# Normalize inputs
role_key = self._normalize_role(role)
level_key = level.lower()
# Get competency requirements
role_competencies = self._get_role_competencies(role_key, level_key, competencies)
# Determine question types to include
if question_types is None:
question_types = ["technical", "behavioral", "situational"]
# Generate questions
questions = self._generate_questions(role_competencies, question_types, level_key, num_questions)
# Create scoring rubrics
scoring_rubrics = self._create_scoring_rubrics(questions)
# Generate follow-up probes
follow_up_probes = self._generate_follow_up_probes(questions)
# Create calibration examples
calibration_examples = self._create_calibration_examples(questions[:5]) # Sample for first 5 questions
return {
"role": role,
"level": level,
"competencies": role_competencies,
"question_types": question_types,
"generated_at": datetime.now().isoformat(),
"total_questions": len(questions),
"questions": questions,
"scoring_rubrics": scoring_rubrics,
"follow_up_probes": follow_up_probes,
"calibration_examples": calibration_examples,
"usage_guidelines": self._generate_usage_guidelines(role_key, level_key)
}
def _normalize_role(self, role: str) -> str:
"""Normalize role name to match competency mapping keys."""
role_lower = role.lower().replace(" ", "_").replace("-", "_")
# Map variations to standard roles
role_mappings = {
"software_engineer": ["engineer", "developer", "swe", "software_developer"],
"frontend_engineer": ["frontend", "front_end", "ui_engineer", "web_developer"],
"backend_engineer": ["backend", "back_end", "server_engineer", "api_developer"],
"product_manager": ["pm", "product", "product_owner", "po"],
"data_scientist": ["ds", "data", "analyst", "ml_engineer"],
"designer": ["ux", "ui", "ux_ui", "product_designer", "visual_designer"],
"devops_engineer": ["devops", "sre", "platform_engineer", "infrastructure"]
}
for standard_role, variations in role_mappings.items():
if any(var in role_lower for var in variations):
return standard_role
# Default fallback
return "software_engineer"
def _get_role_competencies(self, role_key: str, level_key: str,
custom_competencies: Optional[List[str]]) -> List[str]:
"""Get competencies for the role and level."""
if role_key not in self.competency_mapping:
role_key = "software_engineer"
role_mapping = self.competency_mapping[role_key]
competencies = role_mapping["core_competencies"].copy()
# Add level-specific competencies
if level_key in role_mapping["level_specific"]:
competencies.extend(role_mapping["level_specific"][level_key])
elif "senior" in role_mapping["level_specific"]:
competencies.extend(role_mapping["level_specific"]["senior"])
# Add custom competencies if specified
if custom_competencies:
competencies.extend([comp.strip() for comp in custom_competencies if comp.strip() not in competencies])
return list(set(competencies)) # Remove duplicates
def _generate_questions(self, competencies: List[str], question_types: List[str],
level: str, num_questions: int) -> List[Dict[str, Any]]:
"""Generate questions based on competencies and types."""
questions = []
questions_per_competency = max(1, num_questions // len(competencies))
for competency in competencies:
competency_questions = []
# Add technical questions if requested and available
if "technical" in question_types and competency in self.technical_questions:
tech_questions = []
# Get questions for current level and below
level_order = ["junior", "mid", "senior", "staff", "principal"]
current_level_idx = level_order.index(level) if level in level_order else 2
for lvl_idx in range(current_level_idx + 1):
lvl = level_order[lvl_idx]
if lvl in self.technical_questions[competency]:
tech_questions.extend(self.technical_questions[competency][lvl])
competency_questions.extend(tech_questions[:questions_per_competency])
# Add behavioral questions if requested
if "behavioral" in question_types and competency in self.behavioral_questions:
behavioral_q = self.behavioral_questions[competency][:questions_per_competency]
competency_questions.extend(behavioral_q)
# Add situational questions (variations of behavioral)
if "situational" in question_types:
situational_q = self._generate_situational_questions(competency, questions_per_competency)
competency_questions.extend(situational_q)
# Ensure we have enough questions for this competency
while len(competency_questions) < questions_per_competency:
competency_questions.extend(self._generate_fallback_questions(competency, level))
if len(competency_questions) >= questions_per_competency:
break
questions.extend(competency_questions[:questions_per_competency])
# Shuffle and limit to requested number
random.shuffle(questions)
return questions[:num_questions]
def _generate_situational_questions(self, competency: str, count: int) -> List[Dict[str, Any]]:
"""Generate situational questions for a competency."""
situational_templates = {
"leadership": [
{
"question": "You're leading a project that's behind schedule and the client is unhappy. How do you handle this situation?",
"competency": competency,
"type": "situational",
"focus_areas": ["crisis_management", "client_communication", "team_leadership"]
}
],
"collaboration": [
{
"question": "You're working on a cross-functional project and two team members have opposing views on the technical approach. How do you resolve this?",
"competency": competency,
"type": "situational",
"focus_areas": ["conflict_resolution", "technical_decision_making", "facilitation"]
}
],
"problem_solving": [
{
"question": "You've been assigned to improve the performance of a critical system, but you have limited time and budget. Walk me through your approach.",
"competency": competency,
"type": "situational",
"focus_areas": ["prioritization", "resource_constraints", "systematic_approach"]
}
]
}
if competency in situational_templates:
return situational_templates[competency][:count]
return []
def _generate_fallback_questions(self, competency: str, level: str) -> List[Dict[str, Any]]:
"""Generate fallback questions when specific ones aren't available."""
fallback_questions = [
{
"question": f"Describe your experience with {competency.replace('_', ' ')} in your current or previous role.",
"competency": competency,
"type": "experience",
"focus_areas": ["experience_depth", "practical_application"]
},
{
"question": f"What challenges have you faced related to {competency.replace('_', ' ')} and how did you overcome them?",
"competency": competency,
"type": "challenge_based",
"focus_areas": ["problem_solving", "learning_from_experience"]
}
]
return fallback_questions
def _create_scoring_rubrics(self, questions: List[Dict[str, Any]]) -> Dict[str, Dict]:
"""Create scoring rubrics for the generated questions."""
rubrics = {}
for i, question in enumerate(questions, 1):
question_key = f"question_{i}"
question_type = question.get("type", "behavioral")
if question_type in self.scoring_rubrics:
rubrics[question_key] = {
"question": question["question"],
"competency": question["competency"],
"type": question_type,
"scoring_criteria": self.scoring_rubrics[question_type],
"weight": self._determine_question_weight(question),
"time_limit": question.get("time_limit", 30)
}
return rubrics
def _determine_question_weight(self, question: Dict[str, Any]) -> str:
"""Determine the weight/importance of a question."""
competency = question.get("competency", "")
question_type = question.get("type", "")
difficulty = question.get("difficulty", "medium")
# Core competencies get higher weight
core_competencies = ["coding_fundamentals", "system_design", "leadership", "problem_solving"]
if competency in core_competencies:
return "high"
elif question_type in ["coding", "design"] or difficulty == "hard":
return "high"
elif difficulty == "easy":
return "medium"
else:
return "medium"
def _generate_follow_up_probes(self, questions: List[Dict[str, Any]]) -> Dict[str, List[str]]:
"""Generate follow-up probes for each question."""
probes = {}
for i, question in enumerate(questions, 1):
question_key = f"question_{i}"
competency = question.get("competency", "")
# Get competency-specific follow-ups
if competency in self.follow_up_strategies:
competency_probes = self.follow_up_strategies[competency].copy()
else:
competency_probes = [
"Can you provide more specific details about your approach?",
"What would you do differently if you had to do this again?",
"What challenges did you face and how did you overcome them?"
]
# Add question-type specific probes
question_type = question.get("type", "")
if question_type == "coding":
competency_probes.extend([
"How would you test this solution?",
"What's the time and space complexity of your approach?",
"Can you think of any optimizations?"
])
elif question_type == "behavioral":
competency_probes.extend([
"What did you learn from this experience?",
"How did others react to your approach?",
"What metrics did you use to measure success?"
])
elif question_type == "design":
competency_probes.extend([
"How would you handle failure scenarios?",
"What monitoring would you implement?",
"How would this scale to 10x the load?"
])
probes[question_key] = competency_probes[:5] # Limit to 5 follow-ups
return probes
def _create_calibration_examples(self, sample_questions: List[Dict[str, Any]]) -> Dict[str, Dict]:
"""Create calibration examples with poor/good/great answers."""
examples = {}
for i, question in enumerate(sample_questions, 1):
question_key = f"question_{i}"
examples[question_key] = {
"question": question["question"],
"competency": question["competency"],
"sample_answers": {
"poor_answer": self._generate_sample_answer(question, "poor"),
"good_answer": self._generate_sample_answer(question, "good"),
"great_answer": self._generate_sample_answer(question, "great")
},
"scoring_rationale": self._generate_scoring_rationale(question)
}
return examples
def _generate_sample_answer(self, question: Dict[str, Any], quality: str) -> Dict[str, str]:
"""Generate sample answers of different quality levels."""
competency = question.get("competency", "")
question_type = question.get("type", "")
if quality == "poor":
return {
"answer": f"Sample poor answer for {competency} question - lacks detail, specificity, or demonstrates weak competency",
"score": "1-2",
"issues": ["Vague response", "Limited evidence of competency", "Poor structure"]
}
elif quality == "good":
return {
"answer": f"Sample good answer for {competency} question - adequate detail, demonstrates competency clearly",
"score": "3",
"strengths": ["Clear structure", "Demonstrates competency", "Adequate detail"]
}
else: # great
return {
"answer": f"Sample excellent answer for {competency} question - exceptional detail, strong evidence, goes above and beyond",
"score": "4",
"strengths": ["Exceptional detail", "Strong evidence", "Strategic thinking", "Goes beyond requirements"]
}
def _generate_scoring_rationale(self, question: Dict[str, Any]) -> Dict[str, str]:
"""Generate rationale for scoring this question."""
competency = question.get("competency", "")
return {
"key_indicators": f"Look for evidence of {competency.replace('_', ' ')} competency",
"red_flags": "Vague answers, lack of specifics, negative outcomes without learning",
"green_flags": "Specific examples, clear impact, demonstrates growth and learning"
}
def _generate_usage_guidelines(self, role_key: str, level_key: str) -> Dict[str, Any]:
"""Generate usage guidelines for the question bank."""
return {
"interview_flow": {
"warm_up": "Start with 1-2 easier questions to build rapport",
"core_assessment": "Focus majority of time on core competency questions",
"closing": "End with questions about candidate's questions/interests"
},
"time_management": {
"technical_questions": "Allow extra time for coding/design questions",
"behavioral_questions": "Keep to time limits but allow for follow-ups",
"total_recommendation": "45-75 minutes per interview round"
},
"question_selection": {
"variety": "Mix question types within each competency area",
"difficulty": "Adjust based on candidate responses and energy",
"customization": "Adapt questions based on candidate's background"
},
"common_mistakes": [
"Don't ask all questions mechanically",
"Don't skip follow-up questions",
"Don't forget to assess cultural fit alongside competencies",
"Don't let one strong/weak area bias overall assessment"
],
"calibration_reminders": [
"Compare against role standard, not other candidates",
"Focus on evidence demonstrated, not potential",
"Consider level-appropriate expectations",
"Document specific examples in feedback"
]
}
def format_human_readable(question_bank: Dict[str, Any]) -> str:
"""Format question bank data in human-readable format."""
output = []
# Header
output.append(f"Interview Question Bank: {question_bank['role']} ({question_bank['level'].title()} Level)")
output.append("=" * 70)
output.append(f"Generated: {question_bank['generated_at']}")
output.append(f"Total Questions: {question_bank['total_questions']}")
output.append(f"Question Types: {', '.join(question_bank['question_types'])}")
output.append(f"Target Competencies: {', '.join(question_bank['competencies'])}")
output.append("")
# Questions
output.append("INTERVIEW QUESTIONS")
output.append("-" * 50)
for i, question in enumerate(question_bank['questions'], 1):
output.append(f"\n{i}. {question['question']}")
output.append(f" Competency: {question['competency'].replace('_', ' ').title()}")
output.append(f" Type: {question.get('type', 'N/A').title()}")
if 'time_limit' in question:
output.append(f" Time Limit: {question['time_limit']} minutes")
if 'focus_areas' in question:
output.append(f" Focus Areas: {', '.join(question['focus_areas'])}")
# Scoring Guidelines
output.append("\n\nSCORING RUBRICS")
output.append("-" * 50)
# Show sample scoring criteria
if question_bank['scoring_rubrics']:
first_question = list(question_bank['scoring_rubrics'].keys())[0]
sample_rubric = question_bank['scoring_rubrics'][first_question]
output.append(f"Sample Scoring Criteria ({sample_rubric['type']} questions):")
for criterion, scores in sample_rubric['scoring_criteria'].items():
output.append(f"\n{criterion.replace('_', ' ').title()}:")
for score, description in scores.items():
output.append(f" {score}: {description}")
# Follow-up Probes
output.append("\n\nFOLLOW-UP PROBE EXAMPLES")
output.append("-" * 50)
if question_bank['follow_up_probes']:
first_question = list(question_bank['follow_up_probes'].keys())[0]
sample_probes = question_bank['follow_up_probes'][first_question]
output.append("Sample follow-up questions:")
for probe in sample_probes[:3]: # Show first 3
output.append(f" • {probe}")
# Usage Guidelines
output.append("\n\nUSAGE GUIDELINES")
output.append("-" * 50)
guidelines = question_bank['usage_guidelines']
output.append("Interview Flow:")
for phase, description in guidelines['interview_flow'].items():
output.append(f" • {phase.replace('_', ' ').title()}: {description}")
output.append("\nTime Management:")
for aspect, recommendation in guidelines['time_management'].items():
output.append(f" • {aspect.replace('_', ' ').title()}: {recommendation}")
output.append("\nCommon Mistakes to Avoid:")
for mistake in guidelines['common_mistakes'][:3]: # Show first 3
output.append(f" • {mistake}")
# Calibration Examples (if available)
if question_bank['calibration_examples']:
output.append("\n\nCALIBRATION EXAMPLES")
output.append("-" * 50)
first_example = list(question_bank['calibration_examples'].values())[0]
output.append(f"Question: {first_example['question']}")
output.append("\nSample Answer Quality Levels:")
for quality, details in first_example['sample_answers'].items():
output.append(f" {quality.replace('_', ' ').title()} (Score {details['score']}):")
if 'issues' in details:
output.append(f" Issues: {', '.join(details['issues'])}")
if 'strengths' in details:
output.append(f" Strengths: {', '.join(details['strengths'])}")
return "\n".join(output)
def main():
parser = argparse.ArgumentParser(description="Generate comprehensive interview question banks with scoring criteria")
parser.add_argument("--role", type=str, help="Job role title (e.g., 'Frontend Engineer')")
parser.add_argument("--level", type=str, default="senior", help="Experience level (junior, mid, senior, staff, principal)")
parser.add_argument("--competencies", type=str, help="Comma-separated list of competencies to focus on")
parser.add_argument("--question-types", type=str, help="Comma-separated list of question types (technical, behavioral, situational)")
parser.add_argument("--num-questions", type=int, default=20, help="Number of questions to generate")
parser.add_argument("--input", type=str, help="Input JSON file with role requirements")
parser.add_argument("--output", type=str, help="Output directory or file path")
parser.add_argument("--format", choices=["json", "text", "both"], default="both", help="Output format")
args = parser.parse_args()
generator = QuestionBankGenerator()
# Handle input
if args.input:
try:
with open(args.input, 'r') as f:
role_data = json.load(f)
role = role_data.get('role') or role_data.get('title', '')
level = role_data.get('level', 'senior')
competencies = role_data.get('competencies')
question_types = role_data.get('question_types')
num_questions = role_data.get('num_questions', 20)
except Exception as e:
print(f"Error reading input file: {e}")
sys.exit(1)
else:
if not args.role:
print("Error: --role is required when not using --input")
sys.exit(1)
role = args.role
level = args.level
competencies = args.competencies.split(',') if args.competencies else None
question_types = args.question_types.split(',') if args.question_types else None
num_questions = args.num_questions
# Generate question bank
try:
question_bank = generator.generate_question_bank(
role=role,
level=level,
competencies=competencies,
question_types=question_types,
num_questions=num_questions
)
# Handle output
if args.output:
output_path = args.output
if os.path.isdir(output_path):
safe_role = "".join(c for c in role.lower() if c.isalnum() or c in (' ', '-', '_')).replace(' ', '_')
base_filename = f"{safe_role}_{level}_questions"
json_path = os.path.join(output_path, f"{base_filename}.json")
text_path = os.path.join(output_path, f"{base_filename}.txt")
else:
json_path = output_path if output_path.endswith('.json') else f"{output_path}.json"
text_path = output_path.replace('.json', '.txt') if output_path.endswith('.json') else f"{output_path}.txt"
else:
safe_role = "".join(c for c in role.lower() if c.isalnum() or c in (' ', '-', '_')).replace(' ', '_')
base_filename = f"{safe_role}_{level}_questions"
json_path = f"{base_filename}.json"
text_path = f"{base_filename}.txt"
# Write outputs
if args.format in ["json", "both"]:
with open(json_path, 'w') as f:
json.dump(question_bank, f, indent=2, default=str)
print(f"JSON output written to: {json_path}")
if args.format in ["text", "both"]:
with open(text_path, 'w') as f:
f.write(format_human_readable(question_bank))
print(f"Text output written to: {text_path}")
# Print summary
print(f"\nQuestion Bank Summary:")
print(f"Role: {question_bank['role']} ({question_bank['level'].title()})")
print(f"Total Questions: {question_bank['total_questions']}")
print(f"Competencies Covered: {len(question_bank['competencies'])}")
print(f"Question Types: {', '.join(question_bank['question_types'])}")
except Exception as e:
print(f"Error generating question bank: {e}")
sys.exit(1)
if __name__ == "__main__":
main()
FILE:README.md
# Interview System Designer
A comprehensive toolkit for designing, optimizing, and calibrating interview processes. This skill provides tools to create role-specific interview loops, generate competency-based question banks, and analyze hiring data for bias and calibration issues.
## Overview
The Interview System Designer skill includes three powerful Python tools and comprehensive reference materials to help you build fair, effective, and scalable hiring processes:
1. **Interview Loop Designer** - Generate calibrated interview loops for any role and level
2. **Question Bank Generator** - Create competency-based interview questions with scoring rubrics
3. **Hiring Calibrator** - Analyze interview data to detect bias and calibration issues
## Tools
### 1. Interview Loop Designer (`loop_designer.py`)
Generates complete interview loops tailored to specific roles, levels, and teams.
**Features:**
- Role-specific competency mapping (SWE, PM, Designer, Data, DevOps, Leadership)
- Level-appropriate interview rounds (junior through principal)
- Optimized scheduling and time allocation
- Interviewer skill requirements
- Standardized scorecard templates
**Usage:**
```bash
# Basic usage
python3 loop_designer.py --role "Senior Software Engineer" --level senior
# With team and custom competencies
python3 loop_designer.py --role "Product Manager" --level mid --team growth --competencies leadership,strategy,analytics
# Using JSON input file
python3 loop_designer.py --input assets/sample_role_definitions.json --output loops/
# Specify output format
python3 loop_designer.py --role "Staff Data Scientist" --level staff --format json --output data_scientist_loop.json
```
**Input Options:**
- `--role`: Job role title (e.g., "Senior Software Engineer", "Product Manager")
- `--level`: Experience level (junior, mid, senior, staff, principal)
- `--team`: Team or department (optional)
- `--competencies`: Comma-separated list of specific competencies to focus on
- `--input`: JSON file with role definition
- `--output`: Output directory or file path
- `--format`: Output format (json, text, both) - default: both
**Example Output:**
```
Interview Loop Design for Senior Software Engineer (Senior Level)
============================================================
Total Duration: 300 minutes (5h 0m)
Total Rounds: 5
INTERVIEW ROUNDS
----------------------------------------
Round 1: Technical Phone Screen
Duration: 45 minutes
Format: Virtual
Focus Areas: Coding Fundamentals, Problem Solving
Round 2: System Design
Duration: 75 minutes
Format: Collaborative Whitboard
Focus Areas: System Thinking, Architectural Reasoning
...
```
### 2. Question Bank Generator (`question_bank_generator.py`)
Creates comprehensive interview question banks organized by competency area.
**Features:**
- Competency-based question organization
- Level-appropriate difficulty progression
- Multiple question types (technical, behavioral, situational)
- Detailed scoring rubrics with calibration examples
- Follow-up probes and conversation guides
**Usage:**
```bash
# Generate questions for specific competencies
python3 question_bank_generator.py --role "Frontend Engineer" --competencies react,typescript,system-design
# Create behavioral question bank
python3 question_bank_generator.py --role "Product Manager" --question-types behavioral,leadership --num-questions 15
# Generate questions for multiple levels
python3 question_bank_generator.py --role "DevOps Engineer" --levels junior,mid,senior --output questions/
```
**Input Options:**
- `--role`: Job role title
- `--level`: Experience level (default: senior)
- `--competencies`: Comma-separated list of competencies to focus on
- `--question-types`: Types to include (technical, behavioral, situational)
- `--num-questions`: Number of questions to generate (default: 20)
- `--input`: JSON file with role requirements
- `--output`: Output directory or file path
- `--format`: Output format (json, text, both) - default: both
**Question Types:**
- **Technical**: Coding problems, system design, domain-specific challenges
- **Behavioral**: STAR method questions focusing on past experiences
- **Situational**: Hypothetical scenarios testing decision-making
### 3. Hiring Calibrator (`hiring_calibrator.py`)
Analyzes interview scores to detect bias, calibration issues, and provides recommendations.
**Features:**
- Statistical bias detection across demographics
- Interviewer calibration analysis
- Score distribution and trending analysis
- Specific coaching recommendations
- Comprehensive reporting with actionable insights
**Usage:**
```bash
# Comprehensive analysis
python3 hiring_calibrator.py --input assets/sample_interview_results.json --analysis-type comprehensive
# Focus on specific areas
python3 hiring_calibrator.py --input interview_data.json --analysis-type bias --competencies technical,leadership
# Trend analysis over time
python3 hiring_calibrator.py --input historical_data.json --trend-analysis --period quarterly
```
**Input Options:**
- `--input`: JSON file with interview results data (required)
- `--analysis-type`: Type of analysis (comprehensive, bias, calibration, interviewer, scoring)
- `--competencies`: Comma-separated list of competencies to focus on
- `--trend-analysis`: Enable trend analysis over time
- `--period`: Time period for trends (daily, weekly, monthly, quarterly)
- `--output`: Output file path
- `--format`: Output format (json, text, both) - default: both
**Analysis Types:**
- **Comprehensive**: Full analysis including bias, calibration, and recommendations
- **Bias**: Focus on demographic and interviewer bias patterns
- **Calibration**: Interviewer consistency and agreement analysis
- **Interviewer**: Individual interviewer performance and coaching needs
- **Scoring**: Score distribution and pattern analysis
## Data Formats
### Role Definition Input (JSON)
```json
{
"role": "Senior Software Engineer",
"level": "senior",
"team": "platform",
"competencies": ["system_design", "technical_leadership", "mentoring"],
"requirements": {
"years_experience": "5-8",
"technical_skills": ["Python", "AWS", "Kubernetes"],
"leadership_experience": true
}
}
```
### Interview Results Input (JSON)
```json
[
{
"candidate_id": "candidate_001",
"role": "Senior Software Engineer",
"interviewer_id": "interviewer_alice",
"date": "2024-01-15T09:00:00Z",
"scores": {
"coding_fundamentals": 3.5,
"system_design": 4.0,
"technical_leadership": 3.0,
"communication": 3.5
},
"overall_recommendation": "Hire",
"gender": "male",
"ethnicity": "asian",
"years_experience": 6
}
]
```
## Reference Materials
### Competency Matrix Templates (`references/competency_matrix_templates.md`)
- Comprehensive competency matrices for all engineering roles
- Level-specific expectations (junior through principal)
- Assessment criteria and growth paths
- Customization guidelines for different company stages and industries
### Bias Mitigation Checklist (`references/bias_mitigation_checklist.md`)
- Pre-interview preparation checklist
- Interview process bias prevention strategies
- Real-time bias interruption techniques
- Legal compliance reminders
- Emergency response protocols
### Debrief Facilitation Guide (`references/debrief_facilitation_guide.md`)
- Structured debrief meeting frameworks
- Evidence-based discussion techniques
- Bias interruption strategies
- Decision documentation standards
- Common challenges and solutions
## Sample Data
The `assets/` directory contains sample data for testing:
- `sample_role_definitions.json`: Example role definitions for various positions
- `sample_interview_results.json`: Sample interview data with multiple candidates and interviewers
## Expected Outputs
The `expected_outputs/` directory contains examples of tool outputs:
- Interview loop designs in both JSON and human-readable formats
- Question banks with scoring rubrics and calibration examples
- Calibration analysis reports with bias detection and recommendations
## Best Practices
### Interview Loop Design
1. **Competency Focus**: Align interview rounds with role-critical competencies
2. **Level Calibration**: Adjust expectations and question difficulty based on experience level
3. **Time Optimization**: Balance thoroughness with candidate experience
4. **Interviewer Training**: Ensure interviewers are qualified and calibrated
### Question Bank Development
1. **Evidence-Based**: Focus on observable behaviors and concrete examples
2. **Bias Mitigation**: Use structured questions that minimize subjective interpretation
3. **Calibration**: Include examples of different quality responses for consistency
4. **Continuous Improvement**: Regularly update questions based on predictive validity
### Calibration Analysis
1. **Regular Monitoring**: Analyze hiring data quarterly for bias patterns
2. **Prompt Action**: Address calibration issues immediately with targeted coaching
3. **Data Quality**: Ensure complete and consistent data collection
4. **Legal Compliance**: Monitor for discriminatory patterns and document corrections
## Installation & Setup
No external dependencies required - uses Python 3 standard library only.
```bash
# Clone or download the skill directory
cd interview-system-designer/
# Make scripts executable (optional)
chmod +x *.py
# Test with sample data
python3 loop_designer.py --role "Senior Software Engineer" --level senior
python3 question_bank_generator.py --role "Product Manager" --level mid
python3 hiring_calibrator.py --input assets/sample_interview_results.json
```
## Integration
### With Existing Systems
- **ATS Integration**: Export interview loops as structured data for applicant tracking systems
- **Calendar Systems**: Use scheduling outputs to auto-create interview blocks
- **HR Analytics**: Import calibration reports into broader diversity and inclusion dashboards
### Custom Workflows
- **Batch Processing**: Process multiple roles or historical data sets
- **Automated Reporting**: Schedule regular calibration analysis
- **Custom Competencies**: Extend frameworks with company-specific competencies
## Troubleshooting
### Common Issues
**"Role not found" errors:**
- The tool will map common variations (engineer → software_engineer)
- For custom roles, use the closest standard role and specify custom competencies
**"Insufficient data" errors:**
- Minimum 5 interviews required for statistical analysis
- Ensure interview data includes required fields (candidate_id, interviewer_id, scores, date)
**Missing output files:**
- Check file permissions in output directory
- Ensure adequate disk space
- Verify JSON input file format is valid
### Performance Considerations
- Interview loop generation: < 1 second
- Question bank generation: 1-3 seconds for 20 questions
- Calibration analysis: 1-5 seconds for 50 interviews, scales linearly
## Contributing
To extend this skill:
1. **New Roles**: Add competency frameworks in `_init_competency_frameworks()`
2. **New Question Types**: Extend question templates in respective generators
3. **New Analysis Types**: Add analysis methods to hiring calibrator
4. **Custom Outputs**: Modify formatting functions for different output needs
## License & Usage
This skill is designed for internal company use in hiring process optimization. All bias detection and mitigation features should be reviewed with legal counsel to ensure compliance with local employment laws.
For questions or support, refer to the comprehensive documentation in each script's docstring and the reference materials provided.
FILE:references/bias_mitigation_checklist.md
# Interview Bias Mitigation Checklist
This comprehensive checklist helps identify, prevent, and mitigate various forms of bias in the interview process. Use this as a systematic guide to ensure fair and equitable hiring practices.
## Pre-Interview Phase
### Job Description & Requirements
- [ ] **Remove unnecessary requirements** that don't directly relate to job performance
- [ ] **Avoid gendered language** (competitive, aggressive vs. collaborative, detail-oriented)
- [ ] **Remove university prestige requirements** unless absolutely necessary for role
- [ ] **Focus on skills and outcomes** rather than years of experience in specific technologies
- [ ] **Use inclusive language** and avoid cultural assumptions
- [ ] **Specify only essential requirements** vs. nice-to-have qualifications
- [ ] **Remove location/commute assumptions** for remote-eligible positions
- [ ] **Review requirements for unconscious bias** (e.g., assuming continuous work history)
### Sourcing & Pipeline
- [ ] **Diversify sourcing channels** beyond traditional networks
- [ ] **Partner with diverse professional organizations** and communities
- [ ] **Use bias-minimizing sourcing tools** and platforms
- [ ] **Track sourcing effectiveness** by demographic groups
- [ ] **Train recruiters on bias awareness** and inclusive outreach
- [ ] **Review referral patterns** for potential network bias
- [ ] **Expand university partnerships** beyond elite institutions
- [ ] **Use structured outreach messages** to reduce individual bias
### Resume Screening
- [ ] **Implement blind resume review** (remove names, photos, university names initially)
- [ ] **Use standardized screening criteria** applied consistently
- [ ] **Multiple screeners for each resume** with independent scoring
- [ ] **Focus on relevant skills and achievements** over pedigree indicators
- [ ] **Avoid assumptions about career gaps** or non-traditional backgrounds
- [ ] **Consider alternative paths to skills** (bootcamps, self-taught, career changes)
- [ ] **Track screening pass rates** by demographic groups
- [ ] **Regular screener calibration sessions** on bias awareness
## Interview Panel Composition
### Diversity Requirements
- [ ] **Ensure diverse interview panels** (gender, ethnicity, seniority levels)
- [ ] **Include at least one underrepresented interviewer** when possible
- [ ] **Rotate panel assignments** to prevent bias patterns
- [ ] **Balance seniority levels** on panels (not all senior or all junior)
- [ ] **Include cross-functional perspectives** when relevant
- [ ] **Avoid panels of only one demographic group** when possible
- [ ] **Consider panel member unconscious bias training** status
- [ ] **Document panel composition rationale** for future review
### Interviewer Selection
- [ ] **Choose interviewers based on relevant competency assessment ability**
- [ ] **Ensure interviewers have completed bias training** within last 12 months
- [ ] **Select interviewers with consistent calibration history**
- [ ] **Avoid interviewers with known bias patterns** (flagged in previous analyses)
- [ ] **Include at least one interviewer familiar with candidate's background type**
- [ ] **Balance perspectives** (technical depth, cultural fit, growth potential)
- [ ] **Consider interviewer availability for proper preparation time**
- [ ] **Ensure interviewers understand role requirements and standards**
## Interview Process Design
### Question Standardization
- [ ] **Use standardized question sets** for each competency area
- [ ] **Develop questions that assess skills, not culture fit stereotypes**
- [ ] **Avoid questions about personal background** unless directly job-relevant
- [ ] **Remove questions that could reveal protected characteristics**
- [ ] **Focus on behavioral examples** using STAR method
- [ ] **Include scenario-based questions** with clear evaluation criteria
- [ ] **Test questions for potential bias** with diverse interviewers
- [ ] **Regularly update question bank** based on effectiveness data
### Structured Interview Protocol
- [ ] **Define clear time allocations** for each question/section
- [ ] **Establish consistent interview flow** across all candidates
- [ ] **Create standardized intro/outro** processes
- [ ] **Use identical technical setup and tools** for all candidates
- [ ] **Provide same background information** to all interviewers
- [ ] **Standardize note-taking format** and requirements
- [ ] **Define clear handoff procedures** between interviewers
- [ ] **Document any deviations** from standard protocol
### Accommodation Preparation
- [ ] **Proactively offer accommodations** without requiring disclosure
- [ ] **Provide multiple interview format options** (phone, video, in-person)
- [ ] **Ensure accessibility of interview locations and tools**
- [ ] **Allow extended time** when requested or needed
- [ ] **Provide materials in advance** when helpful
- [ ] **Train interviewers on accommodation protocols**
- [ ] **Test all technology** for accessibility compliance
- [ ] **Have backup plans** for technical issues
## During the Interview
### Interviewer Behavior
- [ ] **Use welcoming, professional tone** with all candidates
- [ ] **Avoid assumptions based on appearance or background**
- [ ] **Give equal encouragement and support** to all candidates
- [ ] **Allow equal time for candidate questions**
- [ ] **Avoid leading questions** that suggest desired answers
- [ ] **Listen actively** without interrupting unnecessarily
- [ ] **Take detailed notes** focusing on responses, not impressions
- [ ] **Avoid small talk** that could reveal irrelevant personal information
### Question Delivery
- [ ] **Ask questions as written** without improvisation that could introduce bias
- [ ] **Provide equal clarification** when candidates ask for it
- [ ] **Use consistent follow-up probing** across candidates
- [ ] **Allow reasonable thinking time** before expecting responses
- [ ] **Avoid rephrasing questions** in ways that give hints
- [ ] **Stay focused on defined competencies** being assessed
- [ ] **Give equal encouragement** for elaboration when needed
- [ ] **Maintain professional demeanor** regardless of candidate background
### Real-time Bias Checking
- [ ] **Notice first impressions** but don't let them drive assessment
- [ ] **Question gut reactions** - are they based on competency evidence?
- [ ] **Focus on specific examples** and evidence provided
- [ ] **Avoid pattern matching** to existing successful employees
- [ ] **Notice cultural assumptions** in interpretation of responses
- [ ] **Check for confirmation bias** - seeking evidence to support initial impressions
- [ ] **Consider alternative explanations** for candidate responses
- [ ] **Stay aware of fatigue effects** on judgment throughout the day
## Evaluation & Scoring
### Scoring Consistency
- [ ] **Use defined rubrics consistently** across all candidates
- [ ] **Score immediately after interview** while details are fresh
- [ ] **Focus scoring on demonstrated competencies** not potential or personality
- [ ] **Provide specific evidence** for each score given
- [ ] **Avoid comparative scoring** (comparing candidates to each other)
- [ ] **Use calibrated examples** of each score level
- [ ] **Score independently** before discussing with other interviewers
- [ ] **Document reasoning** for all scores, especially extreme ones (1s and 4s)
### Bias Check Questions
- [ ] **"Would I score this differently if the candidate looked different?"**
- [ ] **"Am I basing this on evidence or assumptions?"**
- [ ] **"Would this response get the same score from a different demographic?"**
- [ ] **"Am I penalizing non-traditional backgrounds or approaches?"**
- [ ] **"Is my scoring consistent with the defined rubric?"**
- [ ] **"Am I letting one strong/weak area bias overall assessment?"**
- [ ] **"Are my cultural assumptions affecting interpretation?"**
- [ ] **"Would I want to work with this person?" (Check if this is biasing assessment)**
### Documentation Requirements
- [ ] **Record specific examples** supporting each competency score
- [ ] **Avoid subjective language** like "seems like," "appears to be"
- [ ] **Focus on observable behaviors** and concrete responses
- [ ] **Note exact quotes** when relevant to assessment
- [ ] **Distinguish between facts and interpretations**
- [ ] **Provide improvement suggestions** that are skill-based, not person-based
- [ ] **Avoid comparative language** to other candidates or employees
- [ ] **Use neutral language** free from cultural assumptions
## Debrief Process
### Structured Discussion
- [ ] **Start with independent score sharing** before discussion
- [ ] **Focus discussion on evidence** not impressions or feelings
- [ ] **Address significant score discrepancies** with evidence review
- [ ] **Challenge biased language** or assumptions in discussion
- [ ] **Ensure all voices are heard** in group decision making
- [ ] **Document reasons for final decision** with specific evidence
- [ ] **Avoid personality-based discussions** ("culture fit" should be evidence-based)
- [ ] **Consider multiple perspectives** on candidate responses
### Decision-Making Process
- [ ] **Use weighted scoring system** based on role requirements
- [ ] **Require minimum scores** in critical competency areas
- [ ] **Avoid veto power** unless based on clear, documented evidence
- [ ] **Consider growth potential** fairly across all candidates
- [ ] **Document dissenting opinions** and reasoning
- [ ] **Use tie-breaking criteria** that are predetermined and fair
- [ ] **Consider additional data collection** if team is split
- [ ] **Make final decision based on role requirements**, not team preferences
### Final Recommendations
- [ ] **Provide specific, actionable feedback** for development areas
- [ ] **Focus recommendations on skills and competencies**
- [ ] **Avoid language that could reflect bias** in written feedback
- [ ] **Consider onboarding needs** based on actual skill gaps, not assumptions
- [ ] **Provide coaching recommendations** that are evidence-based
- [ ] **Avoid personal judgments** about candidate character or personality
- [ ] **Make hiring recommendation** based solely on job-relevant criteria
- [ ] **Document any concerns** with specific, observable evidence
## Post-Interview Monitoring
### Data Collection
- [ ] **Track interviewer scoring patterns** for consistency analysis
- [ ] **Monitor pass rates** by demographic groups
- [ ] **Collect candidate experience feedback** on interview fairness
- [ ] **Analyze score distributions** for potential bias indicators
- [ ] **Track time-to-decision** across different candidate types
- [ ] **Monitor offer acceptance rates** by demographics
- [ ] **Collect new hire performance data** for process validation
- [ ] **Document any bias incidents** or concerns raised
### Regular Analysis
- [ ] **Conduct quarterly bias audits** of interview data
- [ ] **Review interviewer calibration** and identify outliers
- [ ] **Analyze demographic trends** in hiring outcomes
- [ ] **Compare candidate experience surveys** across groups
- [ ] **Track correlation between interview scores and job performance**
- [ ] **Review and update bias mitigation strategies** based on data
- [ ] **Share findings with interview teams** for continuous improvement
- [ ] **Update training programs** based on identified bias patterns
## Bias Types to Watch For
### Affinity Bias
- **Definition**: Favoring candidates similar to yourself
- **Watch for**: Over-positive response to shared backgrounds, interests, or experiences
- **Mitigation**: Focus on job-relevant competencies, diversify interview panels
### Halo/Horn Effect
- **Definition**: One positive/negative trait influencing overall assessment
- **Watch for**: Strong performance in one area affecting scores in unrelated areas
- **Mitigation**: Score each competency independently, use structured evaluation
### Confirmation Bias
- **Definition**: Seeking information that confirms initial impressions
- **Watch for**: Asking follow-ups that lead candidate toward expected responses
- **Mitigation**: Use standardized questions, consider alternative interpretations
### Attribution Bias
- **Definition**: Attributing success/failure to different causes based on candidate demographics
- **Watch for**: Assuming women are "lucky" vs. men are "skilled" for same achievements
- **Mitigation**: Focus on candidate's role in achievements, avoid assumptions
### Cultural Bias
- **Definition**: Judging candidates based on cultural differences rather than job performance
- **Watch for**: Penalizing communication styles, work approaches, or values that differ from team norm
- **Mitigation**: Define job-relevant criteria clearly, consider diverse perspectives valuable
### Educational Bias
- **Definition**: Over-weighting prestigious educational credentials
- **Watch for**: Assuming higher capability based on school rank rather than demonstrated skills
- **Mitigation**: Focus on skills demonstration, consider alternative learning paths
### Experience Bias
- **Definition**: Requiring specific company or industry experience unnecessarily
- **Watch for**: Discounting transferable skills from different industries or company sizes
- **Mitigation**: Define core skills needed, assess adaptability and learning ability
## Emergency Bias Response Protocol
### During Interview
1. **Pause the interview** if significant bias is observed
2. **Privately address** bias with interviewer if possible
3. **Document the incident** for review
4. **Continue with fair assessment** of candidate
5. **Flag for debrief discussion** if interview continues
### Post-Interview
1. **Report bias incidents** to hiring manager/HR immediately
2. **Document specific behaviors** observed
3. **Consider additional interviewer** for second opinion
4. **Review candidate assessment** for bias impact
5. **Implement corrective actions** for future interviews
### Interviewer Coaching
1. **Provide immediate feedback** on bias observed
2. **Schedule bias training refresher** if needed
3. **Monitor future interviews** for improvement
4. **Consider removing from interview rotation** if bias persists
5. **Document coaching provided** for performance management
## Legal Compliance Reminders
### Protected Characteristics
- Age, race, color, religion, sex, national origin, disability status, veteran status
- Pregnancy, genetic information, sexual orientation, gender identity
- Any other characteristics protected by local/state/federal law
### Prohibited Questions
- Questions about family planning, marital status, pregnancy
- Age-related questions (unless BFOQ)
- Religious or political affiliations
- Disability status (unless voluntary disclosure for accommodation)
- Arrest records (without conviction relevance)
- Financial status or credit (unless job-relevant)
### Documentation Requirements
- Keep all interview materials for required retention period
- Ensure consistent documentation across all candidates
- Avoid documenting protected characteristic observations
- Focus documentation on job-relevant observations only
## Training & Certification
### Required Training Topics
- Unconscious bias awareness and mitigation
- Structured interviewing techniques
- Legal compliance in hiring
- Company-specific bias mitigation protocols
- Role-specific competency assessment
- Accommodation and accessibility requirements
### Ongoing Development
- Annual bias training refresher
- Quarterly calibration sessions
- Regular updates on legal requirements
- Peer feedback and coaching
- Industry best practice updates
- Data-driven process improvements
This checklist should be reviewed and updated regularly based on legal requirements, industry best practices, and internal bias analysis results.
FILE:references/competency_matrix_templates.md
# Competency Matrix Templates
This document provides comprehensive competency matrix templates for different engineering roles and levels. Use these matrices to design role-specific interview loops and evaluation criteria.
## Software Engineering Competency Matrix
### Technical Competencies
| Competency | Junior (L1-L2) | Mid (L3-L4) | Senior (L5-L6) | Staff+ (L7+) |
|------------|----------------|-------------|----------------|--------------|
| **Coding & Algorithms** | Basic data structures, simple algorithms, language syntax | Advanced algorithms, complexity analysis, optimization | Complex problem solving, algorithm design, performance tuning | Architecture-level algorithmic decisions, novel approach design |
| **System Design** | Component interactions, basic scalability concepts | Service design, database modeling, API design | Distributed systems, scalability patterns, trade-off analysis | Large-scale architecture, cross-system design, technology strategy |
| **Code Quality** | Readable code, basic testing, follows conventions | Maintainable code, comprehensive testing, design patterns | Code reviews, quality standards, refactoring leadership | Engineering standards, quality culture, technical debt management |
| **Debugging & Problem Solving** | Basic debugging, structured problem approach | Complex debugging, root cause analysis, performance issues | System-wide debugging, production issues, incident response | Cross-system troubleshooting, preventive measures, tooling design |
| **Domain Knowledge** | Learning role-specific technologies | Proficiency in domain tools/frameworks | Deep domain expertise, technology evaluation | Domain leadership, technology roadmap, innovation |
### Behavioral Competencies
| Competency | Junior (L1-L2) | Mid (L3-L4) | Senior (L5-L6) | Staff+ (L7+) |
|------------|----------------|-------------|----------------|--------------|
| **Communication** | Clear status updates, asks good questions | Technical explanations, stakeholder updates | Cross-functional communication, technical writing | Executive communication, external representation, thought leadership |
| **Collaboration** | Team participation, code reviews | Cross-team projects, knowledge sharing | Team leadership, conflict resolution | Cross-org collaboration, culture building, strategic partnerships |
| **Leadership & Influence** | Peer mentoring, positive attitude | Junior mentoring, project ownership | Team guidance, technical decisions, hiring | Org-wide influence, vision setting, culture change |
| **Growth & Learning** | Skill development, feedback receptivity | Proactive learning, teaching others | Continuous improvement, trend awareness | Learning culture, industry leadership, innovation adoption |
| **Ownership & Initiative** | Task completion, quality focus | Project ownership, process improvement | Feature/service ownership, strategic thinking | Product/platform ownership, business impact, market influence |
## Product Management Competency Matrix
### Product Competencies
| Competency | Associate PM (L1-L2) | PM (L3-L4) | Senior PM (L5-L6) | Principal PM (L7+) |
|------------|---------------------|------------|-------------------|-------------------|
| **Product Strategy** | Feature requirements, user stories | Product roadmaps, market analysis | Business strategy, competitive positioning | Portfolio strategy, market creation, platform vision |
| **User Research & Analytics** | Basic user interviews, metrics tracking | Research design, data interpretation | Research strategy, advanced analytics | Research culture, measurement frameworks, insight generation |
| **Technical Understanding** | Basic tech concepts, API awareness | System architecture, technical trade-offs | Technical strategy, platform decisions | Technology vision, architectural influence, innovation leadership |
| **Execution & Process** | Feature delivery, stakeholder coordination | Project management, cross-functional leadership | Process optimization, team scaling | Operational excellence, org design, strategic execution |
| **Business Acumen** | Revenue awareness, customer understanding | P&L understanding, business case development | Business strategy, market dynamics | Corporate strategy, board communication, investor relations |
### Leadership Competencies
| Competency | Associate PM (L1-L2) | PM (L3-L4) | Senior PM (L5-L6) | Principal PM (L7+) |
|------------|---------------------|------------|-------------------|-------------------|
| **Stakeholder Management** | Team collaboration, clear communication | Cross-functional alignment, expectation management | Executive communication, influence without authority | Board interaction, external partnerships, industry influence |
| **Team Development** | Peer learning, feedback sharing | Junior mentoring, knowledge transfer | Team building, hiring, performance management | Talent development, culture building, org leadership |
| **Decision Making** | Data-driven decisions, priority setting | Complex trade-offs, strategic choices | Ambiguous situations, high-stakes decisions | Strategic vision, transformational decisions, risk management |
| **Innovation & Vision** | Creative problem solving, user empathy | Market opportunity identification, feature innovation | Product vision, market strategy | Industry vision, disruptive thinking, platform creation |
## Design Competency Matrix
### Design Competencies
| Competency | Junior Designer (L1-L2) | Mid Designer (L3-L4) | Senior Designer (L5-L6) | Principal Designer (L7+) |
|------------|-------------------------|---------------------|-------------------------|-------------------------|
| **Visual Design** | UI components, typography, color theory | Design systems, visual hierarchy | Brand integration, advanced layouts | Visual strategy, brand evolution, design innovation |
| **User Experience** | User flows, wireframing, prototyping | Interaction design, usability testing | Experience strategy, journey mapping | UX vision, service design, behavioral insights |
| **Research & Validation** | User interviews, usability tests | Research planning, data synthesis | Research strategy, methodology design | Research culture, insight frameworks, market research |
| **Design Systems** | Component usage, style guides | System contribution, pattern creation | System architecture, governance | System strategy, scalable design, platform thinking |
| **Tools & Craft** | Design software proficiency, asset creation | Advanced techniques, workflow optimization | Tool evaluation, process design | Technology integration, future tooling, craft evolution |
### Collaboration Competencies
| Competency | Junior Designer (L1-L2) | Mid Designer (L3-L4) | Senior Designer (L5-L6) | Principal Designer (L7+) |
|------------|-------------------------|---------------------|-------------------------|-------------------------|
| **Cross-functional Partnership** | Engineering collaboration, handoff quality | Product partnership, stakeholder alignment | Leadership collaboration, strategic alignment | Executive partnership, business strategy integration |
| **Communication & Advocacy** | Design rationale, feedback integration | Design presentations, user advocacy | Executive communication, design thinking evangelism | Industry thought leadership, external representation |
| **Mentorship & Growth** | Peer learning, skill sharing | Junior mentoring, critique facilitation | Team development, hiring, career guidance | Design culture, talent strategy, industry leadership |
| **Business Impact** | User-centered thinking, design quality | Feature success, user satisfaction | Business metrics, strategic impact | Market influence, competitive advantage, innovation leadership |
## Data Science Competency Matrix
### Technical Competencies
| Competency | Junior DS (L1-L2) | Mid DS (L3-L4) | Senior DS (L5-L6) | Principal DS (L7+) |
|------------|-------------------|----------------|-------------------|-------------------|
| **Statistical Analysis** | Descriptive stats, hypothesis testing | Advanced statistics, experimental design | Causal inference, advanced modeling | Statistical strategy, methodology innovation |
| **Machine Learning** | Basic ML algorithms, model training | Advanced ML, feature engineering | ML systems, model deployment | ML strategy, AI platform, research direction |
| **Data Engineering** | SQL, basic ETL, data cleaning | Pipeline design, data modeling | Platform architecture, scalable systems | Data strategy, infrastructure vision, governance |
| **Programming & Tools** | Python/R proficiency, visualization | Advanced programming, tool integration | Software engineering, system design | Technology strategy, platform development, innovation |
| **Domain Expertise** | Business understanding, metric interpretation | Domain modeling, insight generation | Strategic analysis, business integration | Market expertise, competitive intelligence, thought leadership |
### Impact & Leadership Competencies
| Competency | Junior DS (L1-L2) | Mid DS (L3-L4) | Senior DS (L5-L6) | Principal DS (L7+) |
|------------|-------------------|----------------|-------------------|-------------------|
| **Business Impact** | Metric improvement, insight delivery | Project leadership, business case development | Strategic initiatives, P&L impact | Business transformation, market advantage, innovation |
| **Communication** | Technical reporting, visualization | Stakeholder presentations, executive briefings | Board communication, external representation | Industry leadership, thought leadership, market influence |
| **Team Leadership** | Peer collaboration, knowledge sharing | Junior mentoring, project management | Team building, hiring, culture development | Organizational leadership, talent strategy, vision setting |
| **Innovation & Research** | Algorithm implementation, experimentation | Research projects, publication | Research strategy, academic partnerships | Research vision, industry influence, breakthrough innovation |
## DevOps Engineering Competency Matrix
### Technical Competencies
| Competency | Junior DevOps (L1-L2) | Mid DevOps (L3-L4) | Senior DevOps (L5-L6) | Principal DevOps (L7+) |
|------------|----------------------|-------------------|----------------------|----------------------|
| **Infrastructure** | Basic cloud services, server management | Infrastructure automation, containerization | Platform architecture, multi-cloud strategy | Infrastructure vision, emerging technologies, industry standards |
| **CI/CD & Automation** | Pipeline basics, script writing | Advanced pipelines, deployment automation | Platform design, workflow optimization | Automation strategy, developer experience, productivity platforms |
| **Monitoring & Observability** | Basic monitoring, log analysis | Advanced monitoring, alerting systems | Observability strategy, SLA/SLI design | Monitoring vision, reliability engineering, performance culture |
| **Security & Compliance** | Security basics, access management | Security automation, compliance frameworks | Security architecture, risk management | Security strategy, governance, industry leadership |
| **Performance & Scalability** | Performance monitoring, basic optimization | Capacity planning, performance tuning | Scalability architecture, cost optimization | Performance strategy, efficiency platforms, innovation |
### Leadership & Impact Competencies
| Competency | Junior DevOps (L1-L2) | Mid DevOps (L3-L4) | Senior DevOps (L5-L6) | Principal DevOps (L7+) |
|------------|----------------------|-------------------|----------------------|----------------------|
| **Developer Experience** | Tool support, documentation | Platform development, self-service tools | Developer productivity, workflow design | Developer platform vision, industry best practices |
| **Incident Management** | Incident response, troubleshooting | Incident coordination, root cause analysis | Incident strategy, prevention systems | Reliability culture, organizational resilience |
| **Team Collaboration** | Cross-team support, knowledge sharing | Process improvement, training delivery | Culture building, practice evangelism | Organizational transformation, industry influence |
| **Strategic Impact** | Operational excellence, cost awareness | Efficiency improvements, platform adoption | Strategic initiatives, business enablement | Technology strategy, competitive advantage, market leadership |
## Engineering Management Competency Matrix
### People Leadership Competencies
| Competency | Manager (L1-L2) | Senior Manager (L3-L4) | Director (L5-L6) | VP+ (L7+) |
|------------|-----------------|------------------------|------------------|----------|
| **Team Building** | Hiring, onboarding, 1:1s | Team culture, performance management | Multi-team coordination, org design | Organizational culture, talent strategy |
| **Performance Management** | Individual development, feedback | Performance systems, coaching | Calibration across teams, promotion standards | Talent development, succession planning |
| **Communication** | Team updates, stakeholder management | Executive communication, cross-functional alignment | Board updates, external communication | Industry representation, thought leadership |
| **Conflict Resolution** | Team conflicts, process improvements | Cross-team issues, organizational friction | Strategic alignment, cultural challenges | Corporate-level conflicts, crisis management |
### Technical Leadership Competencies
| Competency | Manager (L1-L2) | Senior Manager (L3-L4) | Director (L5-L6) | VP+ (L7+) |
|------------|-----------------|------------------------|------------------|----------|
| **Technical Vision** | Team technical decisions, architecture input | Platform strategy, technology choices | Technical roadmap, innovation strategy | Technology vision, industry standards |
| **System Ownership** | Feature/service ownership, quality standards | Platform ownership, scalability planning | System portfolio, technical debt management | Technology strategy, competitive advantage |
| **Process & Practice** | Team processes, development practices | Engineering standards, quality systems | Process innovation, best practices | Engineering culture, industry influence |
| **Technology Strategy** | Tool evaluation, team technology choices | Platform decisions, technical investments | Technology portfolio, strategic architecture | Corporate technology strategy, market leadership |
## Usage Guidelines
### Assessment Approach
1. **Level Calibration**: Use these matrices to calibrate expectations for each level within your organization
2. **Interview Design**: Select competencies most relevant to the specific role and level being hired for
3. **Evaluation Consistency**: Ensure all interviewers understand and apply the same competency standards
4. **Growth Planning**: Use matrices for career development and promotion discussions
### Customization Tips
1. **Industry Adaptation**: Modify competencies based on your industry (fintech, healthcare, etc.)
2. **Company Stage**: Adjust expectations based on startup vs. enterprise environment
3. **Team Needs**: Emphasize competencies most critical for current team challenges
4. **Cultural Fit**: Add company-specific values and cultural competencies
### Common Pitfalls
1. **Unrealistic Expectations**: Don't expect senior-level competencies from junior candidates
2. **One-Size-Fits-All**: Customize competency emphasis based on role requirements
3. **Static Assessment**: Regularly update matrices based on changing business needs
4. **Bias Introduction**: Ensure competencies are measurable and don't introduce unconscious bias
## Matrix Validation Process
### Regular Review Cycle
- **Quarterly**: Review competency relevance and adjust weights
- **Semi-annually**: Update level expectations based on market standards
- **Annually**: Comprehensive review with stakeholder feedback
### Stakeholder Input
- **Hiring Managers**: Validate role-specific competency requirements
- **Current Team Members**: Confirm level expectations match reality
- **Recent Hires**: Gather feedback on assessment accuracy
- **HR Partners**: Ensure legal compliance and bias mitigation
### Continuous Improvement
- **Performance Correlation**: Track new hire performance against competency assessments
- **Market Benchmarking**: Compare standards with industry peers
- **Feedback Integration**: Incorporate interviewer and candidate feedback
- **Bias Monitoring**: Regular analysis of assessment patterns across demographics
FILE:references/debrief_facilitation_guide.md
# Interview Debrief Facilitation Guide
This guide provides a comprehensive framework for conducting effective, unbiased interview debriefs that lead to consistent hiring decisions. Use this to facilitate productive discussions that focus on evidence-based evaluation.
## Pre-Debrief Preparation
### Facilitator Responsibilities
- [ ] **Review all interviewer feedback** before the meeting
- [ ] **Identify significant score discrepancies** that need discussion
- [ ] **Prepare discussion agenda** with time allocations
- [ ] **Gather role requirements** and competency framework
- [ ] **Review any flags or special considerations** noted during interviews
- [ ] **Ensure all required materials** are available (scorecards, rubrics, candidate resume)
- [ ] **Set up meeting logistics** (room, video conference, screen sharing)
- [ ] **Send agenda to participants** 30 minutes before meeting
### Required Materials Checklist
- [ ] Candidate resume and application materials
- [ ] Job description and competency requirements
- [ ] Individual interviewer scorecards
- [ ] Scoring rubrics and competency definitions
- [ ] Interview notes and documentation
- [ ] Any technical assessments or work samples
- [ ] Company hiring standards and calibration examples
- [ ] Bias mitigation reminders and prompts
### Participant Preparation Requirements
- [ ] All interviewers must **complete independent scoring** before debrief
- [ ] **Submit written feedback** with specific evidence for each competency
- [ ] **Review scoring rubrics** to ensure consistent interpretation
- [ ] **Prepare specific examples** to support scoring decisions
- [ ] **Flag any concerns or unusual circumstances** that affected assessment
- [ ] **Avoid discussing candidate** with other interviewers before debrief
- [ ] **Come prepared to defend scores** with concrete evidence
- [ ] **Be ready to adjust scores** based on additional evidence shared
## Debrief Meeting Structure
### Opening (5 minutes)
1. **State meeting purpose**: Make hiring decision based on evidence
2. **Review agenda and time limits**: Keep discussion focused and productive
3. **Remind of bias mitigation principles**: Focus on competencies, not personality
4. **Confirm confidentiality**: Discussion stays within hiring team
5. **Establish ground rules**: One person speaks at a time, evidence-based discussion
### Individual Score Sharing (10-15 minutes)
- **Go around the room systematically** - each interviewer shares scores independently
- **No discussion or challenges yet** - just data collection
- **Record scores on shared document** visible to all participants
- **Note any abstentions** or "insufficient data" responses
- **Identify clear patterns** and discrepancies without commentary
- **Flag any scores requiring explanation** (1s or 4s typically need strong evidence)
### Competency-by-Competency Discussion (30-40 minutes)
#### For Each Core Competency:
**1. Present Score Distribution (2 minutes)**
- Display all scores for this competency
- Note range and any outliers
- Identify if consensus exists or discussion needed
**2. Evidence Sharing (5-8 minutes per competency)**
- Start with interviewers who assessed this competency directly
- Share specific examples and observations
- Focus on what candidate said/did, not interpretations
- Allow questions for clarification (not challenges yet)
**3. Discussion and Calibration (3-5 minutes)**
- Address significant discrepancies (>1 point difference)
- Challenge vague or potentially biased language
- Seek additional evidence if needed
- Allow score adjustments based on new information
- Reach consensus or note dissenting views
#### Structured Discussion Questions:
- **"What specific evidence supports this score?"**
- **"Can you provide the exact example or quote?"**
- **"How does this compare to our rubric definition?"**
- **"Would this response receive the same score regardless of who gave it?"**
- **"Are we evaluating the competency or making assumptions?"**
- **"What would need to change for this to be the next level up/down?"**
### Overall Recommendation Discussion (10-15 minutes)
#### Weighted Score Calculation
1. **Apply competency weights** based on role requirements
2. **Calculate overall weighted average**
3. **Check minimum threshold requirements**
4. **Consider any veto criteria** (critical competency failures)
#### Final Recommendation Options
- **Strong Hire**: Exceeds requirements in most areas, clear value-add
- **Hire**: Meets requirements with growth potential
- **No Hire**: Doesn't meet minimum requirements for success
- **Strong No Hire**: Significant gaps that would impact team/company
#### Decision Rationale Documentation
- **Summarize key strengths** with specific evidence
- **Identify development areas** with specific examples
- **Explain final recommendation** with competency-based reasoning
- **Note any dissenting opinions** and reasoning
- **Document onboarding considerations** if hiring
### Closing and Next Steps (5 minutes)
- **Confirm final decision** and documentation
- **Assign follow-up actions** (feedback delivery, offer preparation, etc.)
- **Schedule any additional interviews** if needed
- **Review timeline** for candidate communication
- **Remind confidentiality** of discussion and decision
## Facilitation Best Practices
### Creating Psychological Safety
- **Encourage honest feedback** without fear of judgment
- **Validate different perspectives** and assessment approaches
- **Address power dynamics** - ensure junior voices are heard
- **Model vulnerability** - admit when evidence changes your mind
- **Focus on learning** and calibration, not winning arguments
- **Thank participants** for thorough preparation and thoughtful input
### Managing Difficult Conversations
#### When Scores Vary Significantly
1. **Acknowledge the discrepancy** without judgment
2. **Ask for specific evidence** from each scorer
3. **Look for different interpretations** of the same data
4. **Consider if different questions** revealed different competency levels
5. **Check for bias patterns** in reasoning
6. **Allow time for reflection** and potential score adjustments
#### When Someone Uses Biased Language
1. **Pause the conversation** gently but firmly
2. **Ask for specific evidence** behind the assessment
3. **Reframe in competency terms** - "What specific skills did this demonstrate?"
4. **Challenge assumptions** - "Help me understand how we know that"
5. **Redirect to rubric** - "How does this align with our scoring criteria?"
6. **Document and follow up** privately if bias persists
#### When the Discussion Gets Off Track
- **Redirect to competencies**: "Let's focus on the technical skills demonstrated"
- **Ask for evidence**: "What specific example supports that assessment?"
- **Reference rubrics**: "How does this align with our level 3 definition?"
- **Manage time**: "We have 5 minutes left on this competency"
- **Table unrelated issues**: "That's important but separate from this hire decision"
### Encouraging Evidence-Based Discussion
#### Good Evidence Examples
- **Direct quotes**: "When asked about debugging, they said..."
- **Specific behaviors**: "They organized their approach by first..."
- **Observable outcomes**: "Their code compiled on first run and handled edge cases"
- **Process descriptions**: "They walked through their problem-solving step by step"
- **Measurable results**: "They identified 3 optimization opportunities"
#### Poor Evidence Examples
- **Gut feelings**: "They just seemed off"
- **Comparisons**: "Not as strong as our last hire"
- **Assumptions**: "Probably wouldn't fit our culture"
- **Vague impressions**: "Didn't seem passionate"
- **Irrelevant factors**: "Their background is different from ours"
### Managing Group Dynamics
#### Ensuring Equal Participation
- **Direct questions** to quieter participants
- **Prevent interrupting** and ensure everyone finishes thoughts
- **Balance speaking time** across all interviewers
- **Validate minority opinions** even if not adopted
- **Check for unheard perspectives** before finalizing decisions
#### Handling Strong Personalities
- **Set time limits** for individual speaking
- **Redirect monopolizers**: "Let's hear from others on this"
- **Challenge confidently stated opinions** that lack evidence
- **Support less assertive voices** in expressing dissenting views
- **Focus on data**, not personality or seniority in decision making
## Bias Interruption Strategies
### Affinity Bias Interruption
- **Notice pattern**: Positive assessment seems based on shared background/interests
- **Interrupt with**: "Let's focus on the job-relevant skills they demonstrated"
- **Redirect to**: Specific competency evidence and measurable outcomes
- **Document**: Note if personal connection affected professional assessment
### Halo/Horn Effect Interruption
- **Notice pattern**: One area strongly influencing assessment of unrelated areas
- **Interrupt with**: "Let's score each competency independently"
- **Redirect to**: Specific evidence for each individual competency area
- **Recalibrate**: Ask for separate examples supporting each score
### Confirmation Bias Interruption
- **Notice pattern**: Only seeking/discussing evidence that supports initial impression
- **Interrupt with**: "What evidence might suggest a different assessment?"
- **Redirect to**: Consider alternative interpretations of the same data
- **Challenge**: "How might we be wrong about this assessment?"
### Attribution Bias Interruption
- **Notice pattern**: Attributing success to luck/help for some demographics, skill for others
- **Interrupt with**: "What role did the candidate play in achieving this outcome?"
- **Redirect to**: Candidate's specific contributions and decision-making
- **Standardize**: Apply same attribution standards across all candidates
## Decision Documentation Framework
### Required Documentation Elements
1. **Final scores** for each assessed competency
2. **Overall recommendation** with supporting rationale
3. **Key strengths** with specific evidence
4. **Development areas** with specific examples
5. **Dissenting opinions** if any, with reasoning
6. **Special considerations** or accommodation needs
7. **Next steps** and timeline for decision communication
### Evidence Quality Standards
- **Specific and observable**: What exactly did the candidate do or say?
- **Job-relevant**: How does this relate to success in the role?
- **Measurable**: Can this be quantified or clearly described?
- **Unbiased**: Would this evidence be interpreted the same way regardless of candidate demographics?
- **Complete**: Does this represent the full picture of their performance in this area?
### Writing Guidelines
- **Use active voice** and specific language
- **Avoid assumptions** about motivations or personality
- **Focus on behaviors** demonstrated during the interview
- **Provide context** for any unusual circumstances
- **Be constructive** in describing development areas
- **Maintain professionalism** and respect for candidate
## Common Debrief Challenges and Solutions
### Challenge: "I just don't think they'd fit our culture"
**Solution**:
- Ask for specific, observable evidence
- Define what "culture fit" means in job-relevant terms
- Challenge assumptions about cultural requirements
- Focus on ability to collaborate and contribute effectively
### Challenge: Scores vary widely with no clear explanation
**Solution**:
- Review if different interviewers assessed different competencies
- Look for question differences that might explain variance
- Consider if candidate performance varied across interviews
- May need additional data gathering or interview
### Challenge: Everyone loved/hated the candidate but can't articulate why
**Solution**:
- Push for specific evidence supporting emotional reactions
- Review competency rubrics together
- Look for halo/horn effects influencing overall impression
- Consider unconscious bias training for team
### Challenge: Technical vs. non-technical interviewers disagree
**Solution**:
- Clarify which competencies each interviewer was assessing
- Ensure technical assessments carry appropriate weight
- Look for different perspectives on same evidence
- Consider specialist input for technical decisions
### Challenge: Senior interviewer dominates decision making
**Solution**:
- Structure discussion to hear from all levels first
- Ask direct questions to junior interviewers
- Challenge opinions that lack supporting evidence
- Remember that assessment ability doesn't correlate with seniority
### Challenge: Team wants to hire but scores don't support it
**Solution**:
- Review if rubrics match actual job requirements
- Check for consistent application of scoring standards
- Consider if additional competencies need assessment
- May indicate need for rubric calibration or role requirement review
## Post-Debrief Actions
### Immediate Actions (Same Day)
- [ ] **Finalize decision documentation** with all evidence
- [ ] **Communicate decision** to recruiting team
- [ ] **Schedule candidate feedback** delivery if applicable
- [ ] **Update interview scheduling** based on decision
- [ ] **Note any process improvements** needed for future
### Follow-up Actions (Within 1 Week)
- [ ] **Deliver candidate feedback** (internal or external)
- [ ] **Update interview feedback** in tracking system
- [ ] **Schedule any additional interviews** if needed
- [ ] **Begin offer process** if hiring
- [ ] **Document lessons learned** for process improvement
### Long-term Actions (Monthly/Quarterly)
- [ ] **Analyze debrief effectiveness** and decision quality
- [ ] **Review interviewer calibration** based on decisions
- [ ] **Update rubrics** based on debrief insights
- [ ] **Provide additional training** if bias patterns identified
- [ ] **Share successful practices** with other hiring teams
## Continuous Improvement Framework
### Debrief Effectiveness Metrics
- **Decision consistency**: Are similar candidates receiving similar decisions?
- **Time to decision**: Are debriefs completing within planned time?
- **Participation quality**: Are all interviewers contributing evidence-based input?
- **Bias incidents**: How often are bias interruptions needed?
- **Decision satisfaction**: Do participants feel good about the process and outcome?
### Regular Review Process
- **Monthly**: Review debrief facilitation effectiveness and interviewer feedback
- **Quarterly**: Analyze decision patterns and potential bias indicators
- **Semi-annually**: Update debrief processes based on hiring outcome data
- **Annually**: Comprehensive review of debrief framework and training needs
### Training and Calibration
- **New facilitators**: Shadow 3-5 debriefs before leading independently
- **All facilitators**: Quarterly calibration sessions on bias interruption
- **Interviewer training**: Include debrief participation expectations
- **Leadership training**: Ensure hiring managers can facilitate effectively
This guide should be adapted to your organization's specific needs while maintaining focus on evidence-based, unbiased decision making.
FILE:references/interview-frameworks.md
# Interview Frameworks
## Loop Design by Level
### Junior/Mid
- Emphasize fundamentals, debugging, and growth potential.
- Keep loops concise with coding + behavioral validation.
### Senior
- Add system design and leadership rounds.
- Evaluate tradeoff quality, mentoring, and cross-team collaboration.
### Staff+
- Focus on architecture direction and organizational impact.
- Assess strategy, influence, and long-term technical judgment.
## Competency Areas
- Technical depth (implementation, design, quality)
- Problem solving (ambiguity handling, prioritization)
- Collaboration (communication, stakeholder alignment)
- Leadership (ownership, mentoring, influence)
## Scoring Rubric Baseline
- `4`: exceeds level expectations with strong evidence
- `3`: meets expectations consistently
- `2`: partial signal with notable gaps
- `1`: does not meet baseline requirements
## Calibration Guidelines
- Run recurring interviewer calibration sessions.
- Compare interviewer scoring variance across rounds.
- Track interview signal against new-hire outcomes.
- Use structured debriefs with independent scoring before discussion.
## Bias-Reduction Baseline
- Standardize question banks per competency area.
- Keep scorecards evidence-based and behavior-specific.
- Use diverse interviewer panels where possible.
- Require written rationale for strong yes/no recommendations.
FILE:scripts/interview_planner.py
#!/usr/bin/env python3
"""Generate an interview loop plan by role and level."""
from __future__ import annotations
import argparse
import json
from typing import Dict, List
BASE_ROUNDS = {
"junior": [
("Screen", 45, "Fundamentals and communication"),
("Coding", 60, "Problem solving and code quality"),
("Behavioral", 45, "Collaboration and growth mindset"),
],
"mid": [
("Screen", 45, "Fundamentals and ownership"),
("Coding", 60, "Implementation quality"),
("System Design", 60, "Service/component design"),
("Behavioral", 45, "Stakeholder collaboration"),
],
"senior": [
("Screen", 45, "Depth and tradeoff reasoning"),
("Coding", 60, "Code quality and testing"),
("System Design", 75, "Scalability and reliability"),
("Leadership", 60, "Mentoring and decision making"),
("Behavioral", 45, "Cross-functional influence"),
],
"staff": [
("Screen", 45, "Strategic and technical depth"),
("Architecture", 90, "Org-level design decisions"),
("Technical Strategy", 60, "Long-term tradeoffs"),
("Influence", 60, "Cross-team leadership"),
("Behavioral", 45, "Values and executive communication"),
],
}
QUESTION_BANK = {
"coding": [
"Walk through your approach before coding and identify tradeoffs.",
"How would you test this implementation for edge cases?",
"What would you refactor if this code became a shared library?",
],
"system": [
"Design this system for 10x traffic growth in 12 months.",
"Where are the main failure modes and how would you detect them?",
"What components would you scale first and why?",
],
"leadership": [
"Describe a time you changed technical direction with incomplete information.",
"How do you raise the bar for code quality across a team?",
"How do you handle disagreement between product and engineering priorities?",
],
"behavioral": [
"Tell me about a high-stakes mistake and what changed afterward.",
"Describe a conflict where you had to influence without authority.",
"How do you support underperforming teammates?",
],
}
def normalize_level(level: str) -> str:
level = level.strip().lower()
if level in {"staff+", "principal", "lead"}:
return "staff"
if level not in BASE_ROUNDS:
raise ValueError(f"Unsupported level: {level}")
return level
def suggested_questions(round_name: str) -> List[str]:
name = round_name.lower()
if "coding" in name:
return QUESTION_BANK["coding"]
if "system" in name or "architecture" in name:
return QUESTION_BANK["system"]
if "lead" in name or "influence" in name or "strategy" in name:
return QUESTION_BANK["leadership"]
return QUESTION_BANK["behavioral"]
def generate_plan(role: str, level: str) -> Dict[str, object]:
normalized = normalize_level(level)
rounds = []
for idx, (name, minutes, focus) in enumerate(BASE_ROUNDS[normalized], start=1):
rounds.append(
{
"round": idx,
"name": name,
"duration_minutes": minutes,
"focus": focus,
"suggested_questions": suggested_questions(name),
}
)
return {
"role": role,
"level": normalized,
"total_rounds": len(rounds),
"total_minutes": sum(r["duration_minutes"] for r in rounds),
"rounds": rounds,
}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Generate an interview loop plan for a role and level.")
parser.add_argument("--role", required=True, help="Role name (e.g., Senior Software Engineer)")
parser.add_argument("--level", required=True, help="Level: junior|mid|senior|staff")
parser.add_argument("--json", action="store_true", help="Output as JSON")
return parser.parse_args()
def main() -> int:
args = parse_args()
plan = generate_plan(args.role, args.level)
if args.json:
print(json.dumps(plan, indent=2))
else:
print(f"Interview Plan: {plan['role']} ({plan['level']})")
print(f"Total rounds: {plan['total_rounds']} | Total time: {plan['total_minutes']} minutes")
print("")
for r in plan["rounds"]:
print(f"Round {r['round']}: {r['name']} ({r['duration_minutes']} min)")
print(f"Focus: {r['focus']}")
for q in r["suggested_questions"]:
print(f"- {q}")
print("")
return 0
if __name__ == "__main__":
raise SystemExit(main())
Kiểm chứng cơ hội sản phẩm, lập bản đồ giả định, lên kế hoạch discovery sprint và thử độ khớp vấn đề-giải pháp trước khi đầu tư phát triển.
---
name: product-discovery
description: Use when validating product opportunities, mapping assumptions, planning discovery sprints, or testing problem-solution fit before committing delivery resources.
---
# Product Discovery
Run structured discovery to identify high-value opportunities and de-risk product bets.
## When To Use
Use this skill for:
- Opportunity Solution Tree facilitation
- Assumption mapping and test planning
- Problem validation interviews and evidence synthesis
- Solution validation with prototypes/experiments
- Discovery sprint planning and outputs
## Core Discovery Workflow
1. Define desired outcome
- Set one measurable outcome to improve.
- Establish baseline and target horizon.
2. Build Opportunity Solution Tree (OST)
- Outcome -> opportunities -> solution ideas -> experiments
- Keep opportunities grounded in user evidence, not internal opinions.
3. Map assumptions
- Identify desirability, viability, feasibility, and usability assumptions.
- Score assumptions by risk and certainty.
Use:
```bash
python3 scripts/assumption_mapper.py assumptions.csv
```
4. Validate the problem
- Conduct interviews and behavior analysis.
- Confirm frequency, severity, and willingness to solve.
- Reject weak opportunities early.
5. Validate the solution
- Prototype before building.
- Run concept, usability, and value tests.
- Measure behavior, not only stated preference.
6. Plan discovery sprint
- 1-2 week cycle with explicit hypotheses
- Daily evidence reviews
- End with decision: proceed, pivot, or stop
## Opportunity Solution Tree (Teresa Torres)
Structure:
- Outcome: metric you want to move
- Opportunities: unmet customer needs/pains
- Solutions: candidate interventions
- Experiments: fastest learning actions
Quality checks:
- At least 3 distinct opportunities before converging.
- At least 2 experiments per top opportunity.
- Tie every branch to evidence source.
## Assumption Mapping
Assumption categories:
- Desirability: users want this
- Viability: business value exists
- Feasibility: team can build/operate it
- Usability: users can successfully use it
Prioritization rule:
- High risk + low certainty assumptions are tested first.
## Problem Validation Techniques
- Problem interviews focused on current behavior
- Journey friction mapping
- Support ticket and sales-call synthesis
- Behavioral analytics triangulation
Evidence threshold examples:
- Same pain repeated across multiple target users
- Observable workaround behavior
- Measurable cost of current pain
## Solution Validation Techniques
- Concept tests (value proposition comprehension)
- Prototype usability tests (task success/time-to-complete)
- Fake door or concierge tests (demand signal)
- Limited beta cohorts (retention/activation signals)
## Discovery Sprint Planning
Suggested 10-day structure:
- Day 1-2: Outcome + opportunity framing
- Day 3-4: Assumption mapping + test design
- Day 5-7: Problem and solution tests
- Day 8-9: Evidence synthesis + decision options
- Day 10: Stakeholder decision review
## Tooling
### `scripts/assumption_mapper.py`
CLI utility that:
- reads assumptions from CSV or inline input
- scores risk/certainty priority
- emits prioritized test plan with suggested test types
See `references/discovery-frameworks.md` for framework details.
FILE:references/discovery-frameworks.md
# Discovery Frameworks
## Opportunity Solution Tree (OST)
Purpose: continuously connect product outcomes to validated opportunities and tested solutions.
Core structure:
- Outcome (metric)
- Opportunity nodes (needs/pains)
- Solution ideas
- Experiments
OST practice tips:
- Keep tree live; update after each interview or test.
- Separate opportunity evidence from solution proposals.
- Avoid single-branch trees that force one solution.
## Jobs-to-be-Done (JTBD)
Use JTBD to understand progress users seek.
JTBD template:
"When [situation], I want to [motivation], so I can [expected outcome]."
JTBD interview focus:
- Trigger moments
- Current alternatives and workarounds
- Purchase/adoption anxieties
- Desired progress and success criteria
## Kano Model
Classify features by impact on satisfaction:
- Must-be: expected baseline features
- Performance: more is better
- Delighters: unexpected value multipliers
- Indifferent: low impact
- Reverse: can reduce satisfaction for some users
Use Kano when prioritizing solution concepts after problem validation.
## Design Sprint Methodology
Typical phases:
1. Understand
2. Sketch
3. Decide
4. Prototype
5. Test
Discovery usage:
- Compress learning cycle into one week.
- Best for high-ambiguity opportunities requiring cross-functional alignment.
## Assumption Prioritization Matrix
Map assumptions on two axes:
- Risk if wrong (low -> high)
- Certainty (low -> high)
Priority order:
1. High risk, low certainty (test first)
2. High risk, high certainty (validate quickly)
3. Low risk, low certainty (defer)
4. Low risk, high certainty (document)
## Discovery Evidence Rules
- One source is not enough for major decisions.
- Triangulate qualitative and quantitative signals.
- Predefine decision criteria before test execution.
- Archive evidence with date, segment, and method.
FILE:scripts/assumption_mapper.py
#!/usr/bin/env python3
"""Prioritize product assumptions and suggest validation tests."""
import argparse
import csv
from dataclasses import dataclass
@dataclass
class Assumption:
statement: str
category: str
risk: float
certainty: float
@property
def priority_score(self) -> float:
# High-risk, low-certainty assumptions should be tested first.
return self.risk * (1.0 - self.certainty)
def parse_float(value: str, field: str) -> float:
number = float(value)
if number < 0 or number > 1:
raise ValueError(f"{field} must be in [0, 1]")
return number
def suggest_test(category: str) -> str:
category = category.lower().strip()
if category == "desirability":
return "problem interviews or fake-door test"
if category == "viability":
return "pricing/willingness-to-pay test"
if category == "feasibility":
return "technical spike or architecture prototype"
if category == "usability":
return "moderated usability test"
return "smallest possible experiment with clear success criteria"
def load_from_csv(path: str) -> list[Assumption]:
assumptions: list[Assumption] = []
with open(path, "r", encoding="utf-8", newline="") as handle:
reader = csv.DictReader(handle)
required = {"assumption", "category", "risk", "certainty"}
missing = required - set(reader.fieldnames or [])
if missing:
missing_str = ", ".join(sorted(missing))
raise ValueError(f"Missing required columns: {missing_str}")
for row in reader:
assumptions.append(
Assumption(
statement=(row.get("assumption") or "").strip(),
category=(row.get("category") or "").strip(),
risk=parse_float(row.get("risk") or "0", "risk"),
certainty=parse_float(row.get("certainty") or "0", "certainty"),
)
)
return assumptions
def parse_inline(items: list[str]) -> list[Assumption]:
assumptions: list[Assumption] = []
for item in items:
# format: statement|category|risk|certainty
parts = [part.strip() for part in item.split("|")]
if len(parts) != 4:
raise ValueError("Inline assumption must be: statement|category|risk|certainty")
assumptions.append(
Assumption(
statement=parts[0],
category=parts[1],
risk=parse_float(parts[2], "risk"),
certainty=parse_float(parts[3], "certainty"),
)
)
return assumptions
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description="Prioritize assumptions and generate test plan.")
parser.add_argument("input", nargs="?", help="CSV file path")
parser.add_argument(
"--assumption",
action="append",
default=[],
help="Inline assumption: statement|category|risk|certainty",
)
parser.add_argument("--top", type=int, default=10, help="Maximum assumptions to print")
return parser
def main() -> int:
parser = build_parser()
args = parser.parse_args()
assumptions: list[Assumption] = []
if args.input:
assumptions.extend(load_from_csv(args.input))
if args.assumption:
assumptions.extend(parse_inline(args.assumption))
if not assumptions:
parser.error("Provide a CSV input file or at least one --assumption value.")
assumptions.sort(key=lambda item: item.priority_score, reverse=True)
print("prioritized_assumption_test_plan")
print("rank,priority_score,category,risk,certainty,test,assumption")
for rank, item in enumerate(assumptions[: args.top], start=1):
test = suggest_test(item.category)
print(
f"{rank},{item.priority_score:.4f},{item.category},{item.risk:.2f},"
f"{item.certainty:.2f},{test},{item.statement}"
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
Chủ động lưu tri thức quan trọng vào bộ nhớ tự động kèm thời gian và ngữ cảnh, khi phát hiện quá quan trọng để dựa vào tự động ghi nhận.
---
name: "remember"
description: "Explicitly save important knowledge to auto-memory with timestamp and context. Use when a discovery is too important to rely on auto-capture."
---
# /si:remember — Save Knowledge Explicitly
Writes an explicit entry to auto-memory when something is important enough that you don't want to rely on Claude noticing it automatically.
## Usage
```
/si:remember <what to remember>
/si:remember "This project's CI requires Node 20 LTS — v22 breaks the build"
/si:remember "The /api/auth endpoint uses a custom JWT library, not passport"
/si:remember "Reza prefers explicit error handling over try-catch-all patterns"
```
## When to Use
| Situation | Example |
|-----------|---------|
| Hard-won debugging insight | "CORS errors on /api/upload are caused by the CDN, not the backend" |
| Project convention not in CLAUDE.md | "We use barrel exports in src/components/" |
| Tool-specific gotcha | "Jest needs `--forceExit` flag or it hangs on DB tests" |
| Architecture decision | "We chose Drizzle over Prisma for type-safe SQL" |
| Preference you want Claude to learn | "Don't add comments explaining obvious code" |
## Workflow
### Step 1: Parse the knowledge
Extract from the user's input:
- **What**: The concrete fact or pattern
- **Why it matters**: Context (if provided)
- **Scope**: Project-specific or global?
### Step 2: Check for duplicates
```bash
MEMORY_DIR="$HOME/.claude/projects/$(pwd | sed 's|/|%2F|g; s|%2F|/|; s|^/||')/memory"
grep -ni "<keywords>" "$MEMORY_DIR/MEMORY.md" 2>/dev/null
```
If a similar entry exists:
- Show it to the user
- Ask: "Update the existing entry or add a new one?"
### Step 3: Write to MEMORY.md
Append to the end of `MEMORY.md`:
```markdown
- {{concise fact or pattern}}
```
Keep entries concise — one line when possible. Auto-memory entries don't need timestamps, IDs, or metadata. They're notes, not database records.
If MEMORY.md is over 180 lines, warn the user:
```
⚠️ MEMORY.md is at {{n}}/200 lines. Consider running /si:review to free space.
```
### Step 4: Suggest promotion
If the knowledge sounds like a rule (imperative, always/never, convention):
```
💡 This sounds like it could be a CLAUDE.md rule rather than a memory entry.
Rules are enforced with higher priority. Want to /si:promote it instead?
```
### Step 5: Confirm
```
✅ Saved to auto-memory
"{{entry}}"
MEMORY.md: {{n}}/200 lines
Claude will see this at the start of every session in this project.
```
## What NOT to use /si:remember for
- **Temporary context**: Use session memory or just tell Claude in conversation
- **Enforced rules**: Use `/si:promote` to write directly to CLAUDE.md
- **Cross-project knowledge**: Use `~/.claude/CLAUDE.md` for global rules
- **Sensitive data**: Never store credentials, tokens, or secrets in memory files
## Tips
- Be concise — one line beats a paragraph
- Include the concrete command or value, not just the concept
- ✅ "Build with `pnpm build`, tests with `pnpm test:e2e`"
- ❌ "The project uses pnpm for building and testing"
- If you're remembering the same thing twice, promote it to CLAUDE.md
Sub-agent đọc nguồn mới, đề xuất tóm tắt và ý chính, xác định trang bị ảnh hưởng, cảnh báo mâu thuẫn rồi ghi vào wiki sau khi xác nhận.
--- name: cs-wiki-ingestor description: Dispatched sub-agent that ingests a new source into an LLM Wiki vault. Reads the source, proposes TL;DR and key claims, identifies which entity/concept/synthesis pages will be touched, flags contradictions with existing pages, and — after user confirmation — writes the source summary, updates cross-references across 5-15 pages, regenerates the index, and appends a standardized log entry. Spawn when the user says "ingest this", "add this paper/article/book to the wiki", or drops a file into raw/. skills: engineering/llm-wiki domain: engineering model: opus tools: [Read, Write, Edit, Bash, Grep, Glob] context: fork --- # wiki-ingestor ## Role You are a disciplined wiki maintainer. A user has dropped a new source into the `raw/` layer of an LLM Wiki vault and asked you to ingest it. Your job is to read it, discuss it with the user, and integrate it into the `wiki/` layer — touching every relevant entity, concept, and synthesis page, flagging contradictions, updating the index, and appending to the log. You are spawned **per-ingest**, not as a long-running agent. You do one source at a time. ## Inputs - Path to a source file (must be inside the vault's `raw/` layer) - The current state of `wiki/` (especially `index.md`) - The vault's `CLAUDE.md` or `AGENTS.md` schema ## Workflow Follow `references/ingest-workflow.md` in the llm-wiki skill. Summary: ### 1. Prep Run `python <plugin>/scripts/ingest_source.py --vault . --source <path> --json` to get the brief (title guess, word count, preview, suggested summary path, whether a summary already exists). ### 2. Read Use the Read tool on the source file directly. For PDFs, use Read's PDF support. For images, use vision. ### 3. Discuss (user in the loop) Before writing anything, report to the user: - Title, authors, date - 2-3 sentence TL;DR - Key claims (3-7 bullets) - **Which existing wiki pages you plan to touch** (bulleted wikilinks) - **Any contradictions** with existing pages - Whether this is a fresh ingest or a **merge** (summary page exists) **Wait for the user to confirm or redirect before writing.** ### 4. Write the source summary Create `wiki/sources/<slug>.md` using the source-summary template from the llm-wiki skill. Required frontmatter: `title`, `category: source`, `summary`, `source_path`, `ingested`, `updated`. If the page exists (merge mode), append a new `## Re-ingest <date>` section at the bottom. ### 5. Update every relevant page For each entity and concept mentioned in the source: - **If the page exists:** update "Key claims", "Appears in" / "Used in", increment `sources:`, set `updated:` to today - **If not:** create a stub page from the appropriate template with at least the minimum (title, summary, one key fact, link back to this source) A typical ingest touches **5-15 pages**. Don't skimp — the wiki's value comes from cross-references. ### 6. Flag contradictions If this source contradicts an existing page, add a `> ⚠️ Contradiction:` callout to **both** pages, linking the disagreeing sources. ### 7. Update synthesis pages If the source meaningfully shifts a `synthesis/` page's thesis, revise the "Thesis" paragraph and append a dated entry under "How this synthesis has changed". ### 8. Regenerate the index Run `python <plugin>/scripts/update_index.py --vault .` OR edit `wiki/index.md` inline for small changes. ### 9. Log the ingest Run `python <plugin>/scripts/append_log.py --vault . --op ingest --title "<title>" --detail "<touched pages summary>"`. ### 10. Report back Give the user a bulleted list of every touched page as wikilinks, plus any contradictions flagged. ## Rules - **`raw/` is immutable.** Never edit files there. Read only. - **Every write goes to `wiki/`.** - **Discuss before writing.** The user is in the loop. - **Minimum 5 file touches per ingest.** (source summary + 2-4 cross-references + index + log) - **Cite aggressively.** Every claim on an entity/concept page links to a source page. - **Flag contradictions** on both sides. - **Update `updated:` frontmatter** on every page you touch. ## Red flags Stop and ask the user before proceeding if: - The source is outside `raw/` - The source appears to duplicate an existing source exactly - Ingesting would require deleting existing wiki pages (only the user decides) - You detect >5 contradictions in one ingest (likely a paradigm-shifting source — worth a conversation)
Theo dõi đối thủ có hệ thống, phục vụ định vị, battlecard bán hàng và quyết định lộ trình sản phẩm.
---
name: "context-engine"
description: "Loads and manages company context for all C-suite advisor skills. Reads ~/.claude/company-context.md, detects stale context (>90 days), enriches context during conversations, and enforces privacy/anonymization rules before external API calls."
license: MIT
metadata:
version: 1.0.0
author: Alireza Rezvani
category: c-level
domain: orchestration
updated: 2026-03-05
frameworks: context-loading, anonymization, context-enrichment
---
# Company Context Engine
The memory layer for C-suite advisors. Every advisor skill loads this first. Context is what turns generic advice into specific insight.
## Keywords
company context, context loading, context engine, company profile, advisor context, stale context, context refresh, privacy, anonymization
---
## Load Protocol (Run at Start of Every C-Suite Session)
**Step 1 — Check for context file:** `~/.claude/company-context.md`
- Exists → proceed to Step 2
- Missing → prompt: *"Run /cs:setup to build your company context — it makes every advisor conversation significantly more useful."*
**Step 2 — Check staleness:** Read `Last updated` field.
- **< 90 days:** Load and proceed.
- **≥ 90 days:** Prompt: *"Your context is [N] days old. Quick 15-min refresh (/cs:update), or continue with what I have?"*
- If continue: load with `[STALE — last updated DATE]` noted internally.
**Step 3 — Parse into working memory.** Always active:
- Company stage (pre-PMF / scaling / optimizing)
- Founder archetype (product / sales / technical / operator)
- Current #1 challenge
- Runway (as risk signal — never share externally)
- Team size
- Unfair advantage
- 12-month target
---
## Context Quality Signals
| Condition | Confidence | Action |
|-----------|-----------|--------|
| < 30 days, full interview | High | Use directly |
| 30–90 days, update done | Medium | Use, flag what may have changed |
| > 90 days | Low | Flag stale, prompt refresh |
| Key fields missing | Low | Ask in-session |
| No file | None | Prompt /cs:setup |
If Low: *"My context is [stale/incomplete] — I'm assuming [X]. Correct me if I'm wrong."*
---
## Context Enrichment
During conversations, you'll learn things not in the file. Capture them.
**Triggers:** New number or timeline revealed, key person mentioned, priority shift, constraint surfaces.
**Protocol:**
1. Note internally: `[CONTEXT UPDATE: {what was learned}]`
2. At session end: *"I picked up a few things to add to your context. Want me to update the file?"*
3. If yes: append to the relevant dimension, update timestamp.
**Never silently overwrite.** Always confirm before modifying the context file.
---
## Privacy Rules
### Never send externally
- Specific revenue or burn figures
- Customer names
- Employee names (unless publicly known)
- Investor names (unless public)
- Specific runway months
- Watch List contents
### Safe to use externally (with anonymization)
- Stage label
- Team size ranges (1–10, 10–50, 50–200+)
- Industry vertical
- Challenge category
- Market position descriptor
### Before any external API call or web search
Apply `references/anonymization-protocol.md`:
- Numbers → ranges or stage-relative descriptors
- Names → roles
- Revenue → percentages or stage labels
- Customers → "Customer A, B, C"
---
## Missing or Partial Context
Handle gracefully — never block the conversation.
- **Missing stage:** "Just to calibrate — are you still finding PMF or scaling what works?"
- **Missing financials:** Use stage + team size to infer. Note the gap.
- **Missing founder profile:** Infer from conversation style. Mark as inferred.
- **Multiple founders:** Context reflects the interviewee. Note co-founder perspective may differ.
---
## Required Context Fields
```
Required:
- Last updated (date)
- Company Identity → What we do
- Stage & Scale → Stage
- Founder Profile → Founder archetype
- Current Challenges → Priority #1
- Goals & Ambition → 12-month target
High-value optional:
- Unfair advantage
- Kill-shot risk
- Avoided decision
- Watch list
```
Missing required fields: note gaps, work around in session, ask in-session only when critical.
---
## References
- `references/anonymization-protocol.md` — detailed rules for stripping sensitive data before external calls
FILE:references/anonymization-protocol.md
# Anonymization Protocol
Rules for stripping sensitive company data before any external API call, web search, or tool invocation that sends data outside the local environment.
---
## When This Protocol Applies
**Trigger:** Any time company context or conversation content will leave the local session.
Examples:
- Web search that includes company specifics
- External API call with company data in the payload
- Any tool call where conversation content is part of the request
**Does NOT apply to:**
- Local file reads/writes (`~/.claude/company-context.md`)
- In-session reasoning and analysis
- Generating advice or documents that stay local
---
## Rule 1: Financial Figures → Relative Ranges
Never send specific financial data externally.
| Raw data | Anonymized version |
|----------|-------------------|
| "$2.4M ARR" | "early-stage ARR (sub-$5M)" |
| "$180K MRR" | "growing MRR, Series A range" |
| "14 months runway" | "runway is healthy for stage" |
| "burn rate is $320K/month" | "burn rate is moderate for stage" |
| "raised $8M Series A" | "Series A company" |
| "customer LTV is $4,200" | "LTV is above industry average for segment" |
| "CAC is $680" | "CAC is in a sustainable range" |
**Rule:** No dollar amounts. No month counts for runway. Use stage-relative descriptors.
---
## Rule 2: Customer Names → Anonymized Labels
Never send customer or client names externally.
| Raw data | Anonymized version |
|----------|-------------------|
| "Acme Corp is our biggest customer" | "Customer A (largest account)" |
| "we're working with NHS England" | "a large public-sector customer" |
| "BMW, Volkswagen, and Stellantis" | "three major automotive OEMs" |
| "10 enterprise customers including..." | "10 enterprise customers" |
**Rule:** Use "Customer A/B/C" for named accounts, or describe by segment without naming.
---
## Rule 3: Revenue Figures → Percentage Changes or Stage Descriptors
Revenue trajectory is safer than absolute numbers.
| Raw data | Anonymized version |
|----------|-------------------|
| "growing from $1M to $2M ARR" | "2x revenue growth year-over-year" |
| "revenue dropped from $500K to $430K" | "revenue declined ~15% in the period" |
| "hit $10M ARR last quarter" | "crossed a significant ARR milestone" |
| "doing $50K MRR" | "pre-Series A revenue, strong growth trajectory" |
**Rule:** Percentages and directional signals (growing / declining / flat) are safe. Absolutes are not.
---
## Rule 4: Employee Names → Roles Only
Never send individual names externally.
| Raw data | Anonymized version |
|----------|-------------------|
| "Our CTO, Sarah Chen, is struggling" | "our CTO is struggling with the transition" |
| "James is the best performer on the team" | "our strongest performer is in the engineering lead role" |
| "we're about to let go of Michael" | "we're about to make a leadership change" |
| "the founding team is me, Alex, and Priya" | "a three-person founding team" |
**Exception:** Publicly known executives (CEO of a public company, named in press releases) can be referenced by name. If in doubt, use role.
---
## Rule 5: Investor Names → Generic Descriptors
| Raw data | Anonymized version |
|----------|-------------------|
| "Sequoia led our round" | "a top-tier VC led our round" |
| "our lead investor is pushing for an exit" | "pressure from investors toward exit" |
| "Y Combinator alumni" | "accelerator alumni" |
**Exception:** YC, Techstars, and similar well-known accelerators are commonly referenced and safe if the founder has publicly disclosed. When in doubt, omit.
---
## Rule 6: Location → Country or Region
| Raw data | Anonymized version |
|----------|-------------------|
| "Berlin-based startup" | "European startup" |
| "we're in San Francisco" | "US-based startup" |
| "expanding to Munich and Vienna" | "expanding in the DACH region" |
**Exception:** Location is less sensitive than financials. Use judgment — if it's on their website, it's fine.
---
## Anonymization Decision Tree
```
Before sending data externally:
1. Does it include a specific dollar amount?
→ YES: Replace with range or relative descriptor
2. Does it include a person's name?
→ YES: Replace with role only (unless publicly known)
3. Does it include a company or customer name?
→ YES: Replace with "Customer A" or segment descriptor
4. Does it include specific headcount or runway months?
→ YES: Replace with range (1–10, 10–50) or "healthy/tight/critical"
5. Does it include proprietary data, roadmap, or unreleased product info?
→ YES: Do not include. Reference only generically ("product expansion planned")
6. Is it publicly available information?
→ YES: Safe to send as-is
```
---
## Required vs Optional Anonymization
### Required (always strip before external calls)
- Revenue figures (absolute)
- Burn rate (absolute)
- Runway (specific months)
- Customer names
- Employee names
- Investor names (unless public)
- Funding amounts (unless public)
### Optional (use judgment based on sensitivity)
- Industry vertical (usually fine)
- Company stage (usually fine)
- Team size ranges (usually fine)
- Geographic region (usually fine)
- General challenge category (usually fine)
---
## What to Do If You're Unsure
Default to stricter anonymization. The cost of over-anonymizing is slightly less useful external results. The cost of under-anonymizing is a privacy breach.
When in doubt: **remove it**.
---
## Audit Log (Internal Only)
When running external calls with company context, note internally:
```
[EXTERNAL CALL: {tool/API used}]
[ANONYMIZED: {fields stripped}]
[RETAINED: {fields kept and why}]
```
This is for internal reasoning only — never included in output to the founder.
Tạo, tối ưu và phân tích chương trình giới thiệu, affiliate và chiến lược truyền miệng: vòng lan truyền, ưu đãi giới thiệu.
---
name: referrals
description: "When the user wants to create, optimize, or analyze a referral program, affiliate program, or word-of-mouth strategy. Also use when the user mentions 'referral,' 'affiliate,' 'ambassador,' 'word of mouth,' 'viral loop,' 'refer a friend,' 'partner program,' 'referral incentive,' 'how to get referrals,' 'customers referring customers,' or 'affiliate payout.' Use this whenever someone wants existing users or partners to bring in new customers. For launch-specific virality, see launch."
metadata:
version: 2.0.1
---
# Referral & Affiliate Programs
You are an expert in viral growth and referral marketing. Your goal is to help design and optimize programs that turn customers into growth engines.
## Before Starting
**Check for product marketing context first:**
If `.agents/product-marketing.md` exists (or `.claude/product-marketing.md`, or the legacy `product-marketing-context.md` filename, in older setups), read it before asking questions. Use that context and only ask for information not already covered or specific to this task.
Gather this context (ask if not provided):
### 1. Program Type
- Customer referral program, affiliate program, or both?
- B2B or B2C?
- What's the average customer LTV?
- What's your current CAC from other channels?
### 2. Current State
- Existing referral/affiliate program?
- Current referral rate (% who refer)?
- What incentives have you tried?
### 3. Product Fit
- Is your product shareable?
- Does it have network effects?
- Do customers naturally talk about it?
### 4. Resources
- Tools/platforms you use or consider?
- Budget for referral incentives?
---
## Should You Engineer Virality First?
Before building a reward-driven program, check whether virality can be **built into the product** — often cheaper and more durable than paid referrals. But **don't force virality where it doesn't naturally fit.**
Place the product on the **Viral Potential Spectrum**:
- **Natural** (build for it): collaboration tools, communication tools, user-facing outputs — every use exposes the product to non-users.
- **Limited** (don't force it): backend, competitive-advantage, internal-only, and infrastructure products. Invest in referral programs, content, and partnerships instead.
If the product is on the natural end, consider **product-embedded viral mechanisms** (Powered By badges, exposure loops, social sharing, embeds, watermarks) before or alongside a reward program.
**For the spectrum diagnostic, the 7 viral mechanisms, value-presentation and timing best practices, and affiliate power-law mechanics**: See [references/viral-mechanisms.md](references/viral-mechanisms.md)
---
## Referral vs. Affiliate
### Customer Referral Programs
**Best for:**
- Existing customers recommending to their network
- Products with natural word-of-mouth
- Lower-ticket or self-serve products
**Characteristics:**
- Referrer is an existing customer
- One-time or limited rewards
- Higher trust, lower volume
### Affiliate Programs
**Best for:**
- Reaching audiences you don't have access to
- Content creators, influencers, bloggers
- Higher-ticket products that justify commissions
**Characteristics:**
- Affiliates may not be customers
- Ongoing commission relationship
- Higher volume, variable trust
---
## Referral Program Design
### The Referral Loop
```
Trigger Moment → Share Action → Convert Referred → Reward → (Loop)
```
### Step 1: Identify Trigger Moments
**High-intent moments:**
- Right after first "aha" moment
- After achieving a milestone
- After exceptional support
- After renewing or upgrading
### Step 2: Design Share Mechanism
**Ranked by effectiveness:**
1. In-product sharing (highest conversion)
2. Personalized link
3. Email invitation
4. Social sharing
5. Referral code (works offline)
### Step 3: Choose Incentive Structure
**Single-sided rewards** (referrer only): Simpler, works for high-value products
**Double-sided rewards** (both parties): Higher conversion, win-win framing
**Tiered rewards**: Gamifies referral process, increases engagement
**Present the reward with the bigger-*feeling* number** — "lead with the larger number" (say "$10 off," not "40% off," on a low-priced product). Reward at the **aha moment or milestone**, not signup. Reduce friction: one-click share, pre-written messages.
**For examples and incentive sizing**: See [references/program-examples.md](references/program-examples.md)
**For product-embedded virality, value-presentation rules, and affiliate power-law mechanics**: See [references/viral-mechanisms.md](references/viral-mechanisms.md)
---
## Program Optimization
### Improving Referral Rate
**If few customers are referring:**
- Ask at better moments
- Simplify sharing process
- Test different incentive types
- Make referral prominent in product
**If referrals aren't converting:**
- Improve landing experience for referred users
- Strengthen incentive for new users
- Ensure referrer's endorsement is visible
### A/B Tests to Run
**Incentive tests:** Amount, type, single vs. double-sided, timing
**Messaging tests:** Program description, CTA copy, landing page copy
**Placement tests:** Where and when the referral prompt appears
### Common Problems & Fixes
| Problem | Fix |
|---------|-----|
| Low awareness | Add prominent in-app prompts |
| Low share rate | Simplify to one click |
| Low conversion | Optimize referred user experience |
| Fraud/abuse | Add verification, limits |
| One-time referrers | Add tiered/gamified rewards |
---
## Measuring Success
### Key Metrics
**Program health:**
- Active referrers (referred someone in last 30 days)
- Referral conversion rate
- Rewards earned/paid
**Business impact:**
- % of new customers from referrals
- CAC via referral vs. other channels
- LTV of referred customers
- Referral program ROI
### Typical Findings
- Referred customers have 16-25% higher LTV
- Referred customers have 18-37% lower churn
- Referred customers refer others at 2-3x rate
---
## Launch Checklist
### Before Launch
- [ ] Define program goals and success metrics
- [ ] Design incentive structure
- [ ] Build or configure referral tool
- [ ] Create referral landing page
- [ ] Set up tracking and attribution
- [ ] Define fraud prevention rules
- [ ] Create terms and conditions
- [ ] Test complete referral flow
### Launch
- [ ] Announce to existing customers
- [ ] Add in-app referral prompts
- [ ] Update website with program details
- [ ] Brief support team
### Post-Launch (First 30 Days)
- [ ] Review conversion funnel
- [ ] Identify top referrers
- [ ] Gather feedback
- [ ] Fix friction points
- [ ] Send reminder emails to non-referrers
---
## Email Sequences
### Referral Program Launch
```
Subject: You can now earn [reward] for sharing [Product]
We just launched our referral program!
Share [Product] with friends and earn [reward] for each signup.
They get [their reward] too.
[Unique referral link]
1. Share your link
2. Friend signs up
3. You both get [reward]
```
### Referral Nurture Sequence
- Day 7: Remind about referral program
- Day 30: "Know anyone who'd benefit?"
- Day 60: Success story + referral prompt
- After milestone: "You achieved [X]—know others who'd want this?"
---
## Affiliate Programs
**For detailed affiliate program design, commission structures, recruitment, and tools**: See [references/affiliate-programs.md](references/affiliate-programs.md)
**For affiliate power-law mechanics (buyout clauses ~12× monthly commission, the 20/80 super-promoter rule, launch-affiliate tactics)**: See [references/viral-mechanisms.md](references/viral-mechanisms.md)
---
## Task-Specific Questions
1. What type of program (referral, affiliate, or both)?
2. What's your customer LTV and current CAC?
3. Existing program or starting from scratch?
4. What tools/platforms are you considering?
5. What's your budget for rewards/commissions?
6. Is your product naturally shareable?
---
## Tool Integrations
For implementation, see the [tools registry](../../tools/REGISTRY.md). Key tools for referral programs:
| Tool | Best For | Guide |
|------|----------|-------|
| **Rewardful** | Stripe-native affiliate programs | [rewardful.md](../../tools/integrations/rewardful.md) |
| **Tolt** | SaaS affiliate programs | [tolt.md](../../tools/integrations/tolt.md) |
| **Mention Me** | Enterprise referral programs | [mention-me.md](../../tools/integrations/mention-me.md) |
| **Dub.co** | Link tracking and attribution | [dub-co.md](../../tools/integrations/dub-co.md) |
| **Stripe** | Payment processing (for commission tracking) | [stripe.md](../../tools/integrations/stripe.md) |
| **Introw** | Channel partner programs with tiers, deal registration, QBRs | [introw.md](../../tools/integrations/introw.md) |
| **PartnerStack** | Enterprise partner and affiliate programs | [partnerstack.md](../../tools/integrations/partnerstack.md) |
---
## Related Skills
- **launch**: For launching referral program effectively
- **emails**: For referral nurture campaigns
- **marketing-psychology**: For understanding referral motivation
- **analytics**: For tracking referral attribution
FILE:evals/evals.json
{
"skill_name": "referrals",
"evals": [
{
"id": 1,
"prompt": "Help me design a referral program for our SaaS product. We're a $49/month project management tool with about 1,000 customers. We want to encourage word-of-mouth growth.",
"expected_output": "Should check for product-marketing.md first. Should distinguish between referral and affiliate programs (this is referral — existing customers referring peers). Should design the referral loop: trigger point (when to ask for referral), share mechanism (unique link, email invite, social share), conversion flow (what the referred person experiences), and reward structure. Should recommend incentive type: double-sided recommended (both referrer and referred get value). Should suggest specific incentives appropriate for $49/month SaaS (e.g., free month for both). Should include the launch checklist. Should recommend tool integrations (Rewardful, Tolt, etc.).",
"assertions": [
"Checks for product-marketing.md",
"Distinguishes referral from affiliate",
"Designs the referral loop (trigger, share, convert, reward)",
"Recommends double-sided incentive structure",
"Suggests specific incentives for the price point",
"Includes launch checklist",
"Recommends tool integrations"
],
"files": []
},
{
"id": 2,
"prompt": "We have a referral program but only 5% of customers have ever referred someone. How do we increase participation?",
"expected_output": "Should apply the program optimization guidance. Should diagnose low participation: are customers aware of the program? Is the trigger point well-timed? Is the incentive compelling enough? Is sharing easy? Should recommend optimization tactics: better placement/visibility, timing referral asks at peak satisfaction moments, improving the incentive, simplifying the share mechanism, adding referral reminders in email and in-app. Should provide specific experiment ideas to test improvements.",
"assertions": [
"Applies program optimization guidance",
"Diagnoses potential causes of low participation",
"Checks awareness, timing, incentive, and friction",
"Recommends optimization tactics",
"Suggests timing referral asks at satisfaction moments",
"Provides experiment ideas"
],
"files": []
},
{
"id": 3,
"prompt": "should we do referral or affiliate? we sell online courses for $199-499 and want to get other creators and influencers to promote us.",
"expected_output": "Should trigger on casual phrasing. Should apply the referral vs affiliate distinction clearly. For this use case (getting creators/influencers to promote), should recommend an affiliate program (not referral — affiliates are third-party promoters, not existing customers). Should apply the affiliate program section guidance: commission structure for digital products (typically 20-40% for courses), cookie duration, payout terms, affiliate onboarding. Should recommend affiliate platforms/tools appropriate for course creators.",
"assertions": [
"Triggers on casual phrasing",
"Clearly distinguishes referral from affiliate",
"Recommends affiliate for this use case",
"Provides commission structure guidance for courses",
"Addresses cookie duration and payout terms",
"Recommends appropriate affiliate platforms"
],
"files": []
},
{
"id": 4,
"prompt": "What incentive structure works best? We've been offering $10 off for referrers but it's not working. Our product is $29/month.",
"expected_output": "Should evaluate the current incentive: $10 off on a $29/month product is significant but only benefits the referrer (single-sided). Should recommend testing double-sided incentives (both parties get value). Should discuss incentive types: account credit, free months, feature upgrades, cash. Should apply the tiered incentive concept (increasing rewards for multiple referrals). Should provide specific alternative incentive structures to test. Should note that incentive alone may not be the problem — placement and timing matter too.",
"assertions": [
"Evaluates current incentive structure",
"Identifies as single-sided and recommends double-sided",
"Discusses multiple incentive types",
"Applies tiered incentive concept",
"Provides specific alternatives to test",
"Notes incentive may not be the only issue"
],
"files": []
},
{
"id": 5,
"prompt": "How do we measure the success of our referral program? What metrics should we track?",
"expected_output": "Should apply the measuring success framework. Should define key metrics: participation rate (% of customers who refer), share rate (referrals sent per participant), conversion rate (referred visitors who become customers), viral coefficient (k-factor), customer acquisition cost via referral vs other channels, referred customer LTV vs organic customer LTV. Should recommend tracking tools and dashboards. Should provide benchmark ranges for each metric.",
"assertions": [
"Applies measuring success framework",
"Defines participation rate, share rate, conversion rate",
"Includes viral coefficient / k-factor",
"Compares referral CAC to other channels",
"Compares referred customer LTV to organic",
"Recommends tracking approach",
"Provides benchmark ranges"
],
"files": []
},
{
"id": 6,
"prompt": "Can you write the referral invitation emails? I need the email that goes out when someone shares their referral link.",
"expected_output": "Should recognize this overlaps with email writing. Should apply the referral email sequence section from the skill for referral-specific emails. However, for detailed email sequence design (multi-email nurture for referred users), should cross-reference the emails skill. Should provide the referral invitation email but note that broader email sequence work is handled by emails.",
"assertions": [
"Applies referral email section from the skill",
"Provides referral invitation email guidance",
"Cross-references emails for broader email work",
"Provides specific referral email copy or template"
],
"files": []
},
{
"id": 7,
"prompt": "We build a collaborative design tool. We keep hearing 'add a referral program' but I want to know if we can just make the product spread on its own. What are our options?",
"expected_output": "Should place the product on the Viral Potential Spectrum: a collaborative design tool is on the natural end (collaboration + user-facing output), so product-embedded virality fits before or instead of a reward program. Should recommend engineering virality through product design rather than defaulting to a reward program, and warn against forcing virality where it doesn't fit. Should walk through applicable viral mechanisms from the 7: exposure loops (invite/collaboration), embeds (Notion/Figma/Loom style), social sharing (e.g. a #MadeWith hashtag), and Powered By / watermark badges on free-tier output. Should note referral programs are the incentive-driven fallback when the product doesn't spread naturally. Should reference viral-mechanisms.md.",
"assertions": [
"Places the product on the Viral Potential Spectrum (natural end)",
"Recommends product-embedded virality before defaulting to a reward program",
"Warns against forcing virality where it doesn't fit",
"Names applicable mechanisms (exposure loops, embeds, social sharing, badges/watermarks)",
"Frames referral programs as the incentive-driven fallback",
"References viral-mechanisms.md"
],
"files": []
}
]
}
FILE:references/affiliate-programs.md
# Affiliate Program Design
Detailed guidance for building and managing affiliate programs.
## Contents
- Commission Structures
- Cookie Duration
- Affiliate Recruitment
- Affiliate Enablement
- Tools & Platforms (Referral Program Tools, Affiliate Program Tools, Choosing a Tool)
- Fraud Prevention (Common Referral Fraud, Prevention Measures)
## Commission Structures
**Percentage of sale:**
- Standard: 10-30% of first sale or first year
- Works for: E-commerce, SaaS with clear pricing
- Example: "Earn 25% of every sale you refer"
**Flat fee per action:**
- Standard: $5-500 depending on value
- Works for: Lead gen, trials, freemium
- Example: "$50 for every qualified demo"
**Recurring commission:**
- Standard: 10-25% of recurring revenue
- Works for: Subscription products
- Example: "20% of subscription for 12 months"
**Tiered commission:**
- Works for: Motivating high performers
- Example: "20% for 1-10 sales, 25% for 11-25, 30% for 26+"
---
## Cookie Duration
How long after click does affiliate get credit?
| Duration | Use Case |
|----------|----------|
| 24 hours | High-volume, low-consideration purchases |
| 7-14 days | Standard e-commerce |
| 30 days | Standard SaaS/B2B |
| 60-90 days | Long sales cycles, enterprise |
| Lifetime | Premium affiliate relationships |
---
## Affiliate Recruitment
### Where to find affiliates:
- Existing customers who create content
- Industry bloggers and reviewers
- YouTubers in your niche
- Newsletter writers
- Complementary tool companies
- Consultants and agencies
### Outreach template:
```
Subject: Partnership opportunity — [Your Product]
Hi [Name],
I've been following your content on [topic] — particularly [specific piece] — and think there could be a great fit for a partnership.
[Your Product] helps [audience] [achieve outcome], and I think your audience would find it valuable.
We offer [commission structure] for partners, plus [additional benefits: early access, co-marketing, etc.].
Would you be open to learning more?
[Your name]
```
---
## Affiliate Enablement
Provide affiliates with:
- [ ] Unique tracking links/codes
- [ ] Product overview and key benefits
- [ ] Target audience description
- [ ] Comparison to competitors
- [ ] Creative assets (logos, banners, images)
- [ ] Sample copy and talking points
- [ ] Case studies and testimonials
- [ ] Demo access or free account
- [ ] FAQ and objection handling
- [ ] Payment terms and schedule
---
## Tools & Platforms
### Referral Program Tools
**Full-featured platforms:**
- ReferralCandy — E-commerce focused
- Ambassador — Enterprise referral programs
- Friendbuy — E-commerce and subscription
- GrowSurf — SaaS and tech companies
- Mention Me — AI-powered referral marketing
- Viral Loops — Template-based campaigns
**Built-in options:**
- Stripe (basic referral tracking)
- HubSpot (CRM-integrated)
- Segment (tracking and analytics)
### Affiliate Program Tools
**Affiliate networks:**
- ShareASale — Large merchant network
- Impact — Enterprise partnerships
- PartnerStack — SaaS focused
- Tapfiliate — Simple SaaS affiliate tracking
- FirstPromoter — SaaS affiliate management
**Partner Relationship Management (PRM):**
- Introw — Full PRM with deal registration, commissions, tiers, QBRs, and partner engagement tracking ([integration guide](../../../tools/integrations/introw.md))
**Self-hosted:**
- Rewardful — Stripe-integrated affiliates
- Refersion — E-commerce affiliates
### Choosing a Tool
Consider:
- Integration with your payment system
- Fraud detection capabilities
- Payout management
- Reporting and analytics
- Customization options
- Price vs. program scale
---
## Fraud Prevention
### Common Referral Fraud
- Self-referrals (creating fake accounts)
- Referral rings (groups referring each other)
- Coupon sites posting referral codes
- Fake email addresses
- VPN/device spoofing
### Prevention Measures
**Technical:**
- Email verification required
- Device fingerprinting
- IP address monitoring
- Delayed reward payout (after activation)
- Minimum activity threshold
**Policy:**
- Clear terms of service
- Maximum referrals per period
- Reward clawback for refunds/chargebacks
- Manual review for suspicious patterns
**Structural:**
- Require referred user to take meaningful action
- Cap lifetime rewards
- Pay rewards in product credit (less attractive to fraudsters)
FILE:references/program-examples.md
# Referral Program Examples
Real-world examples of successful referral programs.
## Contents
- Dropbox (Classic)
- Uber/Lyft
- Morning Brew
- Notion
- Incentive Types Comparison
- Incentive Sizing Framework
- Viral Coefficient & Metrics (Key Metrics, Calculating Referral Program ROI)
## Dropbox (Classic)
**Program:** Give 500MB storage, get 500MB storage
**Why it worked:**
- Reward directly tied to product value
- Low friction (just an email)
- Both parties benefit equally
- Gamified with progress tracking
---
## Uber/Lyft
**Program:** Give $10 ride credit, get $10 when they ride
**Why it worked:**
- Immediate, clear value
- Double-sided incentive
- Easy to share (code/link)
- Triggered at natural moments
---
## Morning Brew
**Program:** Tiered rewards for subscriber referrals
- 3 referrals: Newsletter stickers
- 5 referrals: T-shirt
- 10 referrals: Mug
- 25 referrals: Hoodie
**Why it worked:**
- Gamification drives ongoing engagement
- Physical rewards are shareable (more referrals)
- Low cost relative to subscriber value
- Built status/identity
---
## Notion
**Program:** $10 credit per referral (education)
**Why it worked:**
- Targeted high-sharing audience (students)
- Product naturally spreads in teams
- Credit keeps users engaged
---
## Incentive Types Comparison
| Type | Pros | Cons | Best For |
|------|------|------|----------|
| Cash/credit | Universally valued | Feels transactional | Marketplaces, fintech |
| Product credit | Drives usage | Only valuable if they'll use it | SaaS, subscriptions |
| Free months | Clear value | May attract freebie-seekers | Subscription products |
| Feature unlock | Low cost to you | Only works for gated features | Freemium products |
| Swag/gifts | Memorable, shareable | Logistics complexity | Brand-focused companies |
| Charity donation | Feel-good | Lower personal motivation | Mission-driven brands |
---
## Incentive Sizing Framework
**Calculate your maximum incentive:**
```
Max Referral Reward = (Customer LTV × Gross Margin) - Target CAC
```
**Example:**
- LTV: $1,200
- Gross margin: 70%
- Target CAC: $200
- Max reward: ($1,200 × 0.70) - $200 = $640
**Typical referral rewards:**
- B2C: $10-50 or 10-25% of first purchase
- B2B SaaS: $50-500 or 1-3 months free
- Enterprise: Higher, often custom
---
## Viral Coefficient & Metrics
### Key Metrics
**Viral coefficient (K-factor):**
```
K = Invitations × Conversion Rate
K > 1 = Viral growth (each user brings more than 1 new user)
K < 1 = Amplified growth (referrals supplement other acquisition)
```
**Example:**
- Average customer sends 3 invitations
- 15% of invitations convert
- K = 3 × 0.15 = 0.45
**Referral rate:**
```
Referral Rate = (Customers who refer) / (Total customers)
```
Benchmarks:
- Good: 10-25% of customers refer
- Great: 25-50%
- Exceptional: 50%+
**Referrals per referrer:**
Benchmarks:
- Average: 1-2 referrals per referrer
- Good: 2-5
- Exceptional: 5+
### Calculating Referral Program ROI
```
Referral Program ROI = (Revenue from referred customers - Program costs) / Program costs
Program costs = Rewards paid + Tool costs + Management time
```
**Track separately:**
- Cost per referred customer (CAC via referral)
- LTV of referred customers (often higher than average)
- Payback period for referral rewards
FILE:references/viral-mechanisms.md
# Viral Mechanisms
Virality can be **engineered through product design**, not just bought with reward programs. But don't force it — decide *whether* virality fits your product before building anything.
## Contents
- Viral Potential Spectrum (the diagnostic)
- The 7 Viral Mechanisms
- Referral Best Practices (presentation, timing, friction)
- Affiliate Mechanics (buyout clauses, the 20/80 power law, launch tactics)
---
## Viral Potential Spectrum
Before engineering virality, place your product on the spectrum. **Don't force virality where it doesn't naturally fit.**
**Natural viral potential (build for it):**
- **Collaboration tools** — value grows when you invite others (docs, whiteboards, project management)
- **Communication tools** — you can't use them alone (email, scheduling, messaging)
- **User-facing outputs** — every use produces something others see (design, video, forms, links)
**Limited viral potential (don't force it):**
- **Backend / infrastructure** — invisible to end users
- **Competitive-advantage tools** — users *hide* that they use them (their edge)
- **Internal-only tools** — never leave the org
- **Infrastructure** — plumbing no one talks about
If you're on the limited end, invest in referral programs, content, and partnerships instead of embedding viral loops that won't fire.
---
## The 7 Viral Mechanisms
Most are **non-incentive** — the loop is built into the product, not paid for.
### 1. "Powered By" Badges
A small attributed badge on user-facing output ("Powered by [Product]"). Every page/form/widget a customer ships becomes an ad. Often free-tier only (paid tier removes it).
### 2. Exposure Loops
The product's normal use exposes it to non-users.
- **Calendly / SavvyCal** — every meeting invite you send shows the tool to the recipient, who often becomes a user.
- **Superhuman email signatures** ("Sent via Superhuman") — works as a **status signal**, not just attribution. The signature signaled early-adopter status, so recipients *wanted* it. Exposure loops are strongest when using the product confers status.
### 3. Social Sharing
Make output natively shareable with a branded hook.
- **#MadeWithGlide** — a hashtag turns every user creation into discoverable social proof.
- One-tap "share to X/LinkedIn" on any milestone, result, or artifact.
### 4. Embed Options
Let users embed their content elsewhere; the embed carries your brand and a link back.
- **Notion, Figma, Loom** — embedded docs, designs, and videos spread the product to every viewer on every host site.
### 5. Watermarks / Mandatory Badges
Like "Powered By" but harder to remove — baked into the output itself.
- **OpusClips** watermark on generated clips.
- **"Made in Webflow"** badge on free-plan sites.
Free tier carries the mark; paid tier removes it. The free users become the distribution.
### 6. Referral Programs
Explicit incentives for referring. Covered in detail in [program-examples.md](program-examples.md) and below. The one *incentive-driven* mechanism on this list — use it when the product itself doesn't naturally spread.
### 7. Product-Driven Word-of-Mouth
The purest form: the product is so good, novel, or useful that people tell others unprompted. Not a mechanism you bolt on — it's earned through the product experience. Engineering the other six makes this easier to trigger.
---
## Referral Best Practices
Detail beyond the core referral loop (trigger → share → convert → reward).
### Value Presentation: Lead With the Larger Number
Frame the reward with whichever number *looks* bigger.
- On a $25 product, say **"$10 off"** — not "40% off."
- On a $500 product, say **"20% off"** — not "$100 off" if the percentage frames better... but usually the absolute dollar figure wins for smaller prices.
- Rule of thumb: **under ~$100, lead with the dollar amount; over ~$100, test the percentage.** Always pick the bigger-*feeling* number.
### Reward Timing: Fire at the Aha / Milestone
Trigger the referral ask (and reward) at the moment the user has just felt the product's value — the **aha moment** or a **milestone** (first success, upgrade, streak). Motivation to share peaks right after value is experienced, not at signup.
### Double-Sided Rewards
Both referrer and referred get value. Higher conversion than single-sided, and gives the referrer a generous, non-selfish reason to share ("here's $10 for you too").
### Friction Reduction
Every extra step kills share rate.
- **One-click sharing** — pre-generated link, no form.
- **Pre-written messages** — draft the email/DM/post copy so the user just hits send.
- In-product placement at the trigger moment, not buried in settings.
---
## Affiliate Mechanics
Detail deferred from the partnerships side — for building an affiliate motion into a referral/partner strategy.
### Buyout Clauses (~12× Monthly Commission)
For high-performing affiliates on **recurring** commissions, include a **buyout clause**: the right to buy out the affiliate's future commission stream for a lump sum, commonly around **12× the monthly commission**. Protects margin on a customer the affiliate referred once but earns on forever, and gives the affiliate an attractive cash-out.
### The 20/80 Affiliate Power Law
Roughly **20% of affiliates drive ~80% of results**. Don't spread effort evenly across a long tail of dormant sign-ups. **Identify super-promoters and invest in them** — higher tiers, custom assets, co-marketing, direct relationship, early access. Recruiting 1,000 passive affiliates is worth less than activating 10 great ones.
### Launch-Affiliate Tactic (Cometly / Demio)
Time affiliate promotion around a **launch or a hard deadline** to concentrate volume. Cometly drove **$251K on a single launch day** by mobilizing affiliates simultaneously; Demio ran launch-window affiliate pushes. The mechanic: give affiliates a shared date, shared assets, and a reason for their audience to act *now* (bonus, cohort, closing offer) so promotion stacks instead of trickling.
Xây dựng tệp người theo dõi, viết nội dung lan truyền, phân tích hồ sơ, nghiên cứu đối thủ và tối ưu tương tác trên X/Twitter.
---
name: "x-twitter-growth"
description: "X/Twitter growth engine for building audience, crafting viral content, and analyzing engagement. Use when the user wants to grow on X/Twitter, write tweets or threads, analyze their X profile, research competitors on X, plan a posting strategy, or optimize engagement. Complements social-content (generic multi-platform) with X-specific depth: algorithm mechanics, thread engineering, reply strategy, profile optimization, and competitive intelligence via web search."
license: MIT
metadata:
version: 1.0.0
author: Alireza Rezvani
category: marketing
updated: 2026-03-10
---
# X/Twitter Growth Engine
X-specific growth skill. For general social media content across platforms, see `social-content`. For social strategy and calendar planning, see `social-media-manager`. This skill goes deep on X.
## When to Use This vs Other Skills
| Need | Use |
|------|-----|
| Write a tweet or thread | **This skill** |
| Plan content across LinkedIn + X + Instagram | social-content |
| Analyze engagement metrics across platforms | social-media-analyzer |
| Build overall social strategy | social-media-manager |
| X-specific growth, algorithm, competitive intel | **This skill** |
---
## Step 1 — Profile Audit
Before any growth work, audit the current X presence. Run `scripts/profile_auditor.py` with the handle, or manually assess:
### Bio Checklist
- [ ] Clear value proposition in first line (who you help + how)
- [ ] Specific niche — not "entrepreneur | thinker | builder"
- [ ] Social proof element (followers, title, metric, brand)
- [ ] CTA or link (newsletter, product, site)
- [ ] No hashtags in bio (signals amateur)
### Pinned Tweet
- [ ] Exists and is less than 30 days old
- [ ] Showcases best work or strongest hook
- [ ] Has clear CTA (follow, subscribe, read)
### Recent Activity (last 30 posts)
- [ ] Posting frequency: minimum 1x/day, ideal 3-5x/day
- [ ] Mix of formats: tweets, threads, replies, quotes
- [ ] Reply ratio: >30% of activity should be replies
- [ ] Engagement trend: improving, flat, or declining
Run: `python3 scripts/profile_auditor.py --handle @username`
---
## Step 2 — Competitive Intelligence
Research competitors and successful accounts in your niche using web search.
### Process
1. Search `site:x.com "topic" min_faves:100` via Brave to find high-performing content
2. Identify 5-10 accounts in your niche with strong engagement
3. For each, analyze: posting frequency, content types, hook patterns, engagement rates
4. Run: `python3 scripts/competitor_analyzer.py --handles @acc1 @acc2 @acc3`
### What to Extract
- **Hook patterns** — How do top posts start? Question? Bold claim? Statistic?
- **Content themes** — What 3-5 topics get the most engagement?
- **Format mix** — Ratio of tweets vs threads vs replies vs quotes
- **Posting times** — When do their best posts go out?
- **Engagement triggers** — What makes people reply vs like vs retweet?
---
## Step 3 — Content Creation
### Tweet Types (ordered by growth impact)
#### 1. Threads (highest reach, highest follow conversion)
```
Structure:
- Tweet 1: Hook — must stop the scroll in <7 words
- Tweet 2: Context or promise ("Here's what I learned:")
- Tweets 3-N: One idea per tweet, each standalone-worthy
- Final tweet: Summary + explicit CTA ("Follow @handle for more")
- Reply to tweet 1: Restate hook + "Follow for more [topic]"
Rules:
- 5-12 tweets optimal (under 5 feels thin, over 12 loses people)
- Each tweet should make sense if read alone
- Use line breaks for readability
- No tweet should be a wall of text (3-4 lines max)
- Number the tweets or use "↓" in tweet 1
```
#### 2. Atomic Tweets (breadth, impression farming)
```
Formats that work:
- Observation: "[Thing] is underrated. Here's why:"
- Listicle: "10 tools I use daily:\n\n1. X — for Y"
- Contrarian: "Unpopular opinion: [statement]"
- Lesson: "I [did X] for [time]. Biggest lesson:"
- Framework: "[Concept] explained in 30 seconds:"
Rules:
- Under 200 characters gets more engagement
- One idea per tweet
- No links in tweet body (kills reach — put link in reply)
- Question tweets drive replies (algorithm loves replies)
```
#### 3. Quote Tweets (authority building)
```
Formula: Original tweet + your unique take
- Add data the original missed
- Provide counterpoint or nuance
- Share personal experience that validates/contradicts
- Never just say "This" or "So true"
```
#### 4. Replies (network growth, fastest path to visibility)
```
Strategy:
- Reply to accounts 2-10x your size
- Add genuine value, not "great post!"
- Be first to reply on accounts with large audiences
- Your reply IS your content — make it tweet-worthy
- Controversial/insightful replies get quote-tweeted (free reach)
```
Run: `python3 scripts/tweet_composer.py --type thread --topic "your topic" --audience "your audience"`
---
## Step 4 — Algorithm Mechanics
### What X rewards (2025-2026)
| Signal | Weight | Action |
|--------|--------|--------|
| Replies received | Very high | Write reply-worthy content (questions, debates) |
| Time spent reading | High | Threads, longer tweets with line breaks |
| Profile visits from tweet | High | Curiosity gaps, tease expertise |
| Bookmarks | High | Tactical, save-worthy content (lists, frameworks) |
| Retweets/Quotes | Medium | Shareable insights, bold takes |
| Likes | Low-medium | Easy agreement, relatable content |
| Link clicks | Low (penalized) | Never put links in tweet body — use reply |
### What kills reach
- Links in tweet body (put in first reply instead)
- Editing tweets within 30 min of posting
- Posting and immediately going offline (no early engagement)
- More than 2 hashtags
- Tagging people who don't engage back
- Threads with inconsistent quality (one weak tweet tanks the whole thread)
### Optimal Posting Cadence
| Account size | Tweets/day | Threads/week | Replies/day |
|-------------|------------|--------------|-------------|
| < 1K followers | 2-3 | 1-2 | 10-20 |
| 1K-10K | 3-5 | 2-3 | 5-15 |
| 10K-50K | 3-7 | 2-4 | 5-10 |
| 50K+ | 2-5 | 1-3 | 5-10 |
---
## Step 5 — Growth Playbook
### Week 1-2: Foundation
1. Optimize bio and pinned tweet (Step 1)
2. Identify 20 accounts in your niche to engage with daily
3. Reply 10-20 times per day to larger accounts (genuine value only)
4. Post 2-3 atomic tweets per day testing different formats
5. Publish 1 thread
### Week 3-4: Pattern Recognition
1. Review what formats got most engagement
2. Double down on top 2 content formats
3. Increase to 3-5 posts per day
4. Publish 2-3 threads per week
5. Start quote-tweeting relevant content daily
### Month 2+: Scale
1. Develop 3-5 recurring content series (e.g., "Friday Framework")
2. Cross-pollinate: repurpose threads as LinkedIn posts, newsletter content
3. Build reply relationships with 5-10 accounts your size (mutual engagement)
4. Experiment with spaces/audio if relevant to niche
5. Run: `python3 scripts/growth_tracker.py --handle @username --period 30d`
---
## Step 6 — Content Calendar Generation
Run: `python3 scripts/content_planner.py --niche "your niche" --frequency 5 --weeks 2`
Generates a 2-week posting plan with:
- Daily tweet topics with hook suggestions
- Thread outlines (2-3 per week)
- Reply targets (accounts to engage with)
- Optimal posting times based on niche
---
## Scripts
| Script | Purpose |
|--------|---------|
| `scripts/profile_auditor.py` | Audit X profile: bio, pinned, activity patterns |
| `scripts/tweet_composer.py` | Generate tweets/threads with hook patterns |
| `scripts/competitor_analyzer.py` | Analyze competitor accounts via web search |
| `scripts/content_planner.py` | Generate weekly/monthly content calendars |
| `scripts/growth_tracker.py` | Track follower growth and engagement trends |
## Common Pitfalls
1. **Posting links directly** — Always put links in the first reply, never in the tweet body
2. **Thread tweet 1 is weak** — If the hook doesn't stop scrolling, nothing else matters
3. **Inconsistent posting** — Algorithm rewards daily consistency over occasional bangers
4. **Only broadcasting** — Replies and engagement are 50%+ of growth, not just posting
5. **Generic bio** — "Helping people do things" tells nobody anything
6. **Copying formats without adapting** — What works for tech Twitter doesn't work for marketing Twitter
## Related Skills
- `social-content` — Multi-platform content creation
- `social-media-manager` — Overall social strategy
- `social-media-analyzer` — Cross-platform analytics
- `content-production` — Long-form content that feeds X threads
- `copywriting` — Headline and hook writing techniques
FILE:references/algorithm-signals.md
# X/Twitter Algorithm Signals (2025-2026)
## Ranking Factors by Weight
### Tier 1 — Strongest Signals
| Signal | Impact | How to Optimize |
|--------|--------|----------------|
| Replies received | Very high | Ask questions, make controversial/insightful points |
| Dwell time (time reading) | Very high | Threads, longer tweets with line breaks |
| Profile clicks from tweet | High | Create curiosity gaps, tease expertise |
| Bookmarks | High | Tactical content (lists, frameworks, templates) |
### Tier 2 — Moderate Signals
| Signal | Impact | How to Optimize |
|--------|--------|----------------|
| Retweets/Quotes | Medium | Shareable insights, bold takes, data |
| Likes | Medium-low | Easy agreement, relatable content |
| Follows from tweet | Medium | Thread CTAs, high-value niche content |
### Tier 3 — Negative Signals
| Signal | Impact | How to Avoid |
|--------|--------|-------------|
| Link in tweet body | Reach penalty | Put links in first reply |
| Edit within 30 min | Suppresses | Don't edit — delete and repost if needed |
| Low early engagement | Decay | Stay online 30 min after posting, engage with replies |
| Hashtag spam (3+) | Spam signal | Max 1-2 hashtags, or zero |
| Tagging non-engagers | Negative | Only tag people likely to engage |
## Content Format Performance (ranked)
1. **Threads** — Highest reach potential, best for follower conversion
2. **Image tweets** — 2-3x engagement vs text-only
3. **Quote tweets** — Network effect (appear in two audiences)
4. **Text tweets** — Baseline, best for hot takes and questions
5. **Polls** — High engagement but low follower conversion
6. **Link tweets** — Lowest reach (algorithm penalizes external links)
## Optimal Timing
| Time Slot (UTC) | Why |
|----------------|-----|
| 12:00-14:00 | US East Coast morning, EU afternoon |
| 16:00-18:00 | US afternoon peak |
| 21:00-23:00 | US evening, high scroll time |
| 07:00-08:00 | EU morning, commute scrolling |
Best days: Tuesday-Thursday for B2B. Saturday-Sunday for consumer/lifestyle.
## Thread-Specific Mechanics
- Tweet 1 gets 10-50x the impressions of tweet 5+
- Hook quality determines 90% of thread performance
- "Numbered" threads (1/, 2/, etc.) signal commitment — algorithm boosts
- Self-reply threads perform better than tweetstorm threads
- Last tweet should have CTA + restate hook for people who scroll fast
## Premium/Blue Subscriber Advantages
- Longer tweets (up to 4,000 chars for Premium+)
- Edit button (use sparingly — edits can suppress reach)
- Higher reply ranking
- Revenue sharing eligibility
- Analytics access
## Sources
- X Engineering Blog (algorithm open-source release, 2023)
- Community testing and experimentation (ongoing)
- Creator program documentation
- Third-party analytics platforms (Typefully, Hypefury, Shield)
FILE:scripts/competitor_analyzer.py
#!/usr/bin/env python3
"""
X/Twitter Competitor Analyzer — Analyze competitor profiles for content strategy insights.
Takes competitor handles and available data, produces a competitive
intelligence report with content patterns, engagement strategies, and gaps.
Usage:
python3 competitor_analyzer.py --handles @user1 @user2 @user3
python3 competitor_analyzer.py --handles @user1 --followers 50000 --niche "AI"
python3 competitor_analyzer.py --import data.json
"""
import argparse
import json
import sys
from dataclasses import dataclass, field, asdict
from typing import Optional
@dataclass
class CompetitorProfile:
handle: str
followers: int = 0
following: int = 0
posts_per_week: float = 0
avg_likes: float = 0
avg_replies: float = 0
avg_retweets: float = 0
thread_frequency: str = "" # daily, weekly, rarely
top_topics: list = field(default_factory=list)
content_mix: dict = field(default_factory=dict) # format: percentage
posting_times: list = field(default_factory=list)
bio: str = ""
notes: str = ""
@dataclass
class CompetitiveInsight:
category: str
finding: str
opportunity: str
priority: str # HIGH, MEDIUM, LOW
def calculate_engagement_rate(profile: CompetitorProfile) -> float:
if profile.followers <= 0:
return 0
total_engagement = profile.avg_likes + profile.avg_replies + profile.avg_retweets
return (total_engagement / profile.followers) * 100
def analyze_competitors(competitors: list) -> list:
insights = []
# Engagement comparison
engagement_rates = []
for c in competitors:
er = calculate_engagement_rate(c)
engagement_rates.append((c.handle, er))
if engagement_rates:
top = max(engagement_rates, key=lambda x: x[1])
if top[1] > 0:
insights.append(CompetitiveInsight(
"Engagement", f"Highest engagement: {top[0]} ({top[1]:.2f}%)",
"Study their top posts — what format and topics drive replies?",
"HIGH"
))
# Posting frequency
frequencies = [(c.handle, c.posts_per_week) for c in competitors if c.posts_per_week > 0]
if frequencies:
avg_freq = sum(f for _, f in frequencies) / len(frequencies)
insights.append(CompetitiveInsight(
"Frequency", f"Average posting: {avg_freq:.0f}/week across competitors",
f"Match or exceed {avg_freq:.0f} posts/week to compete for mindshare",
"HIGH"
))
# Thread usage
thread_users = [c.handle for c in competitors if c.thread_frequency in ("daily", "weekly")]
if thread_users:
insights.append(CompetitiveInsight(
"Format", f"Active thread users: {', '.join(thread_users)}",
"Threads are a proven growth lever in your niche. Publish 2-3/week minimum.",
"HIGH"
))
# Reply engagement
reply_heavy = [(c.handle, c.avg_replies) for c in competitors if c.avg_replies > c.avg_likes * 0.3]
if reply_heavy:
names = [h for h, _ in reply_heavy]
insights.append(CompetitiveInsight(
"Community", f"High reply ratios: {', '.join(names)}",
"These accounts build community through conversation. Ask more questions in your tweets.",
"MEDIUM"
))
# Follower/following ratio
for c in competitors:
if c.followers > 0 and c.following > 0:
ratio = c.followers / c.following
if ratio > 10:
insights.append(CompetitiveInsight(
"Authority", f"{c.handle} has {ratio:.0f}x follower/following ratio",
"Strong authority signal — they attract followers without follow-backs",
"LOW"
))
# Topic gaps
all_topics = []
for c in competitors:
all_topics.extend(c.top_topics)
if all_topics:
from collections import Counter
common = Counter(all_topics).most_common(5)
insights.append(CompetitiveInsight(
"Topics", f"Most covered topics: {', '.join(t for t, _ in common)}",
"Cover these topics to compete, but find unique angles. What are they NOT covering?",
"MEDIUM"
))
return insights
def print_report(competitors: list, insights: list):
print(f"\n{'='*70}")
print(f" COMPETITIVE ANALYSIS REPORT")
print(f"{'='*70}")
# Profile summary table
print(f"\n {'Handle':<20} {'Followers':>10} {'Posts/wk':>10} {'Eng Rate':>10}")
print(f" {'─'*20} {'─'*10} {'─'*10} {'─'*10}")
for c in competitors:
er = calculate_engagement_rate(c)
print(f" {c.handle:<20} {c.followers:>10,} {c.posts_per_week:>10.0f} {er:>9.2f}%")
# Insights
if insights:
print(f"\n {'─'*66}")
print(f" KEY INSIGHTS\n")
priority_order = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
sorted_insights = sorted(insights, key=lambda x: priority_order.get(x.priority, 3))
for i in sorted_insights:
icon = {"HIGH": "🔴", "MEDIUM": "🟡", "LOW": "⚪"}.get(i.priority, "❓")
print(f" {icon} [{i.category}] {i.finding}")
print(f" → {i.opportunity}")
print()
# Action items
print(f" {'─'*66}")
print(f" NEXT STEPS\n")
print(f" 1. Search each competitor's profile on X — note their pinned tweet and bio")
print(f" 2. Read their last 20 posts — categorize by format and topic")
print(f" 3. Identify their top 3 performing posts — what made them work?")
print(f" 4. Find gaps — what topics do they NOT cover that you can own?")
print(f" 5. Set engagement targets based on their metrics as benchmarks")
print(f"\n{'='*70}\n")
def main():
parser = argparse.ArgumentParser(
description="Analyze X/Twitter competitors for content strategy insights",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="""
Examples:
%(prog)s --handles @user1 @user2
%(prog)s --import competitors.json
JSON format for --import:
[{"handle": "@user1", "followers": 50000, "posts_per_week": 14, ...}]
""")
parser.add_argument("--handles", nargs="+", default=[], help="Competitor handles")
parser.add_argument("--import", dest="import_file", help="Import from JSON file")
parser.add_argument("--json", action="store_true", help="Output JSON")
args = parser.parse_args()
competitors = []
if args.import_file:
with open(args.import_file) as f:
data = json.load(f)
for item in data:
competitors.append(CompetitorProfile(**item))
elif args.handles:
for handle in args.handles:
if not handle.startswith("@"):
handle = f"@{handle}"
competitors.append(CompetitorProfile(handle=handle))
if all(c.followers == 0 for c in competitors):
print(f"\n ℹ️ Handles registered: {', '.join(c.handle for c in competitors)}")
print(f" To get full analysis, provide data via JSON import:")
print(f" 1. Research each profile on X")
print(f" 2. Create a JSON file with follower counts, posting frequency, etc.")
print(f" 3. Run: {sys.argv[0]} --import data.json")
print(f"\n Example JSON:")
example = [asdict(CompetitorProfile(
handle="@example",
followers=25000,
following=1200,
posts_per_week=14,
avg_likes=150,
avg_replies=30,
avg_retweets=20,
thread_frequency="weekly",
top_topics=["AI", "startups", "engineering"],
))]
print(f" {json.dumps(example, indent=2)}")
print()
return
if not competitors:
print("Error: provide --handles or --import", file=sys.stderr)
sys.exit(1)
insights = analyze_competitors(competitors)
if args.json:
print(json.dumps({
"competitors": [asdict(c) for c in competitors],
"insights": [asdict(i) for i in insights],
}, indent=2))
else:
print_report(competitors, insights)
if __name__ == "__main__":
main()
FILE:scripts/content_planner.py
#!/usr/bin/env python3
"""
X/Twitter Content Planner — Generate weekly posting calendars.
Creates structured content plans with topic suggestions, format mix,
optimal posting times, and engagement targets.
Usage:
python3 content_planner.py --niche "AI engineering" --frequency 5 --weeks 2
python3 content_planner.py --niche "SaaS growth" --frequency 3 --weeks 1 --json
"""
import argparse
import json
import sys
from datetime import datetime, timedelta
from dataclasses import dataclass, field, asdict
CONTENT_FORMATS = {
"atomic_tweet": {"growth_weight": 0.3, "effort": "low", "description": "Single tweet — observation, tip, or hot take"},
"thread": {"growth_weight": 0.35, "effort": "high", "description": "5-12 tweet deep dive — highest reach potential"},
"question": {"growth_weight": 0.15, "effort": "low", "description": "Engagement bait — drives replies"},
"quote_tweet": {"growth_weight": 0.10, "effort": "low", "description": "Add value to someone else's content"},
"reply_session": {"growth_weight": 0.10, "effort": "medium", "description": "30 min focused engagement on target accounts"},
}
OPTIMAL_TIMES = {
"weekday": ["07:00-08:00", "12:00-13:00", "17:00-18:00", "20:00-21:00"],
"weekend": ["09:00-10:00", "14:00-15:00", "19:00-20:00"],
}
TOPIC_ANGLES = [
"Lessons learned (personal experience)",
"Framework/system breakdown",
"Tool recommendation (with honest take)",
"Myth busting (challenge common belief)",
"Behind the scenes (process, workflow)",
"Industry trend analysis",
"Beginner guide (explain like I'm 5)",
"Comparison (X vs Y — which is better?)",
"Prediction (what's coming next)",
"Case study (real example with numbers)",
"Mistake I made (vulnerability + lesson)",
"Quick tip (tactical, immediately useful)",
"Controversial take (spicy but defensible)",
"Curated list (best resources, tools, accounts)",
]
@dataclass
class DayPlan:
date: str
day_of_week: str
posts: list = field(default_factory=list)
engagement_target: str = ""
@dataclass
class PostSlot:
time: str
format: str
topic_angle: str
topic_suggestion: str
notes: str = ""
@dataclass
class WeekPlan:
week_number: int
start_date: str
end_date: str
days: list = field(default_factory=list)
thread_count: int = 0
total_posts: int = 0
focus_theme: str = ""
def generate_plan(niche: str, posts_per_day: int, weeks: int, start_date: datetime) -> list:
plans = []
angle_idx = 0
time_idx = 0
for week in range(weeks):
week_start = start_date + timedelta(weeks=week)
week_end = week_start + timedelta(days=6)
week_plan = WeekPlan(
week_number=week + 1,
start_date=week_start.strftime("%Y-%m-%d"),
end_date=week_end.strftime("%Y-%m-%d"),
focus_theme=TOPIC_ANGLES[week % len(TOPIC_ANGLES)],
)
for day in range(7):
current = week_start + timedelta(days=day)
day_name = current.strftime("%A")
is_weekend = day >= 5
times = OPTIMAL_TIMES["weekend" if is_weekend else "weekday"]
actual_posts = max(1, posts_per_day - (1 if is_weekend else 0))
day_plan = DayPlan(
date=current.strftime("%Y-%m-%d"),
day_of_week=day_name,
engagement_target="15 min reply session" if is_weekend else "30 min reply session",
)
for p in range(actual_posts):
# Determine format based on day position
if day in [1, 3] and p == 0: # Tue/Thu first slot = thread
fmt = "thread"
elif p == actual_posts - 1 and not is_weekend:
fmt = "question" # Last post = engagement driver
elif day == 4 and p == 0: # Friday first = quote tweet
fmt = "quote_tweet"
else:
fmt = "atomic_tweet"
angle = TOPIC_ANGLES[angle_idx % len(TOPIC_ANGLES)]
angle_idx += 1
slot = PostSlot(
time=times[p % len(times)],
format=fmt,
topic_angle=angle,
topic_suggestion=f"{angle} about {niche}",
notes="Pin if performs well" if fmt == "thread" else "",
)
day_plan.posts.append(asdict(slot))
if fmt == "thread":
week_plan.thread_count += 1
week_plan.total_posts += 1
week_plan.days.append(asdict(day_plan))
plans.append(asdict(week_plan))
return plans
def print_plan(plans: list, niche: str):
print(f"\n{'='*70}")
print(f" X/TWITTER CONTENT PLAN — {niche.upper()}")
print(f"{'='*70}")
for week in plans:
print(f"\n WEEK {week['week_number']} ({week['start_date']} to {week['end_date']})")
print(f" Theme: {week['focus_theme']}")
print(f" Posts: {week['total_posts']} | Threads: {week['thread_count']}")
print(f" {'─'*66}")
for day in week['days']:
print(f"\n {day['day_of_week']:9} {day['date']}")
for post in day['posts']:
fmt_icon = {
"thread": "🧵",
"atomic_tweet": "💬",
"question": "❓",
"quote_tweet": "🔄",
"reply_session": "💬",
}.get(post['format'], "📝")
print(f" {fmt_icon} {post['time']:12} [{post['format']:<14}] {post['topic_angle']}")
if post['notes']:
print(f" ℹ️ {post['notes']}")
print(f" 📊 Engagement: {day['engagement_target']}")
print(f"\n{'='*70}")
print(f" WEEKLY TARGETS")
print(f" • Reply to 10+ accounts in your niche daily")
print(f" • Quote tweet 2-3 relevant posts per week")
print(f" • Update pinned tweet if a thread outperforms current pin")
print(f" • Review analytics every Sunday — double down on what works")
print(f"{'='*70}\n")
def main():
parser = argparse.ArgumentParser(
description="Generate X/Twitter content calendars",
formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--niche", required=True, help="Your content niche")
parser.add_argument("--frequency", type=int, default=3, help="Posts per day (default: 3)")
parser.add_argument("--weeks", type=int, default=2, help="Weeks to plan (default: 2)")
parser.add_argument("--start", default="", help="Start date YYYY-MM-DD (default: next Monday)")
parser.add_argument("--json", action="store_true", help="Output JSON")
args = parser.parse_args()
if args.start:
start = datetime.strptime(args.start, "%Y-%m-%d")
else:
today = datetime.now()
days_until_monday = (7 - today.weekday()) % 7
if days_until_monday == 0:
days_until_monday = 7
start = today + timedelta(days=days_until_monday)
plans = generate_plan(args.niche, args.frequency, args.weeks, start)
if args.json:
print(json.dumps(plans, indent=2))
else:
print_plan(plans, args.niche)
if __name__ == "__main__":
main()
FILE:scripts/growth_tracker.py
#!/usr/bin/env python3
"""
X/Twitter Growth Tracker — Track and analyze account growth over time.
Stores periodic snapshots of account metrics and calculates growth trends,
engagement patterns, and milestone projections.
Usage:
python3 growth_tracker.py --record --handle @user --followers 5200 --eng-rate 2.1
python3 growth_tracker.py --report --handle @user
python3 growth_tracker.py --report --handle @user --period 30d --json
python3 growth_tracker.py --milestone --handle @user --target 10000
"""
import argparse
import json
import os
import sys
from datetime import datetime, timedelta
from pathlib import Path
DATA_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", ".growth-data")
def get_data_file(handle: str) -> str:
clean = handle.lstrip("@").lower()
os.makedirs(DATA_DIR, exist_ok=True)
return os.path.join(DATA_DIR, f"{clean}.jsonl")
def record_snapshot(handle: str, followers: int, following: int = 0,
eng_rate: float = 0, posts_week: float = 0, notes: str = ""):
entry = {
"timestamp": datetime.now().isoformat(),
"handle": handle,
"followers": followers,
"following": following,
"engagement_rate": eng_rate,
"posts_per_week": posts_week,
"notes": notes,
}
filepath = get_data_file(handle)
with open(filepath, "a") as f:
f.write(json.dumps(entry) + "\n")
return entry
def load_snapshots(handle: str, period_days: int = 0) -> list:
filepath = get_data_file(handle)
if not os.path.exists(filepath):
return []
entries = []
cutoff = None
if period_days > 0:
cutoff = datetime.now() - timedelta(days=period_days)
with open(filepath) as f:
for line in f:
line = line.strip()
if not line:
continue
entry = json.loads(line)
if cutoff:
ts = datetime.fromisoformat(entry["timestamp"])
if ts < cutoff:
continue
entries.append(entry)
return entries
def generate_report(handle: str, entries: list) -> dict:
if not entries:
return {"handle": handle, "error": "No data found"}
report = {
"handle": handle,
"data_points": len(entries),
"first_record": entries[0]["timestamp"],
"last_record": entries[-1]["timestamp"],
"current_followers": entries[-1]["followers"],
}
if len(entries) >= 2:
first = entries[0]
last = entries[-1]
follower_change = last["followers"] - first["followers"]
days_span = (datetime.fromisoformat(last["timestamp"]) -
datetime.fromisoformat(first["timestamp"])).days
days_span = max(days_span, 1)
report["follower_change"] = follower_change
report["days_tracked"] = days_span
report["daily_growth"] = round(follower_change / days_span, 1)
report["weekly_growth"] = round((follower_change / days_span) * 7, 1)
report["monthly_projection"] = round((follower_change / days_span) * 30)
if first["followers"] > 0:
pct_change = ((last["followers"] - first["followers"]) / first["followers"]) * 100
report["growth_percent"] = round(pct_change, 1)
# Engagement trend
eng_rates = [e["engagement_rate"] for e in entries if e.get("engagement_rate", 0) > 0]
if len(eng_rates) >= 2:
mid = len(eng_rates) // 2
first_half_avg = sum(eng_rates[:mid]) / mid
second_half_avg = sum(eng_rates[mid:]) / (len(eng_rates) - mid)
report["engagement_trend"] = "improving" if second_half_avg > first_half_avg else "declining"
report["avg_engagement_rate"] = round(sum(eng_rates) / len(eng_rates), 2)
return report
def project_milestone(handle: str, entries: list, target: int) -> dict:
if len(entries) < 2:
return {"error": "Need at least 2 data points for projection"}
current = entries[-1]["followers"]
if current >= target:
return {"handle": handle, "target": target, "status": "Already reached!"}
first = entries[0]
last = entries[-1]
days_span = (datetime.fromisoformat(last["timestamp"]) -
datetime.fromisoformat(first["timestamp"])).days
days_span = max(days_span, 1)
daily_growth = (last["followers"] - first["followers"]) / days_span
if daily_growth <= 0:
return {"handle": handle, "target": target, "status": "Not growing — can't project",
"daily_growth": round(daily_growth, 1)}
remaining = target - current
days_needed = remaining / daily_growth
target_date = datetime.now() + timedelta(days=days_needed)
return {
"handle": handle,
"current": current,
"target": target,
"remaining": remaining,
"daily_growth": round(daily_growth, 1),
"days_needed": round(days_needed),
"projected_date": target_date.strftime("%Y-%m-%d"),
}
def print_report(report: dict):
print(f"\n{'='*60}")
print(f" GROWTH REPORT — {report['handle']}")
print(f"{'='*60}")
if "error" in report:
print(f"\n ⚠️ {report['error']}")
print(f" Record data first: python3 growth_tracker.py --record --handle {report['handle']} --followers N")
print()
return
print(f"\n Current followers: {report['current_followers']:,}")
print(f" Data points: {report['data_points']}")
print(f" Tracking since: {report['first_record'][:10]}")
if "follower_change" in report:
change_icon = "📈" if report["follower_change"] > 0 else "📉" if report["follower_change"] < 0 else "➡️"
print(f"\n {change_icon} Change: {report['follower_change']:+,} followers over {report['days_tracked']} days")
print(f" Daily avg: {report.get('daily_growth', 0):+.1f}/day")
print(f" Weekly avg: {report.get('weekly_growth', 0):+.1f}/week")
print(f" 30-day projection: {report.get('monthly_projection', 0):+,}")
if "growth_percent" in report:
print(f" Growth rate: {report['growth_percent']:+.1f}%")
if "engagement_trend" in report:
trend_icon = "📈" if report["engagement_trend"] == "improving" else "📉"
print(f" Engagement: {trend_icon} {report['engagement_trend']} (avg {report['avg_engagement_rate']}%)")
print(f"\n{'='*60}\n")
def main():
parser = argparse.ArgumentParser(
description="Track X/Twitter account growth over time",
formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--record", action="store_true", help="Record a new snapshot")
parser.add_argument("--report", action="store_true", help="Generate growth report")
parser.add_argument("--milestone", action="store_true", help="Project when target will be reached")
parser.add_argument("--handle", required=True, help="X handle")
parser.add_argument("--followers", type=int, default=0, help="Current follower count")
parser.add_argument("--following", type=int, default=0, help="Current following count")
parser.add_argument("--eng-rate", type=float, default=0, help="Current engagement rate (pct)")
parser.add_argument("--posts-week", type=float, default=0, help="Posts per week")
parser.add_argument("--notes", default="", help="Notes for this snapshot")
parser.add_argument("--period", default="all", help="Report period: 7d, 30d, 90d, all")
parser.add_argument("--target", type=int, default=0, help="Follower milestone target")
parser.add_argument("--json", action="store_true", help="Output JSON")
args = parser.parse_args()
if not args.handle.startswith("@"):
args.handle = f"@{args.handle}"
if args.record:
if args.followers <= 0:
print("Error: --followers required for recording", file=sys.stderr)
sys.exit(1)
entry = record_snapshot(args.handle, args.followers, args.following,
args.eng_rate, args.posts_week, args.notes)
if args.json:
print(json.dumps(entry, indent=2))
else:
print(f" ✅ Recorded: {args.handle} — {args.followers:,} followers")
print(f" File: {get_data_file(args.handle)}")
elif args.report:
period_days = 0
if args.period != "all":
period_days = int(args.period.rstrip("d"))
entries = load_snapshots(args.handle, period_days)
report = generate_report(args.handle, entries)
if args.json:
print(json.dumps(report, indent=2))
else:
print_report(report)
elif args.milestone:
if args.target <= 0:
print("Error: --target required for milestone projection", file=sys.stderr)
sys.exit(1)
entries = load_snapshots(args.handle)
result = project_milestone(args.handle, entries, args.target)
if args.json:
print(json.dumps(result, indent=2))
else:
if "error" in result:
print(f" ⚠️ {result['error']}")
elif "status" in result and "days_needed" not in result:
print(f" 🎉 {result['status']}")
else:
print(f"\n 🎯 Milestone Projection: {result['handle']}")
print(f" Current: {result['current']:,}")
print(f" Target: {result['target']:,}")
print(f" Gap: {result['remaining']:,}")
print(f" Growth: {result['daily_growth']:+.1f}/day")
print(f" ETA: {result['projected_date']} (~{result['days_needed']} days)")
print()
else:
parser.print_help()
if __name__ == "__main__":
main()
FILE:scripts/profile_auditor.py
#!/usr/bin/env python3
"""
X/Twitter Profile Auditor — Audit any X profile for growth readiness.
Checks bio quality, pinned tweet, posting patterns, and provides
actionable recommendations. Works without API access by analyzing
profile data you provide or scraping public info via web search.
Usage:
python3 profile_auditor.py --handle @username
python3 profile_auditor.py --handle @username --json
python3 profile_auditor.py --bio "current bio text" --followers 5000 --posts-per-week 10
"""
import argparse
import json
import re
import sys
from dataclasses import dataclass, field, asdict
from typing import Optional
@dataclass
class ProfileData:
handle: str = ""
bio: str = ""
followers: int = 0
following: int = 0
posts_per_week: float = 0
reply_ratio: float = 0 # % of posts that are replies
thread_ratio: float = 0 # % of posts that are threads
has_pinned: bool = False
pinned_age_days: int = 0
has_link: bool = False
has_newsletter: bool = False
avg_engagement_rate: float = 0 # likes+replies+rts / followers
@dataclass
class AuditFinding:
area: str
status: str # GOOD, WARN, CRITICAL
message: str
fix: str = ""
@dataclass
class AuditReport:
handle: str
score: int = 0
max_score: int = 100
grade: str = ""
findings: list = field(default_factory=list)
recommendations: list = field(default_factory=list)
def audit_bio(profile: ProfileData) -> list:
findings = []
bio = profile.bio.strip()
if not bio:
findings.append(AuditFinding("Bio", "CRITICAL", "No bio provided for audit",
"Provide bio text with --bio flag"))
return findings
# Length check
if len(bio) < 30:
findings.append(AuditFinding("Bio", "WARN", f"Bio too short ({len(bio)} chars)",
"Aim for 100-160 characters with clear value prop"))
elif len(bio) > 160:
findings.append(AuditFinding("Bio", "WARN", f"Bio may be too long ({len(bio)} chars)",
"Keep under 160 chars for readability"))
else:
findings.append(AuditFinding("Bio", "GOOD", f"Bio length OK ({len(bio)} chars)"))
# Hashtag check
hashtags = re.findall(r'#\w+', bio)
if hashtags:
findings.append(AuditFinding("Bio", "WARN", f"Hashtags in bio ({', '.join(hashtags)})",
"Remove hashtags — signals amateur. Use plain text."))
else:
findings.append(AuditFinding("Bio", "GOOD", "No hashtags in bio"))
# Buzzword check
buzzwords = ['entrepreneur', 'guru', 'ninja', 'rockstar', 'visionary', 'hustler',
'thought leader', 'serial entrepreneur', 'dreamer', 'doer']
found = [bw for bw in buzzwords if bw.lower() in bio.lower()]
if found:
findings.append(AuditFinding("Bio", "WARN", f"Buzzwords detected: {', '.join(found)}",
"Replace with specific, concrete descriptions of what you do"))
# Specificity check — pipes and slashes often signal unfocused bios
if bio.count('|') >= 3 or bio.count('/') >= 3:
findings.append(AuditFinding("Bio", "WARN", "Bio may lack focus (too many roles/identities)",
"Lead with ONE clear identity. What's the #1 thing you want to be known for?"))
# Social proof check
proof_patterns = [r'\d+[kKmM]?\+?\s*(followers|subscribers|readers|users|customers)',
r'(founder|ceo|cto|vp|head|director|lead)\s+(of|at|@)',
r'(author|writer)\s+of', r'featured\s+in', r'ex-\w+']
has_proof = any(re.search(p, bio, re.IGNORECASE) for p in proof_patterns)
if has_proof:
findings.append(AuditFinding("Bio", "GOOD", "Social proof detected"))
else:
findings.append(AuditFinding("Bio", "WARN", "No obvious social proof in bio",
"Add a credential: title, metric, brand association, or achievement"))
# CTA/Link check
if profile.has_link:
findings.append(AuditFinding("Bio", "GOOD", "Profile has a link"))
else:
findings.append(AuditFinding("Bio", "WARN", "No link in profile",
"Add a link to newsletter, product, or portfolio"))
return findings
def audit_activity(profile: ProfileData) -> list:
findings = []
# Posting frequency
if profile.posts_per_week <= 0:
findings.append(AuditFinding("Activity", "CRITICAL", "No posting data provided",
"Provide --posts-per-week estimate"))
elif profile.posts_per_week < 3:
findings.append(AuditFinding("Activity", "CRITICAL",
f"Very low posting ({profile.posts_per_week:.0f}/week)",
"Minimum 7 posts/week (1/day). Aim for 14-21."))
elif profile.posts_per_week < 7:
findings.append(AuditFinding("Activity", "WARN",
f"Low posting ({profile.posts_per_week:.0f}/week)",
"Aim for 2-3 posts per day for consistent growth"))
elif profile.posts_per_week < 21:
findings.append(AuditFinding("Activity", "GOOD",
f"Good posting cadence ({profile.posts_per_week:.0f}/week)"))
else:
findings.append(AuditFinding("Activity", "GOOD",
f"High posting cadence ({profile.posts_per_week:.0f}/week)"))
# Reply ratio
if profile.reply_ratio > 0:
if profile.reply_ratio < 0.2:
findings.append(AuditFinding("Activity", "WARN",
f"Low reply ratio ({profile.reply_ratio:.0%})",
"Aim for 30%+ replies. Engage with others, don't just broadcast."))
elif profile.reply_ratio >= 0.3:
findings.append(AuditFinding("Activity", "GOOD",
f"Healthy reply ratio ({profile.reply_ratio:.0%})"))
# Follower/following ratio
if profile.followers > 0 and profile.following > 0:
ratio = profile.followers / profile.following
if ratio < 0.5:
findings.append(AuditFinding("Profile", "WARN",
f"Low follower/following ratio ({ratio:.1f}x)",
"Unfollow inactive accounts. Ratio should trend toward 2:1+"))
elif ratio >= 2:
findings.append(AuditFinding("Profile", "GOOD",
f"Healthy follower/following ratio ({ratio:.1f}x)"))
# Pinned tweet
if profile.has_pinned:
if profile.pinned_age_days > 30:
findings.append(AuditFinding("Profile", "WARN",
f"Pinned tweet is {profile.pinned_age_days} days old",
"Update pinned tweet monthly with your latest best content"))
else:
findings.append(AuditFinding("Profile", "GOOD", "Pinned tweet is recent"))
else:
findings.append(AuditFinding("Profile", "WARN", "No pinned tweet",
"Pin your best-performing tweet or thread. It's your landing page."))
return findings
def calculate_score(findings: list) -> tuple:
total = len(findings)
if total == 0:
return 0, "F"
good = sum(1 for f in findings if f.status == "GOOD")
score = int((good / total) * 100)
if score >= 90:
grade = "A"
elif score >= 75:
grade = "B"
elif score >= 60:
grade = "C"
elif score >= 40:
grade = "D"
else:
grade = "F"
return score, grade
def generate_recommendations(findings: list, profile: ProfileData) -> list:
recs = []
criticals = [f for f in findings if f.status == "CRITICAL"]
warns = [f for f in findings if f.status == "WARN"]
for f in criticals:
if f.fix:
recs.append(f"🔴 {f.fix}")
for f in warns[:3]: # Top 3 warnings
if f.fix:
recs.append(f"🟡 {f.fix}")
# Stage-specific advice
if profile.followers < 1000:
recs.append("📈 Growth phase: Focus 70% on replies to larger accounts, 30% on your own posts")
elif profile.followers < 10000:
recs.append("📈 Momentum phase: 2-3 threads/week + daily engagement. Start a recurring series.")
else:
recs.append("📈 Scale phase: Leverage audience with cross-platform repurposing + newsletter growth")
return recs
def main():
parser = argparse.ArgumentParser(
description="Audit an X/Twitter profile for growth readiness",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="""
Examples:
%(prog)s --handle @rezarezvani --bio "CTO building AI products" --followers 5000
%(prog)s --bio "Entrepreneur | Dreamer | Hustle" --followers 200 --posts-per-week 3
%(prog)s --handle @example --followers 50000 --posts-per-week 21 --reply-ratio 0.4 --json
""")
parser.add_argument("--handle", default="@unknown", help="X handle")
parser.add_argument("--bio", default="", help="Current bio text")
parser.add_argument("--followers", type=int, default=0, help="Follower count")
parser.add_argument("--following", type=int, default=0, help="Following count")
parser.add_argument("--posts-per-week", type=float, default=0, help="Average posts per week")
parser.add_argument("--reply-ratio", type=float, default=0, help="Fraction of posts that are replies (0-1)")
parser.add_argument("--has-pinned", action="store_true", help="Has a pinned tweet")
parser.add_argument("--pinned-age-days", type=int, default=0, help="Age of pinned tweet in days")
parser.add_argument("--has-link", action="store_true", help="Has link in profile")
parser.add_argument("--json", action="store_true", help="Output JSON")
args = parser.parse_args()
profile = ProfileData(
handle=args.handle,
bio=args.bio,
followers=args.followers,
following=args.following,
posts_per_week=args.posts_per_week,
reply_ratio=args.reply_ratio,
has_pinned=args.has_pinned,
pinned_age_days=args.pinned_age_days,
has_link=args.has_link,
)
findings = audit_bio(profile) + audit_activity(profile)
score, grade = calculate_score(findings)
recs = generate_recommendations(findings, profile)
report = AuditReport(
handle=profile.handle,
score=score,
grade=grade,
findings=[asdict(f) for f in findings],
recommendations=recs,
)
if args.json:
print(json.dumps(asdict(report), indent=2))
else:
print(f"\n{'='*60}")
print(f" X PROFILE AUDIT — {report.handle}")
print(f"{'='*60}")
print(f"\n Score: {report.score}/100 (Grade: {report.grade})\n")
for f in findings:
icon = {"GOOD": "✅", "WARN": "⚠️", "CRITICAL": "🔴"}.get(f.status, "❓")
print(f" {icon} [{f.area}] {f.message}")
if f.fix and f.status != "GOOD":
print(f" → {f.fix}")
if recs:
print(f"\n {'─'*56}")
print(f" TOP RECOMMENDATIONS\n")
for i, r in enumerate(recs, 1):
print(f" {i}. {r}")
print(f"\n{'='*60}\n")
if __name__ == "__main__":
main()
FILE:scripts/tweet_composer.py
#!/usr/bin/env python3
"""
Tweet Composer — Generate structured tweets and threads with proven hook patterns.
Provides templates, character counting, thread formatting, and hook generation
for different content types. No API required — pure content scaffolding.
Usage:
python3 tweet_composer.py --type tweet --topic "AI in healthcare"
python3 tweet_composer.py --type thread --topic "lessons from scaling" --tweets 8
python3 tweet_composer.py --type hooks --topic "startup mistakes" --count 10
python3 tweet_composer.py --validate "your tweet text here"
"""
import argparse
import json
import sys
import textwrap
from dataclasses import dataclass, field, asdict
from typing import Optional
MAX_TWEET_CHARS = 280
HOOK_PATTERNS = {
"listicle": [
"{n} {topic} that changed how I {verb}:",
"The {n} biggest mistakes in {topic}:",
"{n} {topic} most people don't know about:",
"I spent {time} studying {topic}. Here are {n} lessons:",
"{n} signs your {topic} needs work:",
],
"contrarian": [
"Unpopular opinion: {claim}",
"Hot take: {claim}",
"Everyone says {common_belief}. They're wrong.",
"Stop {common_action}. Here's what to do instead:",
"The {topic} advice you keep hearing is backwards.",
],
"story": [
"I {did_thing} and it completely changed my {outcome}.",
"Last {timeframe}, I made a mistake with {topic}. Here's what happened:",
"3 years ago I was {before_state}. Now I'm {after_state}. Here's the playbook:",
"I almost {near_miss}. Then I discovered {topic}.",
"The best {topic} advice I ever got came from {unexpected_source}.",
],
"observation": [
"{topic} is underrated. Here's why:",
"Nobody talks about this part of {topic}:",
"The gap between {thing_a} and {thing_b} is where the money is.",
"If you're struggling with {topic}, you're probably {mistake}.",
"The secret to {topic} isn't what you think.",
],
"framework": [
"The {name} framework for {topic} (save this):",
"How to {outcome} in {timeframe} (step by step):",
"{topic} explained in 60 seconds:",
"The only {n} things that matter for {topic}:",
"A simple system for {topic} that actually works:",
],
"question": [
"What's the most underrated {topic}?",
"If you could only {do_one_thing} for {topic}, what would it be?",
"What {topic} advice would you give your younger self?",
"Real question: why do most people {common_mistake}?",
"What's one {topic} that completely changed your perspective?",
],
}
THREAD_STRUCTURE = """
Thread Outline: {topic}
{'='*50}
Tweet 1 (HOOK — most important):
Pattern: {hook_pattern}
Draft: {hook_draft}
Chars: {hook_chars}/280
Tweet 2 (CONTEXT):
Purpose: Set up why this matters
Suggestion: "Here's what most people get wrong about {topic}:"
OR: "I spent [time] learning this. Here's the breakdown:"
Tweets 3-{n} (BODY — one idea per tweet):
{body_suggestions}
Tweet {n_plus_1} (CLOSE):
Purpose: Summarize + CTA
Suggestion: "TL;DR:\\n\\n[3 bullet summary]\\n\\nFollow @handle for more on {topic}"
Reply to Tweet 1 (ENGAGEMENT BAIT):
Purpose: Resurface the thread
Suggestion: "What's your experience with {topic}? Drop it below 👇"
"""
@dataclass
class TweetDraft:
text: str
char_count: int
over_limit: bool
warnings: list = field(default_factory=list)
def validate_tweet(text: str) -> TweetDraft:
"""Validate a tweet and return analysis."""
char_count = len(text)
over_limit = char_count > MAX_TWEET_CHARS
warnings = []
if over_limit:
warnings.append(f"Over limit by {char_count - MAX_TWEET_CHARS} characters")
# Check for links in body
import re
if re.search(r'https?://\S+', text):
warnings.append("Contains URL — consider moving link to reply (hurts reach)")
# Check for hashtags
hashtags = re.findall(r'#\w+', text)
if len(hashtags) > 2:
warnings.append(f"Too many hashtags ({len(hashtags)}) — max 1-2, ideally 0")
elif len(hashtags) > 0:
warnings.append(f"Has {len(hashtags)} hashtag(s) — consider removing for cleaner look")
# Check for @mentions at start
if text.startswith('@'):
warnings.append("Starts with @ — will be treated as reply, not shown in timeline")
# Readability
lines = text.strip().split('\n')
long_lines = [l for l in lines if len(l) > 70]
if long_lines:
warnings.append("Long unbroken lines — add line breaks for mobile readability")
return TweetDraft(text=text, char_count=char_count, over_limit=over_limit, warnings=warnings)
def generate_hooks(topic: str, count: int = 10) -> list:
"""Generate hook variations for a topic."""
hooks = []
for pattern_type, patterns in HOOK_PATTERNS.items():
for p in patterns:
hook = p.replace("{topic}", topic).replace("{n}", "7").replace(
"{time}", "6 months").replace("{timeframe}", "month").replace(
"{claim}", f"{topic} is overrated").replace(
"{common_belief}", f"{topic} is simple").replace(
"{common_action}", f"overthinking {topic}").replace(
"{outcome}", "approach").replace("{verb}", "think").replace(
"{name}", "3-Step").replace("{did_thing}", f"changed my {topic} strategy").replace(
"{before_state}", "stuck").replace("{after_state}", "thriving").replace(
"{near_miss}", f"gave up on {topic}").replace(
"{unexpected_source}", "a complete beginner").replace(
"{thing_a}", "theory").replace("{thing_b}", "execution").replace(
"{mistake}", "overcomplicating it").replace(
"{common_mistake}", f"ignore {topic}").replace(
"{do_one_thing}", "change one thing").replace(
"{common_action}", f"overthinking {topic}")
hooks.append({"type": pattern_type, "hook": hook, "chars": len(hook)})
if len(hooks) >= count:
return hooks
return hooks[:count]
def generate_thread_outline(topic: str, num_tweets: int = 8) -> str:
"""Generate a thread structure outline."""
hooks = generate_hooks(topic, 3)
best_hook = hooks[0]["hook"] if hooks else f"Everything I know about {topic}:"
body = []
suggestions = [
"Key insight or surprising fact",
"Common mistake people make",
"The counterintuitive truth",
"A practical example or case study",
"The framework or system",
"Implementation steps",
"Results or evidence",
"The nuance most people miss",
]
for i, s in enumerate(suggestions[:num_tweets - 3], 3):
body.append(f" Tweet {i}: [{s}]")
body_text = "\n".join(body)
return f"""
{'='*60}
THREAD OUTLINE: {topic}
{'='*60}
Tweet 1 (HOOK):
"{best_hook}"
Chars: {len(best_hook)}/280
Tweet 2 (CONTEXT):
"Here's what most people get wrong about {topic}:"
{body_text}
Tweet {num_tweets - 1} (CLOSE):
"TL;DR:
• [Key takeaway 1]
• [Key takeaway 2]
• [Key takeaway 3]
Follow for more on {topic}"
Reply to Tweet 1 (BOOST):
"What's your biggest challenge with {topic}? 👇"
{'='*60}
RULES:
- Each tweet must stand alone (people read out of order)
- Max 3-4 lines per tweet (mobile readability)
- No filler tweets — cut anything that doesn't add value
- Hook tweet determines 90%% of thread performance
{'='*60}
"""
def main():
parser = argparse.ArgumentParser(
description="Generate tweets, threads, and hooks with proven patterns",
formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--type", choices=["tweet", "thread", "hooks", "validate"],
default="hooks", help="Content type to generate")
parser.add_argument("--topic", default="", help="Topic for content generation")
parser.add_argument("--tweets", type=int, default=8, help="Number of tweets in thread")
parser.add_argument("--count", type=int, default=10, help="Number of hooks to generate")
parser.add_argument("--validate", nargs="?", const="", help="Tweet text to validate")
parser.add_argument("--json", action="store_true", help="Output JSON")
args = parser.parse_args()
if args.type == "validate" or args.validate is not None:
text = args.validate or args.topic
if not text:
print("Error: provide tweet text to validate", file=sys.stderr)
sys.exit(1)
result = validate_tweet(text)
if args.json:
print(json.dumps(asdict(result), indent=2))
else:
icon = "🔴" if result.over_limit else "✅"
print(f"\n {icon} {result.char_count}/{MAX_TWEET_CHARS} characters")
if result.warnings:
for w in result.warnings:
print(f" ⚠️ {w}")
else:
print(" No issues found.")
print()
elif args.type == "hooks":
if not args.topic:
print("Error: --topic required for hook generation", file=sys.stderr)
sys.exit(1)
hooks = generate_hooks(args.topic, args.count)
if args.json:
print(json.dumps(hooks, indent=2))
else:
print(f"\n{'='*60}")
print(f" HOOK IDEAS: {args.topic}")
print(f"{'='*60}\n")
for i, h in enumerate(hooks, 1):
print(f" {i:2d}. [{h['type']:<12}] {h['hook']}")
print(f" ({h['chars']} chars)")
print()
elif args.type == "thread":
if not args.topic:
print("Error: --topic required for thread generation", file=sys.stderr)
sys.exit(1)
outline = generate_thread_outline(args.topic, args.tweets)
print(outline)
elif args.type == "tweet":
if not args.topic:
print("Error: --topic required", file=sys.stderr)
sys.exit(1)
hooks = generate_hooks(args.topic, 5)
print(f"\n 5 tweet drafts for: {args.topic}\n")
for i, h in enumerate(hooks, 1):
print(f" {i}. {h['hook']}")
print(f" ({h['chars']} chars)\n")
if __name__ == "__main__":
main()
Phân tích đầu tư và phân bổ vốn: ROI, IRR, NPV, thời gian hoàn vốn, tự xây hay mua, thuê hay mua.
--- name: business-investment-advisor description: "Business investment analysis and capital allocation advisor. Use when evaluating whether to invest in equipment, real estate, a new business, hiring, technology, or any capital expenditure. Also use for ROI calculations, IRR, NPV, payback period, build vs buy decisions, lease vs buy analysis, vendor evaluation, or deciding where to allocate limited budget for maximum return." --- # Business Investment Advisor > Originally contributed by [chad848](https://github.com/chad848) — enhanced and integrated by the claude-skills team. You are a senior business investment analyst and capital allocation advisor. Your job is to help evaluate every dollar that goes out the door — equipment purchases, hiring decisions, technology investments, real estate, vendor contracts, new business opportunities. You show the math, state the assumptions, give a clear recommendation, and flag what could go wrong. You do NOT give personal stock market or securities investment advice. This skill is for business capital allocation decisions. ## Before Starting **Check for context first:** If `company-context.md` exists, read it before asking questions. Gather this context (ask conversationally, not all at once): ### 1. Investment Details - What is the investment? (equipment, hire, software, real estate, new service line) - Total upfront cost? - Expected useful life or contract term? ### 2. Financial Projections - Expected revenue increase OR cost savings per month/year? - Ongoing costs (maintenance, subscription, salary + benefits)? - How confident are you in these estimates? (Low / Medium / High) ### 3. Context - Alternative uses for this capital (opportunity cost)? - Current cost of capital or interest rate on debt? - Any other options you're comparing this against? Work with partial data — state what you're assuming and flag it clearly. --- ## How This Skill Works ### Mode 1: Single Investment Evaluation Analyze one investment decision — calculate ROI, payback, NPV, IRR, run upside and downside scenarios, produce recommendation. ### Mode 2: Compare Multiple Options Rank and compare multiple investment options against a fixed budget — build the allocation framework, score each option, recommend priority order. ### Mode 3: Build vs Buy / Lease vs Buy / Hire vs Automate Framework-driven decision for specific trade-off scenarios with structured comparison matrix. --- ## Core Analysis Framework ### ROI (Return on Investment) `ROI = (Net Gain from Investment / Cost of Investment) × 100` - Net Gain = Total Returns - Total Costs over the analysis period - Use for quick comparisons. Limitation: ignores time value of money. ### Payback Period `Payback = Total Investment ÷ Annual Net Cash Flow` - Target: <3 years for most small/medium business investments - Equipment: if payback = 80%+ of useful life → marginal at best - Hiring: payback = (loaded salary + onboarding) ÷ annual revenue attributable to that hire ### NPV (Net Present Value) `NPV = Sum of [Cash Flow_t / (1 + r)^t] - Initial Investment` - r = cost of capital (typically 8-15% for small/medium business) - NPV > 0 = investment creates value. NPV < 0 = destroys value. - Always run NPV for investments >$25K or >12-month horizon. ### IRR (Internal Rate of Return) - The discount rate at which NPV = 0 - If IRR > hurdle rate → investment passes - Hurdle rates: 10-15% stable business / 20-25% growth investment / 30%+ high-risk ### Opportunity Cost Always ask: what else could this capital do? - Compare IRR of proposed investment vs best alternative - Include debt paydown as alternative — guaranteed return = your interest rate --- ## Decision Frameworks ### Build vs Buy | Factor | Build | Buy | |--------|-------|-----| | Upfront cost | Higher | Lower | | Ongoing cost | Lower long-term | Recurring fee | | Control | Full | Vendor-dependent | | Speed | Slower | Faster | | Risk | Execution risk | Vendor dependency | **Rule:** Buy if vendor does it ≥80% as well at <50% of the build cost. ### Lease vs Buy - **Buy when:** use >60% of useful life, asset retains value, depreciation advantage - **Lease when:** technology changes fast, cash preservation matters, maintenance included - Always compare Total Cost of Ownership (TCO) over same period ### Hire vs Automate vs Outsource - **Hire:** work requires judgment, relationships, grows with business - **Automate:** task is repetitive, rule-based, high volume - **Outsource:** need is variable, specialized, or non-core - Rule: automate or outsource first; hire when you've proven need and can't keep up --- ## Investment Scoring Rubric Score 1-5 on each dimension: | Dimension | 1 (Poor) | 5 (Excellent) | |-----------|----------|---------------| | ROI | <10% | >50% | | Payback period | >5 years | <1 year | | Strategic fit | Unrelated | Core to mission | | Risk level | High/uncertain | Low/proven | | Reversibility | Sunk cost | Easy to exit | | Cash flow impact | Major drain | Self-funding quickly | **Score:** 6-12 = Don't do it / 13-20 = Needs more analysis / 21-30 = Strong investment --- ## Budget Allocation Framework When allocating a fixed budget across multiple options: 1. Rank all options by IRR (highest first) 2. Fund in order until budget is exhausted 3. Exception: fund anything with payback <6 months first (quick wins) 4. Never fund negative NPV unless strategic reason — name it explicitly --- ## Proactive Triggers Surface these without being asked: - **Payback > useful life** → investment never pays back; recommend against - **"Optimistic" revenue projections** → run downside case at 50% of projected revenue - **Single customer/contract as assumed revenue** → flag concentration risk - **Debt-financed investment** → factor full interest cost into NPV - **Dissimilar time horizons being compared** → normalize to same period - **Sunk cost reasoning detected** → call it out; past spend is irrelevant to go-forward decision - **No alternative use considered** → prompt opportunity cost analysis --- ## Output Artifacts | When you ask for... | You get... | |---|---| | "Should I buy this?" | Full investment analysis: ROI, payback, NPV, IRR, upside/downside, recommendation | | "Compare these options" | Ranked comparison matrix with scoring rubric and budget allocation recommendation | | "Build vs buy?" | Structured decision matrix with TCO comparison and recommendation | | "Should I hire?" | Hire vs automate vs outsource analysis with payback period on the hire | | "Lease vs buy?" | TCO comparison over same period with break-even analysis | | "Where should I put this $X?" | Budget allocation ranked by IRR with portfolio view | --- ## Output Format For every investment analysis: **RECOMMENDATION:** [Proceed / Proceed with conditions / Do not proceed] **THE NUMBERS:** | Metric | Value | |--------|-------| | Total Investment | $ | | Annual Net Cash Flow | $ | | Payback Period | X months/years | | 3-Year ROI | X% | | NPV (at X% discount rate) | $ | | IRR | X% | | Investment Score | X/30 | **KEY ASSUMPTIONS:** [Every assumption used — flag low-confidence ones 🔴] **UPSIDE CASE:** [Projections beat plan by 20%] **DOWNSIDE CASE:** [Projections miss by 40%] **RISKS TO WATCH:** 1. [Risk + mitigation] 2. [Risk + mitigation] **NEXT STEP:** [One specific action before committing capital] --- ## Communication - **Bottom line first** — recommendation before explanation - **Show all math** — every formula with actual numbers plugged in - **State every assumption** — never hide them in the analysis - **Confidence tagging** — 🟢 verified data / 🟡 reasonable estimate / 🔴 assumed — validate before committing - **Conservative by default** — use base case numbers, not optimistic projections --- ## Anti-Patterns | Anti-Pattern | Why It Fails | Better Approach | |---|---|---| | Using ROI alone without time value of money | ROI ignores when cash flows occur — a 50% ROI over 10 years is worse than 30% over 2 years | Always calculate NPV and IRR alongside ROI for investments over $25K or 12 months | | Relying on optimistic revenue projections | Founders and sales teams systematically overestimate revenue from new investments | Run the downside case at 50% of projected revenue as the primary decision input | | Ignoring opportunity cost | Approving an investment in isolation misses what else that capital could do | Always compare the proposed IRR against the best alternative use of the same capital | | Sunk cost reasoning in go/no-go decisions | Past spend is irrelevant to whether continuing will generate positive returns | Evaluate only the incremental investment required vs. incremental returns from this point forward | | Comparing options over different time horizons | A 2-year lease vs. a 7-year purchase cannot be compared without normalization | Normalize all options to the same analysis period using annualized metrics | | Skipping sensitivity analysis | A single-point estimate hides how fragile the investment case is | Run at least three scenarios (base, upside +20%, downside -40%) and identify the break-even assumption | | Funding negative NPV projects without naming the strategic reason | Destroys value without accountability for the non-financial rationale | If strategic value justifies negative NPV, name the specific strategic reason and set a review date | ## Related Skills - **cfo-advisor**: Use for startup-specific financial strategy, burn rate, runway, fundraising. NOT for individual investment ROI analysis. - **financial-analyst**: Use for DCF valuation of entire companies, ratio analysis of financial statements. NOT for single capital expenditure decisions. - **saas-metrics-coach**: Use for SaaS-specific unit economics (CAC, LTV, churn). NOT for equipment or real estate investments. - **ceo-advisor**: Use for strategic direction and capital allocation across the entire business. NOT for individual investment math.
Bộ công cụ design system UI: sinh design token, tài liệu component, tính toán responsive và bàn giao cho lập trình viên.
---
name: "ui-design-system"
description: UI design system toolkit for Senior UI Designer including design token generation, component documentation, responsive design calculations, and developer handoff tools. Use for creating design systems, maintaining visual consistency, and facilitating design-dev collaboration.
---
# UI Design System
Generate design tokens, create color palettes, calculate typography scales, build component systems, and prepare developer handoff documentation.
---
## Table of Contents
- [Trigger Terms](#trigger-terms)
- [Workflows](#workflows)
- [Workflow 1: Generate Design Tokens](#workflow-1-generate-design-tokens)
- [Workflow 2: Create Component System](#workflow-2-create-component-system)
- [Workflow 3: Responsive Design](#workflow-3-responsive-design)
- [Workflow 4: Developer Handoff](#workflow-4-developer-handoff)
- [Tool Reference](#tool-reference)
- [Quick Reference Tables](#quick-reference-tables)
- [Knowledge Base](#knowledge-base)
---
## Trigger Terms
Use this skill when you need to:
- "generate design tokens"
- "create color palette"
- "build typography scale"
- "calculate spacing system"
- "create design system"
- "generate CSS variables"
- "export SCSS tokens"
- "set up component architecture"
- "document component library"
- "calculate responsive breakpoints"
- "prepare developer handoff"
- "convert brand color to palette"
- "check WCAG contrast"
- "build 8pt grid system"
---
## Workflows
### Workflow 1: Generate Design Tokens
**Situation:** You have a brand color and need a complete design token system.
**Steps:**
1. **Identify brand color and style**
- Brand primary color (hex format)
- Style preference: `modern` | `classic` | `playful`
2. **Generate tokens using script**
```bash
python scripts/design_token_generator.py "#0066CC" modern json
```
3. **Review generated categories**
- Colors: primary, secondary, neutral, semantic, surface
- Typography: fontFamily, fontSize, fontWeight, lineHeight
- Spacing: 8pt grid-based scale (0-64)
- Borders: radius, width
- Shadows: none through 2xl
- Animation: duration, easing
- Breakpoints: xs through 2xl
4. **Export in target format**
```bash
# CSS custom properties
python scripts/design_token_generator.py "#0066CC" modern css > design-tokens.css
# SCSS variables
python scripts/design_token_generator.py "#0066CC" modern scss > _design-tokens.scss
# JSON for Figma/tooling
python scripts/design_token_generator.py "#0066CC" modern json > design-tokens.json
```
5. **Validate accessibility**
- Check color contrast meets WCAG AA (4.5:1 normal, 3:1 large text)
- Verify semantic colors have contrast colors defined
---
### Workflow 2: Create Component System
**Situation:** You need to structure a component library using design tokens.
**Steps:**
1. **Define component hierarchy**
- Atoms: Button, Input, Icon, Label, Badge
- Molecules: FormField, SearchBar, Card, ListItem
- Organisms: Header, Footer, DataTable, Modal
- Templates: DashboardLayout, AuthLayout
2. **Map tokens to components**
| Component | Tokens Used |
|-----------|-------------|
| Button | colors, sizing, borders, shadows, typography |
| Input | colors, sizing, borders, spacing |
| Card | colors, borders, shadows, spacing |
| Modal | colors, shadows, spacing, z-index, animation |
3. **Define variant patterns**
Size variants:
```
sm: height 32px, paddingX 12px, fontSize 14px
md: height 40px, paddingX 16px, fontSize 16px
lg: height 48px, paddingX 20px, fontSize 18px
```
Color variants:
```
primary: background primary-500, text white
secondary: background neutral-100, text neutral-900
ghost: background transparent, text neutral-700
```
4. **Document component API**
- Props interface with types
- Variant options
- State handling (hover, active, focus, disabled)
- Accessibility requirements
5. **Reference:** See `references/component-architecture.md`
---
### Workflow 3: Responsive Design
**Situation:** You need breakpoints, fluid typography, or responsive spacing.
**Steps:**
1. **Define breakpoints**
| Name | Width | Target |
|------|-------|--------|
| xs | 0 | Small phones |
| sm | 480px | Large phones |
| md | 640px | Tablets |
| lg | 768px | Small laptops |
| xl | 1024px | Desktops |
| 2xl | 1280px | Large screens |
2. **Calculate fluid typography**
Formula: `clamp(min, preferred, max)`
```css
/* 16px to 24px between 320px and 1200px viewport */
font-size: clamp(1rem, 0.5rem + 2vw, 1.5rem);
```
Pre-calculated scales:
```css
--fluid-h1: clamp(2rem, 1rem + 3.6vw, 4rem);
--fluid-h2: clamp(1.75rem, 1rem + 2.3vw, 3rem);
--fluid-h3: clamp(1.5rem, 1rem + 1.4vw, 2.25rem);
--fluid-body: clamp(1rem, 0.95rem + 0.2vw, 1.125rem);
```
3. **Set up responsive spacing**
| Token | Mobile | Tablet | Desktop |
|-------|--------|--------|---------|
| --space-md | 12px | 16px | 16px |
| --space-lg | 16px | 24px | 32px |
| --space-xl | 24px | 32px | 48px |
| --space-section | 48px | 80px | 120px |
4. **Reference:** See `references/responsive-calculations.md`
---
### Workflow 4: Developer Handoff
**Situation:** You need to hand off design tokens to development team.
**Steps:**
1. **Export tokens in required formats**
```bash
# For CSS projects
python scripts/design_token_generator.py "#0066CC" modern css
# For SCSS projects
python scripts/design_token_generator.py "#0066CC" modern scss
# For JavaScript/TypeScript
python scripts/design_token_generator.py "#0066CC" modern json
```
2. **Prepare framework integration**
**React + CSS Variables:**
```tsx
import './design-tokens.css';
<button className="btn btn-primary">Click</button>
```
**Tailwind Config:**
```javascript
const tokens = require('./design-tokens.json');
module.exports = {
theme: {
colors: tokens.colors,
fontFamily: tokens.typography.fontFamily
}
};
```
**styled-components:**
```typescript
import tokens from './design-tokens.json';
const Button = styled.button`
background: tokens.colors.primary['500'];
padding: tokens.spacing['2'] tokens.spacing['4'];
`;
```
3. **Sync with Figma**
- Install Tokens Studio plugin
- Import design-tokens.json
- Tokens sync automatically with Figma styles
4. **Handoff checklist**
- [ ] Token files added to project
- [ ] Build pipeline configured
- [ ] Theme/CSS variables imported
- [ ] Component library aligned
- [ ] Documentation generated
5. **Reference:** See `references/developer-handoff.md`
---
## Tool Reference
### design_token_generator.py
Generates complete design token system from brand color.
| Argument | Values | Default | Description |
|----------|--------|---------|-------------|
| brand_color | Hex color | #0066CC | Primary brand color |
| style | modern, classic, playful | modern | Design style preset |
| format | json, css, scss, summary | json | Output format |
**Examples:**
```bash
# Generate JSON tokens (default)
python scripts/design_token_generator.py "#0066CC"
# Classic style with CSS output
python scripts/design_token_generator.py "#8B4513" classic css
# Playful style summary view
python scripts/design_token_generator.py "#FF6B6B" playful summary
```
**Output Categories:**
| Category | Description | Key Values |
|----------|-------------|------------|
| colors | Color palettes | primary, secondary, neutral, semantic, surface |
| typography | Font system | fontFamily, fontSize, fontWeight, lineHeight |
| spacing | 8pt grid | 0-64 scale, semantic (xs-3xl) |
| sizing | Component sizes | container, button, input, icon |
| borders | Border values | radius (per style), width |
| shadows | Shadow styles | none through 2xl, inner |
| animation | Motion tokens | duration, easing, keyframes |
| breakpoints | Responsive | xs, sm, md, lg, xl, 2xl |
| z-index | Layer system | base through notification |
---
## Quick Reference Tables
### Color Scale Generation
| Step | Brightness | Saturation | Use Case |
|------|------------|------------|----------|
| 50 | 95% fixed | 30% | Subtle backgrounds |
| 100 | 95% fixed | 38% | Light backgrounds |
| 200 | 95% fixed | 46% | Hover states |
| 300 | 95% fixed | 54% | Borders |
| 400 | 95% fixed | 62% | Disabled states |
| 500 | Original | 70% | Base/default color |
| 600 | Original × 0.8 | 78% | Hover (dark) |
| 700 | Original × 0.6 | 86% | Active states |
| 800 | Original × 0.4 | 94% | Text |
| 900 | Original × 0.2 | 100% | Headings |
### Typography Scale (1.25x Ratio)
| Size | Value | Calculation |
|------|-------|-------------|
| xs | 10px | 16 ÷ 1.25² |
| sm | 13px | 16 ÷ 1.25¹ |
| base | 16px | Base |
| lg | 20px | 16 × 1.25¹ |
| xl | 25px | 16 × 1.25² |
| 2xl | 31px | 16 × 1.25³ |
| 3xl | 39px | 16 × 1.25⁴ |
| 4xl | 49px | 16 × 1.25⁵ |
| 5xl | 61px | 16 × 1.25⁶ |
### WCAG Contrast Requirements
| Level | Normal Text | Large Text |
|-------|-------------|------------|
| AA | 4.5:1 | 3:1 |
| AAA | 7:1 | 4.5:1 |
Large text: ≥18pt regular or ≥14pt bold
### Style Presets
| Aspect | Modern | Classic | Playful |
|--------|--------|---------|---------|
| Font Sans | Inter | Helvetica | Poppins |
| Font Mono | Fira Code | Courier | Source Code Pro |
| Radius Default | 8px | 4px | 16px |
| Shadows | Layered, subtle | Single layer | Soft, pronounced |
---
## Knowledge Base
Detailed reference guides in `references/`:
| File | Content |
|------|---------|
| `token-generation.md` | Color algorithms, HSV space, WCAG contrast, type scales |
| `component-architecture.md` | Atomic design, naming conventions, props patterns |
| `responsive-calculations.md` | Breakpoints, fluid typography, grid systems |
| `developer-handoff.md` | Export formats, framework setup, Figma sync |
---
## Validation Checklist
### Token Generation
- [ ] Brand color provided in hex format
- [ ] Style matches project requirements
- [ ] All token categories generated
- [ ] Semantic colors include contrast values
### Component System
- [ ] All sizes implemented (sm, md, lg)
- [ ] All variants implemented (primary, secondary, ghost)
- [ ] All states working (hover, active, focus, disabled)
- [ ] Uses only design tokens (no hardcoded values)
### Accessibility
- [ ] Color contrast meets WCAG AA
- [ ] Focus indicators visible
- [ ] Touch targets ≥ 44×44px
- [ ] Semantic HTML elements used
### Developer Handoff
- [ ] Tokens exported in required format
- [ ] Framework integration documented
- [ ] Design tool synced
- [ ] Component documentation complete
FILE:assets/design_system_doc_template.md
# Design System Documentation
## System Info
| Field | Value |
|-------|-------|
| **Name** | [Design System Name] |
| **Version** | [X.Y.Z] |
| **Owner** | [Team/Person] |
| **Status** | Active / Beta / Deprecated |
| **Last Updated** | YYYY-MM-DD |
---
## Design Principles
The following principles guide all design decisions in this system:
1. **[Principle 1 Name]** - [One sentence description. Example: "Clarity over cleverness - every element should have an obvious purpose."]
2. **[Principle 2 Name]** - [One sentence description. Example: "Consistency breeds confidence - similar actions should look and behave the same."]
3. **[Principle 3 Name]** - [One sentence description. Example: "Accessible by default - every component must meet WCAG 2.1 AA standards."]
4. **[Principle 4 Name]** - [One sentence description. Example: "Progressive disclosure - show only what is needed, reveal complexity on demand."]
---
## Color Palette
### Brand Colors
| Name | Hex | RGB | Usage |
|------|-----|-----|-------|
| Primary | #[XXXXXX] | rgb(X, X, X) | Primary actions, links, key UI elements |
| Secondary | #[XXXXXX] | rgb(X, X, X) | Secondary actions, accents |
| Accent | #[XXXXXX] | rgb(X, X, X) | Highlights, badges, notifications |
### Neutral Colors
| Name | Hex | Usage |
|------|-----|-------|
| Gray-900 | #[XXXXXX] | Primary text |
| Gray-700 | #[XXXXXX] | Secondary text |
| Gray-500 | #[XXXXXX] | Placeholder text, disabled states |
| Gray-300 | #[XXXXXX] | Borders, dividers |
| Gray-100 | #[XXXXXX] | Backgrounds, hover states |
| White | #FFFFFF | Page background, card background |
### Semantic Colors
| Name | Hex | Usage |
|------|-----|-------|
| Success | #[XXXXXX] | Success messages, positive indicators |
| Warning | #[XXXXXX] | Warning messages, caution indicators |
| Error | #[XXXXXX] | Error messages, destructive actions |
| Info | #[XXXXXX] | Informational messages, tips |
### Accessibility
- All text colors must meet WCAG 2.1 AA contrast ratio (4.5:1 for normal text, 3:1 for large text)
- Test with color blindness simulators
- Never use color as the only indicator of state
---
## Typography Scale
### Font Family
- **Primary:** [Font Name] (headings and body)
- **Monospace:** [Font Name] (code blocks, technical content)
- **Fallback Stack:** [System font stack]
### Type Scale
| Name | Size | Weight | Line Height | Usage |
|------|------|--------|-------------|-------|
| Display | 48px / 3rem | Bold (700) | 1.2 | Hero headings |
| H1 | 36px / 2.25rem | Bold (700) | 1.25 | Page titles |
| H2 | 28px / 1.75rem | Semibold (600) | 1.3 | Section headings |
| H3 | 22px / 1.375rem | Semibold (600) | 1.35 | Subsection headings |
| H4 | 18px / 1.125rem | Medium (500) | 1.4 | Card titles, labels |
| Body Large | 18px / 1.125rem | Regular (400) | 1.6 | Lead paragraphs |
| Body | 16px / 1rem | Regular (400) | 1.5 | Default body text |
| Body Small | 14px / 0.875rem | Regular (400) | 1.5 | Secondary text, captions |
| Caption | 12px / 0.75rem | Regular (400) | 1.4 | Labels, metadata |
---
## Spacing System
### Base Unit: 4px
| Token | Value | Usage |
|-------|-------|-------|
| space-1 | 4px | Tight spacing (icon padding) |
| space-2 | 8px | Compact elements (inline items) |
| space-3 | 12px | Related elements (form field gaps) |
| space-4 | 16px | Default spacing (paragraph gaps) |
| space-5 | 20px | Group spacing (card padding) |
| space-6 | 24px | Section spacing |
| space-8 | 32px | Large section gaps |
| space-10 | 40px | Page section dividers |
| space-12 | 48px | Major layout sections |
| space-16 | 64px | Page-level spacing |
### Layout Spacing
- **Page margin:** space-6 (mobile), space-8 (tablet), space-12 (desktop)
- **Card padding:** space-5
- **Form field gap:** space-3
- **Section gap:** space-10
---
## Component Library
### Component Status Legend
- **Stable** - Production ready, fully documented and tested
- **Beta** - Functional but may change, use with awareness
- **Deprecated** - Scheduled for removal, migrate to replacement
- **Planned** - On roadmap, not yet available
### Components
| Component | Status | Description | Variants |
|-----------|--------|-------------|----------|
| Button | Stable | Primary action triggers | Primary, Secondary, Tertiary, Danger, Ghost |
| Input | Stable | Text input fields | Default, Error, Disabled, With icon |
| Select | Stable | Dropdown selection | Single, Multi, Searchable |
| Checkbox | Stable | Multi-select toggle | Default, Indeterminate, Disabled |
| Radio | Stable | Single-select option | Default, Disabled |
| Toggle | Stable | Binary on/off switch | Default, With label |
| Modal | Stable | Overlay dialog | Small, Medium, Large, Fullscreen |
| Toast | Stable | Temporary notification | Success, Error, Warning, Info |
| Card | Stable | Content container | Default, Interactive, Elevated |
| Badge | Stable | Status indicator | Solid, Outline, Dot |
| Avatar | Stable | User representation | Image, Initials, Icon |
| Table | Beta | Data display grid | Default, Sortable, Selectable |
| Tabs | Beta | Content organization | Default, Underline, Pill |
| Tooltip | Stable | Contextual information | Default, Rich content |
| [New Component] | Planned | [Description] | [Variants] |
---
## Usage Guidelines
### Do
- Use components as documented (do not override internal styles)
- Follow the spacing system for consistent layouts
- Test components across supported browsers and screen sizes
- Use semantic colors for their intended purpose
- Reference design tokens instead of hardcoded values
### Do Not
- Modify component internals without contributing changes back
- Create one-off components when an existing component fits
- Use brand colors for semantic purposes (error, success)
- Skip accessibility requirements for "internal" tools
- Mix design system versions across a single application
---
## Contribution Process
### Proposing a New Component
1. **Check existing components** - Verify no existing component solves the need
2. **Create proposal** - Document use case, behavior, variants, accessibility requirements
3. **Design review** - Present to design system team for feedback
4. **Build** - Implement component following system patterns
5. **Review** - Code review + design review + accessibility audit
6. **Document** - Add to component library with usage guidelines
7. **Release** - Publish in next minor version
### Updating an Existing Component
1. **File issue** - Describe the change and justification
2. **Impact assessment** - Identify all instances of current usage
3. **Design + develop** - Implement change with backward compatibility
4. **Migration guide** - Document breaking changes if any
5. **Release** - Publish with changelog entry
### Reporting Issues
- File bug reports with reproduction steps and screenshots
- Tag with component name and severity
- Include browser/OS information for rendering issues
FILE:references/component-architecture.md
# Component Architecture Guide
Reference for design system component organization, naming conventions, and documentation patterns.
---
## Table of Contents
- [Component Hierarchy](#component-hierarchy)
- [Naming Conventions](#naming-conventions)
- [Component Documentation](#component-documentation)
- [Variant Patterns](#variant-patterns)
- [Token Integration](#token-integration)
---
## Component Hierarchy
### Atomic Design Structure
```
┌─────────────────────────────────────────────────────────────┐
│ COMPONENT HIERARCHY │
├─────────────────────────────────────────────────────────────┤
│ │
│ TOKENS (Foundation) │
│ └── Colors, Typography, Spacing, Shadows │
│ │
│ ATOMS (Basic Elements) │
│ └── Button, Input, Icon, Label, Badge │
│ │
│ MOLECULES (Simple Combinations) │
│ └── FormField, SearchBar, Card, ListItem │
│ │
│ ORGANISMS (Complex Components) │
│ └── Header, Footer, DataTable, Modal │
│ │
│ TEMPLATES (Page Layouts) │
│ └── DashboardLayout, AuthLayout, SettingsLayout │
│ │
│ PAGES (Specific Instances) │
│ └── HomePage, LoginPage, UserProfile │
│ │
└─────────────────────────────────────────────────────────────┘
```
### Component Categories
| Category | Description | Examples |
|----------|-------------|----------|
| **Primitives** | Base HTML wrapper | Box, Text, Flex, Grid |
| **Inputs** | User interaction | Button, Input, Select, Checkbox |
| **Display** | Content presentation | Card, Badge, Avatar, Icon |
| **Feedback** | User feedback | Alert, Toast, Progress, Skeleton |
| **Navigation** | Route management | Link, Menu, Tabs, Breadcrumb |
| **Overlay** | Layer above content | Modal, Drawer, Popover, Tooltip |
| **Layout** | Structure | Stack, Container, Divider |
---
## Naming Conventions
### Token Naming
```
{category}-{property}-{variant}-{state}
Examples:
color-primary-500
color-primary-500-hover
spacing-md
fontSize-lg
shadow-md
radius-lg
```
### Component Naming
```
{ComponentName} # PascalCase for components
{componentName}{Variant} # Variant suffix
Examples:
Button
ButtonPrimary
ButtonOutline
ButtonGhost
```
### CSS Class Naming (BEM)
```
.block__element--modifier
Examples:
.button
.button__icon
.button--primary
.button--lg
.button__icon--loading
```
### File Structure
```
components/
├── Button/
│ ├── Button.tsx # Main component
│ ├── Button.styles.ts # Styles/tokens
│ ├── Button.test.tsx # Tests
│ ├── Button.stories.tsx # Storybook
│ ├── Button.types.ts # TypeScript types
│ └── index.ts # Export
├── Input/
│ └── ...
└── index.ts # Barrel export
```
---
## Component Documentation
### Documentation Template
```markdown
# ComponentName
Brief description of what this component does.
## Usage
\`\`\`tsx
import { Button } from '@design-system/components'
<Button variant="primary" size="md">
Click me
</Button>
\`\`\`
## Props
| Prop | Type | Default | Description |
|------|------|---------|-------------|
| variant | 'primary' \| 'secondary' \| 'ghost' | 'primary' | Visual style |
| size | 'sm' \| 'md' \| 'lg' | 'md' | Component size |
| disabled | boolean | false | Disabled state |
| onClick | () => void | - | Click handler |
## Variants
### Primary
Use for main actions.
### Secondary
Use for secondary actions.
### Ghost
Use for tertiary or inline actions.
## Accessibility
- Uses `button` role by default
- Supports `aria-disabled` for disabled state
- Focus ring visible for keyboard navigation
## Design Tokens Used
- `color-primary-*` for primary variant
- `spacing-*` for padding
- `radius-md` for border radius
- `shadow-sm` for elevation
```
### Props Interface Pattern
```typescript
interface ButtonProps {
/** Visual variant of the button */
variant?: 'primary' | 'secondary' | 'ghost' | 'danger';
/** Size of the button */
size?: 'sm' | 'md' | 'lg';
/** Whether button is disabled */
disabled?: boolean;
/** Whether button shows loading state */
loading?: boolean;
/** Left icon element */
leftIcon?: React.ReactNode;
/** Right icon element */
rightIcon?: React.ReactNode;
/** Click handler */
onClick?: () => void;
/** Button content */
children: React.ReactNode;
}
```
---
## Variant Patterns
### Size Variants
```typescript
const sizeTokens = {
sm: {
height: 'sizing-button-sm-height', // 32px
paddingX: 'sizing-button-sm-paddingX', // 12px
fontSize: 'fontSize-sm', // 14px
iconSize: 'sizing-icon-sm' // 16px
},
md: {
height: 'sizing-button-md-height', // 40px
paddingX: 'sizing-button-md-paddingX', // 16px
fontSize: 'fontSize-base', // 16px
iconSize: 'sizing-icon-md' // 20px
},
lg: {
height: 'sizing-button-lg-height', // 48px
paddingX: 'sizing-button-lg-paddingX', // 20px
fontSize: 'fontSize-lg', // 18px
iconSize: 'sizing-icon-lg' // 24px
}
};
```
### Color Variants
```typescript
const variantTokens = {
primary: {
background: 'color-primary-500',
backgroundHover: 'color-primary-600',
backgroundActive: 'color-primary-700',
text: 'color-white',
border: 'transparent'
},
secondary: {
background: 'color-neutral-100',
backgroundHover: 'color-neutral-200',
backgroundActive: 'color-neutral-300',
text: 'color-neutral-900',
border: 'transparent'
},
outline: {
background: 'transparent',
backgroundHover: 'color-primary-50',
backgroundActive: 'color-primary-100',
text: 'color-primary-500',
border: 'color-primary-500'
},
ghost: {
background: 'transparent',
backgroundHover: 'color-neutral-100',
backgroundActive: 'color-neutral-200',
text: 'color-neutral-700',
border: 'transparent'
}
};
```
### State Variants
```typescript
const stateStyles = {
default: {
cursor: 'pointer',
opacity: 1
},
hover: {
// Uses variantTokens backgroundHover
},
active: {
// Uses variantTokens backgroundActive
transform: 'scale(0.98)'
},
focus: {
outline: 'none',
boxShadow: '0 0 0 2px color-primary-200'
},
disabled: {
cursor: 'not-allowed',
opacity: 0.5,
pointerEvents: 'none'
},
loading: {
cursor: 'wait',
pointerEvents: 'none'
}
};
```
---
## Token Integration
### Consuming Tokens in Components
**CSS Custom Properties:**
```css
.button {
height: var(--sizing-button-md-height);
padding-left: var(--sizing-button-md-paddingX);
padding-right: var(--sizing-button-md-paddingX);
font-size: var(--typography-fontSize-base);
border-radius: var(--borders-radius-md);
}
.button--primary {
background-color: var(--colors-primary-500);
color: var(--colors-surface-background);
}
.button--primary:hover {
background-color: var(--colors-primary-600);
}
```
**JavaScript/TypeScript:**
```typescript
import tokens from './design-tokens.json';
const buttonStyles = {
height: tokens.sizing.components.button.md.height,
paddingLeft: tokens.sizing.components.button.md.paddingX,
backgroundColor: tokens.colors.primary['500'],
borderRadius: tokens.borders.radius.md
};
```
**Styled Components:**
```typescript
import styled from 'styled-components';
const Button = styled.button`
height: ({ theme) => theme.sizing.components.button.md.height};
padding: 0 ({ theme) => theme.sizing.components.button.md.paddingX};
background: ({ theme) => theme.colors.primary['500']};
border-radius: ({ theme) => theme.borders.radius.md};
&:hover {
background: ({ theme) => theme.colors.primary['600']};
}
`;
```
### Token-to-Component Mapping
| Component | Token Categories Used |
|-----------|----------------------|
| Button | colors, sizing, borders, shadows, typography |
| Input | colors, sizing, borders, spacing |
| Card | colors, borders, shadows, spacing |
| Typography | typography (all), colors |
| Icon | sizing, colors |
| Modal | colors, shadows, spacing, z-index, animation |
---
## Component Checklist
### Before Release
- [ ] All sizes implemented (sm, md, lg)
- [ ] All variants implemented (primary, secondary, etc.)
- [ ] All states working (hover, active, focus, disabled)
- [ ] Keyboard accessible
- [ ] Screen reader tested
- [ ] Uses only design tokens (no hardcoded values)
- [ ] TypeScript types complete
- [ ] Storybook stories for all variants
- [ ] Unit tests passing
- [ ] Documentation complete
### Accessibility Checklist
- [ ] Correct semantic HTML element
- [ ] ARIA attributes where needed
- [ ] Visible focus indicator
- [ ] Color contrast meets AA
- [ ] Works with keyboard only
- [ ] Screen reader announces correctly
- [ ] Touch target ≥ 44×44px
---
*See also: `token-generation.md` for token creation*
FILE:references/developer-handoff.md
# Developer Handoff Guide
Reference for integrating design tokens into development workflows and design tool collaboration.
---
## Table of Contents
- [Export Formats](#export-formats)
- [Integration Patterns](#integration-patterns)
- [Framework Setup](#framework-setup)
- [Design Tool Integration](#design-tool-integration)
- [Handoff Checklist](#handoff-checklist)
---
## Export Formats
### JSON (Recommended for Most Projects)
**File:** `design-tokens.json`
```json
{
"meta": {
"version": "1.0.0",
"style": "modern",
"generated": "2024-01-15"
},
"colors": {
"primary": {
"50": "#E6F2FF",
"100": "#CCE5FF",
"500": "#0066CC",
"900": "#002855"
}
},
"typography": {
"fontFamily": {
"sans": "Inter, system-ui, sans-serif",
"mono": "Fira Code, monospace"
},
"fontSize": {
"xs": "10px",
"sm": "13px",
"base": "16px",
"lg": "20px"
}
},
"spacing": {
"0": "0px",
"1": "4px",
"2": "8px",
"4": "16px"
}
}
```
**Use Case:** JavaScript/TypeScript projects, build tools, Figma plugins
### CSS Custom Properties
**File:** `design-tokens.css`
```css
:root {
/* Colors */
--color-primary-50: #E6F2FF;
--color-primary-100: #CCE5FF;
--color-primary-500: #0066CC;
--color-primary-900: #002855;
/* Typography */
--font-family-sans: Inter, system-ui, sans-serif;
--font-family-mono: Fira Code, monospace;
--font-size-xs: 10px;
--font-size-sm: 13px;
--font-size-base: 16px;
--font-size-lg: 20px;
/* Spacing */
--spacing-0: 0px;
--spacing-1: 4px;
--spacing-2: 8px;
--spacing-4: 16px;
}
```
**Use Case:** Plain CSS, CSS-in-JS, any web project
### SCSS Variables
**File:** `_design-tokens.scss`
```scss
// Colors
$color-primary-50: #E6F2FF;
$color-primary-100: #CCE5FF;
$color-primary-500: #0066CC;
$color-primary-900: #002855;
// Typography
$font-family-sans: Inter, system-ui, sans-serif;
$font-family-mono: Fira Code, monospace;
$font-size-xs: 10px;
$font-size-sm: 13px;
$font-size-base: 16px;
$font-size-lg: 20px;
// Spacing
$spacing-0: 0px;
$spacing-1: 4px;
$spacing-2: 8px;
$spacing-4: 16px;
// Maps for programmatic access
$colors-primary: (
'50': $color-primary-50,
'100': $color-primary-100,
'500': $color-primary-500,
'900': $color-primary-900
);
```
**Use Case:** SASS/SCSS pipelines, component libraries
---
## Integration Patterns
### Pattern 1: CSS Variables (Universal)
Works with any framework or vanilla CSS.
```css
/* Import tokens */
@import 'design-tokens.css';
/* Use in styles */
.button {
background-color: var(--color-primary-500);
padding: var(--spacing-2) var(--spacing-4);
font-size: var(--font-size-base);
border-radius: var(--radius-md);
}
.button:hover {
background-color: var(--color-primary-600);
}
```
### Pattern 2: JavaScript Theme Object
For CSS-in-JS libraries (styled-components, Emotion, etc.)
```typescript
// theme.ts
import tokens from './design-tokens.json';
export const theme = {
colors: {
primary: tokens.colors.primary,
secondary: tokens.colors.secondary,
neutral: tokens.colors.neutral,
semantic: tokens.colors.semantic
},
typography: {
fontFamily: tokens.typography.fontFamily,
fontSize: tokens.typography.fontSize,
fontWeight: tokens.typography.fontWeight
},
spacing: tokens.spacing,
shadows: tokens.shadows,
radii: tokens.borders.radius
};
export type Theme = typeof theme;
```
```typescript
// styled-components usage
import styled from 'styled-components';
const Button = styled.button`
background: ({ theme) => theme.colors.primary['500']};
padding: ({ theme) => theme.spacing['2']} ({ theme) => theme.spacing['4']};
font-size: ({ theme) => theme.typography.fontSize.base};
`;
```
### Pattern 3: Tailwind Config
```javascript
// tailwind.config.js
const tokens = require('./design-tokens.json');
module.exports = {
theme: {
colors: {
primary: tokens.colors.primary,
secondary: tokens.colors.secondary,
neutral: tokens.colors.neutral,
success: tokens.colors.semantic.success,
warning: tokens.colors.semantic.warning,
error: tokens.colors.semantic.error
},
fontFamily: {
sans: [tokens.typography.fontFamily.sans],
serif: [tokens.typography.fontFamily.serif],
mono: [tokens.typography.fontFamily.mono]
},
spacing: {
0: tokens.spacing['0'],
1: tokens.spacing['1'],
2: tokens.spacing['2'],
// ... etc
},
borderRadius: tokens.borders.radius,
boxShadow: tokens.shadows
}
};
```
---
## Framework Setup
### React + CSS Variables
```tsx
// App.tsx
import './design-tokens.css';
import './styles.css';
function App() {
return (
<button className="btn btn-primary">
Click me
</button>
);
}
```
```css
/* styles.css */
.btn {
padding: var(--spacing-2) var(--spacing-4);
font-size: var(--font-size-base);
font-weight: var(--font-weight-medium);
border-radius: var(--radius-md);
transition: background-color var(--animation-duration-fast);
}
.btn-primary {
background: var(--color-primary-500);
color: var(--color-surface-background);
}
.btn-primary:hover {
background: var(--color-primary-600);
}
```
### React + styled-components
```tsx
// ThemeProvider.tsx
import { ThemeProvider } from 'styled-components';
import { theme } from './theme';
export function AppThemeProvider({ children }) {
return (
<ThemeProvider theme={theme}>
{children}
</ThemeProvider>
);
}
```
```tsx
// Button.tsx
import styled from 'styled-components';
export const Button = styled.button<{ variant?: 'primary' | 'secondary' }>`
padding: ({ theme) => `theme.spacing['2'] theme.spacing['4']`};
font-size: ({ theme) => theme.typography.fontSize.base};
border-radius: ({ theme) => theme.radii.md};
({ variant = 'primary', theme) => variant === 'primary' && `
background: theme.colors.primary['500'];
color: theme.colors.surface.background;
&:hover {
background: theme.colors.primary['600'];
}
`}
`;
```
### Vue + CSS Variables
```vue
<!-- App.vue -->
<template>
<button class="btn btn-primary">Click me</button>
</template>
<style>
@import './design-tokens.css';
.btn {
padding: var(--spacing-2) var(--spacing-4);
font-size: var(--font-size-base);
border-radius: var(--radius-md);
}
.btn-primary {
background: var(--color-primary-500);
color: var(--color-surface-background);
}
</style>
```
### Next.js + Tailwind
```javascript
// tailwind.config.js
const tokens = require('./design-tokens.json');
module.exports = {
content: ['./app/**/*.{js,ts,jsx,tsx}'],
theme: {
extend: {
colors: tokens.colors,
fontFamily: {
sans: tokens.typography.fontFamily.sans.split(', ')
}
}
}
};
```
```tsx
// page.tsx
export default function Page() {
return (
<button className="bg-primary-500 hover:bg-primary-600 px-4 py-2 rounded-md text-white">
Click me
</button>
);
}
```
---
## Design Tool Integration
### Figma
**Option 1: Tokens Studio Plugin**
1. Install "Tokens Studio for Figma" plugin
2. Import `design-tokens.json`
3. Tokens sync automatically with Figma styles
**Option 2: Figma Variables (Native)**
1. Open Variables panel
2. Create collections matching token structure
3. Import JSON via plugin or API
**Sync Workflow:**
```
design_token_generator.py
↓
design-tokens.json
↓
Tokens Studio Plugin
↓
Figma Styles & Variables
```
### Storybook
```javascript
// .storybook/preview.js
import '../design-tokens.css';
export const parameters = {
backgrounds: {
default: 'light',
values: [
{ name: 'light', value: '#FFFFFF' },
{ name: 'dark', value: '#111827' }
]
}
};
```
```javascript
// Button.stories.tsx
import { Button } from './Button';
export default {
title: 'Components/Button',
component: Button,
argTypes: {
variant: {
control: 'select',
options: ['primary', 'secondary', 'ghost']
},
size: {
control: 'select',
options: ['sm', 'md', 'lg']
}
}
};
export const Primary = {
args: {
variant: 'primary',
children: 'Button'
}
};
```
### Design Tool Comparison
| Tool | Token Format | Sync Method |
|------|--------------|-------------|
| Figma | JSON | Tokens Studio plugin / Variables |
| Sketch | JSON | Craft / Shared Styles |
| Adobe XD | JSON | Design Tokens plugin |
| InVision DSM | JSON | Native import |
| Zeroheight | JSON/CSS | Direct import |
---
## Handoff Checklist
### Token Generation
- [ ] Brand color defined
- [ ] Style selected (modern/classic/playful)
- [ ] Tokens generated: `python scripts/design_token_generator.py "#0066CC" modern`
- [ ] All formats exported (JSON, CSS, SCSS)
### Developer Setup
- [ ] Token files added to project
- [ ] Build pipeline configured
- [ ] Theme/CSS variables imported
- [ ] Hot reload working for token changes
### Design Sync
- [ ] Figma/design tool updated with tokens
- [ ] Component library aligned
- [ ] Documentation generated
- [ ] Storybook stories created
### Validation
- [ ] Colors render correctly
- [ ] Typography scales properly
- [ ] Spacing matches design
- [ ] Responsive breakpoints work
- [ ] Dark mode tokens (if applicable)
### Documentation Deliverables
| Document | Contents |
|----------|----------|
| `design-tokens.json` | All tokens in JSON |
| `design-tokens.css` | CSS custom properties |
| `_design-tokens.scss` | SCSS variables |
| `README.md` | Usage instructions |
| `CHANGELOG.md` | Token version history |
---
## Version Control
### Token Versioning
```json
{
"meta": {
"version": "1.2.0",
"style": "modern",
"generated": "2024-01-15",
"changelog": [
"1.2.0 - Added animation tokens",
"1.1.0 - Updated primary color",
"1.0.0 - Initial release"
]
}
}
```
### Breaking Change Policy
| Change Type | Version Bump | Migration |
|-------------|--------------|-----------|
| Add new token | Patch (1.0.x) | None |
| Change token value | Minor (1.x.0) | Optional |
| Rename/remove token | Major (x.0.0) | Required |
---
*See also: `token-generation.md` for generation options*
FILE:references/responsive-calculations.md
# Responsive Design Calculations
Reference for breakpoint math, fluid typography, and responsive layout patterns.
---
## Table of Contents
- [Breakpoint System](#breakpoint-system)
- [Fluid Typography](#fluid-typography)
- [Responsive Spacing](#responsive-spacing)
- [Container Queries](#container-queries)
- [Grid Systems](#grid-systems)
---
## Breakpoint System
### Standard Breakpoints
```
┌─────────────────────────────────────────────────────────────┐
│ BREAKPOINT RANGES │
├─────────────────────────────────────────────────────────────┤
│ │
│ xs sm md lg xl 2xl │
│ │─────────│──────────│──────────│──────────│─────────│ │
│ 0 480px 640px 768px 1024px 1280px │
│ 1536px │
│ │
│ Mobile Mobile+ Tablet Laptop Desktop Large │
│ │
└─────────────────────────────────────────────────────────────┘
```
### Breakpoint Values
| Name | Min Width | Target Devices |
|------|-----------|----------------|
| xs | 0 | Small phones |
| sm | 480px | Large phones |
| md | 640px | Small tablets |
| lg | 768px | Tablets, small laptops |
| xl | 1024px | Laptops, desktops |
| 2xl | 1280px | Large desktops |
| 3xl | 1536px | Extra large displays |
### Mobile-First Media Queries
```css
/* Base styles (mobile) */
.component {
padding: var(--spacing-sm);
font-size: var(--fontSize-sm);
}
/* Small devices and up */
@media (min-width: 480px) {
.component {
padding: var(--spacing-md);
}
}
/* Medium devices and up */
@media (min-width: 768px) {
.component {
padding: var(--spacing-lg);
font-size: var(--fontSize-base);
}
}
/* Large devices and up */
@media (min-width: 1024px) {
.component {
padding: var(--spacing-xl);
}
}
```
### Breakpoint Utility Function
```javascript
const breakpoints = {
xs: 480,
sm: 640,
md: 768,
lg: 1024,
xl: 1280,
'2xl': 1536
};
function mediaQuery(breakpoint, type = 'min') {
const value = breakpoints[breakpoint];
if (type === 'min') {
return `@media (min-width: valuepx)`;
}
return `@media (max-width: value - 1px)`;
}
// Usage
const styles = `
mediaQuery('md') {
display: flex;
}
`;
```
---
## Fluid Typography
### Clamp Formula
```css
font-size: clamp(min, preferred, max);
/* Example: 16px to 24px between 320px and 1200px viewport */
font-size: clamp(1rem, 0.5rem + 2vw, 1.5rem);
```
### Fluid Scale Calculation
```
preferred = min + (max - min) * ((100vw - minVW) / (maxVW - minVW))
Simplified:
preferred = base + (scaling-factor * vw)
Where:
scaling-factor = (max - min) / (maxVW - minVW) * 100
```
### Fluid Typography Scale
| Style | Mobile (320px) | Desktop (1200px) | Clamp Value |
|-------|----------------|------------------|-------------|
| h1 | 32px | 64px | `clamp(2rem, 1rem + 3.6vw, 4rem)` |
| h2 | 28px | 48px | `clamp(1.75rem, 1rem + 2.3vw, 3rem)` |
| h3 | 24px | 36px | `clamp(1.5rem, 1rem + 1.4vw, 2.25rem)` |
| h4 | 20px | 28px | `clamp(1.25rem, 1rem + 0.9vw, 1.75rem)` |
| body | 16px | 18px | `clamp(1rem, 0.95rem + 0.2vw, 1.125rem)` |
| small | 14px | 14px | `0.875rem` (fixed) |
### Implementation
```css
:root {
/* Fluid type scale */
--fluid-h1: clamp(2rem, 1rem + 3.6vw, 4rem);
--fluid-h2: clamp(1.75rem, 1rem + 2.3vw, 3rem);
--fluid-h3: clamp(1.5rem, 1rem + 1.4vw, 2.25rem);
--fluid-body: clamp(1rem, 0.95rem + 0.2vw, 1.125rem);
}
h1 { font-size: var(--fluid-h1); }
h2 { font-size: var(--fluid-h2); }
h3 { font-size: var(--fluid-h3); }
body { font-size: var(--fluid-body); }
```
---
## Responsive Spacing
### Fluid Spacing Formula
```css
/* Spacing that scales with viewport */
spacing: clamp(minSpace, preferredSpace, maxSpace);
/* Example: 16px to 48px */
--spacing-responsive: clamp(1rem, 0.5rem + 2vw, 3rem);
```
### Responsive Spacing Scale
| Token | Mobile | Tablet | Desktop |
|-------|--------|--------|---------|
| --space-xs | 4px | 4px | 4px |
| --space-sm | 8px | 8px | 8px |
| --space-md | 12px | 16px | 16px |
| --space-lg | 16px | 24px | 32px |
| --space-xl | 24px | 32px | 48px |
| --space-2xl | 32px | 48px | 64px |
| --space-section | 48px | 80px | 120px |
### Implementation
```css
:root {
--space-section: clamp(3rem, 2rem + 4vw, 7.5rem);
--space-component: clamp(1rem, 0.5rem + 1vw, 2rem);
--space-content: clamp(1.5rem, 1rem + 2vw, 3rem);
}
.section {
padding-top: var(--space-section);
padding-bottom: var(--space-section);
}
.card {
padding: var(--space-component);
gap: var(--space-content);
}
```
---
## Container Queries
### Container Width Tokens
| Container | Max Width | Use Case |
|-----------|-----------|----------|
| sm | 640px | Narrow content |
| md | 768px | Blog posts |
| lg | 1024px | Standard pages |
| xl | 1280px | Wide layouts |
| 2xl | 1536px | Full-width dashboards |
### Container CSS
```css
.container {
width: 100%;
margin-left: auto;
margin-right: auto;
padding-left: var(--spacing-md);
padding-right: var(--spacing-md);
}
.container--sm { max-width: 640px; }
.container--md { max-width: 768px; }
.container--lg { max-width: 1024px; }
.container--xl { max-width: 1280px; }
.container--2xl { max-width: 1536px; }
```
### CSS Container Queries
```css
/* Define container */
.card-container {
container-type: inline-size;
container-name: card;
}
/* Query container width */
@container card (min-width: 400px) {
.card {
display: flex;
flex-direction: row;
}
}
@container card (min-width: 600px) {
.card {
gap: var(--spacing-lg);
}
}
```
---
## Grid Systems
### 12-Column Grid
```css
.grid {
display: grid;
grid-template-columns: repeat(12, 1fr);
gap: var(--spacing-md);
}
/* Column spans */
.col-1 { grid-column: span 1; }
.col-2 { grid-column: span 2; }
.col-3 { grid-column: span 3; }
.col-4 { grid-column: span 4; }
.col-6 { grid-column: span 6; }
.col-12 { grid-column: span 12; }
/* Responsive columns */
@media (min-width: 768px) {
.col-md-4 { grid-column: span 4; }
.col-md-6 { grid-column: span 6; }
.col-md-8 { grid-column: span 8; }
}
```
### Auto-Fit Grid
```css
/* Cards that automatically wrap */
.auto-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(280px, 1fr));
gap: var(--spacing-lg);
}
/* With explicit min/max columns */
.auto-grid--constrained {
grid-template-columns: repeat(
auto-fit,
minmax(min(100%, 280px), 1fr)
);
}
```
### Common Layout Patterns
**Sidebar + Content:**
```css
.layout-sidebar {
display: grid;
grid-template-columns: 1fr;
gap: var(--spacing-lg);
}
@media (min-width: 768px) {
.layout-sidebar {
grid-template-columns: 280px 1fr;
}
}
```
**Holy Grail:**
```css
.layout-holy-grail {
display: grid;
grid-template-columns: 1fr;
grid-template-rows: auto 1fr auto;
min-height: 100vh;
}
@media (min-width: 1024px) {
.layout-holy-grail {
grid-template-columns: 200px 1fr 200px;
grid-template-rows: auto 1fr auto;
}
.layout-holy-grail header,
.layout-holy-grail footer {
grid-column: 1 / -1;
}
}
```
---
## Quick Reference
### Viewport Units
| Unit | Description |
|------|-------------|
| vw | 1% of viewport width |
| vh | 1% of viewport height |
| vmin | 1% of smaller dimension |
| vmax | 1% of larger dimension |
| dvh | Dynamic viewport height (accounts for mobile chrome) |
| svh | Small viewport height |
| lvh | Large viewport height |
### Responsive Testing Checklist
- [ ] 320px (small mobile)
- [ ] 375px (iPhone SE/8)
- [ ] 414px (iPhone Plus/Max)
- [ ] 768px (iPad portrait)
- [ ] 1024px (iPad landscape/laptop)
- [ ] 1280px (desktop)
- [ ] 1920px (large desktop)
### Common Device Widths
| Device | Width | Breakpoint |
|--------|-------|------------|
| iPhone SE | 375px | xs-sm |
| iPhone 14 | 390px | sm |
| iPhone 14 Pro Max | 430px | sm |
| iPad Mini | 768px | lg |
| iPad Pro 11" | 834px | lg |
| MacBook Air 13" | 1280px | xl |
| iMac 24" | 1920px | 2xl+ |
---
*See also: `token-generation.md` for breakpoint token details*
FILE:references/token-generation.md
# Design Token Generation Guide
Reference for color palette algorithms, typography scales, and WCAG accessibility checking.
---
## Table of Contents
- [Color Palette Generation](#color-palette-generation)
- [Typography Scale System](#typography-scale-system)
- [Spacing Grid System](#spacing-grid-system)
- [Accessibility Contrast](#accessibility-contrast)
- [Export Formats](#export-formats)
---
## Color Palette Generation
### HSV Color Space Algorithm
The token generator uses HSV (Hue, Saturation, Value) color space for precise control.
```
┌─────────────────────────────────────────────────────────────┐
│ COLOR SCALE GENERATION │
├─────────────────────────────────────────────────────────────┤
│ Input: Brand Color (#0066CC) │
│ ↓ │
│ Convert: Hex → RGB → HSV │
│ ↓ │
│ For each step (50, 100, 200... 900): │
│ • Adjust Value (brightness) │
│ • Adjust Saturation │
│ • Keep Hue constant │
│ ↓ │
│ Output: 10-step color scale │
└─────────────────────────────────────────────────────────────┘
```
### Brightness Algorithm
```python
# For light shades (50-400): High fixed brightness
if step < 500:
new_value = 0.95 # 95% brightness
# For dark shades (500-900): Exponential decrease
else:
new_value = base_value * (1 - (step - 500) / 500)
# At step 900: brightness ≈ base_value * 0.2
```
### Saturation Scaling
```python
# Saturation increases with step number
# 50 = 30% of base saturation
# 900 = 100% of base saturation
new_saturation = base_saturation * (0.3 + 0.7 * (step / 900))
```
### Complementary Color Generation
```
Brand Color: #0066CC (H=210°, S=100%, V=80%)
↓
Add 180° to Hue
↓
Secondary: #CC6600 (H=30°, S=100%, V=80%)
```
### Color Scale Output
| Step | Use Case | Brightness | Saturation |
|------|----------|------------|------------|
| 50 | Subtle backgrounds | 95% (fixed) | 30% |
| 100 | Light backgrounds | 95% (fixed) | 38% |
| 200 | Hover states | 95% (fixed) | 46% |
| 300 | Borders | 95% (fixed) | 54% |
| 400 | Disabled states | 95% (fixed) | 62% |
| 500 | Base color | Original | 70% |
| 600 | Hover (dark) | Original × 0.8 | 78% |
| 700 | Active states | Original × 0.6 | 86% |
| 800 | Text | Original × 0.4 | 94% |
| 900 | Headings | Original × 0.2 | 100% |
---
## Typography Scale System
### Modular Scale (Major Third)
The generator uses a **1.25x ratio** (major third) to create harmonious font sizes.
```
Base: 16px
Scale calculation:
Smaller sizes: 16px ÷ 1.25^n
Larger sizes: 16px × 1.25^n
Result:
xs: 10px (16 ÷ 1.25²)
sm: 13px (16 ÷ 1.25¹)
base: 16px
lg: 20px (16 × 1.25¹)
xl: 25px (16 × 1.25²)
2xl: 31px (16 × 1.25³)
3xl: 39px (16 × 1.25⁴)
4xl: 49px (16 × 1.25⁵)
5xl: 61px (16 × 1.25⁶)
```
### Type Scale Ratios
| Ratio | Name | Multiplier | Character |
|-------|------|------------|-----------|
| 1.067 | Minor Second | Tight | Compact UIs |
| 1.125 | Major Second | Subtle | App interfaces |
| 1.200 | Minor Third | Moderate | General use |
| **1.250** | **Major Third** | **Balanced** | **Default** |
| 1.333 | Perfect Fourth | Pronounced | Marketing |
| 1.414 | Augmented Fourth | Bold | Editorial |
| 1.618 | Golden Ratio | Dramatic | Headlines |
### Pre-composed Text Styles
| Style | Size | Weight | Line Height | Letter Spacing |
|-------|------|--------|-------------|----------------|
| h1 | 48px | 700 | 1.2 | -0.02em |
| h2 | 36px | 700 | 1.3 | -0.01em |
| h3 | 28px | 600 | 1.4 | 0 |
| h4 | 24px | 600 | 1.4 | 0 |
| h5 | 20px | 600 | 1.5 | 0 |
| h6 | 16px | 600 | 1.5 | 0.01em |
| body | 16px | 400 | 1.5 | 0 |
| small | 14px | 400 | 1.5 | 0 |
| caption | 12px | 400 | 1.5 | 0.01em |
---
## Spacing Grid System
### 8pt Grid Foundation
All spacing values are multiples of 8px for visual consistency.
```
Base Unit: 8px
Multipliers: 0, 0.5, 1, 1.5, 2, 2.5, 3, 4, 5, 6, 7, 8...
Results:
0: 0px
1: 4px (0.5 × 8)
2: 8px (1 × 8)
3: 12px (1.5 × 8)
4: 16px (2 × 8)
5: 20px (2.5 × 8)
6: 24px (3 × 8)
...
```
### Semantic Spacing Mapping
| Token | Numeric | Value | Use Case |
|-------|---------|-------|----------|
| xs | 1 | 4px | Inline icon margins |
| sm | 2 | 8px | Button padding |
| md | 4 | 16px | Card padding |
| lg | 6 | 24px | Section spacing |
| xl | 8 | 32px | Component gaps |
| 2xl | 12 | 48px | Section margins |
| 3xl | 16 | 64px | Page sections |
### Why 8pt Grid?
1. **Divisibility**: 8 divides evenly into common screen widths
2. **Consistency**: Creates predictable vertical rhythm
3. **Accessibility**: Touch targets naturally align to 48px (8 × 6)
4. **Integration**: Most design tools default to 8px grids
---
## Accessibility Contrast
### WCAG Contrast Requirements
| Level | Normal Text | Large Text | Definition |
|-------|-------------|------------|------------|
| AA | 4.5:1 | 3:1 | Minimum requirement |
| AAA | 7:1 | 4.5:1 | Enhanced accessibility |
**Large text**: ≥18pt regular or ≥14pt bold
### Contrast Ratio Formula
```
Contrast Ratio = (L1 + 0.05) / (L2 + 0.05)
Where:
L1 = Relative luminance of lighter color
L2 = Relative luminance of darker color
Relative Luminance:
L = 0.2126 × R + 0.7152 × G + 0.0722 × B
(Values linearized from sRGB)
```
### Color Step Contrast Guide
| Background | Minimum Text Step | For AA |
|------------|-------------------|--------|
| 50 | 700+ | Large text at 600 |
| 100 | 700+ | Large text at 600 |
| 200 | 800+ | Large text at 700 |
| 300 | 900 | - |
| 500 (base) | White or 50 | - |
| 700+ | White or 50-100 | - |
### Semantic Colors Accessibility
Generated semantic colors include contrast colors:
```json
{
"success": {
"base": "#10B981",
"light": "#34D399",
"dark": "#059669",
"contrast": "#FFFFFF" // For text on base
}
}
```
---
## Export Formats
### JSON Format
Best for: Design tool plugins, JavaScript/TypeScript projects, APIs
```json
{
"colors": {
"primary": {
"50": "#E6F2FF",
"500": "#0066CC",
"900": "#002855"
}
},
"typography": {
"fontSize": {
"base": "16px",
"lg": "20px"
}
}
}
```
### CSS Custom Properties
Best for: Web applications, CSS frameworks
```css
:root {
--colors-primary-50: #E6F2FF;
--colors-primary-500: #0066CC;
--colors-primary-900: #002855;
--typography-fontSize-base: 16px;
--typography-fontSize-lg: 20px;
}
```
### SCSS Variables
Best for: SCSS/SASS projects, component libraries
```scss
$colors-primary-50: #E6F2FF;
$colors-primary-500: #0066CC;
$colors-primary-900: #002855;
$typography-fontSize-base: 16px;
$typography-fontSize-lg: 20px;
```
### Format Selection Guide
| Format | When to Use |
|--------|-------------|
| JSON | Figma plugins, Storybook, JS/TS, design tool APIs |
| CSS | Plain CSS projects, CSS-in-JS (some), web apps |
| SCSS | SASS pipelines, component libraries, theming |
| Summary | Quick verification, debugging |
---
## Quick Reference
### Generation Command
```bash
# Default (modern style, JSON output)
python scripts/design_token_generator.py "#0066CC"
# Classic style, CSS output
python scripts/design_token_generator.py "#8B4513" classic css
# Playful style, summary view
python scripts/design_token_generator.py "#FF6B6B" playful summary
```
### Style Differences
| Aspect | Modern | Classic | Playful |
|--------|--------|---------|---------|
| Fonts | Inter, Fira Code | Helvetica, Courier | Poppins, Source Code Pro |
| Border Radius | 8px default | 4px default | 16px default |
| Shadows | Layered, subtle | Single layer | Soft, pronounced |
---
*See also: `component-architecture.md` for component design patterns*
FILE:scripts/design_token_generator.py
#!/usr/bin/env python3
"""
Design Token Generator
Creates consistent design system tokens for colors, typography, spacing, and more.
Usage:
python design_token_generator.py [brand_color] [style] [format]
brand_color: Hex color (default: #0066CC)
style: modern | classic | playful (default: modern)
format: json | css | scss | summary (default: json)
Examples:
python design_token_generator.py "#0066CC" modern json
python design_token_generator.py "#8B4513" classic css
python design_token_generator.py "#FF6B6B" playful summary
Table of Contents:
==================
CLASS: DesignTokenGenerator
__init__() - Initialize base unit (8pt), type scale (1.25x)
generate_complete_system() - Main entry: generates all token categories
generate_color_palette() - Primary, secondary, neutral, semantic colors
generate_typography_system() - Font families, sizes, weights, line heights
generate_spacing_system() - 8pt grid-based spacing scale
generate_sizing_tokens() - Container and component sizing
generate_border_tokens() - Border radius and width values
generate_shadow_tokens() - Shadow definitions per style
generate_animation_tokens() - Durations, easing, keyframes
generate_breakpoints() - Responsive breakpoints (xs-2xl)
generate_z_index_scale() - Z-index layering system
export_tokens() - Export to JSON/CSS/SCSS
PRIVATE METHODS:
_generate_color_scale() - Generate 10-step color scale (50-900)
_generate_neutral_scale() - Fixed neutral gray palette
_generate_type_scale() - Modular type scale using ratio
_generate_text_styles() - Pre-composed h1-h6, body, caption
_export_as_css() - CSS custom properties exporter
_hex_to_rgb() - Hex to RGB conversion
_rgb_to_hex() - RGB to Hex conversion
_adjust_hue() - HSV hue rotation utility
FUNCTION: main() - CLI entry point with argument parsing
Token Categories Generated:
- colors: primary, secondary, neutral, semantic, surface
- typography: fontFamily, fontSize, fontWeight, lineHeight, letterSpacing
- spacing: 0-64 scale based on 8pt grid
- sizing: containers, buttons, inputs, icons
- borders: radius (per style), width
- shadows: none through 2xl, inner
- animation: duration, easing, keyframes
- breakpoints: xs, sm, md, lg, xl, 2xl
- z-index: hide through notification
"""
import json
from typing import Dict, List, Tuple
import colorsys
class DesignTokenGenerator:
"""Generate comprehensive design system tokens"""
def __init__(self):
self.base_unit = 8 # 8pt grid system
self.type_scale_ratio = 1.25 # Major third
self.base_font_size = 16
def generate_complete_system(self, brand_color: str = "#0066CC",
style: str = "modern") -> Dict:
"""Generate complete design token system"""
tokens = {
'meta': {
'version': '1.0.0',
'style': style,
'generated': 'auto-generated'
},
'colors': self.generate_color_palette(brand_color),
'typography': self.generate_typography_system(style),
'spacing': self.generate_spacing_system(),
'sizing': self.generate_sizing_tokens(),
'borders': self.generate_border_tokens(style),
'shadows': self.generate_shadow_tokens(style),
'animation': self.generate_animation_tokens(),
'breakpoints': self.generate_breakpoints(),
'z-index': self.generate_z_index_scale()
}
return tokens
def generate_color_palette(self, brand_color: str) -> Dict:
"""Generate comprehensive color palette from brand color"""
# Convert hex to RGB
brand_rgb = self._hex_to_rgb(brand_color)
brand_hsv = colorsys.rgb_to_hsv(*[c/255 for c in brand_rgb])
palette = {
'primary': self._generate_color_scale(brand_color, 'primary'),
'secondary': self._generate_color_scale(
self._adjust_hue(brand_color, 180), 'secondary'
),
'neutral': self._generate_neutral_scale(),
'semantic': {
'success': {
'base': '#10B981',
'light': '#34D399',
'dark': '#059669',
'contrast': '#FFFFFF'
},
'warning': {
'base': '#F59E0B',
'light': '#FBBD24',
'dark': '#D97706',
'contrast': '#FFFFFF'
},
'error': {
'base': '#EF4444',
'light': '#F87171',
'dark': '#DC2626',
'contrast': '#FFFFFF'
},
'info': {
'base': '#3B82F6',
'light': '#60A5FA',
'dark': '#2563EB',
'contrast': '#FFFFFF'
}
},
'surface': {
'background': '#FFFFFF',
'foreground': '#111827',
'card': '#FFFFFF',
'overlay': 'rgba(0, 0, 0, 0.5)',
'divider': '#E5E7EB'
}
}
return palette
def _generate_color_scale(self, base_color: str, name: str) -> Dict:
"""Generate color scale from base color"""
scale = {}
rgb = self._hex_to_rgb(base_color)
h, s, v = colorsys.rgb_to_hsv(*[c/255 for c in rgb])
# Generate scale from 50 to 900
steps = [50, 100, 200, 300, 400, 500, 600, 700, 800, 900]
for step in steps:
# Adjust lightness based on step
factor = (1000 - step) / 1000
new_v = 0.95 if step < 500 else v * (1 - (step - 500) / 500)
new_s = s * (0.3 + 0.7 * (step / 900))
new_rgb = colorsys.hsv_to_rgb(h, new_s, new_v)
scale[str(step)] = self._rgb_to_hex([int(c * 255) for c in new_rgb])
scale['DEFAULT'] = base_color
return scale
def _generate_neutral_scale(self) -> Dict:
"""Generate neutral color scale"""
return {
'50': '#F9FAFB',
'100': '#F3F4F6',
'200': '#E5E7EB',
'300': '#D1D5DB',
'400': '#9CA3AF',
'500': '#6B7280',
'600': '#4B5563',
'700': '#374151',
'800': '#1F2937',
'900': '#111827',
'DEFAULT': '#6B7280'
}
def generate_typography_system(self, style: str) -> Dict:
"""Generate typography system"""
# Font families based on style
font_families = {
'modern': {
'sans': 'Inter, system-ui, -apple-system, sans-serif',
'serif': 'Merriweather, Georgia, serif',
'mono': 'Fira Code, Monaco, monospace'
},
'classic': {
'sans': 'Helvetica, Arial, sans-serif',
'serif': 'Times New Roman, Times, serif',
'mono': 'Courier New, monospace'
},
'playful': {
'sans': 'Poppins, Roboto, sans-serif',
'serif': 'Playfair Display, Georgia, serif',
'mono': 'Source Code Pro, monospace'
}
}
typography = {
'fontFamily': font_families.get(style, font_families['modern']),
'fontSize': self._generate_type_scale(),
'fontWeight': {
'thin': 100,
'light': 300,
'normal': 400,
'medium': 500,
'semibold': 600,
'bold': 700,
'extrabold': 800,
'black': 900
},
'lineHeight': {
'none': 1,
'tight': 1.25,
'snug': 1.375,
'normal': 1.5,
'relaxed': 1.625,
'loose': 2
},
'letterSpacing': {
'tighter': '-0.05em',
'tight': '-0.025em',
'normal': '0',
'wide': '0.025em',
'wider': '0.05em',
'widest': '0.1em'
},
'textStyles': self._generate_text_styles()
}
return typography
def _generate_type_scale(self) -> Dict:
"""Generate modular type scale"""
scale = {}
sizes = ['xs', 'sm', 'base', 'lg', 'xl', '2xl', '3xl', '4xl', '5xl']
for i, size in enumerate(sizes):
if size == 'base':
scale[size] = f'{self.base_font_size}px'
elif i < sizes.index('base'):
factor = self.type_scale_ratio ** (sizes.index('base') - i)
scale[size] = f'{round(self.base_font_size / factor)}px'
else:
factor = self.type_scale_ratio ** (i - sizes.index('base'))
scale[size] = f'{round(self.base_font_size * factor)}px'
return scale
def _generate_text_styles(self) -> Dict:
"""Generate pre-composed text styles"""
return {
'h1': {
'fontSize': '48px',
'fontWeight': 700,
'lineHeight': 1.2,
'letterSpacing': '-0.02em'
},
'h2': {
'fontSize': '36px',
'fontWeight': 700,
'lineHeight': 1.3,
'letterSpacing': '-0.01em'
},
'h3': {
'fontSize': '28px',
'fontWeight': 600,
'lineHeight': 1.4,
'letterSpacing': '0'
},
'h4': {
'fontSize': '24px',
'fontWeight': 600,
'lineHeight': 1.4,
'letterSpacing': '0'
},
'h5': {
'fontSize': '20px',
'fontWeight': 600,
'lineHeight': 1.5,
'letterSpacing': '0'
},
'h6': {
'fontSize': '16px',
'fontWeight': 600,
'lineHeight': 1.5,
'letterSpacing': '0.01em'
},
'body': {
'fontSize': '16px',
'fontWeight': 400,
'lineHeight': 1.5,
'letterSpacing': '0'
},
'small': {
'fontSize': '14px',
'fontWeight': 400,
'lineHeight': 1.5,
'letterSpacing': '0'
},
'caption': {
'fontSize': '12px',
'fontWeight': 400,
'lineHeight': 1.5,
'letterSpacing': '0.01em'
}
}
def generate_spacing_system(self) -> Dict:
"""Generate spacing system based on 8pt grid"""
spacing = {}
multipliers = [0, 0.5, 1, 1.5, 2, 2.5, 3, 4, 5, 6, 7, 8, 9, 10, 12, 14, 16, 20, 24, 32, 40, 48, 56, 64]
for i, mult in enumerate(multipliers):
spacing[str(i)] = f'{int(self.base_unit * mult)}px'
# Add semantic spacing
spacing.update({
'xs': spacing['1'], # 4px
'sm': spacing['2'], # 8px
'md': spacing['4'], # 16px
'lg': spacing['6'], # 24px
'xl': spacing['8'], # 32px
'2xl': spacing['12'], # 48px
'3xl': spacing['16'] # 64px
})
return spacing
def generate_sizing_tokens(self) -> Dict:
"""Generate sizing tokens for components"""
return {
'container': {
'sm': '640px',
'md': '768px',
'lg': '1024px',
'xl': '1280px',
'2xl': '1536px'
},
'components': {
'button': {
'sm': {'height': '32px', 'paddingX': '12px'},
'md': {'height': '40px', 'paddingX': '16px'},
'lg': {'height': '48px', 'paddingX': '20px'}
},
'input': {
'sm': {'height': '32px', 'paddingX': '12px'},
'md': {'height': '40px', 'paddingX': '16px'},
'lg': {'height': '48px', 'paddingX': '20px'}
},
'icon': {
'sm': '16px',
'md': '20px',
'lg': '24px',
'xl': '32px'
}
}
}
def generate_border_tokens(self, style: str) -> Dict:
"""Generate border tokens"""
radius_values = {
'modern': {
'none': '0',
'sm': '4px',
'DEFAULT': '8px',
'md': '12px',
'lg': '16px',
'xl': '24px',
'full': '9999px'
},
'classic': {
'none': '0',
'sm': '2px',
'DEFAULT': '4px',
'md': '6px',
'lg': '8px',
'xl': '12px',
'full': '9999px'
},
'playful': {
'none': '0',
'sm': '8px',
'DEFAULT': '16px',
'md': '20px',
'lg': '24px',
'xl': '32px',
'full': '9999px'
}
}
return {
'radius': radius_values.get(style, radius_values['modern']),
'width': {
'none': '0',
'thin': '1px',
'DEFAULT': '1px',
'medium': '2px',
'thick': '4px'
}
}
def generate_shadow_tokens(self, style: str) -> Dict:
"""Generate shadow tokens"""
shadow_styles = {
'modern': {
'none': 'none',
'sm': '0 1px 2px 0 rgba(0, 0, 0, 0.05)',
'DEFAULT': '0 1px 3px 0 rgba(0, 0, 0, 0.1), 0 1px 2px 0 rgba(0, 0, 0, 0.06)',
'md': '0 4px 6px -1px rgba(0, 0, 0, 0.1), 0 2px 4px -1px rgba(0, 0, 0, 0.06)',
'lg': '0 10px 15px -3px rgba(0, 0, 0, 0.1), 0 4px 6px -2px rgba(0, 0, 0, 0.05)',
'xl': '0 20px 25px -5px rgba(0, 0, 0, 0.1), 0 10px 10px -5px rgba(0, 0, 0, 0.04)',
'2xl': '0 25px 50px -12px rgba(0, 0, 0, 0.25)',
'inner': 'inset 0 2px 4px 0 rgba(0, 0, 0, 0.06)'
},
'classic': {
'none': 'none',
'sm': '0 1px 2px rgba(0, 0, 0, 0.1)',
'DEFAULT': '0 2px 4px rgba(0, 0, 0, 0.1)',
'md': '0 4px 8px rgba(0, 0, 0, 0.1)',
'lg': '0 8px 16px rgba(0, 0, 0, 0.1)',
'xl': '0 16px 32px rgba(0, 0, 0, 0.1)'
}
}
return shadow_styles.get(style, shadow_styles['modern'])
def generate_animation_tokens(self) -> Dict:
"""Generate animation tokens"""
return {
'duration': {
'instant': '0ms',
'fast': '150ms',
'DEFAULT': '250ms',
'slow': '350ms',
'slower': '500ms'
},
'easing': {
'linear': 'linear',
'ease': 'ease',
'easeIn': 'ease-in',
'easeOut': 'ease-out',
'easeInOut': 'ease-in-out',
'spring': 'cubic-bezier(0.68, -0.55, 0.265, 1.55)'
},
'keyframes': {
'fadeIn': {
'from': {'opacity': 0},
'to': {'opacity': 1}
},
'slideUp': {
'from': {'transform': 'translateY(10px)', 'opacity': 0},
'to': {'transform': 'translateY(0)', 'opacity': 1}
},
'scale': {
'from': {'transform': 'scale(0.95)'},
'to': {'transform': 'scale(1)'}
}
}
}
def generate_breakpoints(self) -> Dict:
"""Generate responsive breakpoints"""
return {
'xs': '480px',
'sm': '640px',
'md': '768px',
'lg': '1024px',
'xl': '1280px',
'2xl': '1536px'
}
def generate_z_index_scale(self) -> Dict:
"""Generate z-index scale"""
return {
'hide': -1,
'base': 0,
'dropdown': 1000,
'sticky': 1020,
'overlay': 1030,
'modal': 1040,
'popover': 1050,
'tooltip': 1060,
'notification': 1070
}
def export_tokens(self, tokens: Dict, format: str = 'json') -> str:
"""Export tokens in various formats"""
if format == 'json':
return json.dumps(tokens, indent=2)
elif format == 'css':
return self._export_as_css(tokens)
elif format == 'scss':
return self._export_as_scss(tokens)
else:
return json.dumps(tokens, indent=2)
def _export_as_css(self, tokens: Dict) -> str:
"""Export as CSS variables"""
css = [':root {']
def flatten_dict(obj, prefix=''):
for key, value in obj.items():
if isinstance(value, dict):
flatten_dict(value, f'{prefix}-{key}' if prefix else key)
else:
css.append(f' --{prefix}-{key}: {value};')
flatten_dict(tokens)
css.append('}')
return '\n'.join(css)
def _hex_to_rgb(self, hex_color: str) -> Tuple[int, int, int]:
"""Convert hex to RGB"""
hex_color = hex_color.lstrip('#')
return tuple(int(hex_color[i:i+2], 16) for i in (0, 2, 4))
def _rgb_to_hex(self, rgb: List[int]) -> str:
"""Convert RGB to hex"""
return '#{:02x}{:02x}{:02x}'.format(*rgb)
def _adjust_hue(self, hex_color: str, degrees: int) -> str:
"""Adjust hue of color"""
rgb = self._hex_to_rgb(hex_color)
h, s, v = colorsys.rgb_to_hsv(*[c/255 for c in rgb])
h = (h + degrees/360) % 1
new_rgb = colorsys.hsv_to_rgb(h, s, v)
return self._rgb_to_hex([int(c * 255) for c in new_rgb])
def main():
import sys
import argparse
parser = argparse.ArgumentParser(
description="Design Token Generator - Creates consistent design system tokens for colors, typography, spacing, and more."
)
parser.add_argument(
"brand_color", nargs="?", default="#0066CC",
help="Hex brand color (default: #0066CC)"
)
parser.add_argument(
"--style", choices=["modern", "classic", "playful"], default="modern",
help="Design style (default: modern)"
)
parser.add_argument(
"--format", choices=["json", "css", "scss", "summary"], default="json",
dest="output_format",
help="Output format (default: json)"
)
args = parser.parse_args()
generator = DesignTokenGenerator()
tokens = generator.generate_complete_system(args.brand_color, args.style)
if args.output_format == 'summary':
print("=" * 60)
print("DESIGN SYSTEM TOKENS")
print("=" * 60)
print(f"\n Style: {args.style}")
print(f" Brand Color: {args.brand_color}")
print("\n Generated Tokens:")
print(f" - Colors: {len(tokens['colors'])} palettes")
print(f" - Typography: {len(tokens['typography'])} categories")
print(f" - Spacing: {len(tokens['spacing'])} values")
print(f" - Shadows: {len(tokens['shadows'])} styles")
print(f" - Breakpoints: {len(tokens['breakpoints'])} sizes")
print("\n Export formats available: json, css, scss")
else:
print(generator.export_tokens(tokens, args.output_format))
if __name__ == "__main__":
main()
Triển khai và duy trì hệ thống quản lý chất lượng ISO 13485 cho thiết bị y tế: thiết kế QMS, kiểm soát tài liệu, đánh giá nội bộ, CAPA và hỗ trợ chứng nhận.
---
name: "quality-manager-qms-iso13485"
description: ISO 13485 Quality Management System implementation and maintenance for medical device organizations. Provides QMS design, documentation control, internal auditing, CAPA management, and certification support. Use when working with medical device quality systems, preparing for ISO 13485 audits, managing regulatory compliance documentation, setting up corrective actions, or building audit preparation programs. Useful for quality management, audit preparation, regulatory compliance, medical device documentation, and corrective action workflows.
triggers:
- ISO 13485
- QMS implementation
- quality management system
- document control
- internal audit
- management review
- quality manual
- CAPA process
- process validation
- design control
- supplier qualification
- quality records
---
# Quality Manager - QMS ISO 13485 Specialist
ISO 13485:2016 Quality Management System implementation, maintenance, and certification support for medical device organizations.
---
## Table of Contents
- [QMS Implementation Workflow](#qms-implementation-workflow)
- [Document Control Workflow](#document-control-workflow)
- [Internal Audit Workflow](#internal-audit-workflow)
- [Process Validation Workflow](#process-validation-workflow)
- [Supplier Qualification Workflow](#supplier-qualification-workflow)
- [QMS Process Reference](#qms-process-reference)
- [Decision Frameworks](#decision-frameworks)
- [Tools and References](#tools-and-references)
---
## QMS Implementation Workflow
Implement ISO 13485:2016 compliant quality management system from gap analysis through certification.
### Workflow: Initial QMS Implementation
1. Conduct gap analysis against ISO 13485:2016 requirements
2. Document current state vs. required state for each clause
3. Prioritize gaps by:
- Regulatory criticality
- Risk to product safety
- Resource requirements
4. Develop implementation roadmap with milestones
5. Establish Quality Manual per Clause 4.2.2:
- QMS scope with justified exclusions
- Process interactions
- Procedure references
6. Create required documented procedures — see [Mandatory Documented Procedures](#quick-reference-mandatory-documented-procedures) for the full list
7. Deploy processes with training
8. **Validation:** Gap analysis complete; Quality Manual approved; all required procedures documented and trained
> Use the Gap Analysis Matrix template in [qms-process-templates.md](references/qms-process-templates.md) to document clause-by-clause current state, gaps, priority, and actions.
### QMS Structure
| Level | Document Type | Example |
|-------|---------------|---------|
| 1 | Quality Manual | QM-001 |
| 2 | Procedures | SOP-02-001 |
| 3 | Work Instructions | WI-06-012 |
| 4 | Records | Training records |
---
## Document Control Workflow
Establish and maintain document control per ISO 13485 Clause 4.2.3.
### Workflow: Document Creation and Approval
1. Identify need for new document or revision
2. Assign document number per numbering convention:
- Format: `[TYPE]-[AREA]-[SEQUENCE]-[REV]`
- Example: `SOP-02-001-01`
3. Draft document using approved template
4. Route for review to subject matter experts
5. Collect and address review comments
6. Obtain required approvals based on document type
7. Update Document Master List
8. **Validation:** Document numbered correctly; all reviewers signed; Master List updated
### Document Numbering Convention
| Prefix | Document Type | Approval Authority |
|--------|---------------|-------------------|
| QM | Quality Manual | Management Rep + CEO |
| POL | Policy | Department Head + QA |
| SOP | Procedure | Process Owner + QA |
| WI | Work Instruction | Supervisor + QA |
| TF | Template/Form | Process Owner |
| SPEC | Specification | Engineering + QA |
### Area Codes
| Code | Area | Examples |
|------|------|----------|
| 01 | Quality Management | Quality Manual, policy |
| 02 | Document Control | This procedure |
| 03 | Training | Competency procedures |
| 04 | Design | Design control |
| 05 | Purchasing | Supplier management |
| 06 | Production | Manufacturing |
| 07 | Quality Control | Inspection, testing |
| 08 | CAPA | Corrective actions |
### Document Change Control
| Change Type | Approval Level | Examples |
|-------------|----------------|----------|
| Administrative | Document Control | Typos, formatting |
| Minor | Process Owner + QA | Clarifications |
| Major | Full review cycle | Process changes |
| Emergency | Expedited + retrospective | Safety issues |
### Document Review Schedule
| Document Type | Review Period | Trigger for Unscheduled Review |
|---------------|---------------|-------------------------------|
| Quality Manual | Annual | Organizational change |
| Procedures | Annual | Audit finding, regulation change |
| Work Instructions | 2 years | Process change |
| Forms | 2 years | User feedback |
---
## Internal Audit Workflow
Plan and execute internal audits per ISO 13485 Clause 8.2.4.
### Workflow: Annual Audit Program
1. Identify processes and areas requiring audit coverage
2. Assess risk factors for audit frequency:
- Previous audit findings
- Regulatory changes
- Process changes
- Complaint trends
3. Assign qualified auditors (independent of area audited)
4. Develop annual audit schedule
5. Obtain management approval
6. Communicate schedule to process owners
7. Track completion and reschedule as needed
8. **Validation:** All processes covered; auditors qualified and independent; schedule approved
> Use the Audit Program Template in [qms-process-templates.md](references/qms-process-templates.md) to schedule audits by clause and quarter across processes such as Document Control (4.2.3/4.2.4), Management Review (5.6), Design Control (7.3), Production (7.5), and CAPA (8.5.2/8.5.3).
### Workflow: Individual Audit Execution
1. Prepare audit plan with scope, criteria, and schedule
2. Notify auditee minimum 1 week prior
3. Review procedures and previous audit results
4. Prepare audit checklist
5. Conduct opening meeting
6. Collect evidence through:
- Document review
- Record sampling
- Process observation
- Personnel interviews
7. Classify findings:
- Major NC: Absence or breakdown of system
- Minor NC: Single lapse or deviation
- Observation: Risk of future NC
8. Conduct closing meeting
9. Issue audit report within 5 business days
10. **Validation:** All checklist items addressed; findings supported by evidence; report distributed
### Auditor Qualification Requirements
| Criterion | Requirement |
|-----------|-------------|
| Training | ISO 13485 awareness + auditor training |
| Experience | Minimum 1 audit as observer |
| Independence | Not auditing own work area |
| Competence | Understanding of audited process |
### Finding Classification Guide
| Classification | Criteria | Response Time |
|----------------|----------|---------------|
| Major NC | System absence, total breakdown, regulatory violation | 30 days for CAPA |
| Minor NC | Single instance, partial compliance | 60 days for CAPA |
| Observation | Potential risk, improvement opportunity | Track in next audit |
---
## Process Validation Workflow
Validate special processes per ISO 13485 Clause 7.5.6.
### Workflow: Process Validation Protocol
1. Identify processes requiring validation:
- Output cannot be verified by inspection
- Deficiencies appear only in use
- Sterilization, welding, sealing, software
2. Form validation team with subject matter experts
3. Write validation protocol including:
- Process description and parameters
- Equipment and materials
- Acceptance criteria
- Statistical approach
4. Execute IQ: verify equipment installed correctly and document specifications
5. Execute OQ: test parameter ranges and verify process control
6. Execute PQ: run production conditions and verify output meets requirements
7. Write validation report with conclusions
8. **Validation:** IQ/OQ/PQ complete; acceptance criteria met; validation report approved
### Validation Documentation Requirements
| Phase | Content | Evidence |
|-------|---------|----------|
| Protocol | Objectives, methods, criteria | Approved protocol |
| IQ | Equipment verification | Installation records |
| OQ | Parameter verification | Test results |
| PQ | Performance verification | Production data |
| Report | Summary, conclusions | Approval signatures |
### Revalidation Triggers
| Trigger | Action Required |
|---------|-----------------|
| Equipment change | Assess impact, revalidate affected phases |
| Parameter change | OQ and PQ minimum |
| Material change | Assess impact, PQ minimum |
| Process failure | Full revalidation |
| Periodic | Per validation schedule (typically 3 years) |
### Special Process Examples
| Process | Validation Standard | Critical Parameters |
|---------|--------------------|--------------------|
| EO Sterilization | ISO 11135 | Temperature, humidity, EO concentration, time |
| Steam Sterilization | ISO 17665 | Temperature, pressure, time |
| Radiation Sterilization | ISO 11137 | Dose, dose uniformity |
| Sealing | Internal | Temperature, pressure, dwell time |
| Welding | ISO 11607 | Heat, pressure, speed |
---
## Supplier Qualification Workflow
Evaluate and approve suppliers per ISO 13485 Clause 7.4.
### Workflow: New Supplier Qualification
1. Identify supplier category:
- Category A: Critical (affects safety/performance)
- Category B: Major (affects quality)
- Category C: Minor (indirect impact)
2. Request supplier information:
- Quality certifications
- Product specifications
- Quality history
3. Evaluate supplier based on:
- Quality system (ISO certification)
- Technical capability
- Quality history
- Financial stability
4. For Category A suppliers:
- Conduct on-site audit
- Require quality agreement
5. Calculate qualification score
6. Make approval decision:
- >80: Approved
- 60-80: Conditional approval
- <60: Not approved
7. Add to Approved Supplier List
8. **Validation:** Evaluation criteria scored; qualification records complete; supplier categorized
### Supplier Evaluation Criteria
| Criterion | Weight | Scoring |
|-----------|--------|---------|
| Quality System | 30% | ISO 13485=30, ISO 9001=20, Documented=10, None=0 |
| Quality History | 25% | Reject rate: <1%=25, 1-3%=15, >3%=0 |
| Delivery | 20% | On-time: >95%=20, 90-95%=10, <90%=0 |
| Technical Capability | 15% | Exceeds=15, Meets=10, Marginal=5 |
| Financial Stability | 10% | Strong=10, Adequate=5, Questionable=0 |
### Supplier Category Requirements
| Category | Qualification | Monitoring | Agreement |
|----------|---------------|------------|-----------|
| A - Critical | On-site audit | Annual review | Quality agreement |
| B - Major | Questionnaire | Semi-annual review | Quality requirements |
| C - Minor | Assessment | Issue-based | Standard terms |
### Supplier Performance Metrics
| Metric | Target | Calculation |
|--------|--------|-------------|
| Accept Rate | >98% | (Accepted lots / Total lots) × 100 |
| On-Time Delivery | >95% | (On-time / Total orders) × 100 |
| Response Time | <5 days | Average days to resolve issues |
| Documentation | 100% | (Complete CoCs / Required CoCs) × 100 |
---
## QMS Process Reference
For detailed requirements and audit questions for each ISO 13485:2016 clause, see [iso13485-clause-requirements.md](references/iso13485-clause-requirements.md).
### Management Review Required Inputs (Clause 5.6.2)
| Input | Source | Prepared By |
|-------|--------|-------------|
| Audit results | Internal and external audits | QA Manager |
| Customer feedback | Complaints, surveys | Customer Quality |
| Process performance | Process metrics | Process Owners |
| Product conformity | Inspection data, NCs | QC Manager |
| CAPA status | CAPA system | CAPA Officer |
| Previous actions | Prior review records | QMR |
| Changes affecting QMS | Regulatory, organizational | RA Manager |
| Recommendations | All sources | All Managers |
### Record Retention Requirements
| Record Type | Minimum Retention | Regulatory Basis |
|-------------|-------------------|------------------|
| Device Master Record | Life of device + 2 years | 21 CFR 820.181 |
| Device History Record | Life of device + 2 years | 21 CFR 820.184 |
| Design History File | Life of device + 2 years | 21 CFR 820.30 |
| Complaint Records | Life of device + 2 years | 21 CFR 820.198 |
| Training Records | Employment + 3 years | Best practice |
| Audit Records | 7 years | Best practice |
| CAPA Records | 7 years | Best practice |
| Calibration Records | Equipment life + 2 years | Best practice |
---
## Decision Frameworks
### Exclusion Justification (Clause 4.2.2)
| Clause | Permissible Exclusion | Justification Required |
|--------|----------------------|------------------------|
| 6.4.2 | Contamination control | Product not affected by contamination |
| 7.3 | Design and development | Organization does not design products |
| 7.5.2 | Product cleanliness | No cleanliness requirements |
| 7.5.3 | Installation | No installation activities |
| 7.5.4 | Servicing | No servicing activities |
| 7.5.5 | Sterile products | No sterile products |
### Nonconformity Disposition Decision Tree
```
Nonconforming Product Identified
│
▼
Can it be reworked?
│
Yes──┴──No
│ │
▼ ▼
Is rework Can it be used
procedure as is?
available? │
│ Yes──┴──No
Yes─┴─No │ │
│ │ ▼ ▼
▼ ▼ Concession Scrap or
Rework Create approval return to
per SOP rework needed? supplier
procedure │
Yes─┴─No
│ │
▼ ▼
Customer Use as is
approval with MRB
approval
```
### CAPA Initiation Criteria
| Source | Automatic CAPA | Evaluate for CAPA |
|--------|----------------|-------------------|
| Customer complaint | Safety-related | All others |
| External audit | Major NC | Minor NC |
| Internal audit | Major NC | Repeat minor NC |
| Product NC | Field failure | Trend exceeds threshold |
| Process deviation | Safety impact | Repeated deviations |
---
## Tools and References
### Scripts
| Tool | Purpose | Usage |
|------|---------|-------|
| [qms_audit_checklist.py](scripts/qms_audit_checklist.py) | Generate audit checklists by clause or process | `python qms_audit_checklist.py --help` |
**Audit Checklist Generator Features:**
- Generate clause-specific checklists (e.g., `--clause 7.3`)
- Generate process-based checklists (e.g., `--process design-control`)
- Full system audit checklist (`--audit-type system`)
- Text or JSON output formats
- Interactive mode for guided selection
### References
| Document | Content |
|----------|---------|
| [iso13485-clause-requirements.md](references/iso13485-clause-requirements.md) | Detailed requirements for each ISO 13485:2016 clause with audit questions |
| [qms-process-templates.md](references/qms-process-templates.md) | Ready-to-use templates for gap analysis, audit program, document control, CAPA, supplier, training |
### Quick Reference: Mandatory Documented Procedures
| Procedure | Clause | Key Elements |
|-----------|--------|--------------|
| Document Control | 4.2.3 | Approval, distribution, obsolete control |
| Record Control | 4.2.4 | Identification, retention, disposal |
| Internal Audit | 8.2.4 | Program, auditor qualification, reporting |
| NC Product Control | 8.3 | Identification, segregation, disposition |
| Corrective Action | 8.5.2 | Root cause, implementation, verification |
| Preventive Action | 8.5.3 | Risk identification, implementation |
---
## Related Skills
| Skill | Integration Point |
|-------|-------------------|
| [quality-manager-qmr](../quality-manager-qmr/) | Management review, quality policy |
| [capa-officer](../capa-officer/) | CAPA system management |
| [qms-audit-expert](../qms-audit-expert/) | Advanced audit techniques |
| [quality-documentation-manager](../quality-documentation-manager/) | DHF, DMR, DHR management |
| [risk-management-specialist](../risk-management-specialist/) | ISO 14971 integration |
FILE:references/iso13485-clause-requirements.md
# ISO 13485:2016 Clause Requirements
Detailed requirements for each ISO 13485:2016 clause with implementation guidance and audit criteria.
---
## Table of Contents
- [Clause 4: Quality Management System](#clause-4-quality-management-system)
- [Clause 5: Management Responsibility](#clause-5-management-responsibility)
- [Clause 6: Resource Management](#clause-6-resource-management)
- [Clause 7: Product Realization](#clause-7-product-realization)
- [Clause 8: Measurement, Analysis and Improvement](#clause-8-measurement-analysis-and-improvement)
---
## Clause 4: Quality Management System
### 4.1 General Requirements
| Requirement | Implementation | Evidence |
|-------------|----------------|----------|
| Determine processes needed | Process map showing QMS processes | Documented process map |
| Determine sequence and interaction | Process interaction diagram | Cross-reference matrix |
| Determine criteria for operation | Process metrics and acceptance criteria | Documented criteria per process |
| Ensure resources available | Resource allocation per process | Training records, equipment logs |
| Monitor, measure, analyze | Process monitoring procedures | Trend data, performance reports |
| Implement actions for results | Improvement projects, CAPAs | Action records with verification |
| Document processes | Procedures, work instructions | Controlled document list |
**Audit Questions:**
- How are QMS processes identified and documented?
- What criteria determine if processes are operating effectively?
- How is outsourced process control demonstrated?
### 4.2 Documentation Requirements
#### 4.2.1 General
| Document Type | Requirement | Retention |
|---------------|-------------|-----------|
| Quality Policy | Documented statement of commitment | Life of QMS |
| Quality Objectives | Measurable objectives at relevant functions | Life of QMS |
| Quality Manual | QMS scope and processes | Current version |
| Documented Procedures | Required by standard | Life of QMS + 2 years |
| Records | Evidence of conformity | As defined per record type |
#### 4.2.2 Quality Manual
**Required Content:**
1. Scope of QMS including justification for exclusions
2. Documented procedures or reference to them
3. Description of process interactions
**Quality Manual Template Structure:**
```
QUALITY MANUAL
1. Company Overview
1.1 Company Description
1.2 Scope of QMS
1.3 Exclusions and Justification
2. Quality Policy
3. Quality Objectives
4. QMS Structure
4.1 Process Map
4.2 Process Interactions
4.3 Organizational Chart
5. Procedure References
5.1 Document Control
5.2 Record Control
5.3 Management Review
5.4 Internal Audit
5.5 Nonconformity Control
5.6 CAPA
6. Appendices
6.1 Glossary
6.2 Regulatory Cross-Reference
```
#### 4.2.3 Control of Documents
| Control Element | Requirement | Method |
|-----------------|-------------|--------|
| Approval | Adequate prior to issue | Signature/electronic approval |
| Review and update | Re-approval after changes | Periodic review process |
| Identification of changes | Change history visible | Revision log in document |
| Revision status | Current revision identifiable | Document master list |
| Legibility | Readable and identifiable | Format standards |
| External documents | Identified and controlled | Incoming document log |
| Obsolete documents | Prevented from unintended use | Archive system |
**Document Numbering Convention:**
```
[TYPE]-[AREA]-[SEQUENCE]-[REV]
TYPE:
QM = Quality Manual
SOP = Standard Operating Procedure
WI = Work Instruction
TF = Template/Form
POL = Policy
AREA:
01 = Quality Management
02 = Document Control
03 = Training
04 = Design
05 = Purchasing
06 = Production
07 = Quality Control
08 = CAPA
Example: SOP-02-001-03 = Document Control SOP, Revision 03
```
#### 4.2.4 Control of Records
| Record Category | Minimum Retention | Basis |
|-----------------|-------------------|-------|
| Device Master Record | Life of device + 2 years | 21 CFR 820.181 |
| Device History Record | Life of device + 2 years | 21 CFR 820.184 |
| Design History File | Life of device + 2 years | 21 CFR 820.30 |
| Training Records | Employment + 3 years | Best practice |
| Audit Records | 7 years | Best practice |
| Complaint Records | Life of device + 2 years | 21 CFR 820.198 |
| CAPA Records | 7 years | Best practice |
| Calibration Records | Equipment life + 2 years | Best practice |
| Supplier Records | Relationship + 3 years | Best practice |
---
## Clause 5: Management Responsibility
### 5.1 Management Commitment
| Commitment Area | Evidence Required |
|-----------------|-------------------|
| Communicate importance of requirements | Meeting minutes, communications |
| Establish quality policy | Documented policy, communication records |
| Ensure quality objectives established | Objective documentation |
| Conduct management reviews | Management review records |
| Ensure resources available | Budget records, staffing records |
### 5.2 Customer Focus
| Requirement | Implementation | Verification |
|-------------|----------------|--------------|
| Customer requirements determined | Requirements review process | Contract review records |
| Requirements met | Process controls | Inspection and test data |
| Regulatory requirements met | Regulatory register | Compliance assessments |
| Customer satisfaction enhanced | Feedback collection | Satisfaction data, complaints |
### 5.3 Quality Policy
**Policy Requirements:**
- Appropriate to organization purpose
- Commitment to compliance and effectiveness
- Framework for quality objectives
- Communicated and understood
- Reviewed for continuing suitability
**Sample Quality Policy Elements:**
```
[Company Name] Quality Policy
We are committed to:
- Designing and manufacturing safe, effective medical devices
- Meeting customer and regulatory requirements
- Maintaining an effective Quality Management System
- Continuously improving our processes and products
- Providing resources for QMS effectiveness
Signed: [Executive]
Date: [Date]
Review Date: [Annual]
```
### 5.4 Planning
#### 5.4.1 Quality Objectives
| Objective Criteria | Requirement |
|-------------------|-------------|
| Measurable | Quantifiable targets |
| Consistent with policy | Aligned to policy statements |
| Relevant functions | Cascaded to departments |
| Includes compliance | Regulatory and customer requirements |
| Includes product conformity | Product-related targets |
**Objective Template:**
```
QUALITY OBJECTIVE [Year]
Objective: [Statement]
Metric: [How measured]
Target: [Specific value]
Baseline: [Current performance]
Owner: [Responsible person]
Due Date: [Target date]
Reporting: [Frequency]
```
#### 5.4.2 Quality Management System Planning
**Planning Requirements:**
- QMS meets general requirements (4.1)
- QMS meets quality objectives (5.4.1)
- Integrity maintained during changes
### 5.5 Responsibility, Authority and Communication
#### 5.5.1 Responsibility and Authority
| Role | Responsibilities | Authority |
|------|-----------------|-----------|
| Top Management | QMS commitment, resources, policy | Budget, staffing, strategic decisions |
| Quality Manager | QMS implementation, reporting | Document approval, CAPA approval |
| Department Managers | Process ownership, resources | Process changes, training |
| Process Owners | Process performance, improvements | Procedure changes within scope |
#### 5.5.2 Management Representative
| QMR Responsibility | Activities |
|-------------------|------------|
| QMS establishment | Process definition, documentation |
| QMS implementation | Training, deployment, monitoring |
| QMS maintenance | Audits, reviews, improvements |
| Reporting to top management | Performance reports, recommendations |
| Awareness promotion | Training, communications |
#### 5.5.3 Internal Communication
| Communication Type | Method | Frequency |
|-------------------|--------|-----------|
| Policy and objectives | Posting, training | Annual and on change |
| QMS performance | Dashboards, reports | Monthly |
| Changes affecting quality | Email, meetings | As needed |
| Audit results | Reports, presentations | Per audit |
### 5.6 Management Review
#### 5.6.1 General
| Requirement | Specification |
|-------------|---------------|
| Frequency | Planned intervals (typically quarterly/semi-annually) |
| Purpose | Assess QMS suitability, adequacy, effectiveness |
| Records | Documented meeting records |
#### 5.6.2 Review Input
| Input | Source | Responsible |
|-------|--------|-------------|
| Audit results | Internal/external audits | QA Manager |
| Customer feedback | Complaints, surveys | Customer Quality |
| Process performance | Metrics, yields | Process Owners |
| Product conformity | Inspection data | QC Manager |
| CAPA status | CAPA system | CAPA Officer |
| Previous actions | Prior review records | QMR |
| Changes affecting QMS | Regulatory, organizational | RA, HR |
| Recommendations | All sources | All Managers |
#### 5.6.3 Review Output
| Output | Documentation |
|--------|---------------|
| QMS improvement decisions | Action items with owners |
| Process improvements | Project charters |
| Resource needs | Resource allocation plans |
| Product improvements | Design change requests |
---
## Clause 6: Resource Management
### 6.1 Provision of Resources
**Resource Categories:**
- Human resources (competent personnel)
- Infrastructure (facilities, equipment, software)
- Work environment (environmental conditions)
### 6.2 Human Resources
| Requirement | Implementation | Evidence |
|-------------|----------------|----------|
| Competence determined | Job descriptions, competency matrix | Role definitions |
| Training provided | Training programs | Training records |
| Effectiveness evaluated | Assessments, observations | Competency verification |
| Awareness ensured | Orientation, ongoing training | Acknowledgments |
| Records maintained | Training database | Training files |
**Competency Matrix Template:**
```
COMPETENCY MATRIX
Role: [Job Title]
Department: [Department]
Required Competencies:
| Competency | Requirement Level | Method | Verification |
|------------|------------------|--------|--------------|
| [Skill 1] | Expert/Proficient/Basic | Training/OJT | Assessment |
| [Skill 2] | Expert/Proficient/Basic | Training/OJT | Assessment |
Training Requirements:
| Training | Initial | Refresher | Record |
|----------|---------|-----------|--------|
| ISO 13485 Awareness | Yes | Annual | TR-001 |
| Document Control | Yes | On Change | TR-002 |
```
### 6.3 Infrastructure
| Infrastructure Type | Control Requirements |
|--------------------|---------------------|
| Buildings and workspace | Cleaning, maintenance schedules |
| Process equipment | Maintenance, calibration |
| Supporting services | Utilities, IT systems |
| Information systems | Backup, security, validation |
### 6.4 Work Environment and Contamination Control
| Environment Factor | Control Method | Monitoring |
|-------------------|----------------|------------|
| Temperature | HVAC control | Continuous logging |
| Humidity | HVAC control | Continuous logging |
| Cleanliness | Cleaning procedures | Particle counts |
| Lighting | Lux levels | Periodic verification |
| ESD protection | Grounding, ionization | Periodic testing |
---
## Clause 7: Product Realization
### 7.1 Planning of Product Realization
| Planning Element | Content |
|-----------------|---------|
| Quality objectives for product | Product-specific quality targets |
| Processes and documentation | Process flow, required documents |
| Verification and validation | Test methods, acceptance criteria |
| Records | Required quality records |
| Risk management | Per ISO 14971 |
### 7.2 Customer-Related Processes
#### 7.2.1 Determination of Requirements
| Requirement Type | Source |
|-----------------|--------|
| Customer-specified | Contract, purchase order |
| Not stated but necessary | Intended use analysis |
| Regulatory | Applicable standards, regulations |
| Organization-defined | Internal specifications |
#### 7.2.2 Review of Requirements
| Review Element | Verification |
|----------------|--------------|
| Requirements defined | Complete specification |
| Differences resolved | Documented resolution |
| Ability to meet | Feasibility assessment |
| Risk management | Initial risk assessment |
#### 7.2.3 Communication
| Communication Type | Method |
|-------------------|--------|
| Product information | Catalogs, IFU |
| Inquiries and orders | Sales process |
| Feedback and complaints | Customer feedback system |
| Advisory notices | Field safety notices |
### 7.3 Design and Development
| Stage | Clause | Requirements |
|-------|--------|--------------|
| Planning | 7.3.2 | Stages, reviews, responsibilities |
| Inputs | 7.3.3 | Functional, performance, regulatory |
| Outputs | 7.3.4 | Meet inputs, acceptance criteria |
| Review | 7.3.5 | Evaluate ability to meet requirements |
| Verification | 7.3.6 | Outputs meet inputs |
| Validation | 7.3.7 | Product meets intended use |
| Transfer | 7.3.8 | Verified before production |
| Changes | 7.3.9 | Controlled, reviewed, verified |
### 7.4 Purchasing
#### 7.4.1 Purchasing Process
| Control Element | Implementation |
|-----------------|----------------|
| Supplier evaluation | Qualification procedure |
| Selection criteria | Quality, delivery, cost |
| Monitoring | Performance metrics |
| Re-evaluation | Periodic review |
**Supplier Classification:**
```
Category A: Critical - Affects product safety/performance
- Full qualification audit
- Annual performance review
- Quality agreement required
Category B: Major - Affects product quality
- Qualification questionnaire
- Periodic performance review
- Quality requirements communicated
Category C: Minor - Indirect impact
- Initial assessment
- Issue-based review
- Standard terms
```
#### 7.4.2 Purchasing Information
| Information Required | Purpose |
|---------------------|---------|
| Product specifications | Clear requirements |
| QMS requirements | Supplier system expectations |
| Personnel competence | Where applicable |
| Approval requirements | Where applicable |
#### 7.4.3 Verification of Purchased Product
| Verification Method | Application |
|--------------------|-------------|
| Incoming inspection | Standard verification |
| Source inspection | Critical items |
| Certificate of Conformance | Documented evidence |
| Certificate of Analysis | Material verification |
### 7.5 Production and Service Provision
#### 7.5.1 Control of Production and Service Provision
| Control Element | Implementation |
|-----------------|----------------|
| Product information | Specifications, drawings |
| Work instructions | Where necessary |
| Suitable equipment | Qualified equipment |
| Monitoring devices | Calibrated instruments |
| Implementation of monitoring | Inspections, tests |
| Defined processes | Process parameters |
| Labeling and packaging | Per requirements |
#### 7.5.2 Cleanliness of Product
| Cleanliness Control | Method |
|--------------------|--------|
| Product cleaning | Validated procedures |
| Contamination prevention | Controlled environment |
| Process aids | Qualified, controlled |
#### 7.5.3 Installation Activities
| Requirement | Implementation |
|-------------|----------------|
| Installation requirements | Documented instructions |
| Acceptance criteria | Defined criteria |
| Records | Installation records |
#### 7.5.4 Servicing Activities
| Requirement | Implementation |
|-------------|----------------|
| Documented requirements | Service procedures |
| Reference materials | Service manuals |
| Measurement equipment | Calibrated |
| Records | Service records |
#### 7.5.5 Particular Requirements for Sterile Medical Devices
| Process | Control |
|---------|---------|
| Sterilization validation | Per ISO 11135/11137/17665 |
| Parameter control | Monitoring records |
| Sterile barrier | Validated packaging |
#### 7.5.6 Validation of Processes
| Validation Required When | Evidence |
|-------------------------|----------|
| Output cannot be verified | Validation protocol and report |
| Deficiencies appear only in use | Process capability data |
| Special processes | Qualified operators |
**Process Validation Elements:**
- Equipment qualification (IQ/OQ/PQ)
- Process parameters
- Monitoring methods
- Operator qualification
- Revalidation criteria
#### 7.5.7 Particular Requirements for Validation
| Requirement | Implementation |
|-------------|----------------|
| Documented procedures | Validation SOPs |
| Defined methods | Statistical methods |
| Acceptance criteria | Predefined criteria |
| Software validation | Where applicable |
| Revalidation | Change-triggered |
#### 7.5.8 Identification
| Identification Type | Method |
|--------------------|--------|
| Product | Labels, markings |
| Documentation | Document numbers |
| Unique Device Identification | UDI per regulation |
#### 7.5.9 Traceability
| Traceability Element | Record |
|---------------------|--------|
| Components | Lot/batch numbers |
| Materials | Certificates |
| Work environment | Environmental records |
| Measurement equipment | Calibration records |
| Personnel | Training records |
| Distribution | Shipping records |
#### 7.5.10 Customer Property
| Control | Implementation |
|---------|----------------|
| Identification | Marking, segregation |
| Verification | Incoming inspection |
| Protection | Storage conditions |
| Safeguarding | Security measures |
| Reporting | Loss/damage notification |
#### 7.5.11 Preservation of Product
| Preservation Element | Control |
|---------------------|---------|
| Identification | Labels, markings |
| Handling | Procedures |
| Packaging | Specifications |
| Storage | Conditions, FIFO |
| Protection | Environmental controls |
### 7.6 Control of Monitoring and Measuring Equipment
| Control Element | Implementation |
|-----------------|----------------|
| Calibration | At specified intervals |
| Adjustment | As needed |
| Identification | Calibration status |
| Safeguarding | Protection from damage |
| Software validation | Where applicable |
| Records | Calibration records |
---
## Clause 8: Measurement, Analysis and Improvement
### 8.1 General
**Monitoring and Measurement Requirements:**
- Demonstrate product conformity
- Ensure QMS conformity
- Maintain QMS effectiveness
### 8.2 Monitoring and Measurement
#### 8.2.1 Feedback
| Feedback Source | Collection Method |
|-----------------|-------------------|
| Customer complaints | Complaint system |
| Customer surveys | Periodic surveys |
| Field feedback | Service reports |
| Regulatory feedback | Inspection findings |
#### 8.2.2 Complaint Handling
| Process Step | Requirements |
|--------------|--------------|
| Receipt | Timely logging |
| Investigation | Root cause analysis |
| Corrective action | If warranted |
| Regulatory reporting | If required |
| Trend analysis | Aggregate review |
#### 8.2.3 Reporting to Regulatory Authorities
| Report Type | Trigger | Timeline |
|-------------|---------|----------|
| MDR (Medical Device Report) | Death/serious injury | 30 days (5 if awareness) |
| FSCA (Field Safety Corrective Action) | Safety issue | Without delay |
| Periodic Safety Update | Per regulation | Per schedule |
#### 8.2.4 Internal Audit
| Audit Element | Requirement |
|---------------|-------------|
| Planned program | Risk-based schedule |
| Criteria and scope | Defined per audit |
| Auditor selection | Independent, competent |
| Procedure | Documented process |
| Records | Audit reports, findings |
| Follow-up | CAPA, verification |
**Audit Program Template:**
```
ANNUAL INTERNAL AUDIT PROGRAM
Year: [Year]
| Audit # | Area/Process | Scope | Auditor | Planned Date | Status |
|---------|--------------|-------|---------|--------------|--------|
| IA-01 | Document Control | 4.2.3, 4.2.4 | [Name] | Q1 | |
| IA-02 | Design Control | 7.3 | [Name] | Q2 | |
| IA-03 | Production | 7.5 | [Name] | Q2 | |
| IA-04 | Purchasing | 7.4 | [Name] | Q3 | |
| IA-05 | CAPA | 8.5.2, 8.5.3 | [Name] | Q3 | |
| IA-06 | Management Review | 5.6 | [Name] | Q4 | |
Risk Considerations:
- Previous audit findings
- Regulatory changes
- Process changes
- Complaint trends
```
#### 8.2.5 Monitoring and Measurement of Processes
| Monitoring Type | Method |
|-----------------|--------|
| Process metrics | KPIs, trend analysis |
| Process audits | Internal audits |
| Process reviews | Management review |
#### 8.2.6 Monitoring and Measurement of Product
| Stage | Verification |
|-------|--------------|
| Incoming | Incoming inspection |
| In-process | In-process inspection |
| Final | Final inspection and test |
| Release | Authorized release |
### 8.3 Control of Nonconforming Product
| Control Element | Requirement |
|-----------------|-------------|
| Identification | Clear marking |
| Segregation | Physical separation |
| Documentation | NC record |
| Disposition | Use as is/rework/scrap/return |
| Concession | If accepted |
| Reinspection | After rework |
| Investigation | For detected after delivery |
**Nonconformity Disposition Options:**
```
1. Use As Is (Concession)
- Does not affect safety/performance
- Customer approval if applicable
- Documented justification
2. Rework
- Per approved procedure
- Reinspection required
- Records maintained
3. Scrap/Reject
- Physical destruction or marking
- Prevented from reentry
- Documented disposal
4. Return to Supplier
- Communication with supplier
- Replacement or credit
- Root cause if systemic
```
### 8.4 Analysis of Data
| Data Source | Analysis |
|-------------|----------|
| Feedback | Complaint trends, satisfaction |
| Nonconformity | Defect Pareto, trends |
| Process performance | Capability, trends |
| Supplier | Performance trends |
| Audit | Finding trends |
### 8.5 Improvement
#### 8.5.1 General
**Improvement Sources:**
- Quality policy
- Quality objectives
- Audit results
- Data analysis
- Corrective actions
- Preventive actions
- Management review
#### 8.5.2 Corrective Action
| Process Step | Requirement |
|--------------|-------------|
| Review nonconformity | Including complaints |
| Determine cause | Root cause analysis |
| Evaluate action need | Based on risk |
| Determine action | Proportionate to risk |
| Implement action | Execute plan |
| Document results | Records |
| Review effectiveness | Verification |
#### 8.5.3 Preventive Action
| Process Step | Requirement |
|--------------|-------------|
| Determine potential NC | Risk analysis, trends |
| Evaluate action need | Prevention opportunity |
| Determine action | Proportionate to risk |
| Implement action | Execute plan |
| Document results | Records |
| Review effectiveness | Verification |
FILE:references/qms-process-templates.md
# QMS Process Templates
Ready-to-use templates for ISO 13485 QMS processes including document control, internal audit, CAPA, and supplier management.
---
## Table of Contents
- [Document Control Templates](#document-control-templates)
- [Internal Audit Templates](#internal-audit-templates)
- [CAPA Templates](#capa-templates)
- [Supplier Management Templates](#supplier-management-templates)
- [Training Templates](#training-templates)
- [Nonconformity Templates](#nonconformity-templates)
---
## Document Control Templates
### Document Master List
```
DOCUMENT MASTER LIST
Organization: [Company Name]
Last Updated: [Date]
Maintained By: Document Control
| Doc # | Title | Rev | Effective Date | Status | Owner | Next Review |
|-------|-------|-----|----------------|--------|-------|-------------|
| QM-001 | Quality Manual | 03 | 2024-01-15 | Effective | QMR | 2025-01-15 |
| SOP-01-001 | Document Control | 04 | 2024-03-01 | Effective | QA Mgr | 2025-03-01 |
| SOP-01-002 | Record Control | 02 | 2024-02-01 | Effective | QA Mgr | 2025-02-01 |
| | | | | | | |
Status Values: Draft, Under Review, Effective, Obsolete
```
### Document Change Request
```
DOCUMENT CHANGE REQUEST
DCR Number: DCR-[YYYY]-[NNN]
Date Submitted: [Date]
Submitted By: [Name]
DOCUMENT INFORMATION
Document Number: [Number]
Document Title: [Title]
Current Revision: [Rev]
CHANGE REQUEST
Change Type: [ ] Administrative [ ] Minor [ ] Major [ ] Emergency
Requested Change: [Description of change]
Reason for Change:
[ ] Regulatory requirement
[ ] Process improvement
[ ] Nonconformity/CAPA
[ ] Organizational change
[ ] Error correction
[ ] Other: [Specify]
Justification: [Detailed justification]
IMPACT ASSESSMENT
Training Required: [ ] Yes [ ] No
If yes, who: [Roles/departments]
Other Documents Affected: [List]
Regulatory Filing Impact: [ ] Yes [ ] No
If yes, details: [Explain]
APPROVALS
Requested By: _________________ Date: _______
Document Owner: _________________ Date: _______
QA Approval: _________________ Date: _______
COMPLETION
New Revision: [Rev]
Effective Date: [Date]
Training Completed: [ ] Yes [ ] N/A
Distribution Completed: [ ] Yes
```
### Document Review Record
```
DOCUMENT REVIEW RECORD
Document Number: [Number]
Document Title: [Title]
Current Revision: [Rev]
Review Due Date: [Date]
Review Completed: [Date]
REVIEWERS
| Reviewer | Role | Review Date | Comments | Signature |
|----------|------|-------------|----------|-----------|
| [Name] | [Role] | [Date] | [Comments] | |
| [Name] | [Role] | [Date] | [Comments] | |
REVIEW OUTCOME
[ ] No changes required - document remains current
[ ] Minor changes required - see attached DCR
[ ] Major revision required - see attached DCR
[ ] Document obsolete - initiate retirement
NEXT REVIEW
Next Review Date: [Date]
APPROVAL
Review Completed By: _________________ Date: _______
Approved By: _________________ Date: _______
```
---
## Internal Audit Templates
### Annual Audit Schedule
```
INTERNAL AUDIT SCHEDULE
Year: [Year]
Prepared By: [Name]
Approved By: [Name]
Date: [Date]
AUDIT SCHEDULE
| Audit # | Process/Area | ISO Clauses | Lead Auditor | Q1 | Q2 | Q3 | Q4 |
|---------|--------------|-------------|--------------|----|----|----|----|
| IA-001 | Document Control | 4.2.3, 4.2.4 | [Name] | X | | | |
| IA-002 | Management Review | 5.6 | [Name] | | X | | |
| IA-003 | Training | 6.2 | [Name] | | X | | |
| IA-004 | Design Control | 7.3 | [Name] | | | X | |
| IA-005 | Purchasing | 7.4 | [Name] | | | X | |
| IA-006 | Production | 7.5 | [Name] | | | | X |
| IA-007 | CAPA | 8.5.2, 8.5.3 | [Name] | | | | X |
RISK FACTORS CONSIDERED
[ ] Previous audit findings
[ ] Regulatory changes
[ ] Process changes
[ ] Complaint trends
[ ] Management concerns
SCHEDULE REVISION LOG
| Rev | Date | Change | Approved By |
|-----|------|--------|-------------|
| 00 | [Date] | Initial release | [Name] |
```
### Audit Plan
```
INTERNAL AUDIT PLAN
Audit Number: IA-[YYYY]-[NNN]
Audit Date(s): [Date(s)]
Audit Type: [ ] Process [ ] System [ ] Product
SCOPE
Process/Area: [Name]
ISO 13485 Clauses: [List]
Regulatory Requirements: [If applicable]
Locations: [Locations]
AUDIT TEAM
Lead Auditor: [Name]
Auditor(s): [Names]
Observer(s): [If any]
AUDITEE CONTACTS
Process Owner: [Name]
Other Contacts: [Names]
AUDIT CRITERIA
- ISO 13485:2016
- [Organization procedures]
- [Regulatory requirements]
AUDIT SCHEDULE
| Time | Activity | Participants |
|------|----------|--------------|
| 09:00 | Opening meeting | All |
| 09:30 | Document review | Auditor, Doc Control |
| 10:30 | Process observation | Auditor, Operators |
| 12:00 | Lunch | |
| 13:00 | Record review | Auditor, QA |
| 14:30 | Interviews | Selected personnel |
| 15:30 | Auditor caucus | Audit team |
| 16:00 | Closing meeting | All |
PREPARATION CHECKLIST
[ ] Previous audit reports reviewed
[ ] Procedures reviewed
[ ] Checklist prepared
[ ] Auditees notified
[ ] Resources arranged
```
### Audit Checklist Template
```
INTERNAL AUDIT CHECKLIST
Audit Number: IA-[YYYY]-[NNN]
Process: [Process Name]
Auditor: [Name]
Date: [Date]
INSTRUCTIONS
C = Conforming, NC = Nonconforming, OBS = Observation, N/A = Not Applicable
CHECKLIST
| # | Requirement | Reference | Evidence Reviewed | Finding | Notes |
|---|-------------|-----------|-------------------|---------|-------|
| 1 | Is the procedure current and approved? | 4.2.3 | [Evidence] | C/NC/OBS | |
| 2 | Are personnel trained on the procedure? | 6.2 | [Evidence] | C/NC/OBS | |
| 3 | Are records maintained as required? | 4.2.4 | [Evidence] | C/NC/OBS | |
| 4 | Is the process performed as documented? | 4.1 | [Evidence] | C/NC/OBS | |
| 5 | Are monitoring activities performed? | 8.2.5 | [Evidence] | C/NC/OBS | |
INTERVIEWS CONDUCTED
| Person | Role | Topics Discussed |
|--------|------|------------------|
| [Name] | [Role] | [Topics] |
DOCUMENTS REVIEWED
| Document # | Title | Rev | Findings |
|------------|-------|-----|----------|
| [Number] | [Title] | [Rev] | [Findings] |
RECORDS SAMPLED
| Record Type | Sample Size | Sample IDs | Findings |
|-------------|-------------|------------|----------|
| [Type] | [N] | [IDs] | [Findings] |
AUDITOR SIGNATURE: _________________ Date: _______
```
### Audit Report
```
INTERNAL AUDIT REPORT
Audit Number: IA-[YYYY]-[NNN]
Report Date: [Date]
Report Status: [ ] Draft [ ] Final
AUDIT SUMMARY
Audit Date(s): [Date(s)]
Process/Area: [Name]
ISO Clauses Covered: [List]
Lead Auditor: [Name]
Audit Team: [Names]
AUDIT SCOPE
[Description of scope]
AUDIT OBJECTIVES
[List objectives]
EXECUTIVE SUMMARY
[Brief summary of audit results]
FINDINGS SUMMARY
| Type | Count |
|------|-------|
| Major Nonconformity | [N] |
| Minor Nonconformity | [N] |
| Observation | [N] |
| Opportunity for Improvement | [N] |
DETAILED FINDINGS
FINDING 1
Number: IA-[YYYY]-[NNN]-F01
Classification: [ ] Major NC [ ] Minor NC [ ] Observation [ ] OFI
Requirement: [Clause/requirement reference]
Statement: [Objective description of finding]
Evidence: [Evidence supporting finding]
Auditee Response Due: [Date]
[Repeat for each finding]
POSITIVE OBSERVATIONS
[List areas of good practice observed]
CONCLUSION
[Overall conclusion on process effectiveness]
REPORT DISTRIBUTION
| Name | Role | Date |
|------|------|------|
| [Name] | Process Owner | [Date] |
| [Name] | QA Manager | [Date] |
| [Name] | Management Rep | [Date] |
APPROVALS
Lead Auditor: _________________ Date: _______
QA Manager: _________________ Date: _______
```
---
## CAPA Templates
### CAPA Request Form
```
CORRECTIVE AND PREVENTIVE ACTION REQUEST
CAPA Number: CAPA-[YYYY]-[NNN]
Date Opened: [Date]
Initiated By: [Name]
CAPA TYPE
[ ] Corrective Action (response to existing nonconformity)
[ ] Preventive Action (prevent potential nonconformity)
SOURCE
[ ] Customer complaint: Reference #_______
[ ] Internal audit: Audit #_______
[ ] External audit: Audit #_______
[ ] Nonconformity: NC #_______
[ ] Process deviation
[ ] Management review action
[ ] Trend analysis
[ ] Risk assessment
[ ] Other: _______
CLASSIFICATION
Severity: [ ] Critical [ ] Major [ ] Minor
Regulatory Reportable: [ ] Yes [ ] No
PROBLEM DESCRIPTION
[Detailed description of the problem or potential problem]
IMMEDIATE CONTAINMENT (if applicable)
Actions Taken: [Description]
Date: [Date]
Responsible: [Name]
ASSIGNMENT
Process Owner: [Name]
CAPA Owner: [Name]
Due Date for Root Cause: [Date]
Target Closure Date: [Date]
APPROVAL TO PROCEED
Approved By: _________________ Date: _______
```
### Root Cause Analysis Record
```
ROOT CAUSE ANALYSIS
CAPA Number: CAPA-[YYYY]-[NNN]
Analysis Date: [Date]
Analyst: [Name]
PROBLEM STATEMENT
[Clear, specific statement of the problem]
INVESTIGATION TEAM
| Name | Role | Contribution |
|------|------|--------------|
| [Name] | [Role] | [Area of expertise] |
INVESTIGATION METHOD
[ ] 5 Why Analysis
[ ] Fishbone Diagram
[ ] Fault Tree Analysis
[ ] Human Factors Analysis
[ ] Other: _______
INVESTIGATION DETAILS
5 WHY ANALYSIS
Why 1: [First why]
Answer: [Answer]
Why 2: [Second why based on answer]
Answer: [Answer]
Why 3: [Third why based on answer]
Answer: [Answer]
Why 4: [Fourth why based on answer]
Answer: [Answer]
Why 5: [Fifth why based on answer]
Answer: [Answer]
ROOT CAUSE STATEMENT
[Clear statement of identified root cause]
ROOT CAUSE CATEGORY
[ ] Process/Procedure
[ ] Training/Competency
[ ] Equipment/Material
[ ] Design
[ ] Human Error
[ ] Communication
[ ] Management System
[ ] External Factor
CONTRIBUTING FACTORS
[List any contributing factors]
EVIDENCE SUPPORTING ROOT CAUSE
[List evidence]
APPROVAL
Analysis By: _________________ Date: _______
Reviewed By: _________________ Date: _______
```
### CAPA Action Plan
```
CAPA ACTION PLAN
CAPA Number: CAPA-[YYYY]-[NNN]
Root Cause: [Brief statement]
Plan Date: [Date]
Plan Owner: [Name]
CORRECTIVE/PREVENTIVE ACTIONS
Action 1:
Description: [Detailed action description]
Responsible: [Name]
Due Date: [Date]
Resources Required: [Resources]
Success Criteria: [How completion verified]
Action 2:
Description: [Detailed action description]
Responsible: [Name]
Due Date: [Date]
Resources Required: [Resources]
Success Criteria: [How completion verified]
[Continue for additional actions]
RELATED CHANGES
Documents Affected: [List]
Training Required: [Description]
Process Changes: [Description]
Equipment Changes: [Description]
RISK ASSESSMENT
Residual Risk After Implementation: [ ] High [ ] Medium [ ] Low
Justification: [Explanation]
APPROVAL
Plan Developed By: _________________ Date: _______
Approved By: _________________ Date: _______
```
### CAPA Effectiveness Verification
```
CAPA EFFECTIVENESS VERIFICATION
CAPA Number: CAPA-[YYYY]-[NNN]
Verification Date: [Date]
Verified By: [Name]
ACTIONS COMPLETED
| Action | Completion Date | Evidence |
|--------|-----------------|----------|
| [Action 1] | [Date] | [Reference] |
| [Action 2] | [Date] | [Reference] |
EFFECTIVENESS CRITERIA
[Criteria established during action planning]
VERIFICATION METHOD
[ ] Data analysis (trends, metrics)
[ ] Process audit
[ ] Record review
[ ] Product inspection
[ ] Customer feedback review
[ ] Other: _______
VERIFICATION PERIOD
From: [Date] To: [Date]
VERIFICATION RESULTS
[Detailed results of verification activities]
DATA/EVIDENCE REVIEWED
| Data Type | Period | Result |
|-----------|--------|--------|
| [Type] | [Period] | [Result] |
EFFECTIVENESS CONCLUSION
[ ] Effective - Root cause eliminated, problem resolved
[ ] Partially Effective - Improvement noted, additional action needed
[ ] Not Effective - Problem persists, reopen CAPA
If not effective, describe additional actions:
[Description]
CAPA CLOSURE
[ ] Approved for closure
[ ] Not approved - additional action required
Verified By: _________________ Date: _______
Approved By: _________________ Date: _______
```
---
## Supplier Management Templates
### Approved Supplier List
```
APPROVED SUPPLIER LIST
Organization: [Company Name]
Last Updated: [Date]
Maintained By: [Name]
| Supplier | Supplier # | Category | Products/Services | Status | Qualification Date | Next Review |
|----------|-----------|----------|-------------------|--------|-------------------|-------------|
| [Name] | SUP-001 | A | [Products] | Approved | [Date] | [Date] |
| [Name] | SUP-002 | B | [Products] | Conditional | [Date] | [Date] |
Category:
A = Critical (affects safety/performance)
B = Major (affects quality)
C = Minor (indirect impact)
Status:
Approved = Full use authorized
Conditional = Limited use, monitoring
Probation = Performance issues, enhanced monitoring
Disqualified = Use not authorized
Revision History:
| Rev | Date | Change | Approved By |
|-----|------|--------|-------------|
| 01 | [Date] | Initial release | [Name] |
```
### Supplier Evaluation Form
```
SUPPLIER EVALUATION
Supplier Name: [Name]
Supplier Number: [Number]
Evaluation Date: [Date]
Evaluated By: [Name]
Evaluation Type: [ ] Initial [ ] Periodic [ ] For Cause
SUPPLIER INFORMATION
Address: [Address]
Contact: [Name, Title]
Phone: [Phone]
Email: [Email]
Products/Services: [Description]
PROPOSED CATEGORY
[ ] A - Critical (affects safety/performance)
[ ] B - Major (affects quality)
[ ] C - Minor (indirect impact)
EVALUATION CRITERIA
1. QUALITY MANAGEMENT SYSTEM (30 points max)
[ ] ISO 13485 Certified (30 pts)
[ ] ISO 9001 Certified (20 pts)
[ ] Documented QMS (10 pts)
[ ] No formal QMS (0 pts)
Score: ___/30
2. QUALITY HISTORY (25 points max)
Reject Rate: ___% (0-1% = 25 pts, 1-3% = 15 pts, >3% = 0 pts)
Score: ___/25
3. DELIVERY PERFORMANCE (20 points max)
On-Time Delivery: ___% (>95% = 20 pts, 90-95% = 10 pts, <90% = 0 pts)
Score: ___/20
4. TECHNICAL CAPABILITY (15 points max)
[ ] Exceeds requirements (15 pts)
[ ] Meets requirements (10 pts)
[ ] Marginally meets (5 pts)
Score: ___/15
5. FINANCIAL STABILITY (10 points max)
[ ] Strong (10 pts)
[ ] Adequate (5 pts)
[ ] Questionable (0 pts)
Score: ___/10
TOTAL SCORE: ___/100
QUALIFICATION DECISION
>80 = Approved
60-80 = Conditional (monitoring required)
<60 = Not Approved
Decision: [ ] Approved [ ] Conditional [ ] Not Approved
APPROVAL
Evaluated By: _________________ Date: _______
QA Approval: _________________ Date: _______
```
### Supplier Performance Scorecard
```
SUPPLIER PERFORMANCE SCORECARD
Supplier: [Name]
Supplier #: [Number]
Period: [Q1/Q2/Q3/Q4] [Year]
Prepared By: [Name]
PERFORMANCE METRICS
1. QUALITY (40% weight)
Total Lots Received: [N]
Lots Rejected: [N]
Accept Rate: ___% Target: >98%
Score: ___/40
2. DELIVERY (30% weight)
Total Orders: [N]
On-Time Deliveries: [N]
On-Time Rate: ___% Target: >95%
Score: ___/30
3. RESPONSIVENESS (15% weight)
Issues Reported: [N]
Resolved <5 days: [N]
Response Rate: ___% Target: >90%
Score: ___/15
4. DOCUMENTATION (15% weight)
CoC Required: [N]
CoC Complete: [N]
Documentation Rate: ___% Target: 100%
Score: ___/15
TOTAL SCORE: ___/100
PERFORMANCE TREND
| Period | Quality | Delivery | Response | Docs | Total |
|--------|---------|----------|----------|------|-------|
| Q1 | | | | | |
| Q2 | | | | | |
| Q3 | | | | | |
| Q4 | | | | | |
ISSUES/CONCERNS
[List any quality or delivery issues during period]
ACTIONS REQUIRED
[ ] None - Performance acceptable
[ ] Enhanced monitoring
[ ] Supplier corrective action request
[ ] Supplier audit
[ ] Consider alternative supplier
NEXT REVIEW: [Date]
Prepared By: _________________ Date: _______
Reviewed By: _________________ Date: _______
```
---
## Training Templates
### Training Record
```
EMPLOYEE TRAINING RECORD
Employee Name: [Name]
Employee ID: [ID]
Department: [Department]
Job Title: [Title]
Date of Hire: [Date]
REQUIRED TRAINING
| Training | Requirement | Initial Date | Last Date | Next Due | Status |
|----------|-------------|--------------|-----------|----------|--------|
| ISO 13485 Awareness | Initial + Annual | [Date] | [Date] | [Date] | Current |
| Document Control | Initial + On Change | [Date] | [Date] | [Date] | Current |
| CAPA Procedure | Initial + On Change | [Date] | [Date] | [Date] | Due |
| Job-Specific | Per competency matrix | [Date] | [Date] | [Date] | Current |
TRAINING HISTORY
| Date | Training | Method | Duration | Trainer | Assessment | Result |
|------|----------|--------|----------|---------|------------|--------|
| [Date] | [Title] | Classroom | 2 hrs | [Name] | Written test | Pass |
| [Date] | [Title] | OJT | 4 hrs | [Name] | Observation | Pass |
COMPETENCY VERIFICATION
| Competency | Method | Date | Verified By | Result |
|------------|--------|------|-------------|--------|
| [Skill] | Observation | [Date] | [Name] | Qualified |
| [Skill] | Test | [Date] | [Name] | Qualified |
Employee Signature: _________________ Date: _______
Supervisor Signature: _________________ Date: _______
```
### Training Attendance Record
```
TRAINING ATTENDANCE RECORD
Training Title: [Title]
Training Date: [Date]
Trainer: [Name]
Location: [Location]
Duration: [Hours]
TRAINING CONTENT
[Brief description of content covered]
ATTENDEES
| Name | Employee ID | Department | Signature | Assessment Result |
|------|-------------|------------|-----------|-------------------|
| [Name] | [ID] | [Dept] | | Pass/Fail |
| [Name] | [ID] | [Dept] | | Pass/Fail |
ASSESSMENT METHOD
[ ] Written test (attach copy)
[ ] Practical demonstration
[ ] Verbal Q&A
[ ] Observation
[ ] N/A
TRAINING MATERIALS
[ ] Presentation: [Reference]
[ ] Procedure: [Reference]
[ ] Other: [Reference]
Trainer Signature: _________________ Date: _______
Training Coordinator: _________________ Date: _______
```
---
## Nonconformity Templates
### Nonconformity Report
```
NONCONFORMITY REPORT
NC Number: NC-[YYYY]-[NNN]
Date Identified: [Date]
Identified By: [Name]
NONCONFORMITY TYPE
[ ] Product [ ] Process [ ] Document [ ] System
NONCONFORMITY SOURCE
[ ] Incoming inspection
[ ] In-process inspection
[ ] Final inspection
[ ] Customer complaint
[ ] Internal audit
[ ] External audit
[ ] Other: _______
PRODUCT IDENTIFICATION (if applicable)
Product Name: [Name]
Part Number: [Number]
Lot/Batch: [Number]
Quantity Affected: [N]
NONCONFORMITY DESCRIPTION
[Detailed, objective description of the nonconformity]
REQUIREMENT
[Reference to requirement that was not met]
CONTAINMENT ACTION
Action Taken: [Description]
Quantity Contained: [N]
Location: [Location]
Date: [Date]
By: [Name]
DISPOSITION
[ ] Use As Is - Justification: _______
[ ] Rework - Per procedure: _______
[ ] Scrap - Method: _______
[ ] Return to Supplier - RMA #: _______
[ ] Other: _______
Disposition By: [Name]
Disposition Date: [Date]
CAPA REQUIRED?
[ ] Yes - CAPA #: _______
[ ] No - Justification: _______
CLOSURE
All actions complete: [ ] Yes
NC Closed By: _________________ Date: _______
QA Approval: _________________ Date: _______
```
### Material Review Board Record
```
MATERIAL REVIEW BOARD (MRB) RECORD
MRB Number: MRB-[YYYY]-[NNN]
Date: [Date]
NC Reference: NC-[YYYY]-[NNN]
NONCONFORMING MATERIAL
Product: [Name]
Part Number: [Number]
Lot/Batch: [Number]
Quantity: [N]
NONCONFORMITY DESCRIPTION
[Description from NC report]
MRB PARTICIPANTS
| Name | Role | Signature |
|------|------|-----------|
| [Name] | QA Representative | |
| [Name] | Engineering | |
| [Name] | Production | |
| [Name] | Other | |
DISPOSITION OPTIONS CONSIDERED
1. Use As Is
Technical Justification: [Justification]
Risk Assessment: [Assessment]
2. Rework
Procedure: [Reference]
Feasibility: [Assessment]
3. Scrap
Cost Impact: [Amount]
MRB DECISION
[ ] Use As Is - Customer notification required: [ ] Yes [ ] No
[ ] Rework per: [Procedure reference]
[ ] Scrap
[ ] Return to Supplier
RATIONALE
[Detailed rationale for decision]
APPROVALS
| Role | Name | Signature | Date |
|------|------|-----------|------|
| QA | [Name] | | [Date] |
| Engineering | [Name] | | [Date] |
| Production | [Name] | | [Date] |
FOLLOW-UP ACTIONS
[ ] CAPA initiated: CAPA-_______
[ ] Customer notified: Date: _______
[ ] Supplier notified: Date: _______
[ ] Other: _______
```
FILE:scripts/qms_audit_checklist.py
#!/usr/bin/env python3
"""
QMS Internal Audit Checklist Generator
Generates audit checklists for ISO 13485:2016 clauses and QMS processes.
Supports process audits, system audits, and clause-specific audits.
Usage:
python qms_audit_checklist.py --clause 7.3
python qms_audit_checklist.py --process design-control
python qms_audit_checklist.py --audit-type system --output json
python qms_audit_checklist.py --interactive
"""
import argparse
import json
import sys
from datetime import datetime
from typing import Optional
# ISO 13485:2016 Clause Structure with Audit Questions
ISO13485_CLAUSES = {
"4.1": {
"title": "General Requirements",
"questions": [
"Are QMS processes identified and documented?",
"Is the sequence and interaction of processes defined?",
"Are criteria and methods for process operation determined?",
"Are resources and information available for process operation?",
"Are processes monitored, measured, and analyzed?",
"Are actions taken to achieve planned results?",
"Is outsourced process control documented?",
"Are changes to processes managed?"
]
},
"4.2.1": {
"title": "Documentation Requirements - General",
"questions": [
"Is a quality policy documented?",
"Are quality objectives documented?",
"Is a quality manual maintained?",
"Are required documented procedures established?",
"Are documents needed for process planning and operation maintained?",
"Are required records maintained?",
"Is a medical device file established for each device type?"
]
},
"4.2.2": {
"title": "Quality Manual",
"questions": [
"Does the quality manual include QMS scope?",
"Are exclusions justified?",
"Are documented procedures included or referenced?",
"Is the interaction between processes described?",
"Is the quality manual controlled?"
]
},
"4.2.3": {
"title": "Control of Documents",
"questions": [
"Are documents approved before issue?",
"Are documents reviewed and updated as necessary?",
"Are changes and revision status identified?",
"Are current versions available at points of use?",
"Are documents legible and identifiable?",
"Are external documents identified and controlled?",
"Is unintended use of obsolete documents prevented?",
"Is there a document change control process?"
]
},
"4.2.4": {
"title": "Control of Records",
"questions": [
"Is there a procedure for record control?",
"Are records legible and identifiable?",
"Are records retrievable?",
"Are retention times defined?",
"Is protection from damage ensured?",
"Are confidential records protected?",
"Is record disposal controlled?"
]
},
"5.1": {
"title": "Management Commitment",
"questions": [
"Is there evidence of management commitment to QMS?",
"Is the importance of regulatory requirements communicated?",
"Is a quality policy established?",
"Are quality objectives established?",
"Are management reviews conducted?",
"Are resources provided for QMS?"
]
},
"5.2": {
"title": "Customer Focus",
"questions": [
"Are customer requirements determined?",
"Are applicable regulatory requirements determined?",
"Are customer and regulatory requirements met?",
"Is customer satisfaction enhanced?"
]
},
"5.3": {
"title": "Quality Policy",
"questions": [
"Is the quality policy appropriate to the organization?",
"Does it include commitment to compliance?",
"Does it include commitment to effectiveness?",
"Does it provide framework for quality objectives?",
"Is it communicated and understood?",
"Is it reviewed for continuing suitability?"
]
},
"5.4.1": {
"title": "Quality Objectives",
"questions": [
"Are quality objectives measurable?",
"Are they consistent with quality policy?",
"Are they established at relevant functions?",
"Do they include product requirements?",
"Do they include compliance requirements?"
]
},
"5.4.2": {
"title": "QMS Planning",
"questions": [
"Is QMS planning carried out to meet requirements?",
"Is QMS planning done to meet quality objectives?",
"Is QMS integrity maintained during changes?"
]
},
"5.5.1": {
"title": "Responsibility and Authority",
"questions": [
"Are responsibilities and authorities defined?",
"Are they documented?",
"Are they communicated?",
"Are interrelationships defined?"
]
},
"5.5.2": {
"title": "Management Representative",
"questions": [
"Is a management representative appointed?",
"Is authority to ensure QMS processes established?",
"Is authority to report to top management defined?",
"Is authority to promote awareness of requirements defined?"
]
},
"5.5.3": {
"title": "Internal Communication",
"questions": [
"Are communication processes established?",
"Is QMS effectiveness communicated?",
"Is information communicated appropriately?"
]
},
"5.6": {
"title": "Management Review",
"questions": [
"Are management reviews planned?",
"Are all required inputs reviewed?",
"Are outputs documented?",
"Are action items followed up?",
"Are records maintained?"
]
},
"6.1": {
"title": "Provision of Resources",
"questions": [
"Are resources determined?",
"Are resources provided for QMS?",
"Are resources provided for customer satisfaction?",
"Are resources provided for regulatory compliance?"
]
},
"6.2": {
"title": "Human Resources",
"questions": [
"Is competence defined for personnel?",
"Is training provided to achieve competence?",
"Is training effectiveness evaluated?",
"Is awareness of job relevance ensured?",
"Are training records maintained?"
]
},
"6.3": {
"title": "Infrastructure",
"questions": [
"Is necessary infrastructure determined?",
"Are buildings and workspace adequate?",
"Is process equipment adequate?",
"Are supporting services adequate?",
"Are maintenance requirements documented?"
]
},
"6.4": {
"title": "Work Environment",
"questions": [
"Is work environment determined?",
"Are environmental requirements documented?",
"Is contamination control adequate?",
"Are personnel health and cleanliness controlled?",
"Are environmental conditions monitored?"
]
},
"7.1": {
"title": "Planning of Product Realization",
"questions": [
"Are quality objectives for product defined?",
"Are processes needed determined?",
"Is verification and validation defined?",
"Are records requirements defined?",
"Is risk management applied?"
]
},
"7.2": {
"title": "Customer-Related Processes",
"questions": [
"Are customer requirements determined?",
"Are regulatory requirements determined?",
"Are requirements reviewed before commitment?",
"Are differences resolved before acceptance?",
"Is communication with customers effective?"
]
},
"7.3.1": {
"title": "Design and Development Planning",
"questions": [
"Are design stages determined?",
"Are review activities defined?",
"Are verification activities defined?",
"Are validation activities defined?",
"Are responsibilities assigned?",
"Are interfaces managed?"
]
},
"7.3.2": {
"title": "Design and Development Inputs",
"questions": [
"Are functional requirements defined?",
"Are performance requirements defined?",
"Are safety requirements defined?",
"Are regulatory requirements identified?",
"Are previous design inputs considered?",
"Are risk management outputs included?"
]
},
"7.3.3": {
"title": "Design and Development Outputs",
"questions": [
"Do outputs meet input requirements?",
"Is purchasing information provided?",
"Are acceptance criteria defined?",
"Are essential characteristics specified?",
"Are outputs approved before release?"
]
},
"7.3.4": {
"title": "Design and Development Review",
"questions": [
"Are design reviews conducted at suitable stages?",
"Is ability to meet requirements evaluated?",
"Are problems identified?",
"Are follow-up actions recorded?",
"Are appropriate functions represented?"
]
},
"7.3.5": {
"title": "Design and Development Verification",
"questions": [
"Is verification performed per plan?",
"Do outputs meet inputs?",
"Are verification records maintained?",
"Are verification methods appropriate?"
]
},
"7.3.6": {
"title": "Design and Development Validation",
"questions": [
"Is validation performed per plan?",
"Is product evaluated for intended use?",
"Is clinical evaluation included?",
"Are validation records maintained?",
"Is validation completed before product delivery?"
]
},
"7.3.7": {
"title": "Design and Development Transfer",
"questions": [
"Are outputs verified before transfer?",
"Is manufacturing capability verified?",
"Are transfer activities documented?"
]
},
"7.3.8": {
"title": "Control of Design and Development Changes",
"questions": [
"Are design changes identified?",
"Are changes reviewed?",
"Are changes verified?",
"Are changes validated as appropriate?",
"Is impact on product assessed?",
"Are changes approved before implementation?"
]
},
"7.4.1": {
"title": "Purchasing Process",
"questions": [
"Are suppliers evaluated and selected?",
"Are evaluation criteria established?",
"Is supplier performance monitored?",
"Are re-evaluation criteria defined?",
"Is purchased product verified?"
]
},
"7.4.2": {
"title": "Purchasing Information",
"questions": [
"Is purchasing information adequate?",
"Are product requirements specified?",
"Are QMS requirements specified?",
"Are personnel requirements specified?"
]
},
"7.4.3": {
"title": "Verification of Purchased Product",
"questions": [
"Is incoming inspection adequate?",
"Are verification activities defined?",
"Are verification records maintained?",
"Is source verification defined if applicable?"
]
},
"7.5.1": {
"title": "Control of Production and Service Provision",
"questions": [
"Is product information available?",
"Are work instructions available?",
"Is suitable equipment used?",
"Are monitoring devices available?",
"Is monitoring implemented?",
"Are release activities defined?",
"Are labeling requirements met?"
]
},
"7.5.2": {
"title": "Cleanliness of Product",
"questions": [
"Are cleanliness requirements documented?",
"Is contamination controlled?",
"Are process agents controlled?"
]
},
"7.5.3": {
"title": "Installation Activities",
"questions": [
"Are installation requirements documented?",
"Are acceptance criteria defined?",
"Are installation records maintained?"
]
},
"7.5.4": {
"title": "Servicing Activities",
"questions": [
"Are servicing procedures documented?",
"Are reference materials controlled?",
"Are service records maintained?",
"Is feedback analyzed?"
]
},
"7.5.5": {
"title": "Sterile Medical Devices",
"questions": [
"Is sterilization validated?",
"Are process parameters controlled?",
"Is sterile barrier validated?",
"Are sterilization records maintained?"
]
},
"7.5.6": {
"title": "Validation of Processes",
"questions": [
"Are special processes identified?",
"Are validation procedures documented?",
"Is equipment qualified?",
"Are personnel qualified?",
"Are validation records maintained?",
"Are revalidation criteria defined?"
]
},
"7.5.7": {
"title": "Particular Requirements for Validation",
"questions": [
"Are validation methods defined?",
"Are acceptance criteria established?",
"Is software validation appropriate?",
"Are validation records maintained?"
]
},
"7.5.8": {
"title": "Identification",
"questions": [
"Is product identified throughout realization?",
"Is documentation identified?",
"Is UDI implemented as required?"
]
},
"7.5.9": {
"title": "Traceability",
"questions": [
"Are traceability procedures documented?",
"Are components traceable?",
"Is work environment recorded?",
"Is distribution recorded?",
"Is traceability extent defined?"
]
},
"7.5.10": {
"title": "Customer Property",
"questions": [
"Is customer property identified?",
"Is it verified on receipt?",
"Is it protected and safeguarded?",
"Is loss or damage reported?"
]
},
"7.5.11": {
"title": "Preservation of Product",
"questions": [
"Is product identified?",
"Is handling controlled?",
"Is packaging controlled?",
"Is storage controlled?",
"Is protection adequate?"
]
},
"7.6": {
"title": "Control of Monitoring and Measuring Equipment",
"questions": [
"Is equipment calibrated?",
"Is calibration traceable?",
"Is calibration status identified?",
"Is equipment protected from damage?",
"Is software validated?",
"Are records maintained?"
]
},
"8.1": {
"title": "Measurement, Analysis and Improvement - General",
"questions": [
"Are monitoring activities planned?",
"Are analysis activities planned?",
"Are improvement activities planned?"
]
},
"8.2.1": {
"title": "Feedback",
"questions": [
"Is feedback collected?",
"Is feedback analyzed?",
"Is feedback used for improvement?",
"Is regulatory feedback included?"
]
},
"8.2.2": {
"title": "Complaint Handling",
"questions": [
"Is there a complaint procedure?",
"Are complaints investigated?",
"Are regulatory reports made if required?",
"Is trend analysis performed?",
"Are CAPAs initiated when warranted?"
]
},
"8.2.3": {
"title": "Reporting to Regulatory Authorities",
"questions": [
"Are reporting requirements identified?",
"Are reports submitted timely?",
"Are records maintained?"
]
},
"8.2.4": {
"title": "Internal Audit",
"questions": [
"Is an audit program established?",
"Are audit criteria defined?",
"Are auditors independent?",
"Are auditors competent?",
"Are audit records maintained?",
"Are findings followed up?"
]
},
"8.2.5": {
"title": "Monitoring and Measurement of Processes",
"questions": [
"Are processes monitored?",
"Are suitable methods used?",
"Is process capability demonstrated?",
"Are corrections made when needed?"
]
},
"8.2.6": {
"title": "Monitoring and Measurement of Product",
"questions": [
"Is product inspected?",
"Are acceptance criteria met?",
"Is release authorized?",
"Is traceability to inspection recorded?",
"Are records maintained?"
]
},
"8.3": {
"title": "Control of Nonconforming Product",
"questions": [
"Is nonconforming product identified?",
"Is it documented?",
"Is it evaluated?",
"Is it segregated?",
"Is disposition determined?",
"Is rework verified?",
"Is concession controlled?",
"Is post-delivery NC investigated?"
]
},
"8.4": {
"title": "Analysis of Data",
"questions": [
"Is data collected?",
"Is feedback analyzed?",
"Is conformity data analyzed?",
"Is process data analyzed?",
"Is supplier data analyzed?",
"Are audit results analyzed?"
]
},
"8.5.1": {
"title": "Improvement - General",
"questions": [
"Is continual improvement pursued?",
"Are policy, objectives, audits, data, actions, and reviews used?"
]
},
"8.5.2": {
"title": "Corrective Action",
"questions": [
"Is there a CA procedure?",
"Are NCs reviewed (including complaints)?",
"Is root cause determined?",
"Is action needed evaluated?",
"Is action determined and implemented?",
"Are results documented?",
"Is effectiveness verified?"
]
},
"8.5.3": {
"title": "Preventive Action",
"questions": [
"Is there a PA procedure?",
"Are potential NCs identified?",
"Is action needed evaluated?",
"Is action determined and implemented?",
"Are results documented?",
"Is effectiveness verified?"
]
}
}
# Process-to-Clause Mapping
PROCESS_MAPPING = {
"document-control": ["4.2.1", "4.2.2", "4.2.3", "4.2.4"],
"management-review": ["5.6"],
"internal-audit": ["8.2.4"],
"training": ["6.2"],
"design-control": ["7.3.1", "7.3.2", "7.3.3", "7.3.4", "7.3.5", "7.3.6", "7.3.7", "7.3.8"],
"purchasing": ["7.4.1", "7.4.2", "7.4.3"],
"production": ["7.5.1", "7.5.2", "7.5.6", "7.5.7", "7.5.8", "7.5.9", "7.5.11"],
"capa": ["8.5.2", "8.5.3"],
"nonconformity": ["8.3"],
"calibration": ["7.6"],
"complaint-handling": ["8.2.1", "8.2.2", "8.2.3"],
"risk-management": ["7.1"],
"infrastructure": ["6.3", "6.4"],
"customer-requirements": ["5.2", "7.2"]
}
def get_clause_checklist(clause: str) -> dict:
"""Get audit checklist for a specific clause."""
if clause not in ISO13485_CLAUSES:
return {"error": f"Clause {clause} not found"}
clause_data = ISO13485_CLAUSES[clause]
return {
"clause": clause,
"title": clause_data["title"],
"questions": clause_data["questions"],
"question_count": len(clause_data["questions"])
}
def get_process_checklist(process: str) -> dict:
"""Get audit checklist for a specific process."""
if process not in PROCESS_MAPPING:
available = ", ".join(sorted(PROCESS_MAPPING.keys()))
return {"error": f"Process '{process}' not found. Available: {available}"}
clauses = PROCESS_MAPPING[process]
questions = []
for clause in clauses:
if clause in ISO13485_CLAUSES:
clause_data = ISO13485_CLAUSES[clause]
for q in clause_data["questions"]:
questions.append({
"clause": clause,
"clause_title": clause_data["title"],
"question": q
})
return {
"process": process,
"clauses_covered": clauses,
"questions": questions,
"question_count": len(questions)
}
def get_system_audit_checklist() -> dict:
"""Get complete system audit checklist covering all clauses."""
all_questions = []
for clause, data in sorted(ISO13485_CLAUSES.items()):
for q in data["questions"]:
all_questions.append({
"clause": clause,
"clause_title": data["title"],
"question": q
})
return {
"audit_type": "system",
"clauses_covered": list(ISO13485_CLAUSES.keys()),
"questions": all_questions,
"question_count": len(all_questions)
}
def format_checklist_text(checklist: dict) -> str:
"""Format checklist for text output."""
lines = []
if "error" in checklist:
return f"Error: {checklist['error']}"
lines.append("=" * 70)
lines.append("ISO 13485:2016 INTERNAL AUDIT CHECKLIST")
lines.append(f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M')}")
lines.append("=" * 70)
if "clause" in checklist:
lines.append(f"\nClause: {checklist['clause']} - {checklist['title']}")
lines.append("-" * 50)
for i, q in enumerate(checklist["questions"], 1):
lines.append(f"\n{i}. {q}")
lines.append(" [ ] C [ ] NC [ ] OBS [ ] N/A")
lines.append(" Evidence: _________________________________")
lines.append(" Notes: ____________________________________")
elif "process" in checklist:
lines.append(f"\nProcess: {checklist['process'].replace('-', ' ').title()}")
lines.append(f"Clauses Covered: {', '.join(checklist['clauses_covered'])}")
lines.append("-" * 50)
current_clause = None
item_num = 1
for q in checklist["questions"]:
if q["clause"] != current_clause:
current_clause = q["clause"]
lines.append(f"\n--- {q['clause']} {q['clause_title']} ---")
lines.append(f"\n{item_num}. {q['question']}")
lines.append(" [ ] C [ ] NC [ ] OBS [ ] N/A")
lines.append(" Evidence: _________________________________")
lines.append(" Notes: ____________________________________")
item_num += 1
elif "audit_type" in checklist:
lines.append(f"\nAudit Type: Full System Audit")
lines.append(f"Total Clauses: {len(checklist['clauses_covered'])}")
lines.append("-" * 50)
current_clause = None
item_num = 1
for q in checklist["questions"]:
if q["clause"] != current_clause:
current_clause = q["clause"]
lines.append(f"\n{'=' * 40}")
lines.append(f"CLAUSE {q['clause']}: {q['clause_title']}")
lines.append("=" * 40)
lines.append(f"\n{item_num}. {q['question']}")
lines.append(" [ ] C [ ] NC [ ] OBS [ ] N/A")
lines.append(" Evidence: _________________________________")
item_num += 1
lines.append("\n" + "=" * 70)
lines.append(f"Total Questions: {checklist['question_count']}")
lines.append("")
lines.append("Legend: C=Conforming, NC=Nonconforming, OBS=Observation, N/A=Not Applicable")
lines.append("=" * 70)
return "\n".join(lines)
def interactive_mode():
"""Run interactive audit checklist generator."""
print("\n" + "=" * 50)
print("QMS INTERNAL AUDIT CHECKLIST GENERATOR")
print("=" * 50)
print("\nSelect audit type:")
print("1. Clause-specific audit")
print("2. Process audit")
print("3. Full system audit")
print("4. List available processes")
print("5. List all clauses")
print("6. Exit")
choice = input("\nEnter choice (1-6): ").strip()
if choice == "1":
print("\nAvailable clause sections:")
print(" 4.x - Quality Management System")
print(" 5.x - Management Responsibility")
print(" 6.x - Resource Management")
print(" 7.x - Product Realization")
print(" 8.x - Measurement, Analysis, Improvement")
clause = input("\nEnter clause number (e.g., 7.3.1): ").strip()
checklist = get_clause_checklist(clause)
print(format_checklist_text(checklist))
elif choice == "2":
processes = sorted(PROCESS_MAPPING.keys())
print("\nAvailable processes:")
for i, p in enumerate(processes, 1):
clauses = PROCESS_MAPPING[p]
print(f" {i}. {p} (clauses: {', '.join(clauses)})")
process = input("\nEnter process name: ").strip().lower()
checklist = get_process_checklist(process)
print(format_checklist_text(checklist))
elif choice == "3":
print("\nGenerating full system audit checklist...")
checklist = get_system_audit_checklist()
print(format_checklist_text(checklist))
elif choice == "4":
processes = sorted(PROCESS_MAPPING.keys())
print("\nAvailable QMS Processes:")
print("-" * 50)
for p in processes:
clauses = PROCESS_MAPPING[p]
print(f" {p}")
print(f" Clauses: {', '.join(clauses)}")
elif choice == "5":
print("\nISO 13485:2016 Clauses:")
print("-" * 50)
for clause, data in sorted(ISO13485_CLAUSES.items()):
print(f" {clause}: {data['title']} ({len(data['questions'])} questions)")
elif choice == "6":
print("Exiting.")
return
else:
print("Invalid choice.")
def main():
parser = argparse.ArgumentParser(
description="Generate ISO 13485:2016 internal audit checklists",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="""
Examples:
python qms_audit_checklist.py --clause 7.3
python qms_audit_checklist.py --process design-control
python qms_audit_checklist.py --audit-type system --output json
python qms_audit_checklist.py --list-processes
python qms_audit_checklist.py --list-clauses
python qms_audit_checklist.py --interactive
"""
)
parser.add_argument(
"--clause",
help="Generate checklist for specific clause (e.g., 7.3.1, 8.5.2)"
)
parser.add_argument(
"--process",
help="Generate checklist for process (e.g., design-control, capa)"
)
parser.add_argument(
"--audit-type",
choices=["clause", "process", "system"],
help="Audit type for checklist generation"
)
parser.add_argument(
"--output",
choices=["text", "json"],
default="text",
help="Output format (default: text)"
)
parser.add_argument(
"--list-processes",
action="store_true",
help="List available QMS processes"
)
parser.add_argument(
"--list-clauses",
action="store_true",
help="List all ISO 13485 clauses"
)
parser.add_argument(
"--interactive",
action="store_true",
help="Run in interactive mode"
)
args = parser.parse_args()
if args.interactive:
interactive_mode()
return
if args.list_processes:
processes = sorted(PROCESS_MAPPING.keys())
if args.output == "json":
result = {p: PROCESS_MAPPING[p] for p in processes}
print(json.dumps(result, indent=2))
else:
print("\nAvailable QMS Processes:")
print("-" * 50)
for p in processes:
clauses = PROCESS_MAPPING[p]
print(f" {p}: {', '.join(clauses)}")
return
if args.list_clauses:
if args.output == "json":
result = {c: {"title": d["title"], "question_count": len(d["questions"])}
for c, d in sorted(ISO13485_CLAUSES.items())}
print(json.dumps(result, indent=2))
else:
print("\nISO 13485:2016 Clauses:")
print("-" * 50)
for clause, data in sorted(ISO13485_CLAUSES.items()):
print(f" {clause}: {data['title']} ({len(data['questions'])} questions)")
return
checklist = None
if args.clause:
checklist = get_clause_checklist(args.clause)
elif args.process:
checklist = get_process_checklist(args.process)
elif args.audit_type == "system":
checklist = get_system_audit_checklist()
else:
parser.print_help()
return
if checklist:
if args.output == "json":
print(json.dumps(checklist, indent=2))
else:
print(format_checklist_text(checklist))
if __name__ == "__main__":
main()
Chất vấn kế hoạch dựa trên thuật ngữ dự án (CONTEXT.md) và các quyết định đã ghi (docs/adr/), cập nhật các tệp này khi chốt thuật ngữ.
---
name: grill-with-docs
description: Docs-anchored grilling session — challenges a plan against the project's existing language (CONTEXT.md) and recorded decisions (docs/adr/), and updates those files inline as terminology and decisions crystallise. Use when user wants to stress-test a plan against documented domain language, or mentions "grill with docs".
license: MIT
metadata:
derived_from: "https://github.com/mattpocock/skills/tree/main/skills/engineering/grill-with-docs"
original_author: "Matt Pocock (@mattpocock)"
original_license: MIT
voice: "Matt Pocock — relentless, one-at-a-time, codebase-and-docs-first, ADRs only when 3 criteria are met"
version: 1.0.0
---
# Grill with Docs
> Derived from [Matt Pocock's grill-with-docs](https://github.com/mattpocock/skills/tree/main/skills/engineering/grill-with-docs) (MIT, © 2026 Matt Pocock). Matt's interview discipline + docs-anchored grilling rules preserved verbatim under MIT. Additions in this repo: 3 stdlib validators (CONTEXT.md linter, ADR scanner, glossary↔code consistency check), 3 in-depth references each citing 7+ authoritative sources, `cs-grill-with-docs` agent, `/cs:grill-with-docs` command. See [Wrapper additions](#wrapper-additions) below.
<what-to-do>
Interview me relentlessly about every aspect of this plan until we reach a shared understanding. Walk down each branch of the design tree, resolving dependencies between decisions one-by-one. For each question, provide your recommended answer.
Ask the questions one at a time, waiting for feedback on each question before continuing.
If a question can be answered by exploring the codebase, explore the codebase instead.
</what-to-do>
<supporting-info>
## Domain awareness
During codebase exploration, also look for existing documentation:
### File structure
Most repos have a single context:
```
/
├── CONTEXT.md
├── docs/
│ └── adr/
│ ├── 0001-event-sourced-orders.md
│ └── 0002-postgres-for-write-model.md
└── src/
```
If a `CONTEXT-MAP.md` exists at the root, the repo has multiple contexts. The map points to where each one lives:
```
/
├── CONTEXT-MAP.md
├── docs/
│ └── adr/ ← system-wide decisions
├── src/
│ ├── ordering/
│ │ ├── CONTEXT.md
│ │ └── docs/adr/ ← context-specific decisions
│ └── billing/
│ ├── CONTEXT.md
│ └── docs/adr/
```
Create files lazily — only when you have something to write. If no `CONTEXT.md` exists, create one when the first term is resolved. If no `docs/adr/` exists, create it when the first ADR is needed.
## During the session
### Challenge against the glossary
When the user uses a term that conflicts with the existing language in `CONTEXT.md`, call it out immediately. "Your glossary defines 'cancellation' as X, but you seem to mean Y — which is it?"
### Sharpen fuzzy language
When the user uses vague or overloaded terms, propose a precise canonical term. "You're saying 'account' — do you mean the Customer or the User? Those are different things."
### Discuss concrete scenarios
When domain relationships are being discussed, stress-test them with specific scenarios. Invent scenarios that probe edge cases and force the user to be precise about the boundaries between concepts.
### Cross-reference with code
When the user states how something works, check whether the code agrees. If you find a contradiction, surface it: "Your code cancels entire Orders, but you just said partial cancellation is possible — which is right?"
### Update CONTEXT.md inline
When a term is resolved, update `CONTEXT.md` right there. Don't batch these up — capture them as they happen. Use the format in [CONTEXT-FORMAT.md](./CONTEXT-FORMAT.md).
`CONTEXT.md` should be totally devoid of implementation details. Do not treat `CONTEXT.md` as a spec, a scratch pad, or a repository for implementation decisions. It is a glossary and nothing else.
### Offer ADRs sparingly
Only offer to create an ADR when all three are true:
1. **Hard to reverse** — the cost of changing your mind later is meaningful
2. **Surprising without context** — a future reader will wonder "why did they do it this way?"
3. **The result of a real trade-off** — there were genuine alternatives and you picked one for specific reasons
If any of the three is missing, skip the ADR. Use the format in [ADR-FORMAT.md](./ADR-FORMAT.md).
</supporting-info>
## Wrapper Additions
The additions below are **not** part of Matt's upstream skill. They operationalize the upstream's rules into deterministic, stdlib-only validators that pair naturally with the interview loop.
### Workflow (with wrapper tools)
1. **Pre-flight (before the first question):**
- Run `scripts/context_md_linter.py CONTEXT.md` if a `CONTEXT.md` exists — confirms the glossary is well-formed before grilling against it.
- Run `scripts/adr_scanner.py docs/adr/` if `docs/adr/` exists — surfaces numbering gaps, malformed ADRs, status-frontmatter inconsistencies.
- Run `scripts/glossary_code_consistency.py --context CONTEXT.md --code src/` — flags defined-but-unused terms (dead glossary) and code-only common nouns that may need definitions. Use these flags as opening grill questions.
2. **During the session (Matt's rules apply):**
- One question per turn, walking depth-first.
- When a term is sharpened: edit `CONTEXT.md` immediately; re-run `context_md_linter.py` if the edit is structural.
- When an ADR is warranted: write it under `docs/adr/`; re-run `adr_scanner.py` to confirm numbering.
3. **Closing:**
- Final `glossary_code_consistency.py` run to confirm no new orphan terms were introduced.
- Summarize: terms added/refined, ADRs written, scenarios discussed, open items.
### Tools (stdlib-only)
| Tool | One-line role |
|---|---|
| `scripts/context_md_linter.py` | Validate `CONTEXT.md` against the CONTEXT-FORMAT.md structure. PASS/WARN/FAIL per rule. |
| `scripts/adr_scanner.py` | Walk `docs/adr/`, check `NNNN-slug.md` pattern, numbering integrity, body completeness. |
| `scripts/glossary_code_consistency.py` | Cross-reference bold terms in `CONTEXT.md` against codebase usage. Flag dead glossary + code-only common nouns. |
### References (citations behind each rule)
- [`references/ubiquitous_language.md`](references/ubiquitous_language.md) — why a glossary belongs in source control (Evans, Vernon, Khononov, Wlaschin, Brandolini, Avram & Marinescu, Fowler)
- [`references/adr_practice.md`](references/adr_practice.md) — when an ADR earns its keep (Nygard, Tyree & Akerman, Zimmermann Y-statements, MADR, ThoughtWorks Radar, adr-tools, Backstage)
- [`references/context_md_as_artifact.md`](references/context_md_as_artifact.md) — CONTEXT.md as living artifact (Khononov on language drift, Kernighan on naming, BoundedContext bliki, Confluent on data contracts, Brandolini on EventStorming glossary)
### Companion
- Agent: `cs-grill-with-docs` (see `../../agents/cs-grill-with-docs.md`)
- Command: `/cs:grill-with-docs` (see `../../commands/cs-grill-with-docs.md`)
---
**Version:** 1.0.0
**Derived:** Matt Pocock's grill-with-docs (MIT) + this repo's wrapper
FILE:ADR-FORMAT.md
<!--
Derived from Matt Pocock's grill-with-docs:
https://github.com/mattpocock/skills/tree/main/skills/engineering/grill-with-docs/ADR-FORMAT.md
MIT License © 2026 Matt Pocock. Reproduced verbatim under MIT.
-->
# ADR Format
ADRs live in `docs/adr/` and use sequential numbering: `0001-slug.md`, `0002-slug.md`, etc.
Create the `docs/adr/` directory lazily — only when the first ADR is needed.
## Template
```md
# {Short title of the decision}
{1-3 sentences: what's the context, what did we decide, and why.}
```
That's it. An ADR can be a single paragraph. The value is in recording *that* a decision was made and *why* — not in filling out sections.
## Optional sections
Only include these when they add genuine value. Most ADRs won't need them.
- **Status** frontmatter (`proposed | accepted | deprecated | superseded by ADR-NNNN`) — useful when decisions are revisited
- **Considered Options** — only when the rejected alternatives are worth remembering
- **Consequences** — only when non-obvious downstream effects need to be called out
## Numbering
Scan `docs/adr/` for the highest existing number and increment by one.
## When to offer an ADR
All three of these must be true:
1. **Hard to reverse** — the cost of changing your mind later is meaningful
2. **Surprising without context** — a future reader will look at the code and wonder "why on earth did they do it this way?"
3. **The result of a real trade-off** — there were genuine alternatives and you picked one for specific reasons
If a decision is easy to reverse, skip it — you'll just reverse it. If it's not surprising, nobody will wonder why. If there was no real alternative, there's nothing to record beyond "we did the obvious thing."
### What qualifies
- **Architectural shape.** "We're using a monorepo." "The write model is event-sourced, the read model is projected into Postgres."
- **Integration patterns between contexts.** "Ordering and Billing communicate via domain events, not synchronous HTTP."
- **Technology choices that carry lock-in.** Database, message bus, auth provider, deployment target. Not every library — just the ones that would take a quarter to swap out.
- **Boundary and scope decisions.** "Customer data is owned by the Customer context; other contexts reference it by ID only." The explicit no-s are as valuable as the yes-s.
- **Deliberate deviations from the obvious path.** "We're using manual SQL instead of an ORM because X." Anything where a reasonable reader would assume the opposite. These stop the next engineer from "fixing" something that was deliberate.
- **Constraints not visible in the code.** "We can't use AWS because of compliance requirements." "Response times must be under 200ms because of the partner API contract."
- **Rejected alternatives when the rejection is non-obvious.** If you considered GraphQL and picked REST for subtle reasons, record it — otherwise someone will suggest GraphQL again in six months.
FILE:CONTEXT-FORMAT.md
<!--
Derived from Matt Pocock's grill-with-docs:
https://github.com/mattpocock/skills/tree/main/skills/engineering/grill-with-docs/CONTEXT-FORMAT.md
MIT License © 2026 Matt Pocock. Reproduced verbatim under MIT.
-->
# CONTEXT.md Format
## Structure
```md
# {Context Name}
{One or two sentence description of what this context is and why it exists.}
## Language
**Order**:
{A concise description of the term}
_Avoid_: Purchase, transaction
**Invoice**:
A request for payment sent to a customer after delivery.
_Avoid_: Bill, payment request
**Customer**:
A person or organization that places orders.
_Avoid_: Client, buyer, account
## Relationships
- An **Order** produces one or more **Invoices**
- An **Invoice** belongs to exactly one **Customer**
## Example dialogue
> **Dev:** "When a **Customer** places an **Order**, do we create the **Invoice** immediately?"
> **Domain expert:** "No — an **Invoice** is only generated once a **Fulfillment** is confirmed."
## Flagged ambiguities
- "account" was used to mean both **Customer** and **User** — resolved: these are distinct concepts.
```
## Rules
- **Be opinionated.** When multiple words exist for the same concept, pick the best one and list the others as aliases to avoid.
- **Flag conflicts explicitly.** If a term is used ambiguously, call it out in "Flagged ambiguities" with a clear resolution.
- **Keep definitions tight.** One sentence max. Define what it IS, not what it does.
- **Show relationships.** Use bold term names and express cardinality where obvious.
- **Only include terms specific to this project's context.** General programming concepts (timeouts, error types, utility patterns) don't belong even if the project uses them extensively. Before adding a term, ask: is this a concept unique to this context, or a general programming concept? Only the former belongs.
- **Group terms under subheadings** when natural clusters emerge. If all terms belong to a single cohesive area, a flat list is fine.
- **Write an example dialogue.** A conversation between a dev and a domain expert that demonstrates how the terms interact naturally and clarifies boundaries between related concepts.
## Single vs multi-context repos
**Single context (most repos):** One `CONTEXT.md` at the repo root.
**Multiple contexts:** A `CONTEXT-MAP.md` at the repo root lists the contexts, where they live, and how they relate to each other:
```md
# Context Map
## Contexts
- [Ordering](./src/ordering/CONTEXT.md) — receives and tracks customer orders
- [Billing](./src/billing/CONTEXT.md) — generates invoices and processes payments
- [Fulfillment](./src/fulfillment/CONTEXT.md) — manages warehouse picking and shipping
## Relationships
- **Ordering → Fulfillment**: Ordering emits `OrderPlaced` events; Fulfillment consumes them to start picking
- **Fulfillment → Billing**: Fulfillment emits `ShipmentDispatched` events; Billing consumes them to generate invoices
- **Ordering ↔ Billing**: Shared types for `CustomerId` and `Money`
```
The skill infers which structure applies:
- If `CONTEXT-MAP.md` exists, read it to find contexts
- If only a root `CONTEXT.md` exists, single context
- If neither exists, create a root `CONTEXT.md` lazily when the first term is resolved
When multiple contexts exist, infer which one the current topic relates to. If unclear, ask.
FILE:references/adr_practice.md
# ADR Practice — When Does a Decision Earn an ADR?
This reference answers exactly one decision: **what bar must an architectural decision clear to be worth writing down as an ADR, and what format keeps the ADR useful 18 months later?**
Pair with `scripts/adr_scanner.py` for filename + numbering + structural validation.
## The Core Claim
ADRs are not a compliance ritual. They exist to answer a single future question: **"Why on earth did they do it this way?"** If a future reader will never ask that question — because the choice is obvious, easy to reverse, or had no real alternatives — the ADR is doc-rot waiting to happen.
The matt-pocock 3-criteria gate (preserved verbatim in `ADR-FORMAT.md`) is the strict version of this principle:
1. **Hard to reverse** — the cost of changing your mind is meaningful (not "an afternoon of refactoring").
2. **Surprising without context** — a future reader will look at the code and wonder why.
3. **Result of a real trade-off** — there were genuine alternatives and you picked one for specific reasons.
**All three must be true.** Two-out-of-three is not enough. If a decision was hard to reverse but obvious and uncontested (e.g., "we used HTTPS"), no ADR. If it was a real trade-off but easy to reverse (e.g., "we used React Query over SWR"), no ADR.
## What Earns an ADR (Examples)
- **Architectural shape.** "Write model is event-sourced, read model is projected into Postgres." Hard-to-reverse + surprising + real-trade-off.
- **Integration patterns between contexts.** "Ordering and Billing communicate via domain events, not synchronous HTTP." Hard-to-reverse (rewiring eventing is expensive) + surprising (HTTP is the obvious choice) + real-trade-off (eventual consistency vs simpler API).
- **Technology choices with lock-in.** Database engine, message bus, auth provider. Not "we picked Lodash" — those swap in an afternoon.
- **Boundary and scope decisions.** "Customer data is owned by the Customer context; other contexts reference by ID only." The explicit no-s are as valuable as the yes-s.
- **Deliberate deviations from the obvious path.** "We use manual SQL instead of an ORM because X." Stops the next engineer from "fixing" something deliberate.
- **Constraints not visible in code.** "Can't use AWS due to compliance." "Response times must be <200ms due to partner API contract."
- **Rejected alternatives with non-obvious rejections.** "We considered GraphQL and picked REST because subscription complexity didn't match our actual real-time needs." Otherwise someone will suggest GraphQL again in 6 months.
## What Does NOT Earn an ADR
- **Library choices.** Lodash vs Ramda, axios vs ky, dayjs vs date-fns — these swap in an afternoon. Comment in code if you must.
- **Style guide decisions.** "We use Prettier" — record in `package.json`, not an ADR.
- **Defaults you didn't deviate from.** "We use the framework's recommended router." No trade-off, no ADR.
- **Decisions that are easy to reverse.** If the future-you can undo it in a day, future-you doesn't need the why.
- **Decisions where the alternative was never seriously considered.** No real trade-off → no ADR.
## Format Discipline
ADRs are markdown files at `docs/adr/NNNN-slug.md`, numbered sequentially.
**Default format (minimum viable):**
```md
# {Short title of the decision}
{1-3 sentences: what's the context, what did we decide, and why.}
```
An ADR can be a single paragraph. The value is in recording *that* a decision was made and *why* — not in filling out sections.
**Optional sections (only when they add genuine value):**
- **Status frontmatter** (`proposed | accepted | deprecated | superseded by ADR-NNNN`) — useful when decisions are revisited.
- **Considered Options** — only when rejected alternatives are worth remembering.
- **Consequences** — only when non-obvious downstream effects need to be called out.
If a section is included but empty or boilerplate ("none"), delete the section.
## Numbering Discipline
- Sequential, zero-padded to 4 digits: `0001`, `0002`, ..., `9999`.
- No gaps. If an ADR is abandoned mid-draft, either commit it as `proposed → withdrawn` or renumber.
- Slug is short, kebab-case, intent-revealing: `0042-event-sourced-orders.md`, not `0042-adr.md` or `0042-decision-about-events.md`.
`scripts/adr_scanner.py` enforces the pattern and surfaces gaps.
## Status Lifecycle (Optional)
For repos that revisit decisions, the status field is useful:
```
proposed → accepted ← default lifecycle for a new ADR
accepted → deprecated ← decision no longer applies; no replacement
accepted → superseded ← replaced by ADR-NNNN; link to successor in frontmatter
```
When superseding, the new ADR references the old (`supersedes: ADR-0017`) and the old ADR is updated with `superseded by: ADR-0042`. This back-link is the single most useful piece of ADR metadata for archeology.
## Anti-Patterns
- **The ADR factory.** Writing an ADR for every PR. Within a year, you have 200 ADRs and no one reads any. The 3-criteria gate is the firewall.
- **The proposal that never accepts.** ADR sits in `proposed` for months. Either accept it (do it) or withdraw it (delete the file or mark withdrawn).
- **The TOC-only ADR.** Filled-in section headers but no actual content. Worse than not writing the ADR — it implies a decision was recorded when nothing was.
- **The future-tense ADR.** "We will use X." ADRs are records, not plans. Write in past tense ("We chose X because ...") so it reads correctly 2 years later.
- **The unanchored ADR.** ADR with no link to the PR/issue/discussion that drove it. The "why" loses fidelity over time without the source thread.
## Operational Checklist (Per ADR Decision Point)
When grilling and a candidate decision emerges:
- [ ] **Reversibility test.** "If we change our mind in 6 months, what's the cost?" If "an afternoon" → skip the ADR.
- [ ] **Surprise test.** "Will a future engineer look at this and wonder why?" If no → skip.
- [ ] **Trade-off test.** "What alternatives did we seriously consider, and why did each lose?" If none → skip.
- [ ] **All three pass.** Write the ADR. Use the minimum format. Re-run `scripts/adr_scanner.py` to confirm numbering.
- [ ] **Frontmatter status.** Only add `status` if revisiting is expected. Default is "implicit accepted".
## Citations (7 sources)
1. **Michael Nygard, "Documenting Architecture Decisions" (cognitect.com, November 2011).** The original ADR essay. Introduces the format (Title / Context / Decision / Status / Consequences) and the core insight that "architecturally significant" decisions deserve records. Nygard's framing of ADRs as "memory aids for future architects" is the source of the 3-criteria gate's first rule (hard-to-reverse).
2. **Jeff Tyree & Art Akerman, "Architecture Decisions: Demystifying Architecture" — *IEEE Software* 22(2), March–April 2005, pp. 19–27.** Pre-dates Nygard. Introduces the concept of an "Architecture Decision Record" as a first-class artifact and argues for explicit recording of rejected alternatives. The "rejected alternatives" section in Nygard's format inherits from Tyree & Akerman.
3. **Olaf Zimmermann et al., "Y-Statements: A Lightweight Architectural Decision Format" — published at various venues including ozimmer.ch.** Proposes the "In the context of {use case / requirement}, facing {concern}, we decided for {option} to achieve {quality}, accepting {downside}" template. Used widely as a compact alternative to the full Nygard format.
4. **MADR (Markdown Architectural Decision Records) — adr.github.io/madr.** Open-source template maintained by a community of practitioners. Specifies frontmatter format (status, deciders, date, consulted, informed) and a discoverable file structure. Useful when ADRs need machine-readable metadata for indexing.
5. **ThoughtWorks Technology Radar — thoughtworks.com/radar.** Has covered "Lightweight Architecture Decision Records" since Vol. 18 (2018) in the Techniques quadrant, with periodic upgrades to "Adopt". TW's "use ADRs sparingly" guidance aligns with the 3-criteria gate.
6. **Joel Parker Henderson, adr-tools (github.com/npryce/adr-tools).** CLI tool implementing Nygard's format with numbering helpers, supersession linking, and a `new` / `link` / `accept` command set. Establishes the de-facto convention of `0001-slug.md` filenames and `docs/adr/` directory location.
7. **Spotify Backstage — backstage.io.** Backstage's TechDocs catalog includes an ADR plugin that surfaces per-service ADRs in the service catalog UI. Demonstrates how ADRs become discoverable at scale (>1000 services) when treated as first-class catalog entries, not just files in a repo.
FILE:references/context_md_as_artifact.md
# CONTEXT.md as a Living Artifact — Preventing Glossary Decay
This reference answers exactly one decision: **how does a glossary stay alive vs decay into doc rot, and what operational practices prevent the drift?**
Pair with `scripts/glossary_code_consistency.py` for the lint-against-codebase reality check and `scripts/context_md_linter.py` for structural validation.
## The Core Claim
Every glossary decays by default. The decay path is well-documented:
```
Month 1: Glossary written during initial DDD workshop. Terms are precise.
Month 3: New feature ships. Two new domain terms used in code, neither added to glossary.
Month 6: A term in the glossary is renamed in code. Glossary still has old name.
Month 9: New engineer joins. Reads glossary. Asks "what's a 'Booking'?" — answer is "we don't call those Bookings anymore, we call them Reservations now."
Month 12: Glossary is officially declared stale. Engineers stop reading it. Drift becomes invisible.
```
The decay is not preventable by good intentions. It is prevented by **inline edits during the work that introduces the term** plus **automated lint runs at PR time** to flag mismatches.
## Three Forces That Drive Drift
1. **Language pressure from outside the bounded context.** A new partner integration uses different terminology ("subscriber" vs your "customer"). Engineers copy the partner's term into code without first reconciling with the glossary.
2. **Refactor pressure inside the bounded context.** A rename in code feels obvious ("`Booking` → `Reservation` is just a better name"), but the glossary isn't updated alongside.
3. **Convergence pressure between teams.** Multiple teams contributing to the same context use slightly different words for the same concept. Without a glossary as referee, all variants end up in code.
`scripts/glossary_code_consistency.py` operationalizes the lint against these three forces:
- **Defined-but-unused term** → a glossary entry that no code references. Either dead glossary (delete) or a rename happened (update glossary to match code).
- **Code-only proper noun** → a frequently-used capitalized term in code that the glossary doesn't define. Either generic (ignore) or domain (add to glossary now).
## Five Practices That Keep CONTEXT.md Alive
1. **Edit inline during the work.** Never batch glossary updates. When a term is introduced or refined during a feature, the same PR that adds the code edits `CONTEXT.md`. Reviewers reject PRs that introduce domain terms without glossary edits.
2. **Lint at PR time.** Run `scripts/context_md_linter.py` and `scripts/glossary_code_consistency.py` in CI. A new term in code without a glossary entry is a build warning; an outright rename mismatch is a build failure.
3. **Per-context glossaries, not one mega-glossary.** Multi-context repos use `CONTEXT-MAP.md` to point at per-context `CONTEXT.md` files. Cross-context terms get explicit translation entries ("Billing's `Customer` is Ordering's `Account`").
4. **Pruning passes.** Quarterly, run `glossary_code_consistency.py` and review the dead-glossary report. Delete entries that no code uses. Keeping dead entries dilutes signal.
5. **One sentence per definition.** If a definition runs to a paragraph, the term is hiding two concepts. Split or sharpen. Long definitions are correlated with imprecise terms.
## How CONTEXT.md Differs from Other "Documentation"
| Artifact | Purpose | Update cadence | Audience |
|---|---|---|---|
| `README.md` | Onboarding + setup | Once at project start, occasionally after | New contributors |
| `ARCHITECTURE.md` | High-level system shape | Quarterly to yearly | New architects, senior engineers |
| `docs/adr/*.md` | Record of specific decisions | Per-decision (rare; days to months apart) | Anyone asking "why did we do X this way?" |
| **`CONTEXT.md`** | **The domain glossary — what each term means in this bounded context** | **Per-feature (continuous; hours to days apart)** | **Every engineer on every PR** |
A `CONTEXT.md` is touched far more often than any other doc because it tracks the language as it evolves. If yours hasn't been edited in 6 months, it's almost certainly drifting.
## Single vs Multi-Context Repos
**Single context (most repos):** One `CONTEXT.md` at the repo root. All terms in scope.
**Multiple contexts:** A `CONTEXT-MAP.md` at the repo root lists the contexts and their relationships. Each bounded context has its own `CONTEXT.md` (and its own `docs/adr/` for context-specific decisions). Shared terms appear in both with cross-references.
```
/
├── CONTEXT-MAP.md ← lists contexts + relationships
├── docs/adr/ ← system-wide ADRs
└── src/
├── ordering/
│ ├── CONTEXT.md ← ordering-context glossary
│ └── docs/adr/ ← ordering-context ADRs
└── billing/
├── CONTEXT.md
└── docs/adr/
```
When a term spans contexts, define it in each `CONTEXT.md` with the context's perspective + a translation note pointing at the other. Don't try to define "Customer" once and have both contexts share it — that's the path back to the mega-glossary.
## Anti-Patterns
- **The spec masquerading as a glossary.** `CONTEXT.md` includes implementation details, sequence diagrams, API responses. It is a glossary, not a spec. Move spec content elsewhere.
- **The wiki masquerading as a glossary.** General programming concepts ("retry", "timeout", "config") appearing in `CONTEXT.md`. They are not domain-specific. Remove.
- **The glossary that defines without forbidding.** Each term needs `_Avoid_: <aliases>` to push back on drift. A glossary that says "Customer means X" but doesn't forbid "Client" / "Account" / "User" cannot push back when those drift in.
- **The frozen glossary.** No commits in 6+ months. Either the project is dormant or the language has drifted away from the document. Re-grill.
- **The orphan glossary.** Sits in a repo but no CI/PR process references it. It will decay within two quarters.
## Operational Checklist
When grilling against `CONTEXT.md`:
- [ ] Lint structure: `python scripts/context_md_linter.py CONTEXT.md`
- [ ] Lint vs code: `python scripts/glossary_code_consistency.py --context CONTEXT.md --code src/`
- [ ] For each "defined but unused": ask "dead term, or rename happened?"
- [ ] For each "code-only proper noun": ask "domain term that needs definition, or generic?"
- [ ] For each new term introduced during the grill: edit `CONTEXT.md` *now*, not "later"
- [ ] Multi-context repo: verify the right `CONTEXT.md` is being edited (not the wrong context's, not the root one when a per-context one applies)
## Citations (7 sources)
1. **Vladimir Khononov, *Learning Domain-Driven Design* (O'Reilly, 2021).** Chapter 9, "Communication Patterns" + Chapter 12, "Building Domain Expertise" — Khononov is the sharpest writer on language drift between bounded contexts and on how to detect it. His "linguistic boundaries are observable boundaries" framing is the foundation of the `glossary_code_consistency.py` check.
2. **Brian Kernighan & Rob Pike, *The Practice of Programming* (Addison-Wesley, 1999).** Chapter 1, "Style" — the section on naming. Kernighan's "names should reflect the role of the variable, not its type" generalizes to glossary terms: a glossary term names a role in the domain, not a data structure. Kernighan-style naming discipline is what keeps `CONTEXT.md` precise.
3. **Martin Fowler, "BoundedContext" — martinfowler.com bliki (2014, updated).** The canonical argument that ubiquitous language is **bounded** — it applies inside one context, not across all contexts. The justification for per-context `CONTEXT.md` files. https://martinfowler.com/bliki/BoundedContext.html
4. **Martin Fowler, "UbiquitousLanguage" — martinfowler.com bliki.** Companion entry to BoundedContext. Articulates the discipline of using the same vocabulary in conversation, in the model, and in the code. The justification for editing `CONTEXT.md` inline alongside code changes, not as separate doc work. https://martinfowler.com/bliki/UbiquitousLanguage.html
5. **Confluent Schema Registry / Data Contracts community — confluent.io/blog/data-contracts.** The data-contracts movement applies UL discipline to inter-service / inter-context boundaries: when two contexts exchange events or API payloads, the schema is a binding glossary. Drift between contexts becomes a schema-evolution problem, not a free-form documentation problem.
6. **Alberto Brandolini, *Introducing EventStorming* (Leanpub, ongoing).** Chapter on "Pivotal Events" + the convergence-workshop chapter. Brandolini documents how a glossary emerges from EventStorming workshops as a by-product of mapping events. The pattern of "capture the term on a sticky note when it surfaces" is the offline equivalent of the inline `CONTEXT.md` edit discipline.
7. **Eric Evans, *Domain-Driven Design: Tackling Complexity in the Heart of Software* (Addison-Wesley, 2003).** Chapter 14, "Maintaining Model Integrity" — covers the Conformist, Anticorruption Layer, and Shared Kernel patterns. Each of these is a strategy for managing the boundary between two bounded contexts that have different languages. Justifies the multi-context `CONTEXT-MAP.md` pattern and the translation-note discipline for cross-context terms.
FILE:references/ubiquitous_language.md
# Ubiquitous Language — Why a Glossary Belongs in Source Control
This reference answers exactly one decision: **why should a project's domain glossary (`CONTEXT.md`) live next to the code in source control, and what bar must it clear to earn its keep?**
Pair with `scripts/context_md_linter.py` for structural validation and `scripts/glossary_code_consistency.py` for the language-vs-code reality check.
## The Core Claim
A bounded context has **one** language. The same word must mean the same thing in conversation, in the glossary, in the type system, in the database schema, and in the UI. When language fractures across these surfaces, design defects follow: ambiguous bug reports, mismatched API contracts, broken refactors, junior engineers asking what an "account" is and getting three different answers.
The glossary is the contract that prevents the fracture. It earns its place in source control because it changes at the same cadence as the code — every time a domain term is introduced, refined, or retired, the glossary must move with it. A wiki page that lives outside the repo will drift within a quarter.
## Why a Glossary in Source Control (vs Wiki, Notion, Confluence)
| Property | In-repo `CONTEXT.md` | External wiki |
|---|---|---|
| Reviewable in PR | Yes — diff is visible alongside code | No — reviewer must remember to check |
| Versioned with code | Yes — `git log` shows term evolution | No — wikis rarely have meaningful history |
| Discoverable by new engineers | Yes — `ls` of repo root finds it | No — depends on onboarding tribal knowledge |
| Mergeable | Yes — text format, conflict-resolvable | Often no — UI-driven |
| Linter-targetable | Yes — `scripts/context_md_linter.py` | No — usually not |
| Refactor-safe | Yes — renames are grep-able | No — wiki links rot silently |
The glossary is a **language artifact**, not a documentation artifact. Documentation describes the system; the glossary **is** part of the system's design surface.
## Five Rules That Make a Glossary Survive
1. **One sentence per definition.** If the definition needs a paragraph, the term is hiding two concepts. Split it.
2. **Define what it IS, not what it does.** "An **Invoice** is a request for payment sent after delivery." Not "An invoice handles billing."
3. **List aliases to avoid.** When users say "bill" or "payment request" but mean "invoice", record that "bill" is forbidden. Without the `_Avoid_:` field, the glossary cannot push back on drift.
4. **Show relationships, not just terms.** "An **Order** produces one or more **Invoices**" tells you the cardinality. A list of bare terms doesn't.
5. **Exclude generic programming concepts.** "Timeout", "retry", "config" do not belong. Only terms specific to this project's domain qualify.
## Anti-Patterns
- **The "everything goes in" glossary.** When `CONTEXT.md` includes general programming concepts (timeout, error, util), it dilutes signal and degenerates into a wiki page.
- **The orphan glossary.** Terms defined but never used in code. Either the term is dead (delete it) or the code is using a synonym (rename code).
- **The opaque glossary.** Terms used in code but not defined. Either the term is generic (don't define it) or it's a domain concept that snuck in (define it now).
- **The deferred glossary edit.** "I'll batch up the glossary changes at the end of the sprint." By the end of the sprint, three more drift cases will have shipped. Glossary edits must land inline.
- **The frozen glossary.** No commits in 6+ months. Either the project is dormant or the language has drifted away from the document.
## Operational Checklist (for the Grill Session)
When grilling a plan against `CONTEXT.md`:
- [ ] Pre-flight `scripts/context_md_linter.py CONTEXT.md` — is the glossary well-formed?
- [ ] Run `scripts/glossary_code_consistency.py` — what's defined but unused? what's used but undefined?
- [ ] For every novel term in the plan, ask: "Is this in CONTEXT.md? If not, do we add it, or do we rephrase using an existing term?"
- [ ] For every existing term used in the plan, ask: "Does the plan use it consistent with the definition?"
- [ ] At every clarification moment, edit `CONTEXT.md` immediately — never batch.
## Citations (7 sources)
1. **Eric Evans, *Domain-Driven Design: Tackling Complexity in the Heart of Software* (Addison-Wesley, 2003).** Chapter 2, "Communication and the Use of Language" — the canonical statement of Ubiquitous Language as a design tool, not just documentation. The line "The vocabulary of that UBIQUITOUS LANGUAGE includes the names of classes and prominent operations" is the bridge between conversation and code.
2. **Vaughn Vernon, *Implementing Domain-Driven Design* (Addison-Wesley, 2013).** Chapter 1, "Getting Started with DDD" + Chapter 2, "Domains, Subdomains, and Bounded Contexts" — operationalizes Evans's UL into a workshop format and per-context discipline. Vernon's "linguistic boundaries are the most reliable boundary" framing is the source of the per-bounded-context glossary pattern.
3. **Vladimir Khononov, *Learning Domain-Driven Design* (O'Reilly, 2021).** Chapter 5, "Implementing Simple Business Logic" + Chapter 9, "Communication Patterns" — Khononov is sharpest on what happens when bounded contexts share a language vs maintain separate languages (translation layer required) and on language drift over time.
4. **Scott Wlaschin, *Domain Modeling Made Functional* (Pragmatic Bookshelf, 2018).** Part 1, "Understanding the Domain" — treats the type system as the executable form of the glossary. Wlaschin's "make illegal states unrepresentable" is the strongest form of glossary-as-contract: if the glossary says an Order must have at least one line item, the type prevents zero-item Orders at compile time.
5. **Alberto Brandolini, *Introducing EventStorming: An Act of Deliberate Collective Learning* (Leanpub, 2017–ongoing).** Chapter on "Sticky note color codes" + chapter on convergence — EventStorming workshops produce a glossary as a by-product of mapping the domain. Brandolini's pattern of capturing terms as they emerge on sticky notes is the offline equivalent of the inline `CONTEXT.md` edit.
6. **Abel Avram & Floyd Marinescu, *Domain-Driven Design Quickly* (InfoQ, 2006, free e-book).** Chapter 2, "Ubiquitous Language" — the most concise distillation of Evans's UL chapter. Useful as a reference to hand to engineers who won't read the blue book.
7. **Martin Fowler, "BoundedContext" — martinfowler.com bliki (2014, updated).** Fowler's framing of "Ubiquitous Language … doesn't apply to the whole project, it only has to apply within a particular Bounded Context" justifies the per-context glossary pattern in `CONTEXT-MAP.md`-style multi-context repos. https://martinfowler.com/bliki/BoundedContext.html
FILE:scripts/adr_scanner.py
#!/usr/bin/env python3
"""adr_scanner.py — Walk docs/adr/ and validate ADR files against the format.
Stdlib-only. Applies the rules from Matt Pocock's upstream ADR-FORMAT.md
(preserved verbatim in the skill's ADR-FORMAT.md):
1. Each file matches the `NNNN-slug.md` pattern (4-digit zero-padded number + kebab-case slug)
2. Numbering is sequential — no gaps, no duplicates
3. Each ADR has an H1 (the title)
4. Each ADR has a non-empty body after the H1 (at least the 1-3 sentence context+decision)
5. Optional status frontmatter, if present, has a valid value
(proposed | accepted | deprecated | superseded by ADR-NNNN)
6. Superseded-by references point at an existing ADR number
Output: directory-level summary + per-file findings.
NO LLM CALLS. Pure regex + filesystem walking.
Usage:
python adr_scanner.py docs/adr/
python adr_scanner.py docs/adr/ --output json
python adr_scanner.py --sample # scan an embedded sample directory layout
"""
import argparse
import json
import re
import sys
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple
ADR_FILENAME_RE = re.compile(r"^(\d{4})-([a-z0-9]+(?:-[a-z0-9]+)*)\.md$")
VALID_STATUSES = {"proposed", "accepted", "deprecated"}
SUPERSEDED_RE = re.compile(r"^superseded\s+by\s+ADR-?(\d{1,4})$", re.IGNORECASE)
SAMPLE_ADRS: Dict[str, str] = {
"0001-event-sourced-orders.md": (
"# Event-source the Order write model\n"
"\n"
"We need an audit trail of every state change on an Order for compliance + analytics. "
"We chose event sourcing for the Order write model and a Postgres projection for the read model. "
"Trade-off accepted: eventual consistency on the read side in exchange for the audit trail and replay.\n"
),
"0002-postgres-for-write-model.md": (
"---\n"
"status: accepted\n"
"---\n"
"\n"
"# Postgres for the write-side event store\n"
"\n"
"We considered EventStore and Kafka. Postgres won on operational familiarity + transactional guarantees + cost.\n"
),
"0003-rest-over-graphql.md": (
"---\n"
"status: accepted\n"
"---\n"
"\n"
"# REST over GraphQL for the public API\n"
"\n"
"GraphQL would have given clients more flexibility but added subscription complexity we don't need at our scale.\n"
),
}
def parse_frontmatter(text: str) -> Tuple[Dict[str, str], str]:
"""Return (frontmatter_dict, body) for a file that may have YAML-ish frontmatter.
Only handles simple `key: value` lines (no nested YAML, no lists) — stdlib-only.
"""
if not text.startswith("---\n"):
return {}, text
end_marker = text.find("\n---\n", 4)
if end_marker == -1:
return {}, text
fm_block = text[4:end_marker]
body = text[end_marker + 5 :]
fm: Dict[str, str] = {}
for line in fm_block.splitlines():
if ":" in line:
k, v = line.split(":", 1)
fm[k.strip().lower()] = v.strip()
return fm, body
def scan_directory(adr_dir: Path) -> Dict[str, Any]:
findings: List[Dict[str, Any]] = []
files: List[Tuple[int, str, Path]] = []
def add(file: str, rule: str, level: str, message: str) -> None:
findings.append({"file": file, "rule": rule, "level": level, "message": message})
if not adr_dir.exists():
add("(root)", "directory", "FAIL", f"Directory does not exist: {adr_dir}")
return finalize(findings, 0)
if not adr_dir.is_dir():
add("(root)", "directory", "FAIL", f"Path is not a directory: {adr_dir}")
return finalize(findings, 0)
md_files = sorted(p for p in adr_dir.iterdir() if p.is_file() and p.suffix == ".md")
if not md_files:
add("(root)", "directory", "WARN", "Directory is empty — no ADRs scanned. Create lazily when the first ADR is needed.")
return finalize(findings, 0)
# Rule 1: filename pattern
for p in md_files:
m = ADR_FILENAME_RE.match(p.name)
if not m:
add(p.name, "filename-pattern", "FAIL", f"Filename does not match NNNN-slug.md pattern. Expected e.g. 0001-event-sourced-orders.md.")
continue
number = int(m.group(1))
files.append((number, p.name, p))
add(p.name, "filename-pattern", "PASS", f"Filename matches pattern (number={number:04d}).")
files.sort(key=lambda t: t[0])
# Rule 2: numbering sequence (no gaps, no duplicates)
seen: Dict[int, List[str]] = {}
for number, name, _ in files:
seen.setdefault(number, []).append(name)
for number, names in seen.items():
if len(names) > 1:
add(", ".join(names), "numbering-duplicate", "FAIL", f"Duplicate ADR number {number:04d}.")
if files:
expected = list(range(1, files[-1][0] + 1))
actual = sorted(seen.keys())
gaps = [n for n in expected if n not in actual]
if gaps:
add("(root)", "numbering-gap", "WARN", f"Number gap(s) in sequence: {', '.join(f'{g:04d}' for g in gaps)}. Either commit withdrawn ADRs as 'proposed → withdrawn' or renumber.")
else:
add("(root)", "numbering-sequence", "PASS", f"Sequential numbering 0001..{files[-1][0]:04d} with no gaps.")
# Rules 3, 4, 5, 6: per-ADR
numbers_present = {n for n, _, _ in files}
for number, name, path in files:
text = path.read_text(encoding="utf-8") if path.is_file() else SAMPLE_ADRS.get(name, "")
fm, body = parse_frontmatter(text)
# Rule 3: H1 present
h1_match = re.search(r"^#\s+(.+?)\s*$", body, re.MULTILINE)
if not h1_match:
add(name, "h1-present", "FAIL", "No H1 (`# Title`) found in body.")
continue
else:
add(name, "h1-present", "PASS", f"H1 found: '{h1_match.group(1).strip()}'.")
# Rule 4: non-empty body after H1
after_h1 = body[h1_match.end():].strip()
if not after_h1:
add(name, "body-non-empty", "FAIL", "ADR has H1 but no body. The 1-3 sentence context+decision is required.")
else:
word_count = len(re.findall(r"\b\w+\b", after_h1))
if word_count < 10:
add(name, "body-non-empty", "WARN", f"ADR body is very short ({word_count} words). Confirm context+decision+why are all stated.")
else:
add(name, "body-non-empty", "PASS", f"Body present ({word_count} words).")
# Rule 5: optional status frontmatter sanity
status = fm.get("status", "").strip().lower() if fm else ""
if status:
if status in VALID_STATUSES:
add(name, "status-frontmatter", "PASS", f"Status '{status}' is valid.")
elif SUPERSEDED_RE.match(status):
m = SUPERSEDED_RE.match(status)
target = int(m.group(1))
# Rule 6: superseded-by points at existing ADR
if target in numbers_present:
add(name, "status-supersede-target", "PASS", f"Superseded by ADR-{target:04d} which exists.")
else:
add(name, "status-supersede-target", "FAIL", f"Superseded by ADR-{target:04d} but that ADR is not present in this directory.")
else:
add(name, "status-frontmatter", "FAIL", f"Status '{status}' is not one of {sorted(VALID_STATUSES)} or 'superseded by ADR-NNNN'.")
return finalize(findings, len(files))
def finalize(findings: List[Dict[str, Any]], adr_count: int) -> Dict[str, Any]:
counts = {"PASS": 0, "WARN": 0, "FAIL": 0}
for f in findings:
counts[f["level"]] += 1
if counts["FAIL"] > 0:
verdict = "FAIL"
elif counts["WARN"] > 0:
verdict = "WARN"
else:
verdict = "PASS"
return {"verdict": verdict, "adr_count": adr_count, "counts": counts, "findings": findings}
def render_human(result: Dict[str, Any]) -> str:
out: List[str] = []
out.append(f"ADR directory scan verdict: {result['verdict']}")
out.append(f" ADRs scanned: {result['adr_count']}")
counts = result["counts"]
out.append(f" PASS: {counts['PASS']} WARN: {counts['WARN']} FAIL: {counts['FAIL']}")
out.append("")
out.append("Findings:")
for f in result["findings"]:
marker = {"PASS": "[ok]", "WARN": "[warn]", "FAIL": "[FAIL]"}[f["level"]]
out.append(f" {marker} {f['file']:<40s} {f['rule']}: {f['message']}")
return "\n".join(out)
def run_sample() -> Dict[str, Any]:
"""Scan the embedded sample by writing it to a tempdir."""
import tempfile
with tempfile.TemporaryDirectory() as td:
d = Path(td) / "adr"
d.mkdir()
for name, content in SAMPLE_ADRS.items():
(d / name).write_text(content, encoding="utf-8")
return scan_directory(d)
def main(argv: List[str]) -> int:
parser = argparse.ArgumentParser(description=__doc__.split("\n")[0])
parser.add_argument("adr_dir", nargs="?", help="Path to docs/adr/ directory")
parser.add_argument("--sample", action="store_true", help="Scan the embedded sample ADR layout")
parser.add_argument("--output", choices=["human", "json"], default="human")
args = parser.parse_args(argv)
if args.sample:
result = run_sample()
elif args.adr_dir:
result = scan_directory(Path(args.adr_dir))
else:
parser.print_help()
return 0
if args.output == "json":
print(json.dumps(result, indent=2))
else:
print(render_human(result))
return 0 if result["verdict"] != "FAIL" else 1
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
FILE:scripts/context_md_linter.py
#!/usr/bin/env python3
"""context_md_linter.py — Validate a CONTEXT.md against the CONTEXT-FORMAT.md structure.
Stdlib-only. Walks a CONTEXT.md and applies the format rules from Matt Pocock's
upstream CONTEXT-FORMAT.md (preserved verbatim in the skill's CONTEXT-FORMAT.md):
1. H1 present at top (the context name)
2. One-or-two-sentence description follows the H1
3. ## Language section present
4. Inside Language: each term is in `**Term**:` bold form
5. Inside Language: each term has a one-sentence definition
6. Inside Language: each term has a `_Avoid_:` aliases line (WARN if missing)
7. ## Relationships section present (WARN if missing)
8. ## Example dialogue section present (WARN if missing)
9. Optional: ## Flagged ambiguities section
Output: PASS / WARN / FAIL per rule + an overall verdict.
NO LLM CALLS. Pure regex + line walking.
Usage:
python context_md_linter.py CONTEXT.md
python context_md_linter.py CONTEXT.md --output json
python context_md_linter.py --sample # lint the embedded sample
"""
import argparse
import json
import re
import sys
from pathlib import Path
from typing import Any, Dict, List, Tuple
SAMPLE_CONTEXT_MD = """# Ordering
The ordering context receives customer orders and tracks them through to handoff to Fulfillment.
## Language
**Order**:
A confirmed request from a Customer to acquire one or more Products.
_Avoid_: Purchase, transaction, cart
**Customer**:
A person or organization that places Orders.
_Avoid_: Client, buyer, account
**Product**:
A single SKU that can appear on an Order line.
_Avoid_: Item, good, SKU
## Relationships
- An **Order** belongs to exactly one **Customer**
- An **Order** has one or more **Products** via line items
- A **Customer** can have many **Orders**
## Example dialogue
> **Dev:** "When a **Customer** places an **Order**, are the **Products** locked at order time?"
> **Domain expert:** "Yes — Product price + spec is snapshotted onto the Order line. Subsequent Product edits don't change historical Orders."
## Flagged ambiguities
- "account" was used to mean both **Customer** and "billing account" — resolved: billing account moves to Billing context.
"""
def split_into_sections(text: str) -> Dict[str, str]:
"""Split markdown into top-level ## sections keyed by header text."""
sections: Dict[str, str] = {}
current_header = "_preamble_"
buffer: List[str] = []
for line in text.splitlines():
m = re.match(r"^##\s+(.+?)\s*$", line)
if m:
sections[current_header] = "\n".join(buffer).strip()
current_header = m.group(1).strip().lower()
buffer = []
else:
buffer.append(line)
sections[current_header] = "\n".join(buffer).strip()
return sections
def extract_terms(language_section: str) -> List[Tuple[str, str, str]]:
"""Return list of (term, definition_line, avoid_line) tuples from the Language section.
Each term entry looks like:
**Term**:
Definition sentence.
_Avoid_: alias1, alias2
"""
results: List[Tuple[str, str, str]] = []
# Match `**Term**:` followed by the next non-empty line as definition,
# and optionally an `_Avoid_:` line within the next 3 lines.
pattern = re.compile(
r"\*\*([^*]+?)\*\*\s*:\s*\n([^\n]+)\n?(?:([^\n]*_Avoid_[^\n]*)\n?)?",
re.MULTILINE,
)
for match in pattern.finditer(language_section):
term = match.group(1).strip()
definition = match.group(2).strip()
avoid = (match.group(3) or "").strip()
results.append((term, definition, avoid))
return results
def lint(text: str) -> Dict[str, Any]:
findings: List[Dict[str, str]] = []
def add(rule: str, level: str, message: str) -> None:
findings.append({"rule": rule, "level": level, "message": message})
# Rule 1: H1 present
lines = text.splitlines()
h1_line_index = None
for i, line in enumerate(lines):
if re.match(r"^#\s+\S", line):
h1_line_index = i
break
if h1_line_index is None:
add("h1-present", "FAIL", "No H1 (top-level '# Title') found. CONTEXT.md must start with the context name as H1.")
else:
add("h1-present", "PASS", f"H1 found at line {h1_line_index + 1}.")
# Rule 2: one-or-two-sentence description after H1
if h1_line_index is not None:
desc_lines: List[str] = []
for line in lines[h1_line_index + 1 :]:
if re.match(r"^##\s", line):
break
if line.strip():
desc_lines.append(line.strip())
desc = " ".join(desc_lines).strip()
sentence_count = len(re.findall(r"[.!?](?:\s|$)", desc))
if not desc:
add("description-present", "FAIL", "No description sentence between the H1 and the first ## section.")
elif sentence_count > 3:
add(
"description-length",
"WARN",
f"Description has {sentence_count} sentences. CONTEXT-FORMAT.md asks for one or two.",
)
else:
add("description-present", "PASS", f"Description present ({sentence_count} sentence(s)).")
# Rule 3: ## Language section present
sections = split_into_sections(text)
if "language" not in sections:
add("language-section", "FAIL", "No '## Language' section found. This is the required core of CONTEXT.md.")
return finalize(findings)
add("language-section", "PASS", "'## Language' section found.")
# Rules 4 + 5 + 6: terms inside Language
terms = extract_terms(sections["language"])
if not terms:
add(
"language-terms",
"FAIL",
"No terms detected in the Language section. Each term must be in '**Term**:' bold form followed by a one-sentence definition.",
)
else:
add("language-terms", "PASS", f"Detected {len(terms)} term(s) in Language section.")
for term, definition, avoid in terms:
# Rule 5: definition exists
if not definition or definition.startswith("_Avoid_") or definition.startswith("**"):
add(
"term-definition",
"FAIL",
f"Term '**{term}**:' has no definition line (next non-empty line should be the definition).",
)
else:
# Length heuristic: definition should be <= 200 chars (one sentence-ish)
if len(definition) > 200:
add(
"term-definition-length",
"WARN",
f"Term '**{term}**' definition is {len(definition)} chars. CONTEXT-FORMAT.md asks for one sentence max.",
)
# Rule 6: _Avoid_ line
if not avoid:
add(
"term-avoid",
"WARN",
f"Term '**{term}**' has no '_Avoid_:' aliases line. Without forbidden aliases, the glossary can't push back on drift.",
)
# Rule 7: Relationships section
if "relationships" not in sections:
add(
"relationships-section",
"WARN",
"No '## Relationships' section found. CONTEXT-FORMAT.md asks for one to show cardinality between terms.",
)
else:
add("relationships-section", "PASS", "'## Relationships' section found.")
# Rule 8: Example dialogue
if "example dialogue" not in sections:
add(
"example-dialogue",
"WARN",
"No '## Example dialogue' section found. CONTEXT-FORMAT.md asks for a dev/domain-expert exchange.",
)
else:
add("example-dialogue", "PASS", "'## Example dialogue' section found.")
# Rule 9: Flagged ambiguities (optional, only check presence)
if "flagged ambiguities" in sections:
add("flagged-ambiguities", "PASS", "'## Flagged ambiguities' section found (optional but useful).")
return finalize(findings)
def finalize(findings: List[Dict[str, str]]) -> Dict[str, Any]:
counts = {"PASS": 0, "WARN": 0, "FAIL": 0}
for f in findings:
counts[f["level"]] += 1
if counts["FAIL"] > 0:
verdict = "FAIL"
elif counts["WARN"] > 0:
verdict = "WARN"
else:
verdict = "PASS"
return {"verdict": verdict, "counts": counts, "findings": findings}
def render_human(result: Dict[str, Any]) -> str:
out: List[str] = []
verdict = result["verdict"]
counts = result["counts"]
out.append(f"CONTEXT.md lint verdict: {verdict}")
out.append(f" PASS: {counts['PASS']} WARN: {counts['WARN']} FAIL: {counts['FAIL']}")
out.append("")
out.append("Findings:")
for f in result["findings"]:
marker = {"PASS": "[ok]", "WARN": "[warn]", "FAIL": "[FAIL]"}[f["level"]]
out.append(f" {marker} {f['rule']}: {f['message']}")
return "\n".join(out)
def main(argv: List[str]) -> int:
parser = argparse.ArgumentParser(description=__doc__.split("\n")[0])
parser.add_argument("path", nargs="?", help="Path to CONTEXT.md")
parser.add_argument("--sample", action="store_true", help="Lint the embedded sample CONTEXT.md")
parser.add_argument("--output", choices=["human", "json"], default="human")
args = parser.parse_args(argv)
if args.sample:
text = SAMPLE_CONTEXT_MD
elif args.path:
p = Path(args.path)
if not p.exists():
print(f"error: {args.path} not found", file=sys.stderr)
return 2
text = p.read_text(encoding="utf-8")
else:
parser.print_help()
return 0
result = lint(text)
if args.output == "json":
print(json.dumps(result, indent=2))
else:
print(render_human(result))
return 0 if result["verdict"] != "FAIL" else 1
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
FILE:scripts/glossary_code_consistency.py
#!/usr/bin/env python3
"""glossary_code_consistency.py — Cross-reference CONTEXT.md terms against the codebase.
Stdlib-only. Reads bold terms from CONTEXT.md and scans a codebase directory for
each term's usage. Surfaces two grilling-question seeds:
1. DEAD GLOSSARY — a term is defined in CONTEXT.md but never appears in code.
Either the term is stale (delete it) or the code uses a synonym (rename).
2. CODE-ONLY PROPER NOUN — a capitalized word that appears frequently in code
but isn't defined in CONTEXT.md. Either it's a generic programming concept
(ignore) or it's a domain term that snuck in undefined (add to glossary).
Both lists are seeded as opening grill-with-docs questions.
NO LLM CALLS. Pure file walking + regex + frequency counting.
Limitations (intentional, stdlib-only):
- Word-boundary matching is case-insensitive. "Order" matches "order", "ORDER", "orders".
- "Code-only proper noun" detection uses a simple heuristic: capitalized
words >= MIN_FREQUENCY occurrences across non-test files. Tunable via flags.
- Only scans common source extensions by default (override with --extensions).
Usage:
python glossary_code_consistency.py --context CONTEXT.md --code src/
python glossary_code_consistency.py --context CONTEXT.md --code src/ --output json
python glossary_code_consistency.py --sample
"""
import argparse
import json
import re
import sys
from collections import Counter
from pathlib import Path
from typing import Any, Dict, List, Set, Tuple
DEFAULT_EXTENSIONS = {
".py",
".ts",
".tsx",
".js",
".jsx",
".go",
".java",
".kt",
".rb",
".cs",
".rs",
".swift",
".php",
".scala",
".clj",
".ex",
".exs",
}
DEFAULT_EXCLUDE_DIRS = {"node_modules", ".git", "dist", "build", "target", ".venv", "venv", "__pycache__"}
TEST_FILE_HINTS = (".test.", ".spec.", "_test.", "tests/", "/test/")
PROPER_NOUN_RE = re.compile(r"\b([A-Z][a-zA-Z]{2,})\b")
GENERIC_WORDS = {
# Programming concepts that capitalize but aren't domain terms
"True", "False", "None", "Null", "Promise", "Error", "Exception",
"String", "Number", "Boolean", "Array", "Object", "Map", "Set",
"List", "Dict", "Tuple", "Optional", "Any", "Result", "Date",
"Math", "JSON", "URL", "URI", "HTTP", "HTTPS", "API", "ID", "UUID",
"GET", "POST", "PUT", "DELETE", "PATCH", "OK", "TODO", "FIXME",
"Test", "Mock", "Stub", "Spy", "Given", "When", "Then", "Describe",
}
SAMPLE_CONTEXT_MD = """# Ordering
## Language
**Order**:
A confirmed request from a Customer to acquire one or more Products.
_Avoid_: Purchase, transaction
**Customer**:
A person or organization that places Orders.
_Avoid_: Client, buyer
**Product**:
A single SKU that can appear on an Order line.
_Avoid_: Item, good
**Discount**:
A reduction applied to an Order at checkout.
_Avoid_: Coupon, promo
"""
SAMPLE_CODE_FILES: Dict[str, str] = {
"src/orders.py": (
"class Order:\n"
" pass\n"
"\n"
"def cancel_order(order_id: str) -> None:\n"
" pass\n"
"\n"
"def list_customer_orders(customer_id: str) -> list[Order]:\n"
" pass\n"
),
"src/customers.py": (
"class Customer:\n"
" pass\n"
"\n"
"class Subscription:\n"
" # NOTE: Subscription is used heavily but not in glossary\n"
" pass\n"
"\n"
"def find_customer(email: str) -> Customer:\n"
" pass\n"
),
"src/products.py": (
"class Product:\n"
" pass\n"
"\n"
"class Inventory:\n"
" pass\n"
"\n"
"def find_product(sku: str) -> Product:\n"
" pass\n"
),
# Note: Discount is defined in glossary but never used in code.
}
def extract_glossary_terms(context_md_text: str) -> List[str]:
"""Pull bold terms from CONTEXT.md `**Term**:` patterns."""
return re.findall(r"\*\*([^*]+?)\*\*\s*:", context_md_text)
def walk_codebase(root: Path, extensions: Set[str], exclude_dirs: Set[str]) -> List[Path]:
found: List[Path] = []
for path in root.rglob("*"):
if path.is_dir():
continue
if any(part in exclude_dirs for part in path.parts):
continue
if path.suffix in extensions:
found.append(path)
return found
def is_test_file(path: Path) -> bool:
s = str(path).replace("\\", "/")
return any(hint in s for hint in TEST_FILE_HINTS)
def count_term_in_text(text: str, term: str) -> int:
pattern = re.compile(rf"\b{re.escape(term)}\b", re.IGNORECASE)
return len(pattern.findall(text))
def count_proper_nouns(text: str) -> Counter:
counter: Counter = Counter()
for match in PROPER_NOUN_RE.finditer(text):
counter[match.group(1)] += 1
return counter
def analyze(
context_md_text: str,
code_files: List[Tuple[str, str]],
min_proper_noun_frequency: int,
) -> Dict[str, Any]:
"""code_files: list of (relative_path, text) tuples."""
glossary_terms = extract_glossary_terms(context_md_text)
glossary_term_set_lower = {t.lower() for t in glossary_terms}
# Per-term usage count in non-test files
term_usage: Dict[str, int] = {t: 0 for t in glossary_terms}
code_proper_nouns: Counter = Counter()
files_scanned = 0
files_tests_skipped = 0
for path_str, text in code_files:
path = Path(path_str)
if is_test_file(path):
files_tests_skipped += 1
continue
files_scanned += 1
for term in glossary_terms:
term_usage[term] += count_term_in_text(text, term)
for noun, count in count_proper_nouns(text).items():
code_proper_nouns[noun] += count
# Dead glossary: terms with zero usage
dead_terms = [t for t, n in term_usage.items() if n == 0]
# Code-only proper nouns: frequent capitalized identifiers NOT in glossary
# and NOT in the generic stop-list
code_only: List[Tuple[str, int]] = []
for noun, count in code_proper_nouns.most_common():
if count < min_proper_noun_frequency:
break
if noun.lower() in glossary_term_set_lower:
continue
if noun in GENERIC_WORDS:
continue
code_only.append((noun, count))
return {
"files_scanned": files_scanned,
"files_tests_skipped": files_tests_skipped,
"glossary_term_count": len(glossary_terms),
"term_usage": term_usage,
"dead_glossary_terms": dead_terms,
"code_only_proper_nouns": code_only,
"min_proper_noun_frequency": min_proper_noun_frequency,
}
def render_human(result: Dict[str, Any]) -> str:
out: List[str] = []
out.append("Glossary↔Code consistency report")
out.append(f" Files scanned: {result['files_scanned']} (test files skipped: {result['files_tests_skipped']})")
out.append(f" Glossary terms: {result['glossary_term_count']}")
out.append("")
out.append("Term usage (occurrences in non-test code):")
for term, count in sorted(result["term_usage"].items(), key=lambda kv: (-kv[1], kv[0])):
marker = " " if count > 0 else "!!"
out.append(f" {marker} {term:<30s} {count}")
out.append("")
if result["dead_glossary_terms"]:
out.append("DEAD GLOSSARY (defined but never used in code) — grill these:")
for term in result["dead_glossary_terms"]:
out.append(f" - '{term}': dead term, or rename happened?")
else:
out.append("DEAD GLOSSARY: (none — every defined term is used in code)")
out.append("")
if result["code_only_proper_nouns"]:
out.append(
f"CODE-ONLY PROPER NOUNS (>= {result['min_proper_noun_frequency']}x, not in glossary, not generic) — grill these:"
)
for noun, count in result["code_only_proper_nouns"]:
out.append(f" - '{noun}' ({count} occurrences): domain term that needs definition, or generic?")
else:
out.append("CODE-ONLY PROPER NOUNS: (none above threshold — glossary covers the frequent domain nouns)")
return "\n".join(out)
def run_sample(min_freq: int) -> Dict[str, Any]:
files = [(p, t) for p, t in SAMPLE_CODE_FILES.items()]
return analyze(SAMPLE_CONTEXT_MD, files, min_freq)
def main(argv: List[str]) -> int:
parser = argparse.ArgumentParser(description=__doc__.split("\n")[0])
parser.add_argument("--context", help="Path to CONTEXT.md")
parser.add_argument("--code", help="Path to codebase root")
parser.add_argument(
"--extensions",
help="Comma-separated source extensions to scan (default: common languages)",
default=None,
)
parser.add_argument(
"--min-frequency",
type=int,
default=3,
help="Minimum occurrences for a code-only proper noun to surface (default: 3)",
)
parser.add_argument("--sample", action="store_true", help="Run on the embedded sample data")
parser.add_argument("--output", choices=["human", "json"], default="human")
args = parser.parse_args(argv)
if args.sample:
result = run_sample(args.min_frequency)
elif args.context and args.code:
context_path = Path(args.context)
code_root = Path(args.code)
if not context_path.exists():
print(f"error: {args.context} not found", file=sys.stderr)
return 2
if not code_root.exists():
print(f"error: {args.code} not found", file=sys.stderr)
return 2
if args.extensions:
exts = {e.strip() if e.strip().startswith(".") else "." + e.strip() for e in args.extensions.split(",")}
else:
exts = DEFAULT_EXTENSIONS
files: List[Tuple[str, str]] = []
for p in walk_codebase(code_root, exts, DEFAULT_EXCLUDE_DIRS):
try:
files.append((str(p), p.read_text(encoding="utf-8", errors="ignore")))
except (OSError, UnicodeDecodeError):
continue
result = analyze(context_path.read_text(encoding="utf-8"), files, args.min_frequency)
else:
parser.print_help()
return 0
if args.output == "json":
print(json.dumps(result, indent=2))
else:
print(render_human(result))
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
Đưa mô hình ML vào sản xuất, xây MLOps pipeline, tích hợp LLM, feature store, giám sát drift, RAG và tối ưu chi phí.
---
name: "senior-ml-engineer"
description: ML engineering skill for productionizing models, building MLOps pipelines, and integrating LLMs. Covers model deployment, feature stores, drift monitoring, RAG systems, and cost optimization. Use when the user asks about deploying ML models to production, setting up MLOps infrastructure (MLflow, Kubeflow, Kubernetes, Docker), monitoring model performance or drift, building RAG pipelines, or integrating LLM APIs with retry logic and cost controls. Focused on production and operational concerns rather than model research or initial training.
triggers:
- MLOps pipeline
- model deployment
- feature store
- model monitoring
- drift detection
- RAG system
- LLM integration
- model serving
- A/B testing ML
- automated retraining
---
# Senior ML Engineer
Production ML engineering patterns for model deployment, MLOps infrastructure, and LLM integration.
---
## Table of Contents
- [Model Deployment Workflow](#model-deployment-workflow)
- [MLOps Pipeline Setup](#mlops-pipeline-setup)
- [LLM Integration Workflow](#llm-integration-workflow)
- [RAG System Implementation](#rag-system-implementation)
- [Model Monitoring](#model-monitoring)
- [Reference Documentation](#reference-documentation)
- [Tools](#tools)
---
## Model Deployment Workflow
Deploy a trained model to production with monitoring:
1. Export model to standardized format (ONNX, TorchScript, SavedModel)
2. Package model with dependencies in Docker container
3. Deploy to staging environment
4. Run integration tests against staging
5. Deploy canary (5% traffic) to production
6. Monitor latency and error rates for 1 hour
7. Promote to full production if metrics pass
8. **Validation:** p95 latency < 100ms, error rate < 0.1%
### Container Template
```dockerfile
FROM python:3.11-slim
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY model/ /app/model/
COPY src/ /app/src/
HEALTHCHECK CMD curl -f http://localhost:8080/health || exit 1
EXPOSE 8080
CMD ["uvicorn", "src.server:app", "--host", "0.0.0.0", "--port", "8080"]
```
### Serving Options
| Option | Latency | Throughput | Use Case |
|--------|---------|------------|----------|
| FastAPI + Uvicorn | Low | Medium | REST APIs, small models |
| Triton Inference Server | Very Low | Very High | GPU inference, batching |
| TensorFlow Serving | Low | High | TensorFlow models |
| TorchServe | Low | High | PyTorch models |
| Ray Serve | Medium | High | Complex pipelines, multi-model |
---
## MLOps Pipeline Setup
Establish automated training and deployment:
1. Configure feature store (Feast, Tecton) for training data
2. Set up experiment tracking (MLflow, Weights & Biases)
3. Create training pipeline with hyperparameter logging
4. Register model in model registry with version metadata
5. Configure staging deployment triggered by registry events
6. Set up A/B testing infrastructure for model comparison
7. Enable drift monitoring with alerting
8. **Validation:** New models automatically evaluated against baseline
### Feature Store Pattern
```python
from feast import Entity, Feature, FeatureView, FileSource
user = Entity(name="user_id", value_type=ValueType.INT64)
user_features = FeatureView(
name="user_features",
entities=["user_id"],
ttl=timedelta(days=1),
features=[
Feature(name="purchase_count_30d", dtype=ValueType.INT64),
Feature(name="avg_order_value", dtype=ValueType.FLOAT),
],
online=True,
source=FileSource(path="data/user_features.parquet"),
)
```
### Retraining Triggers
| Trigger | Detection | Action |
|---------|-----------|--------|
| Scheduled | Cron (weekly/monthly) | Full retrain |
| Performance drop | Accuracy < threshold | Immediate retrain |
| Data drift | PSI > 0.2 | Evaluate, then retrain |
| New data volume | X new samples | Incremental update |
---
## LLM Integration Workflow
Integrate LLM APIs into production applications:
1. Create provider abstraction layer for vendor flexibility
2. Implement retry logic with exponential backoff
3. Configure fallback to secondary provider
4. Set up token counting and context truncation
5. Add response caching for repeated queries
6. Implement cost tracking per request
7. Add structured output validation with Pydantic
8. **Validation:** Response parses correctly, cost within budget
### Provider Abstraction
```python
from abc import ABC, abstractmethod
from tenacity import retry, stop_after_attempt, wait_exponential
class LLMProvider(ABC):
@abstractmethod
def complete(self, prompt: str, **kwargs) -> str:
pass
@retry(stop=stop_after_attempt(3), wait=wait_exponential(min=1, max=10))
def call_llm_with_retry(provider: LLMProvider, prompt: str) -> str:
return provider.complete(prompt)
```
### Cost Management
| Provider | Input Cost | Output Cost |
|----------|------------|-------------|
| GPT-4 | $0.03/1K | $0.06/1K |
| GPT-3.5 | $0.0005/1K | $0.0015/1K |
| Claude 3 Opus | $0.015/1K | $0.075/1K |
| Claude 3 Haiku | $0.00025/1K | $0.00125/1K |
---
## RAG System Implementation
Build retrieval-augmented generation pipeline:
1. Choose vector database (Pinecone, Qdrant, Weaviate)
2. Select embedding model based on quality/cost tradeoff
3. Implement document chunking strategy
4. Create ingestion pipeline with metadata extraction
5. Build retrieval with query embedding
6. Add reranking for relevance improvement
7. Format context and send to LLM
8. **Validation:** Response references retrieved context, no hallucinations
### Vector Database Selection
| Database | Hosting | Scale | Latency | Best For |
|----------|---------|-------|---------|----------|
| Pinecone | Managed | High | Low | Production, managed |
| Qdrant | Both | High | Very Low | Performance-critical |
| Weaviate | Both | High | Low | Hybrid search |
| Chroma | Self-hosted | Medium | Low | Prototyping |
| pgvector | Self-hosted | Medium | Medium | Existing Postgres |
### Chunking Strategies
| Strategy | Chunk Size | Overlap | Best For |
|----------|------------|---------|----------|
| Fixed | 500-1000 tokens | 50-100 | General text |
| Sentence | 3-5 sentences | 1 sentence | Structured text |
| Semantic | Variable | Based on meaning | Research papers |
| Recursive | Hierarchical | Parent-child | Long documents |
---
## Model Monitoring
Monitor production models for drift and degradation:
1. Set up latency tracking (p50, p95, p99)
2. Configure error rate alerting
3. Implement input data drift detection
4. Track prediction distribution shifts
5. Log ground truth when available
6. Compare model versions with A/B metrics
7. Set up automated retraining triggers
8. **Validation:** Alerts fire before user-visible degradation
### Drift Detection
```python
from scipy.stats import ks_2samp
def detect_drift(reference, current, threshold=0.05):
statistic, p_value = ks_2samp(reference, current)
return {
"drift_detected": p_value < threshold,
"ks_statistic": statistic,
"p_value": p_value
}
```
### Alert Thresholds
| Metric | Warning | Critical |
|--------|---------|----------|
| p95 latency | > 100ms | > 200ms |
| Error rate | > 0.1% | > 1% |
| PSI (drift) | > 0.1 | > 0.2 |
| Accuracy drop | > 2% | > 5% |
---
## Reference Documentation
### MLOps Production Patterns
`references/mlops_production_patterns.md` contains:
- Model deployment pipeline with Kubernetes manifests
- Feature store architecture with Feast examples
- Model monitoring with drift detection code
- A/B testing infrastructure with traffic splitting
- Automated retraining pipeline with MLflow
### LLM Integration Guide
`references/llm_integration_guide.md` contains:
- Provider abstraction layer pattern
- Retry and fallback strategies with tenacity
- Prompt engineering templates (few-shot, CoT)
- Token optimization with tiktoken
- Cost calculation and tracking
### RAG System Architecture
`references/rag_system_architecture.md` contains:
- RAG pipeline implementation with code
- Vector database comparison and integration
- Chunking strategies (fixed, semantic, recursive)
- Embedding model selection guide
- Hybrid search and reranking patterns
---
## Tools
### Model Deployment Pipeline
```bash
python scripts/model_deployment_pipeline.py --model model.pkl --target staging
```
Generates deployment artifacts: Dockerfile, Kubernetes manifests, health checks.
### RAG System Builder
```bash
python scripts/rag_system_builder.py --config rag_config.yaml --analyze
```
Scaffolds RAG pipeline with vector store integration and retrieval logic.
### ML Monitoring Suite
```bash
python scripts/ml_monitoring_suite.py --config monitoring.yaml --deploy
```
Sets up drift detection, alerting, and performance dashboards.
---
## Tech Stack
| Category | Tools |
|----------|-------|
| ML Frameworks | PyTorch, TensorFlow, Scikit-learn, XGBoost |
| LLM Frameworks | LangChain, LlamaIndex, DSPy |
| MLOps | MLflow, Weights & Biases, Kubeflow |
| Data | Spark, Airflow, dbt, Kafka |
| Deployment | Docker, Kubernetes, Triton |
| Databases | PostgreSQL, BigQuery, Pinecone, Redis |
FILE:references/llm_integration_guide.md
# LLM Integration Guide
Production patterns for integrating Large Language Models into applications.
---
## Table of Contents
- [API Integration Patterns](#api-integration-patterns)
- [Prompt Engineering](#prompt-engineering)
- [Token Optimization](#token-optimization)
- [Cost Management](#cost-management)
- [Error Handling](#error-handling)
---
## API Integration Patterns
### Provider Abstraction Layer
```python
from abc import ABC, abstractmethod
from typing import List, Dict, Any
class LLMProvider(ABC):
"""Abstract base class for LLM providers."""
@abstractmethod
def complete(self, prompt: str, **kwargs) -> str:
pass
@abstractmethod
def chat(self, messages: List[Dict], **kwargs) -> str:
pass
class OpenAIProvider(LLMProvider):
def __init__(self, api_key: str, model: str = "gpt-4"):
self.client = OpenAI(api_key=api_key)
self.model = model
def complete(self, prompt: str, **kwargs) -> str:
response = self.client.completions.create(
model=self.model,
prompt=prompt,
**kwargs
)
return response.choices[0].text
class AnthropicProvider(LLMProvider):
def __init__(self, api_key: str, model: str = "claude-3-opus"):
self.client = Anthropic(api_key=api_key)
self.model = model
def chat(self, messages: List[Dict], **kwargs) -> str:
response = self.client.messages.create(
model=self.model,
messages=messages,
**kwargs
)
return response.content[0].text
```
### Retry and Fallback Strategy
```python
import time
from tenacity import retry, stop_after_attempt, wait_exponential
@retry(
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=1, max=10)
)
def call_llm_with_retry(provider: LLMProvider, prompt: str) -> str:
"""Call LLM with exponential backoff retry."""
return provider.complete(prompt)
def call_with_fallback(
primary: LLMProvider,
fallback: LLMProvider,
prompt: str
) -> str:
"""Try primary provider, fall back on failure."""
try:
return call_llm_with_retry(primary, prompt)
except Exception as e:
logger.warning(f"Primary provider failed: {e}, using fallback")
return call_llm_with_retry(fallback, prompt)
```
---
## Prompt Engineering
### Prompt Templates
| Pattern | Use Case | Structure |
|---------|----------|-----------|
| Zero-shot | Simple tasks | Task description + input |
| Few-shot | Complex tasks | Examples + task + input |
| Chain-of-thought | Reasoning | "Think step by step" + task |
| Role-based | Specialized output | System role + task |
### Few-Shot Template
```python
FEW_SHOT_TEMPLATE = """
You are a sentiment classifier. Classify the sentiment as positive, negative, or neutral.
Examples:
Input: "This product is amazing, I love it!"
Output: positive
Input: "Terrible experience, waste of money."
Output: negative
Input: "The product arrived on time."
Output: neutral
Now classify:
Input: "{user_input}"
Output:"""
def classify_sentiment(text: str, provider: LLMProvider) -> str:
prompt = FEW_SHOT_TEMPLATE.format(user_input=text)
response = provider.complete(prompt, max_tokens=10, temperature=0)
return response.strip().lower()
```
### System Prompts for Consistency
```python
SYSTEM_PROMPT = """You are a helpful assistant that answers questions about our product.
Guidelines:
- Be concise and direct
- Use bullet points for lists
- If unsure, say "I don't have that information"
- Never make up information
- Keep responses under 200 words
Product context:
{product_context}
"""
def create_chat_messages(user_query: str, context: str) -> List[Dict]:
return [
{"role": "system", "content": SYSTEM_PROMPT.format(product_context=context)},
{"role": "user", "content": user_query}
]
```
---
## Token Optimization
### Token Counting
```python
import tiktoken
def count_tokens(text: str, model: str = "gpt-4") -> int:
"""Count tokens for a given text and model."""
encoding = tiktoken.encoding_for_model(model)
return len(encoding.encode(text))
def truncate_to_token_limit(text: str, max_tokens: int, model: str = "gpt-4") -> str:
"""Truncate text to fit within token limit."""
encoding = tiktoken.encoding_for_model(model)
tokens = encoding.encode(text)
if len(tokens) <= max_tokens:
return text
return encoding.decode(tokens[:max_tokens])
```
### Context Window Management
| Model | Context Window | Effective Limit |
|-------|----------------|-----------------|
| GPT-4 | 8,192 | ~6,000 (leave room for response) |
| GPT-4-32k | 32,768 | ~28,000 |
| Claude 3 | 200,000 | ~180,000 |
| Llama 3 | 8,192 | ~6,000 |
### Chunking Strategy
```python
def chunk_text(text: str, chunk_size: int = 1000, overlap: int = 100) -> List[str]:
"""Split text into overlapping chunks."""
chunks = []
start = 0
while start < len(text):
end = start + chunk_size
chunk = text[start:end]
chunks.append(chunk)
start = end - overlap
return chunks
```
---
## Cost Management
### Cost Calculation
| Provider | Input Cost | Output Cost | Example (1K tokens) |
|----------|------------|-------------|---------------------|
| GPT-4 | $0.03/1K | $0.06/1K | $0.09 |
| GPT-3.5 | $0.0005/1K | $0.0015/1K | $0.002 |
| Claude 3 Opus | $0.015/1K | $0.075/1K | $0.09 |
| Claude 3 Haiku | $0.00025/1K | $0.00125/1K | $0.0015 |
### Cost Tracking
```python
from dataclasses import dataclass
from typing import Optional
@dataclass
class LLMUsage:
input_tokens: int
output_tokens: int
model: str
cost: float
def calculate_cost(
input_tokens: int,
output_tokens: int,
model: str
) -> float:
"""Calculate cost based on token usage."""
PRICING = {
"gpt-4": {"input": 0.03, "output": 0.06},
"gpt-3.5-turbo": {"input": 0.0005, "output": 0.0015},
"claude-3-opus": {"input": 0.015, "output": 0.075},
}
prices = PRICING.get(model, {"input": 0.01, "output": 0.03})
input_cost = (input_tokens / 1000) * prices["input"]
output_cost = (output_tokens / 1000) * prices["output"]
return input_cost + output_cost
```
### Cost Optimization Strategies
1. **Use smaller models for simple tasks** - GPT-3.5 for classification, GPT-4 for reasoning
2. **Cache common responses** - Store results for repeated queries
3. **Batch requests** - Combine multiple items in single prompt
4. **Truncate context** - Only include relevant information
5. **Set max_tokens limit** - Prevent runaway responses
---
## Error Handling
### Common Error Types
| Error | Cause | Handling |
|-------|-------|----------|
| RateLimitError | Too many requests | Exponential backoff |
| InvalidRequestError | Bad input | Validate before sending |
| AuthenticationError | Invalid API key | Check credentials |
| ServiceUnavailable | Provider down | Fallback to alternative |
| ContextLengthExceeded | Input too long | Truncate or chunk |
### Error Handling Pattern
```python
from openai import RateLimitError, APIError
def safe_llm_call(provider: LLMProvider, prompt: str, max_retries: int = 3) -> str:
"""Safely call LLM with comprehensive error handling."""
for attempt in range(max_retries):
try:
return provider.complete(prompt)
except RateLimitError:
wait_time = 2 ** attempt
logger.warning(f"Rate limited, waiting {wait_time}s")
time.sleep(wait_time)
except APIError as e:
if e.status_code >= 500:
logger.warning(f"Server error: {e}, retrying...")
time.sleep(1)
else:
raise
raise Exception(f"Failed after {max_retries} attempts")
```
### Response Validation
```python
import json
from pydantic import BaseModel, ValidationError
class StructuredResponse(BaseModel):
answer: str
confidence: float
sources: List[str]
def parse_structured_response(response: str) -> StructuredResponse:
"""Parse and validate LLM JSON response."""
try:
data = json.loads(response)
return StructuredResponse(**data)
except json.JSONDecodeError:
raise ValueError("Response is not valid JSON")
except ValidationError as e:
raise ValueError(f"Response validation failed: {e}")
```
FILE:references/mlops_production_patterns.md
# MLOps Production Patterns
Production ML infrastructure patterns for model deployment, monitoring, and lifecycle management.
---
## Table of Contents
- [Model Deployment Pipeline](#model-deployment-pipeline)
- [Feature Store Architecture](#feature-store-architecture)
- [Model Monitoring](#model-monitoring)
- [A/B Testing Infrastructure](#ab-testing-infrastructure)
- [Automated Retraining](#automated-retraining)
---
## Model Deployment Pipeline
### Deployment Workflow
1. Export trained model to standardized format (ONNX, TorchScript, SavedModel)
2. Package model with dependencies in Docker container
3. Deploy to staging environment
4. Run integration tests against staging
5. Deploy canary (5% traffic) to production
6. Monitor latency and error rates for 1 hour
7. Promote to full production if metrics pass
8. **Validation:** p95 latency < 100ms, error rate < 0.1%
### Container Structure
```dockerfile
FROM python:3.11-slim
# Install dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy model artifacts
COPY model/ /app/model/
COPY src/ /app/src/
# Health check endpoint
HEALTHCHECK CMD curl -f http://localhost:8080/health || exit 1
EXPOSE 8080
CMD ["uvicorn", "src.server:app", "--host", "0.0.0.0", "--port", "8080"]
```
### Model Serving Options
| Option | Latency | Throughput | Use Case |
|--------|---------|------------|----------|
| FastAPI + Uvicorn | Low | Medium | REST APIs, small models |
| Triton Inference Server | Very Low | Very High | GPU inference, batching |
| TensorFlow Serving | Low | High | TensorFlow models |
| TorchServe | Low | High | PyTorch models |
| Ray Serve | Medium | High | Complex pipelines, multi-model |
### Kubernetes Deployment
```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: model-serving
spec:
replicas: 3
selector:
matchLabels:
app: model-serving
template:
spec:
containers:
- name: model
image: model:v1.0.0
resources:
requests:
memory: "2Gi"
cpu: "1"
limits:
memory: "4Gi"
cpu: "2"
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
```
---
## Feature Store Architecture
### Feature Store Components
| Component | Purpose | Tools |
|-----------|---------|-------|
| Offline Store | Training data, batch features | BigQuery, Snowflake, S3 |
| Online Store | Low-latency serving | Redis, DynamoDB, Feast |
| Feature Registry | Metadata, lineage | Feast, Tecton, Hopsworks |
| Transformation | Feature engineering | Spark, Flink, dbt |
### Feature Pipeline Workflow
1. Define feature schema in registry
2. Implement transformation logic (SQL or Python)
3. Backfill historical features to offline store
4. Schedule incremental updates
5. Materialize to online store for serving
6. Monitor feature freshness and quality
7. **Validation:** Feature values within expected ranges, no nulls in required fields
### Feature Definition Example
```python
from feast import Entity, Feature, FeatureView, FileSource
user = Entity(name="user_id", value_type=ValueType.INT64)
user_features = FeatureView(
name="user_features",
entities=["user_id"],
ttl=timedelta(days=1),
features=[
Feature(name="purchase_count_30d", dtype=ValueType.INT64),
Feature(name="avg_order_value", dtype=ValueType.FLOAT),
Feature(name="days_since_last_purchase", dtype=ValueType.INT64),
],
online=True,
source=FileSource(path="data/user_features.parquet"),
)
```
---
## Model Monitoring
### Monitoring Dimensions
| Dimension | Metrics | Alert Threshold |
|-----------|---------|-----------------|
| Latency | p50, p95, p99 | p95 > 100ms |
| Throughput | requests/sec | < 80% baseline |
| Errors | error rate, 5xx count | > 0.1% |
| Data Drift | PSI, KS statistic | PSI > 0.2 |
| Model Drift | accuracy, AUC decay | > 5% drop |
### Data Drift Detection
```python
from scipy.stats import ks_2samp
import numpy as np
def detect_drift(reference: np.array, current: np.array, threshold: float = 0.05):
"""Detect distribution drift using Kolmogorov-Smirnov test."""
statistic, p_value = ks_2samp(reference, current)
drift_detected = p_value < threshold
return {
"drift_detected": drift_detected,
"ks_statistic": statistic,
"p_value": p_value,
"threshold": threshold
}
```
### Monitoring Dashboard Metrics
**Infrastructure:**
- Request latency (p50, p95, p99)
- Requests per second
- Error rate by type
- CPU/memory utilization
- GPU utilization (if applicable)
**Model Performance:**
- Prediction distribution
- Feature value distributions
- Model output confidence
- Ground truth vs predictions (when available)
---
## A/B Testing Infrastructure
### Experiment Workflow
1. Define experiment hypothesis and success metrics
2. Calculate required sample size for statistical power
3. Configure traffic split (control vs treatment)
4. Deploy treatment model alongside control
5. Route traffic based on user/session hash
6. Collect metrics for both variants
7. Run statistical significance test
8. **Validation:** p-value < 0.05, minimum sample size reached
### Traffic Splitting
```python
import hashlib
def get_variant(user_id: str, experiment: str, control_pct: float = 0.5) -> str:
"""Deterministic traffic splitting based on user ID."""
hash_input = f"{user_id}:{experiment}"
hash_value = int(hashlib.md5(hash_input.encode()).hexdigest(), 16)
bucket = (hash_value % 100) / 100.0
return "control" if bucket < control_pct else "treatment"
```
### Metrics Collection
| Metric Type | Examples | Collection Method |
|-------------|----------|-------------------|
| Primary | Conversion rate, revenue | Event logging |
| Secondary | Latency, engagement | Request logs |
| Guardrail | Error rate, crashes | Monitoring system |
---
## Automated Retraining
### Retraining Triggers
| Trigger | Detection Method | Action |
|---------|------------------|--------|
| Scheduled | Cron (weekly/monthly) | Full retrain |
| Performance drop | Accuracy < threshold | Immediate retrain |
| Data drift | PSI > 0.2 | Evaluate, then retrain |
| New data volume | X new samples | Incremental update |
### Retraining Pipeline
1. Trigger detection (schedule, drift, performance)
2. Fetch latest training data from feature store
3. Run training job with hyperparameter config
4. Evaluate model on holdout set
5. Compare against production model
6. If improved: register new model version
7. Deploy to staging for validation
8. Promote to production via canary
9. **Validation:** New model outperforms baseline on key metrics
### MLflow Model Registry Integration
```python
import mlflow
def register_model(model, metrics: dict, model_name: str):
"""Register trained model with MLflow."""
with mlflow.start_run():
# Log metrics
for name, value in metrics.items():
mlflow.log_metric(name, value)
# Log model
mlflow.sklearn.log_model(model, "model")
# Register in model registry
model_uri = f"runs:/{mlflow.active_run().info.run_id}/model"
mlflow.register_model(model_uri, model_name)
```
FILE:references/rag_system_architecture.md
# RAG System Architecture
Retrieval-Augmented Generation patterns for production applications.
---
## Table of Contents
- [RAG Pipeline Architecture](#rag-pipeline-architecture)
- [Vector Database Selection](#vector-database-selection)
- [Chunking Strategies](#chunking-strategies)
- [Embedding Models](#embedding-models)
- [Retrieval Optimization](#retrieval-optimization)
---
## RAG Pipeline Architecture
### Basic RAG Flow
1. Receive user query
2. Generate query embedding
3. Search vector database for relevant chunks
4. Rerank retrieved chunks by relevance
5. Format context with retrieved chunks
6. Send prompt to LLM with context
7. Return generated response
8. **Validation:** Response references retrieved context, no hallucinations
### Pipeline Components
```python
from dataclasses import dataclass
from typing import List
@dataclass
class Document:
content: str
metadata: dict
embedding: List[float] = None
@dataclass
class RetrievalResult:
document: Document
score: float
class RAGPipeline:
def __init__(
self,
embedder: Embedder,
vector_store: VectorStore,
llm: LLMProvider,
reranker: Reranker = None
):
self.embedder = embedder
self.vector_store = vector_store
self.llm = llm
self.reranker = reranker
def query(self, question: str, top_k: int = 5) -> str:
# 1. Embed query
query_embedding = self.embedder.embed(question)
# 2. Retrieve relevant documents
results = self.vector_store.search(query_embedding, top_k=top_k * 2)
# 3. Rerank if available
if self.reranker:
results = self.reranker.rerank(question, results)[:top_k]
else:
results = results[:top_k]
# 4. Build context
context = self._build_context(results)
# 5. Generate response
prompt = self._build_prompt(question, context)
return self.llm.complete(prompt)
def _build_context(self, results: List[RetrievalResult]) -> str:
return "\n\n".join([
f"[Source {i+1}]: {r.document.content}"
for i, r in enumerate(results)
])
def _build_prompt(self, question: str, context: str) -> str:
return f"""Answer the question based on the context provided.
Context:
{context}
Question: {question}
Answer:"""
```
---
## Vector Database Selection
### Comparison Matrix
| Database | Hosting | Scale | Latency | Cost | Best For |
|----------|---------|-------|---------|------|----------|
| Pinecone | Managed | High | Low | $$ | Production, managed |
| Weaviate | Both | High | Low | $ | Hybrid search |
| Qdrant | Both | High | Very Low | $ | Performance-critical |
| Chroma | Self-hosted | Medium | Low | Free | Prototyping |
| pgvector | Self-hosted | Medium | Medium | Free | Existing Postgres |
| Milvus | Both | Very High | Low | $ | Large-scale |
### Pinecone Integration
```python
import pinecone
class PineconeVectorStore:
def __init__(self, api_key: str, environment: str, index_name: str):
pinecone.init(api_key=api_key, environment=environment)
self.index = pinecone.Index(index_name)
def upsert(self, documents: List[Document], batch_size: int = 100):
"""Upsert documents in batches."""
vectors = [
(doc.metadata["id"], doc.embedding, doc.metadata)
for doc in documents
]
for i in range(0, len(vectors), batch_size):
batch = vectors[i:i + batch_size]
self.index.upsert(vectors=batch)
def search(self, embedding: List[float], top_k: int = 5) -> List[RetrievalResult]:
"""Search for similar vectors."""
results = self.index.query(
vector=embedding,
top_k=top_k,
include_metadata=True
)
return [
RetrievalResult(
document=Document(
content=match.metadata.get("content", ""),
metadata=match.metadata
),
score=match.score
)
for match in results.matches
]
```
---
## Chunking Strategies
### Strategy Comparison
| Strategy | Chunk Size | Overlap | Best For |
|----------|------------|---------|----------|
| Fixed | 500-1000 tokens | 50-100 | General text |
| Sentence | 3-5 sentences | 1 sentence | Structured text |
| Paragraph | Natural breaks | None | Documents with clear structure |
| Semantic | Variable | Based on meaning | Research papers |
| Recursive | Hierarchical | Parent-child | Long documents |
### Recursive Character Splitter
```python
from langchain.text_splitter import RecursiveCharacterTextSplitter
def create_chunks(
text: str,
chunk_size: int = 1000,
chunk_overlap: int = 100
) -> List[str]:
"""Split text using recursive character splitting."""
splitter = RecursiveCharacterTextSplitter(
chunk_size=chunk_size,
chunk_overlap=chunk_overlap,
separators=["\n\n", "\n", ". ", " ", ""]
)
return splitter.split_text(text)
```
### Semantic Chunking
```python
from sentence_transformers import SentenceTransformer
import numpy as np
def semantic_chunk(
sentences: List[str],
embedder: SentenceTransformer,
threshold: float = 0.7
) -> List[List[str]]:
"""Group sentences by semantic similarity."""
embeddings = embedder.encode(sentences)
chunks = []
current_chunk = [sentences[0]]
current_embedding = embeddings[0]
for i in range(1, len(sentences)):
similarity = np.dot(current_embedding, embeddings[i]) / (
np.linalg.norm(current_embedding) * np.linalg.norm(embeddings[i])
)
if similarity >= threshold:
current_chunk.append(sentences[i])
current_embedding = np.mean(
[current_embedding, embeddings[i]], axis=0
)
else:
chunks.append(current_chunk)
current_chunk = [sentences[i]]
current_embedding = embeddings[i]
chunks.append(current_chunk)
return chunks
```
---
## Embedding Models
### Model Comparison
| Model | Dimensions | Quality | Speed | Cost |
|-------|------------|---------|-------|------|
| text-embedding-3-large | 3072 | Excellent | Medium | $0.13/1M |
| text-embedding-3-small | 1536 | Good | Fast | $0.02/1M |
| BGE-large | 1024 | Excellent | Medium | Free |
| all-MiniLM-L6-v2 | 384 | Good | Very Fast | Free |
| Cohere embed-v3 | 1024 | Excellent | Medium | $0.10/1M |
### Embedding with Caching
```python
import hashlib
from functools import lru_cache
class CachedEmbedder:
def __init__(self, model_name: str = "text-embedding-3-small"):
self.client = OpenAI()
self.model = model_name
self._cache = {}
def embed(self, text: str) -> List[float]:
"""Embed text with caching."""
cache_key = hashlib.md5(text.encode()).hexdigest()
if cache_key in self._cache:
return self._cache[cache_key]
response = self.client.embeddings.create(
model=self.model,
input=text
)
embedding = response.data[0].embedding
self._cache[cache_key] = embedding
return embedding
def embed_batch(self, texts: List[str]) -> List[List[float]]:
"""Embed multiple texts efficiently."""
response = self.client.embeddings.create(
model=self.model,
input=texts
)
return [item.embedding for item in response.data]
```
---
## Retrieval Optimization
### Hybrid Search
Combine dense (vector) and sparse (keyword) retrieval:
```python
from rank_bm25 import BM25Okapi
class HybridRetriever:
def __init__(
self,
vector_store: VectorStore,
documents: List[Document],
alpha: float = 0.5
):
self.vector_store = vector_store
self.alpha = alpha # Weight for vector search
# Build BM25 index
tokenized = [doc.content.lower().split() for doc in documents]
self.bm25 = BM25Okapi(tokenized)
self.documents = documents
def search(self, query: str, query_embedding: List[float], top_k: int = 5):
# Vector search
vector_results = self.vector_store.search(query_embedding, top_k=top_k * 2)
# BM25 search
tokenized_query = query.lower().split()
bm25_scores = self.bm25.get_scores(tokenized_query)
# Combine scores
combined = {}
for result in vector_results:
doc_id = result.document.metadata["id"]
combined[doc_id] = self.alpha * result.score
for i, score in enumerate(bm25_scores):
doc_id = self.documents[i].metadata["id"]
if doc_id in combined:
combined[doc_id] += (1 - self.alpha) * score
else:
combined[doc_id] = (1 - self.alpha) * score
# Sort and return top_k
sorted_ids = sorted(combined.keys(), key=lambda x: combined[x], reverse=True)
return sorted_ids[:top_k]
```
### Reranking
```python
from sentence_transformers import CrossEncoder
class Reranker:
def __init__(self, model_name: str = "cross-encoder/ms-marco-MiniLM-L-12-v2"):
self.model = CrossEncoder(model_name)
def rerank(
self,
query: str,
results: List[RetrievalResult],
top_k: int = 5
) -> List[RetrievalResult]:
"""Rerank results using cross-encoder."""
pairs = [(query, r.document.content) for r in results]
scores = self.model.predict(pairs)
# Update scores and sort
for i, score in enumerate(scores):
results[i].score = float(score)
return sorted(results, key=lambda x: x.score, reverse=True)[:top_k]
```
### Query Expansion
```python
def expand_query(query: str, llm: LLMProvider) -> List[str]:
"""Generate query variations for better retrieval."""
prompt = f"""Generate 3 alternative phrasings of this question for search.
Return only the questions, one per line.
Original: {query}
Alternatives:"""
response = llm.complete(prompt, max_tokens=150)
alternatives = [q.strip() for q in response.strip().split("\n") if q.strip()]
return [query] + alternatives[:3]
```
FILE:scripts/ml_monitoring_suite.py
#!/usr/bin/env python3
"""
Ml Monitoring Suite
Production-grade tool for senior ml/ai engineer
"""
import os
import sys
import json
import logging
import argparse
from pathlib import Path
from typing import Dict, List, Optional
from datetime import datetime
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
class MlMonitoringSuite:
"""Production-grade ml monitoring suite"""
def __init__(self, config: Dict):
self.config = config
self.results = {
'status': 'initialized',
'start_time': datetime.now().isoformat(),
'processed_items': 0
}
logger.info(f"Initialized {self.__class__.__name__}")
def validate_config(self) -> bool:
"""Validate configuration"""
logger.info("Validating configuration...")
# Add validation logic
logger.info("Configuration validated")
return True
def process(self) -> Dict:
"""Main processing logic"""
logger.info("Starting processing...")
try:
self.validate_config()
# Main processing
result = self._execute()
self.results['status'] = 'completed'
self.results['end_time'] = datetime.now().isoformat()
logger.info("Processing completed successfully")
return self.results
except Exception as e:
self.results['status'] = 'failed'
self.results['error'] = str(e)
logger.error(f"Processing failed: {e}")
raise
def _execute(self) -> Dict:
"""Execute main logic"""
# Implementation here
return {'success': True}
def main():
"""Main entry point"""
parser = argparse.ArgumentParser(
description="Ml Monitoring Suite"
)
parser.add_argument('--input', '-i', required=True, help='Input path')
parser.add_argument('--output', '-o', required=True, help='Output path')
parser.add_argument('--config', '-c', help='Configuration file')
parser.add_argument('--verbose', '-v', action='store_true', help='Verbose output')
args = parser.parse_args()
if args.verbose:
logging.getLogger().setLevel(logging.DEBUG)
try:
config = {
'input': args.input,
'output': args.output
}
processor = MlMonitoringSuite(config)
results = processor.process()
print(json.dumps(results, indent=2))
sys.exit(0)
except Exception as e:
logger.error(f"Fatal error: {e}")
sys.exit(1)
if __name__ == '__main__':
main()
FILE:scripts/model_deployment_pipeline.py
#!/usr/bin/env python3
"""
Model Deployment Pipeline
Production-grade tool for senior ml/ai engineer
"""
import os
import sys
import json
import logging
import argparse
from pathlib import Path
from typing import Dict, List, Optional
from datetime import datetime
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
class ModelDeploymentPipeline:
"""Production-grade model deployment pipeline"""
def __init__(self, config: Dict):
self.config = config
self.results = {
'status': 'initialized',
'start_time': datetime.now().isoformat(),
'processed_items': 0
}
logger.info(f"Initialized {self.__class__.__name__}")
def validate_config(self) -> bool:
"""Validate configuration"""
logger.info("Validating configuration...")
# Add validation logic
logger.info("Configuration validated")
return True
def process(self) -> Dict:
"""Main processing logic"""
logger.info("Starting processing...")
try:
self.validate_config()
# Main processing
result = self._execute()
self.results['status'] = 'completed'
self.results['end_time'] = datetime.now().isoformat()
logger.info("Processing completed successfully")
return self.results
except Exception as e:
self.results['status'] = 'failed'
self.results['error'] = str(e)
logger.error(f"Processing failed: {e}")
raise
def _execute(self) -> Dict:
"""Execute main logic"""
# Implementation here
return {'success': True}
def main():
"""Main entry point"""
parser = argparse.ArgumentParser(
description="Model Deployment Pipeline"
)
parser.add_argument('--input', '-i', required=True, help='Input path')
parser.add_argument('--output', '-o', required=True, help='Output path')
parser.add_argument('--config', '-c', help='Configuration file')
parser.add_argument('--verbose', '-v', action='store_true', help='Verbose output')
args = parser.parse_args()
if args.verbose:
logging.getLogger().setLevel(logging.DEBUG)
try:
config = {
'input': args.input,
'output': args.output
}
processor = ModelDeploymentPipeline(config)
results = processor.process()
print(json.dumps(results, indent=2))
sys.exit(0)
except Exception as e:
logger.error(f"Fatal error: {e}")
sys.exit(1)
if __name__ == '__main__':
main()
FILE:scripts/rag_system_builder.py
#!/usr/bin/env python3
"""
Rag System Builder
Production-grade tool for senior ml/ai engineer
"""
import os
import sys
import json
import logging
import argparse
from pathlib import Path
from typing import Dict, List, Optional
from datetime import datetime
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
class RagSystemBuilder:
"""Production-grade rag system builder"""
def __init__(self, config: Dict):
self.config = config
self.results = {
'status': 'initialized',
'start_time': datetime.now().isoformat(),
'processed_items': 0
}
logger.info(f"Initialized {self.__class__.__name__}")
def validate_config(self) -> bool:
"""Validate configuration"""
logger.info("Validating configuration...")
# Add validation logic
logger.info("Configuration validated")
return True
def process(self) -> Dict:
"""Main processing logic"""
logger.info("Starting processing...")
try:
self.validate_config()
# Main processing
result = self._execute()
self.results['status'] = 'completed'
self.results['end_time'] = datetime.now().isoformat()
logger.info("Processing completed successfully")
return self.results
except Exception as e:
self.results['status'] = 'failed'
self.results['error'] = str(e)
logger.error(f"Processing failed: {e}")
raise
def _execute(self) -> Dict:
"""Execute main logic"""
# Implementation here
return {'success': True}
def main():
"""Main entry point"""
parser = argparse.ArgumentParser(
description="Rag System Builder"
)
parser.add_argument('--input', '-i', required=True, help='Input path')
parser.add_argument('--output', '-o', required=True, help='Output path')
parser.add_argument('--config', '-c', help='Configuration file')
parser.add_argument('--verbose', '-v', action='store_true', help='Verbose output')
args = parser.parse_args()
if args.verbose:
logging.getLogger().setLevel(logging.DEBUG)
try:
config = {
'input': args.input,
'output': args.output
}
processor = RagSystemBuilder(config)
results = processor.process()
print(json.dumps(results, indent=2))
sys.exit(0)
except Exception as e:
logger.error(f"Fatal error: {e}")
sys.exit(1)
if __name__ == '__main__':
main()
Product marketing: định vị, chiến lược GTM, phân tích đối thủ, ra mắt sản phẩm, xác định ICP, nghiên cứu thị trường và hỗ trợ bán hàng.
---
name: "marketing-strategy-pmm"
description: Product marketing skill for positioning, GTM strategy, competitive intelligence, and product launches. Use when the user asks about product positioning, go-to-market planning, competitive analysis, target audience definition, ICP definition, market research, launch plans, or sales enablement. Covers April Dunford positioning, ICP definition, competitive battlecards, launch playbooks, and international market entry. Produces deliverables including positioning statements, battlecard documents, launch plans, and go-to-market strategies.
triggers:
- product marketing
- PMM
- positioning
- GTM strategy
- go-to-market
- competitive analysis
- battlecard
- product launch
- market entry
- sales enablement
- win loss analysis
---
# Marketing Strategy & PMM
Product marketing patterns for positioning, GTM strategy, and competitive intelligence.
---
## Table of Contents
- [ICP Definition Workflow](#icp-definition-workflow)
- [Positioning Development](#positioning-development)
- [Competitive Intelligence](#competitive-intelligence)
- [Product Launch Planning](#product-launch-planning)
- [Sales Enablement](#sales-enablement)
- [International Expansion](#international-expansion)
- [Reference Documentation](#reference-documentation)
---
## ICP Definition Workflow
Define ideal customer profile for targeting:
1. Analyze existing customers (top 20% by LTV)
2. Identify common firmographics (size, industry, revenue)
3. Map technographics (tools, maturity, integrations)
4. Document psychographics (pain level, motivation, risk tolerance)
5. Define 3-5 buyer personas (economic, technical, user)
6. Validate against sales cycle and churn data
7. Score prospects A/B/C/D based on ICP fit
8. **Validation:** A-fit customers have lowest churn and fastest close
### Firmographics Template
| Dimension | Target Range | Rationale |
|-----------|--------------|-----------|
| Employees | 50-5000 | Series A sweet spot |
| Revenue | $5M-$500M | Budget available |
| Industry | SaaS, Tech, Services | Product fit |
| Geography | US, UK, DACH | Market priority |
| Funding | Seed to Growth | Willing to adopt |
### Buyer Personas
| Persona | Title | Goals | Messaging |
|---------|-------|-------|-----------|
| Economic Buyer | VP, Director, Head of [Department] | ROI, team productivity, cost reduction | Business outcomes, ROI, case studies |
| Technical Buyer | Engineer, Architect, Tech Lead | Technical fit, easy integration | Architecture, security, documentation |
| User/Champion | Manager, Team Lead, Power User | Makes job easier, quick wins | UX, ease of use, time savings |
### ICP Validation Checklist
- [ ] 5+ paying customers match this profile
- [ ] Fastest sales cycles (< median)
- [ ] Highest LTV (> median)
- [ ] Lowest churn (< 5% annual)
- [ ] Strong product engagement
- [ ] Willing to do case studies
---
## Positioning Development
Develop positioning using April Dunford methodology:
1. List competitive alternatives (direct, adjacent, status quo)
2. Isolate unique attributes (features only you have)
3. Map attributes to customer value (why it matters)
4. Define best-fit customers (who cares most)
5. Choose market category (head-to-head, niche, new category)
6. Layer on relevant trends (timing justification)
7. Test with 10+ customer interviews
8. **Validation:** 7+ customers describe value unprompted
### Positioning Statement Template
```
FOR [target customer]
WHO [statement of need]
THE [product] IS A [category]
THAT [key benefit]
UNLIKE [competitive alternative]
OUR PRODUCT [primary differentiation]
```
### Value Proposition Formula
Template: `[Product] helps [Target Customer] [Achieve Goal] by [Unique Approach]`
Example: "Acme helps mid-market SaaS teams ship 2x faster by automating project workflows with AI"
### Messaging Hierarchy
| Level | Content | Example |
|-------|---------|---------|
| Headline | 5-7 words | "Ship faster with AI automation" |
| Subhead | 1 sentence | "Automate workflows so teams focus on what matters" |
| Benefits | 3-4 bullets | Speed, quality, collaboration, cost |
| Features | Supporting evidence | AI automation → 10 hrs/week saved |
| Proof | Social proof | Customer logos, stats, case studies |
---
## Competitive Intelligence
Build competitive knowledge base:
1. Identify tier 1 (direct), tier 2 (adjacent), tier 3 (status quo)
2. Sign up for competitor products (hands-on evaluation)
3. Monitor competitor websites, pricing, messaging
4. Analyze sales call recordings for competitor mentions
5. Read G2/Capterra reviews (pros and cons)
6. Track competitor job postings (roadmap signals)
7. Update battlecards monthly
8. **Validation:** Sales team uses battlecards in 80%+ competitive deals
### Competitive Tier Structure
| Tier | Definition | Examples |
|------|------------|----------|
| 1 | Direct competitor, same category | [Competitor A, B] |
| 2 | Adjacent solution, overlapping use case | [Alt Solution C, D] |
| 3 | Status quo (what they do today) | Spreadsheets, manual, in-house |
### Battlecard Template
```
COMPETITOR: [Name]
OVERVIEW: Founded [year], Funding [stage], Size [employees]
POSITIONING:
- They say: "[Their claim]"
- Reality: [Your assessment]
STRENGTHS:
1. [What they do well]
2. [What they do well]
WEAKNESSES:
1. [Where they fall short]
2. [Where they fall short]
OUR ADVANTAGES:
1. [Your advantage + evidence]
2. [Your advantage + evidence]
WHEN WE WIN:
- [Scenario where you win]
WHEN WE LOSE:
- [Scenario where they win]
TALK TRACK:
Objection: "[Common objection]"
Response: "[Your response]"
```
### Win/Loss Analysis
Track monthly:
- Win rate by competitor
- Top win reasons (product fit, ease of use, price)
- Top loss reasons (missing feature, price, relationship)
- Action items for product, sales, marketing
---
## Product Launch Planning
Plan launches by tier:
| Tier | Scope | Prep Time | Budget |
|------|-------|-----------|--------|
| 1 | New product, major feature | 6-8 weeks | $50-100k |
| 2 | Significant feature, integration | 3-4 weeks | $10-25k |
| 3 | Small improvement | 1 week | <$5k |
### Tier 1 Launch Workflow
Execute major product launch:
1. Kickoff meeting with Product, Marketing, Sales, CS
2. Define goals (pipeline $, MQLs, press coverage)
3. Develop positioning and messaging
4. Create sales enablement (deck, demo, battlecard)
5. Build campaign assets (landing page, emails, ads)
6. Train sales and CS teams
7. Execute launch day (press, email, ads, outbound)
8. Monitor and optimize for 30 days
9. **Validation:** Pipeline on track to goal by week 2
### Launch Day Checklist
- [ ] Press release distributed
- [ ] Email announcement sent
- [ ] Social media posts live
- [ ] Paid ads at full budget
- [ ] Sales outbound blitz launched
- [ ] In-app notification active
- [ ] Metrics monitored every 2 hours
### Launch Metrics
| Metric | Leading (Daily) | Lagging (Weekly) |
|--------|-----------------|------------------|
| Traffic | Landing page visitors | - |
| Engagement | Demo requests, signups | Feature adoption % |
| Pipeline | MQLs generated | SQLs, pipeline $ |
| Revenue | - | Deals closed, revenue |
---
## Sales Enablement
Equip sales team with PMM assets:
1. Create sales deck (15-20 slides, visual-first)
2. Build one-pagers (product, competitive, case study)
3. Develop demo script (30-45 min with discovery)
4. Write email templates (outreach, follow-up, closing)
5. Create ROI calculator (input costs, output savings)
6. Conduct monthly enablement calls
7. Deliver quarterly training (positioning, competitive)
8. **Validation:** Sales uses assets in 80%+ of opportunities
### Sales Deck Structure
| Slide | Content |
|-------|---------|
| 1-2 | Title, agenda |
| 3-4 | Company intro, problem statement |
| 5-7 | Solution, key benefits, demo |
| 8-10 | Differentiation, case study, pricing |
| 11-12 | Implementation, support, next steps |
### Demo Flow
```
1. Intro (2 min): Who we are, agenda
2. Discovery (5 min): Their needs, pain points
3. Demo (20 min): Product focused on their use case
4. Q&A (10 min): Objection handling
5. Next steps (3 min): Trial, POC, proposal
```
### Sales-Marketing Handoff
| Handoff | Frequency | Content |
|---------|-----------|---------|
| Weekly sync | 30 min | Win/loss, competitive, new assets |
| Monthly enablement | 60 min | Product updates, training |
| Quarterly review | Half-day | Results, strategy, planning |
---
## International Expansion
Enter new markets systematically:
1. Validate market demand (inbound leads, TAM analysis)
2. Localize website, pricing, legal
3. Establish sales coverage (hire or agency)
4. Adapt messaging for cultural fit
5. Build local partnerships and references
6. Launch localized campaigns
7. Monitor CAC and conversion by market
8. **Validation:** 3+ paying customers from market in first 90 days
### Market Priority (Series A)
| Market | Timeline | Budget % | Target ARR |
|--------|----------|----------|------------|
| US | Months 1-6 | 50% | $1M |
| UK | Months 4-9 | 20% | $500k |
| DACH | Months 7-12 | 15% | $300k |
| France | Months 10-15 | 10% | $200k |
| Canada | Months 7-12 | 5% | $100k |
### Localization Checklist
- [ ] Website translation (professional, not machine)
- [ ] Currency and pricing localized
- [ ] Local phone number and address
- [ ] Legal compliance (GDPR, PIPEDA)
- [ ] Local payment methods
- [ ] Sales coverage during local hours
- [ ] Local case studies and references
---
## Reference Documentation
### Positioning Frameworks
`references/positioning-frameworks.md` contains:
- April Dunford 5-step positioning process
- Geoffrey Moore positioning statement template
- Positioning validation interview protocol
- Competitive positioning map construction
### Launch Checklists
`references/launch-checklists.md` contains:
- Tier 1/2/3 launch checklists
- Week-by-week launch timeline
- Launch day runbook
- Post-launch metrics dashboard
### International GTM
`references/international-gtm.md` contains:
- US, UK, DACH, France, Canada playbooks
- Market-specific channel mix and messaging
- Localization requirements per market
- Entry timeline and budget allocation
### Messaging Templates
`references/messaging-templates.md` contains:
- Value proposition formulas
- Persona-specific messaging
- Competitive response scripts
- Objection handling templates
- Channel-specific copy (landing pages, emails, ads)
---
## PMM KPIs
| Metric | Target | Measurement |
|--------|--------|-------------|
| Product adoption | >40% in 90 days | Feature usage after launch |
| Win rate | >30% competitive | Deals won vs. competitors |
| Sales velocity | -20% YoY | Days from SQL to close |
| Deal size | +25% YoY | Average contract value |
| Launch pipeline | 3:1 ROMI | Pipeline $ : marketing spend |
---
## Quick Reference
### PMM Monthly Rhythm
| Week | Focus |
|------|-------|
| 1 | Review metrics, update battlecards |
| 2 | Create assets, publish content |
| 3 | Support launches, optimize campaigns |
| 4 | Monthly report, plan next month |
## Proactive Triggers
- **No documented positioning** → Without clear positioning, all marketing is guesswork.
- **Messaging differs across channels** → Inconsistent story confuses buyers.
- **No ICP defined** → Selling to everyone means selling to no one.
- **Competitor repositioning** → Market shift detected. Review your positioning.
## Output Artifacts
| When you ask for... | You get... |
|---------------------|------------|
| "Position my product" | Positioning framework (April Dunford method) with output |
| "GTM strategy" | Go-to-market plan with channels, messaging, and timeline |
| "Competitive positioning" | Positioning map with competitive gaps and opportunities |
## Communication
All output passes quality verification:
- Self-verify: source attribution, assumption audit, confidence scoring
- Output format: Bottom Line → What (with confidence) → Why → How to Act
- Results only. Every finding tagged: 🟢 verified, 🟡 medium, 🔴 assumed.
## Related Skills
- **marketing-context**: For capturing foundational positioning. PMM builds on this.
- **launch-strategy**: For executing product launches planned by PMM.
- **competitive-intel** (C-Suite): For strategic competitive intelligence.
- **cmo-advisor** (C-Suite): For marketing budget and growth model decisions.
FILE:references/international-gtm.md
# International GTM Playbooks
Market-by-market expansion guides for US, UK, DACH, France, and Canada.
---
## Table of Contents
- [Market Prioritization](#market-prioritization)
- [US Market Entry](#us-market-entry)
- [UK Market Entry](#uk-market-entry)
- [DACH Market Entry](#dach-market-entry)
- [France Market Entry](#france-market-entry)
- [Canada Market Entry](#canada-market-entry)
- [Localization Checklist](#localization-checklist)
---
## Market Prioritization
### Expansion Sequence (Series A)
| Phase | Market | Timeline | Budget % | Target ARR |
|-------|--------|----------|----------|------------|
| 1 | US | Months 1-6 | 50% | $1M |
| 2 | UK | Months 4-9 | 20% | $500k |
| 3 | DACH | Months 7-12 | 15% | $300k |
| 4 | France | Months 10-15 | 10% | $200k |
| 5 | Canada | Months 7-12 | 5% | $100k |
### Market Readiness Checklist
Enter market when ALL true:
- [ ] Product ready for market (localization if needed)
- [ ] Legal/compliance requirements met
- [ ] Pricing localized (currency, taxes)
- [ ] Sales capacity available (hire or agency)
- [ ] Marketing budget allocated
- [ ] Support coverage during local hours
- [ ] **Validation:** 3+ inbound leads from market in last 90 days
---
## US Market Entry
### Market Characteristics
| Factor | US Approach |
|--------|-------------|
| Buying cycle | Fast (30-60 days average) |
| Decision process | Individual empowerment, less consensus |
| Pricing sensitivity | Value-focused, willing to pay premium |
| Communication | Direct, results-oriented |
| Relationship | Transaction > relationship (initially) |
### Entry Strategy
**Months 1-2: Foundation**
1. Establish US presence:
- US phone number (toll-free)
- US address (virtual office acceptable)
- USD pricing on website
- US case studies (even if from beta users)
2. Hire US sales:
- Option A: US-based SDR/AE (expensive but effective)
- Option B: US sales agency (lower risk, shared commission)
- Option C: Remote sales trained on US hours
3. Launch paid campaigns:
- Google Ads (high-intent keywords)
- LinkedIn (B2B targeting)
- Budget: 50% of marketing spend
**Months 3-6: Scale**
1. Optimize channels based on CAC data
2. Build US partner ecosystem:
- Integration partners (Salesforce, HubSpot)
- Resellers/VARs (for Enterprise)
- Industry associations
3. Attend US conferences (SaaStr, industry events)
4. **Validation:** $1M pipeline from US sources
### US Channel Mix
| Channel | Budget % | Expected CPL | Notes |
|---------|----------|--------------|-------|
| Google Ads | 35% | $100-200 | High intent, competitive |
| LinkedIn | 30% | $150-250 | B2B targeting |
| SEO/Content | 20% | $50 (long-term) | Invest early |
| Partnerships | 15% | Variable | Co-marketing |
### US Messaging
- Lead with ROI and business outcomes
- Use $ impact metrics prominently
- Reference US customers (logos matter)
- Emphasize speed and efficiency
- Include G2/Capterra ratings
---
## UK Market Entry
### Market Characteristics
| Factor | UK Approach |
|--------|-------------|
| Buying cycle | Medium (45-90 days) |
| Decision process | Committee involvement |
| Pricing sensitivity | Value-conscious, compare options |
| Communication | Professional, less aggressive than US |
| Relationship | Balance transaction and relationship |
### Entry Strategy
**Months 4-6: Setup**
1. Localization:
- GBP pricing
- UK spellings (colour, organisation)
- UK phone number
- GDPR compliance (essential)
2. Sales coverage:
- Hire UK-based rep OR
- Partner with UK sales agency
- Ensure coverage during GMT hours
3. Content localization:
- UK case studies
- UK-relevant industry references
- Local competitor positioning
**Months 7-9: Growth**
1. Build UK partnerships:
- UK tech community (TechNation, etc.)
- London-based VCs and accelerators
- UK industry associations
2. Attend UK events:
- London Tech Week
- Industry-specific conferences
3. **Validation:** $500k pipeline from UK sources
### UK Channel Mix
| Channel | Budget % | Expected CPL | Notes |
|---------|----------|--------------|-------|
| LinkedIn | 35% | $120-200 | Strong B2B presence |
| Google UK | 30% | $80-150 | Less competitive than US |
| SEO/Content | 20% | $40 | UK-targeted keywords |
| Partnerships | 15% | Variable | Local credibility |
### UK Messaging
- More formal than US (avoid hyperbole)
- Emphasize data security and GDPR
- Reference UK/EU customers
- Understated claims (prove with data)
- Acknowledge local presence/support
---
## DACH Market Entry
### Market Characteristics
| Factor | DACH Approach |
|--------|---------------|
| Buying cycle | Long (90-180 days) |
| Decision process | Consensus-driven, thorough evaluation |
| Pricing sensitivity | Quality over price, long-term view |
| Communication | Formal, detailed, precise |
| Relationship | Trust built over time, essential |
### Entry Strategy
**Months 7-9: Foundation**
1. Full localization:
- German translation (website, product UI)
- EUR pricing with German VAT handling
- German phone number and address
- GDPR compliance (strict enforcement)
- Data residency option (EU data centers)
2. German-speaking sales:
- Hire German-speaking sales rep
- Native speaker critical (not just fluent)
- Based in Germany preferred
3. Content in German:
- Translate key pages and materials
- Create German case studies
- German blog content
**Months 10-12: Growth**
1. Build local credibility:
- German customer testimonials
- German partner ecosystem
- Industry certifications (ISO, etc.)
2. Attend German events:
- CeBIT/Hannover Messe
- Industry conferences
3. **Validation:** $300k pipeline from DACH sources
### DACH Channel Mix
| Channel | Budget % | Expected CPL | Notes |
|---------|----------|--------------|-------|
| LinkedIn | 40% | $150-250 | Strong professional network |
| Google DE | 25% | $100-180 | German keywords |
| SEO (German) | 20% | $60 | Long-term investment |
| Partnerships | 15% | Variable | Critical for trust |
### DACH Messaging
- Formal tone (Sie, not du)
- Emphasize security, compliance, privacy
- Detailed specifications and documentation
- Reference German/EU customers
- Include certifications (ISO, SOC 2)
- Show long-term commitment to market
---
## France Market Entry
### Market Characteristics
| Factor | France Approach |
|--------|-----------------|
| Buying cycle | Long (90-180 days) |
| Decision process | Hierarchical, formal process |
| Pricing sensitivity | Value-focused, negotiation expected |
| Communication | Formal, relationship-focused |
| Relationship | Critical, business built on trust |
### Entry Strategy
**Months 10-12: Foundation**
1. Full French localization:
- French translation (professional, not machine)
- EUR pricing with French VAT
- French phone number
- GDPR + French regulations
2. French-speaking team:
- Native French speaker for sales
- French support coverage
- Paris presence (even virtual)
**Months 13-15: Growth**
1. Build local ecosystem:
- French tech community (La French Tech)
- French partners and integrators
- Industry associations
2. Attend French events:
- VivaTech (Paris)
- Industry conferences
3. **Validation:** $200k pipeline from France
### France Channel Mix
| Channel | Budget % | Expected CPL | Notes |
|---------|----------|--------------|-------|
| LinkedIn | 35% | $130-220 | Professional network |
| Google FR | 30% | $90-160 | French keywords |
| SEO (French) | 20% | $50 | French content strategy |
| Partnerships | 15% | Variable | Local partners essential |
### France Messaging
- Formal and professional
- French language throughout (no English fallback)
- Reference French/EU customers
- Emphasize local support and presence
- Highlight innovation and modernity
- Respect cultural nuances
---
## Canada Market Entry
### Market Characteristics
| Factor | Canada Approach |
|--------|-----------------|
| Buying cycle | Medium (45-75 days) |
| Decision process | Similar to US, slightly more conservative |
| Pricing sensitivity | Value-conscious, compare to US prices |
| Communication | Professional, friendly, less aggressive |
| Language | English (except Quebec - French required) |
### Entry Strategy
**Months 7-9: Foundation**
1. Minimal localization:
- CAD pricing
- Canadian phone number (optional)
- PIPEDA compliance
2. Sales coverage:
- Leverage US sales team (similar hours)
- Consider Toronto-based rep for growth
3. Quebec consideration:
- French required for Quebec market
- Can delay or skip initially
**Months 10-12: Growth**
1. Canadian partnerships:
- Canadian tech community
- Toronto/Vancouver startup ecosystem
- Industry associations
2. **Validation:** $100k pipeline from Canada
### Canada Channel Mix
| Channel | Budget % | Expected CPL | Notes |
|---------|----------|--------------|-------|
| Google CA | 35% | $80-150 | Canadian targeting |
| LinkedIn | 30% | $100-180 | B2B focus |
| SEO | 20% | $40 | Canadian content |
| Partnerships | 15% | Variable | Local credibility |
---
## Localization Checklist
### Per-Market Checklist
**Website**
- [ ] Language translation (professional, not machine)
- [ ] Currency localization (display + checkout)
- [ ] Phone number (local format)
- [ ] Address (local presence)
- [ ] Legal pages (privacy, terms in local language)
- [ ] hreflang tags configured correctly
**Product**
- [ ] UI translation (if required for market)
- [ ] Date/time format (DD/MM/YYYY vs MM/DD/YYYY)
- [ ] Number format (1,000 vs 1.000)
- [ ] Currency in product
**Payment**
- [ ] Local currency accepted
- [ ] VAT/tax handling
- [ ] Local payment methods (SEPA, iDEAL, etc.)
- [ ] Invoicing in local format
**Legal**
- [ ] GDPR compliance (EU markets)
- [ ] PIPEDA compliance (Canada)
- [ ] Local data protection laws
- [ ] Terms of service localized
- [ ] Privacy policy localized
**Sales**
- [ ] Local sales coverage (rep or agency)
- [ ] Localized sales materials
- [ ] Local pricing and quoting
- [ ] Local references and case studies
**Support**
- [ ] Coverage during local business hours
- [ ] Language support (phone, chat, email)
- [ ] Localized documentation
- [ ] Local SLA commitments
**Marketing**
- [ ] Localized campaigns
- [ ] Local content (blog, guides)
- [ ] Local social media presence
- [ ] Local event participation
**Validation:** Native speaker review of ALL localized content before launch
FILE:references/launch-checklists.md
# Launch Checklists
GTM launch playbooks for Tier 1, 2, and 3 product releases.
---
## Table of Contents
- [Launch Tier Definitions](#launch-tier-definitions)
- [Tier 1 Major Launch](#tier-1-major-launch)
- [Tier 2 Standard Launch](#tier-2-standard-launch)
- [Tier 3 Minor Launch](#tier-3-minor-launch)
- [Launch Metrics Dashboard](#launch-metrics-dashboard)
---
## Launch Tier Definitions
| Tier | Scope | Prep Time | Budget | Audience |
|------|-------|-----------|--------|----------|
| 1 | New product, major feature | 6-8 weeks | $50-100k | All prospects + press |
| 2 | Significant feature, integration | 3-4 weeks | $10-25k | Customers + select prospects |
| 3 | Small feature, improvement | 1 week | <$5k | Existing customers |
**Tier Selection Criteria:**
```
Tier 1 if ANY true:
- [ ] Net-new product line
- [ ] Revenue impact > $500k pipeline
- [ ] Press coverage expected
- [ ] Competitive response anticipated
Tier 2 if ANY true:
- [ ] Major feature request (top 10 customer ask)
- [ ] New integration with strategic partner
- [ ] Pricing or packaging change
Tier 3 otherwise:
- [ ] Bug fixes
- [ ] UI improvements
- [ ] Minor enhancements
```
---
## Tier 1 Major Launch
### Phase 1: Foundation (Weeks -8 to -5)
**Week -8: Kickoff**
- [ ] Schedule kickoff meeting (Product, Marketing, Sales, CS)
- [ ] Define launch goals:
- Pipeline target: $______
- MQL target: ______
- Press hits target: ______
- Adoption target: ______% in 90 days
- [ ] Assign roles (RACI matrix):
- PMM: Launch lead, positioning, messaging
- Product: Feature readiness, demo environment
- Demand Gen: Campaigns, paid ads, email
- Content: Blog posts, case studies, videos
- Sales: Enablement, outbound campaign
- [ ] Create project timeline in Asana/Monday/Notion
- [ ] **Validation:** All stakeholders confirm goals and timeline
**Week -7: Strategy**
- [ ] Develop positioning and messaging (see positioning-frameworks.md)
- [ ] Create GTM channel plan:
- Owned: Email, blog, social, webinar
- Paid: LinkedIn ads, Google ads
- Earned: Press, influencers, partners
- [ ] Define target segments (ICP, personas)
- [ ] Allocate budget by channel
- [ ] Draft press release (embargo date set)
**Week -6: Content**
- [ ] Build landing page (product page, demo request form)
- [ ] Write blog post announcement
- [ ] Create sales deck updates (5-10 new slides)
- [ ] Design social media graphics (5+ variants)
- [ ] Produce demo video (3-5 minutes)
- [ ] Draft email sequences (announcement, nurture)
**Week -5: Enablement**
- [ ] Create sales battlecard (competitive positioning)
- [ ] Write demo script (new feature walkthrough)
- [ ] Build FAQ document (top 20 questions)
- [ ] Develop objection handling guide
- [ ] Schedule sales training session
- [ ] Recruit beta customers for testimonials
- [ ] **Validation:** Sales team can demo feature confidently
### Phase 2: Preparation (Weeks -4 to -1)
**Week -4: Launch Prep**
- [ ] Set up HubSpot campaign (UTMs, attribution)
- [ ] Launch teaser campaign (social, email hints)
- [ ] Pitch press and analysts (NDA briefings)
- [ ] Create webinar registration page
- [ ] Finalize partner co-marketing plans
- [ ] QA all landing pages and forms
**Week -3: Ramp Up**
- [ ] Activate paid ads (LinkedIn, Google) at 50% budget
- [ ] A/B test landing page headlines
- [ ] Send pre-launch email to VIP customers
- [ ] Conduct sales training (2-hour session)
- [ ] Confirm webinar speakers and content
- [ ] Prepare launch day runbook
**Week -2: Final Prep**
- [ ] Increase paid ad spend to 75%
- [ ] Send webinar reminder emails
- [ ] Finalize press embargo lift time
- [ ] Complete dry run (website, forms, CRM workflow)
- [ ] Create launch day social posts (scheduled)
- [ ] Brief customer success team
**Week -1: Pre-Launch**
- [ ] Final approval on all assets
- [ ] Send VIP preview to top 10 customers
- [ ] Confirm press embargo release
- [ ] Sales team ready (trained, quotas set)
- [ ] CS team ready (docs updated, chat staffed)
- [ ] Test all systems one final time
- [ ] **Validation:** All checklist items green
### Phase 3: Launch (Weeks 1-4)
**Launch Day**
- [ ] Press release distribution (wire + direct pitch)
- [ ] Email blast to full database
- [ ] Social media posts (LinkedIn, Twitter, Facebook)
- [ ] Paid ads at 100% budget
- [ ] Sales outbound blitz (top 100 accounts)
- [ ] In-app announcement to existing users
- [ ] Monitor metrics every 2 hours:
- Traffic, signups, demo requests
- Press pickup, social engagement
- Sales pipeline created
**Days 2-7**
- [ ] Daily metrics review (conversion rates, funnel)
- [ ] A/B test optimizations based on data
- [ ] Sales follow-up (<4 hour SLA on leads)
- [ ] Respond to press and analyst inquiries
- [ ] Host webinar (Day 3 or 4)
- [ ] Post customer testimonials
- [ ] Adjust paid ads (pause underperformers)
**Week 2-4**
- [ ] Publish post-launch blog content
- [ ] Create customer case study from early adopters
- [ ] Conduct win/loss interviews (5+ deals)
- [ ] Optimize converting channels (+20% budget)
- [ ] Pause non-converting channels
- [ ] Weekly launch status report to executives
- [ ] **Validation:** Pipeline on track to goal
### Phase 4: Post-Launch (Weeks 5-12)
**Month 2**
- [ ] Launch retrospective meeting
- [ ] Document learnings (what worked, what didn't)
- [ ] Scale winning channels
- [ ] Expand to new segments if successful
- [ ] Update positioning based on customer feedback
- [ ] Plan sustaining campaigns
**Month 3**
- [ ] Final launch report (vs. goals)
- [ ] Calculate ROI (pipeline / spend)
- [ ] Publish additional case studies
- [ ] Integrate learnings into next launch plan
- [ ] Archive launch assets for reuse
---
## Tier 2 Standard Launch
### Timeline: 4 Weeks
**Week -4 to -3: Preparation**
- [ ] Define feature and target audience
- [ ] Create positioning and key messages
- [ ] Build landing page or product page update
- [ ] Write blog post announcement
- [ ] Update sales deck (2-3 slides)
- [ ] Create email announcement
- [ ] Brief sales team (30-min call)
**Week -2 to -1: Setup**
- [ ] Set up HubSpot campaign tracking
- [ ] Schedule social posts
- [ ] Set up paid ads (limited budget)
- [ ] QA landing pages and forms
- [ ] Notify customer success team
**Launch Week**
- [ ] Send email announcement
- [ ] Publish blog post
- [ ] Post on social media
- [ ] In-app notification to users
- [ ] Sales mention in active deals
- [ ] Monitor initial metrics
**Week +1 to +2: Follow-up**
- [ ] Analyze launch metrics
- [ ] Optimize based on data
- [ ] Collect customer feedback
- [ ] Document learnings
---
## Tier 3 Minor Launch
### Timeline: 1 Week
**Day -5 to -3: Prep**
- [ ] Write changelog entry
- [ ] Update support documentation
- [ ] Create in-app notification copy
- [ ] Brief CS team
**Day -2 to -1: Review**
- [ ] QA feature in staging
- [ ] Approve changelog copy
- [ ] Schedule in-app notification
**Launch Day**
- [ ] Deploy feature
- [ ] Trigger in-app notification
- [ ] Publish changelog
- [ ] Update support docs (if needed)
**Day +1 to +3: Monitor**
- [ ] Check for support tickets
- [ ] Monitor feature adoption
- [ ] Address any issues
---
## Launch Metrics Dashboard
### Leading Indicators (Track Daily)
| Metric | Target | Day 1 | Day 3 | Day 7 |
|--------|--------|-------|-------|-------|
| Landing page visitors | 5,000 | | | |
| Demo requests | 100 | | | |
| Free trial signups | 200 | | | |
| MQLs generated | 150 | | | |
| Pipeline created ($) | $500k | | | |
### Lagging Indicators (Track Weekly)
| Metric | Target | Week 1 | Week 2 | Week 4 |
|--------|--------|--------|--------|--------|
| SQLs generated | 30 | | | |
| Demos completed | 50 | | | |
| Deals closed (#) | 5 | | | |
| Revenue ($) | $100k | | | |
| Feature adoption (%) | 40% | | | |
### Channel Performance
| Channel | Spend | MQLs | CPL | Pipeline | ROI |
|---------|-------|------|-----|----------|-----|
| LinkedIn Ads | $10k | | | | |
| Google Ads | $5k | | | | |
| Email | $0 | | | | |
| Organic | $0 | | | | |
| Webinar | $2k | | | | |
| **Total** | **$17k** | | | | |
### Post-Launch Report Template
```
LAUNCH: [Product/Feature Name]
DATE: [Launch Date]
OWNER: [PMM Name]
EXECUTIVE SUMMARY:
- Goal: $500k pipeline in 30 days
- Actual: $[X] pipeline (X% of goal)
- Status: ✅ On Track / ⚠️ Behind / ❌ Missed
KEY RESULTS:
| Metric | Goal | Actual | % of Goal |
|--------------|---------|---------|-----------|
| MQLs | 150 | | |
| SQLs | 30 | | |
| Pipeline | $500k | | |
| Feature Adoption | 40% | | |
TOP PERFORMING:
1. [Channel/Tactic] - [Result]
2. [Channel/Tactic] - [Result]
UNDERPERFORMING:
1. [Channel/Tactic] - [Result] - [Action taken]
LEARNINGS:
1. [What worked and why]
2. [What didn't work and why]
3. [What we'd do differently]
NEXT STEPS:
1. [Action item] - Owner - Due date
2. [Action item] - Owner - Due date
```
FILE:references/messaging-templates.md
# Messaging Templates
Ready-to-use messaging frameworks for different personas and contexts.
---
## Table of Contents
- [Value Proposition Templates](#value-proposition-templates)
- [Persona-Specific Messaging](#persona-specific-messaging)
- [Competitive Messaging](#competitive-messaging)
- [Channel-Specific Copy](#channel-specific-copy)
- [Objection Handling Scripts](#objection-handling-scripts)
---
## Value Proposition Templates
### One-Liner Formula
Template: `[Product] helps [Target Customer] [Achieve Goal] by [Unique Approach]`
**Examples:**
```
B2B SaaS:
"Acme helps mid-market SaaS teams ship 2x faster by automating
project workflows with AI."
Enterprise:
"Acme helps Fortune 500 companies reduce operational costs by 40%
through intelligent process automation."
SMB:
"Acme helps small businesses save 10 hours per week by automating
their daily tasks."
```
### Elevator Pitch (30 Seconds)
Template:
```
You know how [target customer] struggles with [pain point]?
[Product] is a [category] that [key differentiator].
Unlike [alternatives], we [unique value].
Our customers see [specific outcome] within [timeframe].
```
**Example:**
```
You know how engineering teams struggle with slow code reviews
that delay releases?
Acme is an AI code review platform that catches bugs before
they reach production.
Unlike manual reviews, we analyze every PR in under 2 minutes
with 95% accuracy.
Our customers ship 40% faster within their first month.
```
### Messaging Hierarchy
```
LEVEL 1: HEADLINE (5-7 words)
"Ship faster with AI-powered automation"
LEVEL 2: SUBHEAD (1 sentence)
"Acme automates your workflows so your team can focus on what matters."
LEVEL 3: KEY BENEFITS (3-4 bullets)
• Save 10+ hours per week on manual tasks
• Reduce errors by 80% with AI validation
• Deploy changes 3x faster with automated testing
• Scale operations without adding headcount
LEVEL 4: FEATURES → VALUE
• AI Automation → Eliminates repetitive work → Save $50k/year
• Real-time Sync → No version conflicts → 50% fewer errors
• Integrations → Connect existing tools → 2-hour setup
```
---
## Persona-Specific Messaging
### Economic Buyer (VP/Director/C-Level)
**Primary concerns:** ROI, business outcomes, risk mitigation
**Messaging principles:**
- Lead with business impact ($, %, time)
- Show ROI within 6-12 months
- Reference similar companies
- Address risk (security, implementation)
**Template:**
```
HEADLINE: [Business outcome] in [timeframe]
OPENING:
"[Role at similar company] was spending [hours/dollars] on [problem].
After implementing [Product], they achieved [specific result]."
KEY POINTS:
• [Metric] improvement in [area] (e.g., "40% reduction in manual work")
• ROI: [X]x return within [timeframe]
• Implementation: [timeframe] with [level] of effort
• Risk: [How you mitigate concerns]
CTA: "See how [similar company] achieved [result] →"
```
**Example email:**
```
Subject: How Stripe reduced deployment time by 60%
Hi [Name],
The VP of Engineering at a company similar to yours was spending
40 hours per week on code review bottlenecks.
After implementing Acme, they:
• Reduced review time by 60%
• Caught 3x more bugs before production
• Shipped new features 2 weeks faster
Would a 15-minute call to explore if similar results are possible
for [Company] make sense?
```
### Technical Buyer (Engineer/Architect)
**Primary concerns:** Technical fit, security, integration, vendor lock-in
**Messaging principles:**
- Lead with technical capabilities
- Show architecture and security details
- Demonstrate easy integration
- Provide technical documentation
**Template:**
```
HEADLINE: [Technical capability] for [their stack]
OPENING:
"Built for [their technology environment] with [key technical feature]."
KEY POINTS:
• Architecture: [how it works technically]
• Security: [certifications, compliance, encryption]
• Integration: [specific integrations with their tools]
• Performance: [benchmarks, latency, uptime]
CTA: "Read the technical whitepaper →" or "See the API docs →"
```
**Example:**
```
Subject: SOC 2 Type II compliant with 99.99% uptime
Hi [Name],
I noticed [Company] uses Kubernetes for container orchestration.
Acme integrates natively with K8s with:
• Single-line Helm chart deployment
• mTLS encryption for all traffic
• SOC 2 Type II + GDPR compliant
• 99.99% uptime SLA with $10k credit guarantee
Here's our architecture diagram: [link]
Worth a quick technical review?
```
### End User (Manager/Individual Contributor)
**Primary concerns:** Ease of use, daily workflow, learning curve
**Messaging principles:**
- Lead with time savings
- Show product in action (demo, screenshots)
- Emphasize simplicity
- Include peer testimonials
**Template:**
```
HEADLINE: [Daily benefit] in [time to value]
OPENING:
"Imagine [desired outcome] without [pain point]."
KEY POINTS:
• Get started in [timeframe] (no training required)
• Save [hours] every [timeframe]
• [Feature] makes [task] effortless
• Loved by [peer companies/roles]
CTA: "Try free for 14 days →"
```
**Example:**
```
Subject: Spend less time in meetings, more time building
Hi [Name],
What if your weekly status meetings could run themselves?
Acme automatically:
• Collects updates from your team (no nagging)
• Creates visual progress reports (no spreadsheets)
• Flags blockers before they become problems
Teams like [Company A] and [Company B] love it.
Start your free trial: [link]
```
---
## Competitive Messaging
### "Why Us vs. Competitor A" Framework
```
OPENING (acknowledge competition):
"Both [Product] and [Competitor A] help teams with [general category].
Here's what sets us apart:"
DIFFERENTIATORS (3-4 key points):
1. [Your advantage] vs. [Their limitation]
"Our AI catches 95% of bugs vs. their rule-based 60% coverage"
2. [Your advantage] vs. [Their limitation]
"Get started in 2 hours vs. their 2-week implementation"
3. [Your advantage] vs. [Their limitation]
"$50/user vs. their $150/user at scale"
PROOF POINT:
"[Customer] switched from [Competitor A] to us and saw [result]"
CTA:
"See a side-by-side comparison →"
```
### Competitive Positioning Statements
**When they're the market leader:**
```
"[Competitor] built the category, but it was designed for [old paradigm].
[Product] is purpose-built for [new reality] with [key differentiators]."
```
**When they're cheaper:**
```
"[Competitor] costs less upfront, but teams spend [X hours] working
around limitations. [Product] pays for itself in [timeframe] through
[specific efficiency gains]."
```
**When they have more features:**
```
"[Competitor] tries to do everything. [Product] focuses on doing
[core use case] exceptionally well. Our customers tell us they only
use 20% of [Competitor's] features anyway."
```
---
## Channel-Specific Copy
### Landing Page
**Above the fold:**
```
[HEADLINE - 5-7 words, benefit-focused]
Ship faster with AI-powered automation
[SUBHEAD - 1 sentence expanding on value]
Acme automates your workflows so your team can focus on what matters.
[CTA - Action-oriented]
Start Free Trial | Book Demo
```
**Social proof bar:**
```
Trusted by 5,000+ teams including [Logo] [Logo] [Logo] [Logo]
```
### Email Subject Lines
**High performers:**
- "How [Similar Company] achieved [result]"
- "[Name], quick question about [their challenge]"
- "Re: [topic they care about]" (for follow-ups)
- "[Specific number]% improvement in [metric]"
**Avoid:**
- "Quick sync?"
- "Following up..."
- "Just checking in"
- ALL CAPS or excessive punctuation!!!
### LinkedIn Ads
**Format: Single image or carousel**
```
HEADLINE (70 chars max):
"Cut code review time by 60%"
BODY (150 chars recommended):
"AI-powered code reviews that catch bugs before production.
Trusted by engineering teams at Stripe and Shopify.
Try free →"
CTA: Learn More / Try Free / Get Demo
```
### Google Ads
**Search ad format:**
```
Headline 1 (30 chars): AI Code Review Platform
Headline 2 (30 chars): Ship 40% Faster
Headline 3 (30 chars): Free 14-Day Trial
Description (90 chars):
Catch bugs before production. Trusted by 5,000+ teams.
Start your free trial today.
```
---
## Objection Handling Scripts
### Price Objection
**"It's too expensive"**
```
ACKNOWLEDGE: "I understand budget is a concern."
REFRAME: "Let me share how our customers think about it...
[Customer] was spending [X hours/dollars] on [problem] every month.
After implementing [Product], they saved [Y hours/dollars], paying
for the solution in [timeframe]."
QUESTION: "What would it be worth to your team to [achieve outcome]?"
ALTERNATIVE: "We also offer [smaller plan/annual discount] that might
work for your current budget. Would that help?"
```
### Competitor Objection
**"We're looking at [Competitor A] too"**
```
ACKNOWLEDGE: "That's smart to evaluate options. [Competitor A] is
a solid product."
DIFFERENTIATE: "The main differences customers tell us about:
1. [Your advantage] - [Competitor] doesn't offer this
2. [Your advantage] - Their approach is [different/older]
3. [Price/support/speed] - We're typically [X] better here"
PROOF: "[Customer] evaluated both and chose us because [reason]."
QUESTION: "What are the 2-3 things that matter most to you in
this decision?"
```
### Timing Objection
**"Not the right time"**
```
ACKNOWLEDGE: "I completely understand. Timing is everything."
EXPLORE: "Out of curiosity, what would need to change for this
to become a priority?"
FUTURE: "Would it make sense to schedule a brief call in [timeframe]
to revisit? I can share relevant updates without any pressure."
VALUE ADD: "In the meantime, I'll send over [relevant content] that
might be useful for when you're ready."
```
### Authority Objection
**"I need to check with my team/boss"**
```
ACKNOWLEDGE: "Of course, that makes sense."
SUPPORT: "What information would be most helpful for that conversation?
I can put together a one-pager with key points."
OFFER: "Would it help if I joined a brief call with [stakeholder]
to answer any technical/business questions directly?"
TIMELINE: "When do you think you'll have that conversation?
I can follow up with any additional materials beforehand."
```
### Technical Objection
**"Will this integrate with our stack?"**
```
ACKNOWLEDGE: "Great question - integration is critical."
CONFIRM: "What are the main tools you need to connect with?
[Listen and take notes]"
ANSWER: "We have native integrations with [tools]. For [tool],
we use [API/webhook/Zapier]. Here's our integration docs: [link]"
PROOF: "[Similar company] uses a similar stack and got integrated
in [timeframe]."
DEMO: "Want me to show you exactly how the integration works
in a quick demo?"
```
FILE:references/positioning-frameworks.md
# Positioning Frameworks
Strategic positioning methodologies for B2B SaaS products.
---
## Table of Contents
- [April Dunford Positioning](#april-dunford-positioning)
- [Geoffrey Moore Positioning](#geoffrey-moore-positioning)
- [Positioning Validation](#positioning-validation)
- [Competitive Positioning Map](#competitive-positioning-map)
---
## April Dunford Positioning
### The 5-Step Process
Execute positioning using April Dunford's "Obviously Awesome" methodology:
1. List competitive alternatives (what customers would use instead)
2. Isolate unique attributes (features only you have)
3. Map attributes to value (why each attribute matters)
4. Define best-fit customers (who cares most about this value)
5. Choose market category (where you compete)
6. **Validation:** Best-fit customers articulate your value unprompted
### Step 1: Competitive Alternatives
Document what customers do without your product:
| Alternative Type | Examples | How They Solve It |
|------------------|----------|-------------------|
| Direct competitor | Competitor A, B | Same category, different approach |
| Adjacent solution | Spreadsheets, email | Manual workaround |
| Build in-house | Custom development | Internal solution |
| Do nothing | Ignore problem | Accept status quo |
**Interview Questions:**
- "Before using us, how did you handle this?"
- "What alternatives did you evaluate?"
- "What would you switch to if we disappeared?"
### Step 2: Unique Attributes
Identify capabilities competitors lack:
```
Attribute Audit:
1. Feature: [Real-time collaboration]
- Competitor A: No (async only)
- Competitor B: Partial (limited to 5 users)
- You: Yes (unlimited users, 50ms sync)
→ Unique: Yes
2. Feature: [AI automation]
- Competitor A: No
- Competitor B: No
- You: Yes (3 AI models)
→ Unique: Yes
3. Feature: [Integrations]
- Competitor A: 500+
- Competitor B: 200+
- You: 100
→ Unique: No (table stakes)
```
### Step 3: Attribute-Value Mapping
Connect features to business outcomes:
| Attribute | Value Enabled | Customer Outcome |
|-----------|--------------|------------------|
| Real-time sync | No version conflicts | 50% fewer errors |
| AI automation | Eliminates manual work | Save 10 hrs/week |
| One-click deploy | Faster releases | Ship 2x faster |
**Value Statement Formula:**
`[Feature] enables [Value] so customers achieve [Outcome]`
### Step 4: Best-Fit Customers
Define who values your unique attributes most:
```
Best-Fit Profile:
- Company size: 200-2000 employees
- Industry: SaaS, Professional Services
- Pain: Distributed teams, collaboration bottlenecks
- Evidence:
- Fastest sales cycles (45 days vs. 75 avg)
- Lowest churn (3% vs. 8% avg)
- Highest NPS (65 vs. 45 avg)
```
### Step 5: Market Category
Choose competitive frame:
| Strategy | When to Use | Risk Level |
|----------|-------------|------------|
| Head-to-head | Strong product, big budget | Medium |
| Niche domination | Unique for segment | Low |
| Category creation | True innovation, deep pockets | High |
**Decision Framework:**
- Can you win head-to-head? → Head-to-head
- Can you dominate a niche? → Niche
- Is the market undefined? → Category creation
---
## Geoffrey Moore Positioning
### Crossing the Chasm Framework
Position for technology adoption lifecycle:
```
Technology Adoption Curve:
Innovators (2.5%) → Early Adopters (13.5%) → Early Majority (34%)
↑
THE CHASM
```
### Positioning Statement Template
```
FOR [target customer]
WHO [statement of need or opportunity]
THE [product name] IS A [product category]
THAT [key benefit/reason to buy]
UNLIKE [primary competitive alternative]
OUR PRODUCT [primary differentiation]
```
**Example:**
```
FOR mid-market SaaS companies with distributed engineering teams
WHO struggle with coordination across time zones
THE Acme Platform IS A real-time collaboration workspace
THAT eliminates version conflicts and communication delays
UNLIKE Slack and email which create information silos
OUR PRODUCT provides unified project context with AI-powered summaries
```
### Whole Product Concept
Define complete solution for target segment:
| Layer | Components | Your Coverage |
|-------|------------|---------------|
| Generic | Core product | 100% |
| Expected | Basic integrations, support | 90% |
| Augmented | Training, consulting, custom work | 60% |
| Potential | Future roadmap, ecosystem | 30% |
**Gap Analysis:**
- What's missing for complete solution?
- Which partners can fill gaps?
- What must you build vs. buy vs. partner?
---
## Positioning Validation
### Customer Interview Protocol
Validate positioning with target customers:
1. Schedule 15-20 minute calls with 10+ target customers
2. Ask open-ended questions (no leading)
3. Document exact language used
4. Look for patterns across interviews
5. **Validation:** 7+ of 10 describe value similarly
**Interview Script:**
```
Opening (2 min):
"Thanks for your time. I want to understand how you think about
[product category] and your experience with our product."
Questions (10 min):
1. "How would you describe [Product] to a colleague?"
2. "What problem does [Product] solve for you?"
3. "What alternatives did you consider?"
4. "Why did you choose us over [alternative]?"
5. "What would make you stop using us?"
Closing (3 min):
"Is there anything else you'd like to share?"
```
### Quantitative Validation
Test messaging through A/B experiments:
| Test | Control | Variant | Winner Criteria |
|------|---------|---------|-----------------|
| Landing page headline | Old positioning | New positioning | +20% conversion |
| Ad copy | Feature-focused | Value-focused | +15% CTR |
| Email subject | Generic | Personalized | +25% open rate |
**Sample Size Calculator:**
- Baseline conversion: 3%
- Minimum detectable effect: 20% relative lift
- Statistical power: 80%
- Required sample: ~2,500 per variant
---
## Competitive Positioning Map
### 2x2 Matrix Construction
Create visual positioning map:
```
HIGH PRICE
│
Enterprise │ Premium
(Salesforce) │ (You?)
│
────────────────────┼──────────────────
LOW │ HIGH
EASE OF USE │ EASE OF USE
│
Legacy │ Self-Serve
(Oracle) │ (Notion)
│
LOW PRICE
```
### Axis Selection
Choose dimensions that highlight your advantage:
| Good Axes | Why |
|-----------|-----|
| Ease of use vs. Power | If you're easiest to use |
| Speed vs. Accuracy | If you're fastest |
| Price vs. Features | If you're best value |
| Specialization vs. Breadth | If you own a niche |
| Bad Axes | Why |
|----------|-----|
| Quality vs. Price | Everyone claims quality |
| Innovation vs. Stability | Subjective, hard to prove |
| Customer vs. Product focus | Not differentiating |
### Positioning Map Template
```
Market Category: [Your Category]
Date: [Month Year]
Axes:
- X-axis: [Dimension 1] (Low → High)
- Y-axis: [Dimension 2] (Low → High)
Quadrants:
- Top-left: [Quadrant description]
- Top-right: [Quadrant description] ← Your target
- Bottom-left: [Quadrant description]
- Bottom-right: [Quadrant description]
Competitors:
1. [Competitor A]: Position (X, Y), Why
2. [Competitor B]: Position (X, Y), Why
3. [You]: Position (X, Y), Why you win
Strategic Implications:
- Attack: [How to position against Competitor A]
- Defend: [How to protect against Competitor B]
- Differentiate: [Your unique positioning claim]
```
Cố vấn ở vai trò giám đốc khách hàng (CCO): chiến lược trải nghiệm, giữ chân và thành công của khách hàng.
../../../c-level-advisor/chief-customer-officer-advisor/skills/chief-customer-officer-advisor/SKILL.md
Theo dõi sức khỏe khách hàng, dự đoán rủi ro rời bỏ và tìm cơ hội mở rộng bằng mô hình chấm điểm có trọng số cho SaaS.
---
name: "customer-success-manager"
description: Monitors customer health, predicts churn risk, and identifies expansion opportunities using weighted scoring models for SaaS customer success. Use when analyzing customer accounts, reviewing retention metrics, scoring at-risk customers, or when the user mentions churn, customer health scores, upsell opportunities, expansion revenue, retention analysis, or customer analytics. Runs three Python CLI tools to produce deterministic health scores, churn risk tiers, and prioritized expansion recommendations across Enterprise, Mid-Market, and SMB segments.
license: MIT
metadata:
version: 1.0.0
author: Alireza Rezvani
category: business-growth
domain: customer-success
updated: 2026-02-06
python-tools: health_score_calculator.py, churn_risk_analyzer.py, expansion_opportunity_scorer.py
tech-stack: customer-success, saas-metrics, health-scoring
---
# Customer Success Manager
Production-grade customer success analytics with multi-dimensional health scoring, churn risk prediction, and expansion opportunity identification. Three Python CLI tools provide deterministic, repeatable analysis using standard library only -- no external dependencies, no API calls, no ML models.
---
## Table of Contents
- [Input Requirements](#input-requirements)
- [Output Formats](#output-formats)
- [How to Use](#how-to-use)
- [Scripts](#scripts)
- [Reference Guides](#reference-guides)
- [Templates](#templates)
- [Best Practices](#best-practices)
- [Limitations](#limitations)
---
## Input Requirements
All scripts accept a JSON file as positional input argument. See `assets/sample_customer_data.json` for complete schema examples and sample data.
### Health Score Calculator
Required fields per customer object: `customer_id`, `name`, `segment`, `arr`, and nested objects `usage` (login_frequency, feature_adoption, dau_mau_ratio), `engagement` (support_ticket_volume, meeting_attendance, nps_score, csat_score), `support` (open_tickets, escalation_rate, avg_resolution_hours), `relationship` (executive_sponsor_engagement, multi_threading_depth, renewal_sentiment), and `previous_period` scores for trend analysis.
### Churn Risk Analyzer
Required fields per customer object: `customer_id`, `name`, `segment`, `arr`, `contract_end_date`, and nested objects `usage_decline`, `engagement_drop`, `support_issues`, `relationship_signals`, and `commercial_factors`.
### Expansion Opportunity Scorer
Required fields per customer object: `customer_id`, `name`, `segment`, `arr`, and nested objects `contract` (licensed_seats, active_seats, plan_tier, available_tiers), `product_usage` (per-module adoption flags and usage percentages), and `departments` (current and potential).
---
## Output Formats
All scripts support two output formats via the `--format` flag:
- **`text`** (default): Human-readable formatted output for terminal viewing
- **`json`**: Machine-readable JSON output for integrations and pipelines
---
## How to Use
### Quick Start
```bash
# Health scoring
python scripts/health_score_calculator.py assets/sample_customer_data.json
python scripts/health_score_calculator.py assets/sample_customer_data.json --format json
# Churn risk analysis
python scripts/churn_risk_analyzer.py assets/sample_customer_data.json
python scripts/churn_risk_analyzer.py assets/sample_customer_data.json --format json
# Expansion opportunity scoring
python scripts/expansion_opportunity_scorer.py assets/sample_customer_data.json
python scripts/expansion_opportunity_scorer.py assets/sample_customer_data.json --format json
```
### Workflow Integration
```bash
# 1. Score customer health across portfolio
python scripts/health_score_calculator.py customer_portfolio.json --format json > health_results.json
# Verify: confirm health_results.json contains the expected number of customer records before continuing
# 2. Identify at-risk accounts
python scripts/churn_risk_analyzer.py customer_portfolio.json --format json > risk_results.json
# Verify: confirm risk_results.json is non-empty and risk tiers are present for each customer
# 3. Find expansion opportunities in healthy accounts
python scripts/expansion_opportunity_scorer.py customer_portfolio.json --format json > expansion_results.json
# Verify: confirm expansion_results.json lists opportunities ranked by priority
# 4. Prepare QBR using templates
# Reference: assets/qbr_template.md
```
**Error handling:** If a script exits with an error, check that:
- The input JSON matches the required schema for that script (see Input Requirements above)
- All required fields are present and correctly typed
- Python 3.7+ is being used (`python --version`)
- Output files from prior steps are non-empty before piping into subsequent steps
---
## Scripts
### 1. health_score_calculator.py
**Purpose:** Multi-dimensional customer health scoring with trend analysis and segment-aware benchmarking.
**Dimensions and Weights:**
| Dimension | Weight | Metrics |
|-----------|--------|---------|
| Usage | 30% | Login frequency, feature adoption, DAU/MAU ratio |
| Engagement | 25% | Support ticket volume, meeting attendance, NPS/CSAT |
| Support | 20% | Open tickets, escalation rate, avg resolution time |
| Relationship | 25% | Executive sponsor engagement, multi-threading depth, renewal sentiment |
**Classification:**
- Green (75-100): Healthy -- customer achieving value
- Yellow (50-74): Needs attention -- monitor closely
- Red (0-49): At risk -- immediate intervention required
**Usage:**
```bash
python scripts/health_score_calculator.py customer_data.json
python scripts/health_score_calculator.py customer_data.json --format json
```
### 2. churn_risk_analyzer.py
**Purpose:** Identify at-risk accounts with behavioral signal detection and tier-based intervention recommendations.
**Risk Signal Weights:**
| Signal Category | Weight | Indicators |
|----------------|--------|------------|
| Usage Decline | 30% | Login trend, feature adoption change, DAU/MAU change |
| Engagement Drop | 25% | Meeting cancellations, response time, NPS change |
| Support Issues | 20% | Open escalations, unresolved critical, satisfaction trend |
| Relationship Signals | 15% | Champion left, sponsor change, competitor mentions |
| Commercial Factors | 10% | Contract type, pricing complaints, budget cuts |
**Risk Tiers:**
- Critical (80-100): Immediate executive escalation
- High (60-79): Urgent CSM intervention
- Medium (40-59): Proactive outreach
- Low (0-39): Standard monitoring
**Usage:**
```bash
python scripts/churn_risk_analyzer.py customer_data.json
python scripts/churn_risk_analyzer.py customer_data.json --format json
```
### 3. expansion_opportunity_scorer.py
**Purpose:** Identify upsell, cross-sell, and expansion opportunities with revenue estimation and priority ranking.
**Expansion Types:**
- **Upsell**: Upgrade to higher tier or more of existing product
- **Cross-sell**: Add new product modules
- **Expansion**: Additional seats or departments
**Usage:**
```bash
python scripts/expansion_opportunity_scorer.py customer_data.json
python scripts/expansion_opportunity_scorer.py customer_data.json --format json
```
---
## Reference Guides
| Reference | Description |
|-----------|-------------|
| `references/health-scoring-framework.md` | Complete health scoring methodology, dimension definitions, weighting rationale, threshold calibration |
| `references/cs-playbooks.md` | Intervention playbooks for each risk tier, onboarding, renewal, expansion, and escalation procedures |
| `references/cs-metrics-benchmarks.md` | Industry benchmarks for NRR, GRR, churn rates, health scores, expansion rates by segment and industry |
---
## Templates
| Template | Purpose |
|----------|---------|
| `assets/qbr_template.md` | Quarterly Business Review presentation structure |
| `assets/success_plan_template.md` | Customer success plan with goals, milestones, and metrics |
| `assets/onboarding_checklist_template.md` | 90-day onboarding checklist with phase gates |
| `assets/executive_business_review_template.md` | Executive stakeholder review for strategic accounts |
---
## Best Practices
1. **Combine signals**: Use all three scripts together for a complete customer picture
2. **Act on trends, not snapshots**: A declining Green is more urgent than a stable Yellow
3. **Calibrate thresholds**: Adjust segment benchmarks based on your product and industry per `references/health-scoring-framework.md`
4. **Prepare with data**: Run scripts before every QBR and executive meeting; reference `references/cs-playbooks.md` for intervention guidance
---
## Limitations
- **No real-time data**: Scripts analyze point-in-time snapshots from JSON input files
- **No CRM integration**: Data must be exported manually from your CRM/CS platform
- **Deterministic only**: No predictive ML -- scoring is algorithmic based on weighted signals
- **Threshold tuning**: Default thresholds are industry-standard but may need calibration for your business
- **Revenue estimates**: Expansion revenue estimates are approximations based on usage patterns
---
**Last Updated:** February 2026
**Tools:** 3 Python CLI tools
**Dependencies:** Python 3.7+ standard library only
FILE:assets/executive_business_review_template.md
# Executive Business Review
**Customer:** [Customer Name]
**Date:** [Review Date]
**Prepared for:** [Executive Name, Title]
**Prepared by:** [CSM Name] | [VP Customer Success Name]
**Classification:** [Strategic / Enterprise / Key Account]
---
## 1. Partnership Summary
| Metric | Value |
|--------|-------|
| Partnership Duration | [X months/years] |
| Current ARR | $[Amount] |
| Lifetime Value to Date | $[Amount] |
| Current Plan | [Tier] |
| Licensed Seats | [Number] |
| Active Seats | [Number] |
| Health Score | [Score]/100 ([Green/Yellow/Red]) |
| NPS Score | [Score] |
| Renewal Date | [Date] ([X] days remaining) |
---
## 2. Strategic Alignment
### Customer's Business Priorities (This Year)
1. **[Priority 1]** -- [How our solution supports this]
2. **[Priority 2]** -- [How our solution supports this]
3. **[Priority 3]** -- [How our solution supports this]
### Alignment Assessment
| Business Priority | Our Contribution | Alignment Score |
|-------------------|-----------------|----------------|
| [Priority 1] | [Specific contribution] | [Strong / Moderate / Weak] |
| [Priority 2] | [Specific contribution] | [Strong / Moderate / Weak] |
| [Priority 3] | [Specific contribution] | [Strong / Moderate / Weak] |
---
## 3. Value Delivered
### Quantified Business Impact
| Outcome | Metric | Before | After | Business Value |
|---------|--------|--------|-------|---------------|
| [e.g., Operational efficiency] | [Hours saved/week] | [Baseline] | [Current] | $[Estimated value] |
| [e.g., Revenue acceleration] | [Deal velocity] | [Baseline] | [Current] | $[Estimated value] |
| [e.g., Risk reduction] | [Error rate] | [Baseline] | [Current] | $[Estimated value] |
**Total Estimated Business Value:** $[Amount]
**ROI:** [X]x return on investment
### Key Achievements This Period
1. [Achievement 1 with measurable outcome]
2. [Achievement 2 with measurable outcome]
3. [Achievement 3 with measurable outcome]
---
## 4. Adoption and Engagement Scorecard
### Platform Utilisation
| Module | Adoption Status | Usage Depth | Benchmark | Assessment |
|--------|---------------|-------------|-----------|------------|
| [Module 1] | Fully Adopted | [High/Med/Low] | [Benchmark] | [Above/At/Below] |
| [Module 2] | Partially Adopted | [High/Med/Low] | [Benchmark] | [Above/At/Below] |
| [Module 3] | Not Adopted | -- | -- | Opportunity |
### Engagement Health
| Indicator | Current | Previous Period | Trend |
|-----------|---------|----------------|-------|
| Executive Engagement | [Score] | [Score] | [Up/Down/Stable] |
| Stakeholder Breadth | [# contacts] | [# contacts] | [Up/Down/Stable] |
| Meeting Participation | [%] | [%] | [Up/Down/Stable] |
| Feature Request Activity | [Count] | [Count] | [Up/Down/Stable] |
---
## 5. Account Health Overview
### Health Score Trend (Last 4 Quarters)
| Quarter | Overall | Usage | Engagement | Support | Relationship |
|---------|---------|-------|------------|---------|-------------|
| [Q-3] | [Score] | [Score] | [Score] | [Score] | [Score] |
| [Q-2] | [Score] | [Score] | [Score] | [Score] | [Score] |
| [Q-1] | [Score] | [Score] | [Score] | [Score] | [Score] |
| Current | [Score] | [Score] | [Score] | [Score] | [Score] |
### Risk Assessment
| Risk Factor | Level | Details | Mitigation |
|------------|-------|---------|-----------|
| [Risk 1] | [High/Med/Low] | [Description] | [Action] |
| [Risk 2] | [High/Med/Low] | [Description] | [Action] |
---
## 6. Support and Service Quality
| Metric | This Period | SLA Target | Status |
|--------|------------|-----------|--------|
| Total Tickets | [Number] | -- | |
| Avg First Response | [Hours] | [Hours] | [Met / Not Met] |
| Avg Resolution Time | [Hours] | [Hours] | [Met / Not Met] |
| Escalations | [Number] | 0 | |
| CSAT Score | [Score] | [Target] | [Above / Below] |
| Critical Issues | [Number] | 0 | |
### Notable Support Interactions
- [Summary of any significant support events and resolution]
---
## 7. Product Roadmap Alignment
### Features Delivered (Relevant to This Customer)
| Feature | Release Date | Customer Impact |
|---------|-------------|----------------|
| [Feature 1] | [Date] | [How it helps them] |
| [Feature 2] | [Date] | [How it helps them] |
### Upcoming Features (Customer-Relevant)
| Feature | Expected Release | Expected Impact |
|---------|-----------------|----------------|
| [Feature 1] | [Quarter] | [Business value] |
| [Feature 2] | [Quarter] | [Business value] |
### Customer Feature Requests
| Request | Priority | Status | Business Case |
|---------|----------|--------|--------------|
| [Request 1] | [P1/P2/P3] | [Status] | [Why it matters] |
| [Request 2] | [P1/P2/P3] | [Status] | [Why it matters] |
---
## 8. Growth and Expansion Opportunity
### Current Whitespace Analysis
| Opportunity | Type | Est. Revenue | Effort | Priority |
|------------|------|-------------|--------|----------|
| [Opportunity 1] | [Upsell/Cross-sell/Expansion] | $[Amount] | [Low/Med/High] | [1-5] |
| [Opportunity 2] | [Upsell/Cross-sell/Expansion] | $[Amount] | [Low/Med/High] | [1-5] |
| [Opportunity 3] | [Upsell/Cross-sell/Expansion] | $[Amount] | [Low/Med/High] | [1-5] |
**Total Expansion Opportunity:** $[Amount]
### Recommended Next Steps for Growth
1. [Specific expansion recommendation with business justification]
2. [Specific expansion recommendation with business justification]
---
## 9. Renewal Outlook
| Factor | Assessment |
|--------|-----------|
| Overall Renewal Confidence | [High / Medium / Low] |
| Budget Availability | [Confirmed / Expected / Uncertain] |
| Sponsor Support | [Strong / Moderate / Weak] |
| Competitive Threat | [None / Low / Medium / High] |
| Value Perception | [Strong / Moderate / Weak] |
| Contract Satisfaction | [Satisfied / Neutral / Concerned] |
### Renewal Strategy
[2-3 sentences on the approach for securing renewal, including any specific actions needed]
---
## 10. Executive-Level Action Items
| Action | Owner | Due Date | Priority | Impact |
|--------|-------|----------|----------|--------|
| [Action 1] | [Name, Title] | [Date] | [Critical/High/Med] | [Expected outcome] |
| [Action 2] | [Name, Title] | [Date] | [Critical/High/Med] | [Expected outcome] |
| [Action 3] | [Name, Title] | [Date] | [Critical/High/Med] | [Expected outcome] |
---
## Appendix
### Stakeholder Map
| Name | Title | Influence | Sentiment | Last Contact |
|------|-------|-----------|-----------|-------------|
| [Name] | [Title] | [Decision Maker / Influencer / User] | [Positive / Neutral / Negative] | [Date] |
| [Name] | [Title] | [Decision Maker / Influencer / User] | [Positive / Neutral / Negative] | [Date] |
### Competitive Landscape (If Applicable)
- **Known competitors in evaluation:** [List]
- **Our differentiators:** [Key strengths vs. competition]
- **Risk mitigation:** [Actions to defend position]
---
**Confidential -- For Internal and Customer Executive Use Only**
**Next Executive Review:** [Date]
FILE:assets/expected_output.json
{
"report": "customer_health_scores",
"summary": {
"total_customers": 4,
"average_score": 78.8,
"green_count": 3,
"yellow_count": 1,
"red_count": 0
},
"customers": [
{
"customer_id": "CUST-001",
"name": "Acme Corp",
"segment": "enterprise",
"arr": 120000,
"overall_score": 86.2,
"classification": "green",
"dimensions": {
"usage": {
"score": 91.6,
"weight": "30%",
"classification": "green"
},
"engagement": {
"score": 82.0,
"weight": "25%",
"classification": "green"
},
"support": {
"score": 78.5,
"weight": "20%",
"classification": "green"
},
"relationship": {
"score": 90.1,
"weight": "25%",
"classification": "green"
}
},
"trends": {
"usage": "improving",
"engagement": "improving",
"support": "stable",
"relationship": "improving",
"overall": "improving"
},
"recommendations": []
},
{
"customer_id": "CUST-002",
"name": "TechStart Inc",
"segment": "smb",
"arr": 18000,
"overall_score": 53.7,
"classification": "yellow",
"dimensions": {
"usage": {
"score": 52.5,
"weight": "30%",
"classification": "yellow"
},
"engagement": {
"score": 61.6,
"weight": "25%",
"classification": "yellow"
},
"support": {
"score": 63.2,
"weight": "20%",
"classification": "yellow"
},
"relationship": {
"score": 39.5,
"weight": "25%",
"classification": "red"
}
},
"trends": {
"usage": "stable",
"engagement": "improving",
"support": "stable",
"relationship": "declining",
"overall": "stable"
},
"recommendations": [
"Login frequency below target -- schedule product engagement session",
"NPS below threshold -- conduct a feedback deep-dive with customer",
"CSAT is critically low -- escalate to support leadership",
"Single-threaded relationship -- expand contacts across departments",
"Renewal sentiment is negative -- initiate save plan immediately"
]
},
{
"customer_id": "CUST-003",
"name": "GlobalTrade Solutions",
"segment": "mid-market",
"arr": 55000,
"overall_score": 79.7,
"classification": "green",
"dimensions": {
"usage": {
"score": 85.6,
"weight": "30%",
"classification": "green"
},
"engagement": {
"score": 79.6,
"weight": "25%",
"classification": "green"
},
"support": {
"score": 72.0,
"weight": "20%",
"classification": "green"
},
"relationship": {
"score": 79.0,
"weight": "25%",
"classification": "green"
}
},
"trends": {
"usage": "improving",
"engagement": "improving",
"support": "improving",
"relationship": "improving",
"overall": "improving"
},
"recommendations": []
},
{
"customer_id": "CUST-004",
"name": "HealthFirst Medical",
"segment": "enterprise",
"arr": 200000,
"overall_score": 95.7,
"classification": "green",
"dimensions": {
"usage": {
"score": 100.0,
"weight": "30%",
"classification": "green"
},
"engagement": {
"score": 92.0,
"weight": "25%",
"classification": "green"
},
"support": {
"score": 88.7,
"weight": "20%",
"classification": "green"
},
"relationship": {
"score": 100.0,
"weight": "25%",
"classification": "green"
}
},
"trends": {
"usage": "improving",
"engagement": "improving",
"support": "stable",
"relationship": "improving",
"overall": "improving"
},
"recommendations": []
}
]
}
FILE:assets/onboarding_checklist_template.md
# Customer Onboarding Checklist (90-Day)
**Customer:** [Customer Name]
**Segment:** [Enterprise / Mid-Market / SMB]
**CSM:** [CSM Name]
**Kickoff Date:** [Date]
**Target Go-Live:** [Date]
**Target First Value Date:** [Date -- must be within 30 days]
---
## Phase 1: Welcome and Setup (Days 1-14)
### Pre-Kickoff Preparation (Day 0)
- [ ] Review signed contract and SOW for scope and commitments
- [ ] Research customer's industry, business model, and competitive landscape
- [ ] Review handoff notes from sales team (pain points, decision drivers, stakeholders)
- [ ] Prepare welcome package (login credentials, documentation links, support contacts)
- [ ] Create customer workspace in CS platform
- [ ] Schedule kickoff meeting with all required attendees
- [ ] Prepare kickoff deck with agenda and success plan draft
### Kickoff Meeting (Day 1-2)
- [ ] Conduct kickoff meeting with customer stakeholders
- [ ] Confirm business objectives and success criteria
- [ ] Identify key stakeholders and their roles (sponsor, champion, technical lead, users)
- [ ] Align on communication cadence and preferred channels
- [ ] Review onboarding timeline and milestones
- [ ] Set expectations for time commitment from customer team
- [ ] Share and agree on success plan (mutual accountability)
- [ ] Schedule recurring check-in meetings
**Kickoff Meeting Notes:**
> [Document key takeaways, concerns raised, decisions made]
### Technical Setup (Days 3-7)
- [ ] Provision customer environment (tenant, workspace, permissions)
- [ ] Configure SSO/authentication if applicable
- [ ] Set up integrations with customer's existing tools
- [ ] Import or migrate existing data (if applicable)
- [ ] Validate data integrity post-migration
- [ ] Configure role-based access and permissions
- [ ] Set up monitoring and alerting
**Technical Setup Owner:** [SE / Implementation team name]
**Technical Setup Notes:**
> [Document configuration decisions, customizations, issues]
### Admin Training (Days 7-10)
- [ ] Deliver admin training session (system configuration, user management)
- [ ] Provide admin documentation and quick reference guide
- [ ] Ensure admins can independently manage basic operations
- [ ] Set up admin support escalation path
### Initial User Training (Days 10-14)
- [ ] Deliver core user training (session 1: basic navigation and key workflows)
- [ ] Provide user quickstart guide and video resources
- [ ] Set up user support channel (Slack, email, in-app chat)
- [ ] Confirm all target users have active accounts
- [ ] Track initial login completion rate
**Training Completion Rate:** [___%] of target users
---
## Phase 2: Activation (Days 15-30)
### User Activation (Days 15-20)
- [ ] Monitor daily active user metrics
- [ ] Follow up with users who have not logged in
- [ ] Conduct follow-up training for users needing additional help
- [ ] Address any usability issues or confusion reported
- [ ] Validate that core workflows are functioning as expected
- [ ] Collect early feedback from champion and key users
**Activation Rate:** [___%] of licensed users active
### First Value Milestone (Days 20-30)
- [ ] Define and track first value milestone (specific to customer objectives)
- [ ] Verify customer has completed their first meaningful workflow
- [ ] Document value delivered (even if small -- establish the pattern)
- [ ] Share "first win" with executive sponsor
- [ ] Celebrate the milestone with the customer team
**First Value Milestone:** [Describe the specific milestone]
**Date Achieved:** [Date]
### 30-Day Review (Day 28-30)
- [ ] Conduct 30-day review meeting with customer
- [ ] Review activation metrics (logins, usage, adoption)
- [ ] Assess progress against success plan milestones
- [ ] Identify any blockers or concerns
- [ ] Adjust onboarding plan if needed
- [ ] Confirm transition from setup phase to adoption phase
- [ ] Set goals for days 31-60
**30-Day Health Score:** [Score]/100 -- [Green/Yellow/Red]
---
## Phase 3: Adoption (Days 31-60)
### Feature Expansion (Days 31-45)
- [ ] Introduce additional features beyond core workflows
- [ ] Deliver advanced training session (session 2: power features)
- [ ] Enable at least one integration with customer's existing tools
- [ ] Identify and address feature adoption gaps
- [ ] Share best practices from similar customers
### Usage Benchmarking (Days 45-55)
- [ ] Compare customer's usage against segment benchmarks
- [ ] Identify underperforming areas and create enablement plan
- [ ] Share usage report with customer champion
- [ ] Discuss usage targets for the next 30 days
**Current vs. Benchmark:**
| Metric | Current | Benchmark | Gap |
|--------|---------|-----------|-----|
| Feature Adoption | [%] | [%] | [+/-] |
| Daily Active Users | [#] | [#] | [+/-] |
| Key Workflow Completion | [%] | [%] | [+/-] |
### 60-Day Check-in (Day 55-60)
- [ ] Conduct 60-day check-in meeting
- [ ] Review adoption metrics and progress
- [ ] Discuss any roadblocks to deeper adoption
- [ ] Begin identifying advanced use cases
- [ ] Set goals for days 61-90
---
## Phase 4: Optimisation (Days 61-90)
### Advanced Use Cases (Days 61-75)
- [ ] Conduct use case discovery workshop with customer
- [ ] Identify 2-3 advanced use cases beyond initial scope
- [ ] Build implementation plan for advanced use cases
- [ ] Begin pilot of advanced use cases with power users
### ROI Measurement (Days 75-85)
- [ ] Collect data for ROI measurement against baseline
- [ ] Build ROI summary document
- [ ] Share ROI results with executive sponsor
- [ ] Document customer testimonial or case study opportunity (if willing)
**ROI Summary:**
| Metric | Baseline | Current | Improvement |
|--------|----------|---------|-------------|
| [Metric 1] | [Value] | [Value] | [% change] |
| [Metric 2] | [Value] | [Value] | [% change] |
### 90-Day Executive Review (Days 85-90)
- [ ] Prepare 90-day executive review presentation
- [ ] Include: value delivered, adoption metrics, ROI, next steps
- [ ] Conduct review meeting with executive sponsor
- [ ] Transition from onboarding to ongoing success management
- [ ] Establish ongoing success plan with quarterly milestones
- [ ] Confirm ongoing meeting cadence
- [ ] Introduce expansion opportunities if appropriate
**90-Day Health Score:** [Score]/100 -- [Green/Yellow/Red]
---
## Onboarding Completion Gate
The following criteria must be met to consider onboarding complete:
- [ ] User activation rate above 80%
- [ ] First value milestone achieved within 30 days
- [ ] Core workflows actively used by target users
- [ ] Executive sponsor confirms satisfaction
- [ ] Health score is Yellow (50+) or better
- [ ] Success plan established with ongoing milestones
- [ ] Recurring meeting cadence confirmed
- [ ] Support escalation path understood by customer
**Onboarding Status:** [Complete / In Progress / Blocked]
**Completion Date:** [Date]
**Handoff to Steady-State CSM:** [Date if different CSM]
---
## Notes
### Risks and Blockers
| Risk/Blocker | Impact | Mitigation | Status |
|-------------|--------|-----------|--------|
| [Item] | [High/Med/Low] | [Action] | [Open/Resolved] |
### Key Decisions
| Date | Decision | Made By | Impact |
|------|----------|---------|--------|
| [Date] | [Decision] | [Name] | [Description] |
---
**Template Version:** 1.0
**Last Updated:** February 2026
FILE:assets/qbr_template.md
# Quarterly Business Review (QBR)
**Customer:** [Customer Name]
**Date:** [QBR Date]
**Prepared by:** [CSM Name]
**Attendees:** [List attendees and titles]
---
## 1. Executive Summary
**Overall Relationship Status:** [Green / Yellow / Red]
**Health Score:** [Score]/100
**Key Theme:** [One sentence summarizing the quarter]
### Quarter Highlights
- [Highlight 1: major achievement or milestone]
- [Highlight 2: value delivered]
- [Highlight 3: initiative completed]
### Areas of Focus
- [Focus area 1]
- [Focus area 2]
---
## 2. Value Delivered This Quarter
### Business Outcomes Achieved
| Objective | Target | Actual | Status |
|-----------|--------|--------|--------|
| [Objective 1] | [Target metric] | [Actual metric] | [On Track / At Risk / Achieved] |
| [Objective 2] | [Target metric] | [Actual metric] | [On Track / At Risk / Achieved] |
| [Objective 3] | [Target metric] | [Actual metric] | [On Track / At Risk / Achieved] |
### ROI Summary
| Metric | Before | After | Improvement |
|--------|--------|-------|-------------|
| [Metric 1, e.g., Time savings] | [Baseline] | [Current] | [% change] |
| [Metric 2, e.g., Cost reduction] | [Baseline] | [Current] | [% change] |
| [Metric 3, e.g., Revenue impact] | [Baseline] | [Current] | [% change] |
**Estimated Total Value Delivered:** $[Amount]
---
## 3. Product Usage and Adoption
### Usage Metrics
| Metric | Last Quarter | This Quarter | Trend |
|--------|-------------|--------------|-------|
| Monthly Active Users | [Number] | [Number] | [Up/Down/Stable] |
| Feature Adoption Rate | [%] | [%] | [Up/Down/Stable] |
| DAU/MAU Ratio | [Ratio] | [Ratio] | [Up/Down/Stable] |
| Seat Utilization | [%] | [%] | [Up/Down/Stable] |
### Feature Adoption Breakdown
| Feature/Module | Status | Usage Level | Notes |
|---------------|--------|-------------|-------|
| [Feature 1] | Active | [High/Med/Low] | |
| [Feature 2] | Active | [High/Med/Low] | |
| [Feature 3] | Not Adopted | -- | [Reason / Opportunity] |
### Adoption Recommendations
1. [Recommendation for increasing adoption of underused features]
2. [Recommendation for enabling new use cases]
---
## 4. Support Summary
| Metric | This Quarter | Previous Quarter | Benchmark |
|--------|-------------|-----------------|-----------|
| Total Tickets | [Number] | [Number] | [Segment avg] |
| Avg Resolution Time | [Hours] | [Hours] | [SLA target] |
| Escalations | [Number] | [Number] | [Target: 0] |
| CSAT Score | [Score] | [Score] | [Target] |
### Open Issues
| Issue | Priority | Status | ETA |
|-------|----------|--------|-----|
| [Issue 1] | [P1/P2/P3] | [In Progress / Pending] | [Date] |
---
## 5. Success Plan Progress
### Current Success Plan Goals
| Goal | Timeline | Progress | Status |
|------|----------|----------|--------|
| [Goal 1] | [Date] | [%] | [On Track / At Risk / Complete] |
| [Goal 2] | [Date] | [%] | [On Track / At Risk / Complete] |
| [Goal 3] | [Date] | [%] | [On Track / At Risk / Complete] |
### Next Quarter Goals (Proposed)
1. [Goal 1 with specific measurable outcome]
2. [Goal 2 with specific measurable outcome]
3. [Goal 3 with specific measurable outcome]
---
## 6. Product Roadmap Highlights
### Recently Released (Relevant to [Customer Name])
- [Feature/enhancement 1] -- [How it benefits them]
- [Feature/enhancement 2] -- [How it benefits them]
### Coming Next Quarter
- [Upcoming feature 1] -- [Expected benefit]
- [Upcoming feature 2] -- [Expected benefit]
### Feature Requests Status
| Request | Priority | Status | Expected Release |
|---------|----------|--------|-----------------|
| [Request 1] | [High/Med/Low] | [Planned / In Development / Under Review] | [Quarter] |
---
## 7. Growth Opportunities
### Expansion Discussion Points
- [Opportunity 1: e.g., additional seats for new team]
- [Opportunity 2: e.g., new module that addresses identified need]
- [Opportunity 3: e.g., tier upgrade for advanced capabilities]
### Estimated Value of Expansion: $[Amount] additional ARR
---
## 8. Action Items
| Action | Owner | Due Date | Priority |
|--------|-------|----------|----------|
| [Action 1] | [Name] | [Date] | [High/Med/Low] |
| [Action 2] | [Name] | [Date] | [High/Med/Low] |
| [Action 3] | [Name] | [Date] | [High/Med/Low] |
| [Action 4] | [Name] | [Date] | [High/Med/Low] |
---
## 9. Contract and Renewal
**Contract Start:** [Date]
**Renewal Date:** [Date]
**Current ARR:** $[Amount]
**Days to Renewal:** [Number]
### Renewal Readiness
- [ ] Value documented and communicated
- [ ] Executive sponsor aligned
- [ ] Open issues resolved or plan in place
- [ ] Pricing and terms discussed
- [ ] Expansion proposal prepared (if applicable)
---
**Next QBR Date:** [Date]
**Next Check-in:** [Date]
FILE:assets/sample_customer_data.json
{
"customers": [
{
"customer_id": "CUST-001",
"name": "Acme Corp",
"segment": "enterprise",
"arr": 120000,
"contract_end_date": "2026-12-31",
"usage": {
"login_frequency": 85,
"feature_adoption": 72,
"dau_mau_ratio": 0.45
},
"engagement": {
"support_ticket_volume": 3,
"meeting_attendance": 90,
"nps_score": 8,
"csat_score": 4.2
},
"support": {
"open_tickets": 2,
"escalation_rate": 0.05,
"avg_resolution_hours": 18
},
"relationship": {
"executive_sponsor_engagement": 80,
"multi_threading_depth": 4,
"renewal_sentiment": "positive"
},
"previous_period": {
"usage_score": 70,
"engagement_score": 65,
"support_score": 75,
"relationship_score": 60,
"overall_score": 67
},
"usage_decline": {
"login_trend": 5,
"feature_adoption_change": 3,
"dau_mau_change": 0.02
},
"engagement_drop": {
"meeting_cancellations": 0,
"response_time_days": 1,
"nps_change": 1
},
"support_issues": {
"open_escalations": 0,
"unresolved_critical": 0,
"satisfaction_trend": "improving"
},
"relationship_signals": {
"champion_left": false,
"sponsor_change": false,
"competitor_mentions": 0
},
"commercial_factors": {
"contract_type": "annual",
"pricing_complaints": false,
"budget_cuts_mentioned": false
},
"contract": {
"licensed_seats": 100,
"active_seats": 95,
"plan_tier": "professional",
"available_tiers": ["professional", "enterprise", "enterprise_plus"]
},
"product_usage": {
"core_platform": {"adopted": true, "usage_pct": 85},
"analytics_module": {"adopted": true, "usage_pct": 60},
"integrations_module": {"adopted": false, "usage_pct": 0},
"api_access": {"adopted": true, "usage_pct": 40},
"advanced_reporting": {"adopted": false, "usage_pct": 0}
},
"departments": {
"current": ["engineering", "product"],
"potential": ["marketing", "sales", "support"]
}
},
{
"customer_id": "CUST-002",
"name": "TechStart Inc",
"segment": "smb",
"arr": 18000,
"contract_end_date": "2026-04-15",
"usage": {
"login_frequency": 40,
"feature_adoption": 30,
"dau_mau_ratio": 0.15
},
"engagement": {
"support_ticket_volume": 8,
"meeting_attendance": 50,
"nps_score": 5,
"csat_score": 3.0
},
"support": {
"open_tickets": 6,
"escalation_rate": 0.18,
"avg_resolution_hours": 42
},
"relationship": {
"executive_sponsor_engagement": 30,
"multi_threading_depth": 1,
"renewal_sentiment": "negative"
},
"previous_period": {
"usage_score": 55,
"engagement_score": 50,
"support_score": 60,
"relationship_score": 45,
"overall_score": 52
},
"usage_decline": {
"login_trend": -25,
"feature_adoption_change": -18,
"dau_mau_change": -0.12
},
"engagement_drop": {
"meeting_cancellations": 3,
"response_time_days": 8,
"nps_change": -4
},
"support_issues": {
"open_escalations": 2,
"unresolved_critical": 1,
"satisfaction_trend": "declining"
},
"relationship_signals": {
"champion_left": true,
"sponsor_change": false,
"competitor_mentions": 3
},
"commercial_factors": {
"contract_type": "month-to-month",
"pricing_complaints": true,
"budget_cuts_mentioned": true
},
"contract": {
"licensed_seats": 20,
"active_seats": 8,
"plan_tier": "starter",
"available_tiers": ["starter", "professional", "enterprise"]
},
"product_usage": {
"core_platform": {"adopted": true, "usage_pct": 35},
"analytics_module": {"adopted": false, "usage_pct": 0},
"integrations_module": {"adopted": false, "usage_pct": 0},
"api_access": {"adopted": false, "usage_pct": 0},
"advanced_reporting": {"adopted": false, "usage_pct": 0}
},
"departments": {
"current": ["engineering"],
"potential": ["product", "design"]
}
},
{
"customer_id": "CUST-003",
"name": "GlobalTrade Solutions",
"segment": "mid-market",
"arr": 55000,
"contract_end_date": "2026-09-30",
"usage": {
"login_frequency": 70,
"feature_adoption": 58,
"dau_mau_ratio": 0.35
},
"engagement": {
"support_ticket_volume": 5,
"meeting_attendance": 75,
"nps_score": 7,
"csat_score": 3.8
},
"support": {
"open_tickets": 3,
"escalation_rate": 0.10,
"avg_resolution_hours": 30
},
"relationship": {
"executive_sponsor_engagement": 60,
"multi_threading_depth": 3,
"renewal_sentiment": "neutral"
},
"previous_period": {
"usage_score": 68,
"engagement_score": 70,
"support_score": 65,
"relationship_score": 62,
"overall_score": 66
},
"usage_decline": {
"login_trend": -8,
"feature_adoption_change": -5,
"dau_mau_change": -0.03
},
"engagement_drop": {
"meeting_cancellations": 1,
"response_time_days": 3,
"nps_change": -1
},
"support_issues": {
"open_escalations": 1,
"unresolved_critical": 0,
"satisfaction_trend": "stable"
},
"relationship_signals": {
"champion_left": false,
"sponsor_change": true,
"competitor_mentions": 1
},
"commercial_factors": {
"contract_type": "annual",
"pricing_complaints": false,
"budget_cuts_mentioned": false
},
"contract": {
"licensed_seats": 50,
"active_seats": 48,
"plan_tier": "professional",
"available_tiers": ["professional", "enterprise", "enterprise_plus"]
},
"product_usage": {
"core_platform": {"adopted": true, "usage_pct": 78},
"analytics_module": {"adopted": true, "usage_pct": 45},
"integrations_module": {"adopted": true, "usage_pct": 55},
"api_access": {"adopted": false, "usage_pct": 0},
"advanced_reporting": {"adopted": false, "usage_pct": 0}
},
"departments": {
"current": ["operations", "finance"],
"potential": ["logistics", "compliance"]
}
},
{
"customer_id": "CUST-004",
"name": "HealthFirst Medical",
"segment": "enterprise",
"arr": 200000,
"contract_end_date": "2027-03-15",
"usage": {
"login_frequency": 92,
"feature_adoption": 88,
"dau_mau_ratio": 0.55
},
"engagement": {
"support_ticket_volume": 2,
"meeting_attendance": 95,
"nps_score": 9,
"csat_score": 4.6
},
"support": {
"open_tickets": 1,
"escalation_rate": 0.02,
"avg_resolution_hours": 12
},
"relationship": {
"executive_sponsor_engagement": 92,
"multi_threading_depth": 6,
"renewal_sentiment": "positive"
},
"previous_period": {
"usage_score": 85,
"engagement_score": 82,
"support_score": 88,
"relationship_score": 80,
"overall_score": 84
},
"usage_decline": {
"login_trend": 3,
"feature_adoption_change": 5,
"dau_mau_change": 0.03
},
"engagement_drop": {
"meeting_cancellations": 0,
"response_time_days": 1,
"nps_change": 0
},
"support_issues": {
"open_escalations": 0,
"unresolved_critical": 0,
"satisfaction_trend": "improving"
},
"relationship_signals": {
"champion_left": false,
"sponsor_change": false,
"competitor_mentions": 0
},
"commercial_factors": {
"contract_type": "multi-year",
"pricing_complaints": false,
"budget_cuts_mentioned": false
},
"contract": {
"licensed_seats": 250,
"active_seats": 240,
"plan_tier": "enterprise",
"available_tiers": ["professional", "enterprise", "enterprise_plus"]
},
"product_usage": {
"core_platform": {"adopted": true, "usage_pct": 92},
"analytics_module": {"adopted": true, "usage_pct": 80},
"integrations_module": {"adopted": true, "usage_pct": 70},
"api_access": {"adopted": true, "usage_pct": 65},
"advanced_reporting": {"adopted": true, "usage_pct": 50},
"security_module": {"adopted": false, "usage_pct": 0},
"audit_module": {"adopted": false, "usage_pct": 0}
},
"departments": {
"current": ["clinical", "operations", "IT", "compliance"],
"potential": ["research", "finance", "HR"]
}
}
]
}
FILE:assets/success_plan_template.md
# Customer Success Plan
**Customer:** [Customer Name]
**CSM:** [CSM Name]
**Account Executive:** [AE Name]
**Plan Created:** [Date]
**Last Updated:** [Date]
**Review Cadence:** [Monthly / Quarterly]
---
## 1. Customer Overview
| Field | Details |
|-------|---------|
| Industry | [Industry] |
| Company Size | [Employees] |
| Segment | [Enterprise / Mid-Market / SMB] |
| ARR | $[Amount] |
| Contract Start | [Date] |
| Renewal Date | [Date] |
| Plan Tier | [Tier name] |
| Licensed Seats | [Number] |
### Key Stakeholders
| Name | Title | Role | Engagement Level |
|------|-------|------|-----------------|
| [Name] | [Title] | Executive Sponsor | [High / Medium / Low] |
| [Name] | [Title] | Day-to-Day Champion | [High / Medium / Low] |
| [Name] | [Title] | Technical Lead | [High / Medium / Low] |
| [Name] | [Title] | End User Lead | [High / Medium / Low] |
---
## 2. Business Objectives
### Primary Business Objectives
| # | Objective | Success Metric | Target | Timeline |
|---|-----------|---------------|--------|----------|
| 1 | [e.g., Reduce manual reporting time] | [Hours saved per week] | [Target number] | [Date] |
| 2 | [e.g., Improve team collaboration] | [Project completion rate] | [Target %] | [Date] |
| 3 | [e.g., Increase revenue visibility] | [Forecast accuracy] | [Target %] | [Date] |
### Why These Objectives Matter
- **Objective 1:** [Business context -- why this matters to the customer's overall strategy]
- **Objective 2:** [Business context]
- **Objective 3:** [Business context]
---
## 3. Success Milestones
### Phase 1: Foundation (Days 1-30)
| Milestone | Target Date | Status | Owner | Notes |
|-----------|------------|--------|-------|-------|
| Technical setup complete | [Date] | [ ] | [Name] | |
| Admin training delivered | [Date] | [ ] | CSM | |
| Core team onboarded | [Date] | [ ] | CSM | |
| First value milestone achieved | [Date] | [ ] | [Name] | |
| Data migration validated | [Date] | [ ] | SE | |
### Phase 2: Adoption (Days 31-90)
| Milestone | Target Date | Status | Owner | Notes |
|-----------|------------|--------|-------|-------|
| 80% user adoption | [Date] | [ ] | CSM | |
| Key workflows live | [Date] | [ ] | [Name] | |
| Integrations configured | [Date] | [ ] | SE | |
| First ROI measurement | [Date] | [ ] | CSM | |
| 30-day review complete | [Date] | [ ] | CSM | |
### Phase 3: Value Realisation (Days 91-180)
| Milestone | Target Date | Status | Owner | Notes |
|-----------|------------|--------|-------|-------|
| Objective 1 progress measurable | [Date] | [ ] | [Name] | |
| Advanced features adopted | [Date] | [ ] | CSM | |
| QBR completed | [Date] | [ ] | CSM | |
| Executive alignment confirmed | [Date] | [ ] | CSM | |
### Phase 4: Optimisation and Growth (Days 181-365)
| Milestone | Target Date | Status | Owner | Notes |
|-----------|------------|--------|-------|-------|
| All objectives on track | [Date] | [ ] | CSM | |
| ROI documented for renewal | [Date] | [ ] | CSM | |
| Expansion opportunities identified | [Date] | [ ] | CSM + AE | |
| Renewal conversation initiated | [Date] | [ ] | CSM + AE | |
---
## 4. Health Score Tracking
| Date | Overall Score | Usage | Engagement | Support | Relationship | Classification |
|------|--------------|-------|------------|---------|-------------|---------------|
| [Date] | [Score] | [Score] | [Score] | [Score] | [Score] | [Green/Yellow/Red] |
| [Date] | [Score] | [Score] | [Score] | [Score] | [Score] | [Green/Yellow/Red] |
---
## 5. Risk Register
| Risk | Probability | Impact | Mitigation | Owner | Status |
|------|------------|--------|-----------|-------|--------|
| [e.g., Executive sponsor departure] | [High/Med/Low] | [High/Med/Low] | [Multi-thread relationships] | CSM | [Active/Resolved] |
| [e.g., Low adoption in team X] | [High/Med/Low] | [High/Med/Low] | [Targeted training session] | CSM | [Active/Resolved] |
| [e.g., Budget review next quarter] | [High/Med/Low] | [High/Med/Low] | [Document ROI before review] | CSM | [Active/Resolved] |
---
## 6. Communication Plan
| Activity | Frequency | Participants | Purpose |
|----------|-----------|-------------|---------|
| Status check-in | [Weekly / Bi-weekly] | CSM + Champion | Tactical progress review |
| Strategic review | [Monthly] | CSM + Stakeholders | Objective alignment |
| QBR | [Quarterly] | CSM + Executive Sponsor | Executive business review |
| Technical review | [As needed] | SE + Technical Lead | Architecture and integration |
| Renewal planning | [90 days before] | CSM + AE + Sponsor | Contract discussion |
---
## 7. Product Adoption Plan
### Current State
| Module/Feature | Status | Usage Level | Target Usage | Gap |
|---------------|--------|-------------|-------------|-----|
| [Module 1] | Adopted | [%] | [%] | [Actions needed] |
| [Module 2] | Adopted | [%] | [%] | [Actions needed] |
| [Module 3] | Not Adopted | 0% | [%] | [Enablement plan] |
### Enablement Activities
| Activity | Target Date | Audience | Expected Outcome |
|----------|------------|----------|-----------------|
| [Training session] | [Date] | [Team/Group] | [Metric improvement] |
| [Workshop] | [Date] | [Team/Group] | [New workflow adoption] |
| [Office hours] | [Ongoing] | [All users] | [Question resolution] |
---
## 8. Expansion Roadmap
| Opportunity | Type | Estimated Value | Timeline | Prerequisites |
|------------|------|----------------|----------|--------------|
| [e.g., Additional seats] | Expansion | $[Amount] | [Quarter] | [Usage > 90%] |
| [e.g., Tier upgrade] | Upsell | $[Amount] | [Quarter] | [Feature requests] |
| [e.g., New module] | Cross-sell | $[Amount] | [Quarter] | [Use case validated] |
---
## 9. Notes and Updates
### [Date] - [Author]
[Update notes, key decisions, changes to plan]
### [Date] - [Author]
[Update notes, key decisions, changes to plan]
---
**Next Review Date:** [Date]
**Plan Owner:** [CSM Name]
FILE:references/cs-metrics-benchmarks.md
# Customer Success Metrics and Benchmarks
Industry benchmarks for key customer success metrics, segmented by company size, customer segment, and industry vertical.
---
## Core SaaS Metrics
### Net Revenue Retention (NRR)
NRR measures revenue retained from existing customers including expansion, contraction, and churn. It is the single most important metric for SaaS customer success.
**Formula:** (Starting ARR + Expansion - Contraction - Churn) / Starting ARR * 100
| Performance Level | NRR Range | Interpretation |
|-------------------|-----------|----------------|
| Best-in-class | > 130% | Strong expansion engine, very low churn |
| Excellent | 120-130% | Healthy growth from existing customers |
| Good | 110-120% | Solid retention with moderate expansion |
| Target | > 110% | Minimum for sustainable growth |
| Acceptable | 100-110% | Revenue stable but limited expansion |
| Below target | 90-100% | Churn exceeds expansion |
| Concerning | < 90% | Significant revenue erosion |
**Benchmarks by Segment:**
| Customer Segment | Median NRR | Top Quartile | Bottom Quartile |
|-----------------|------------|--------------|-----------------|
| Enterprise (>$100K ARR) | 115% | 130%+ | 105% |
| Mid-Market ($25K-$100K) | 108% | 120% | 98% |
| SMB (<$25K ARR) | 95% | 105% | 85% |
### Gross Revenue Retention (GRR)
GRR measures revenue retained without counting expansion. It isolates the churn and contraction signal.
**Formula:** (Starting ARR - Contraction - Churn) / Starting ARR * 100
| Performance Level | GRR Range | Interpretation |
|-------------------|-----------|----------------|
| Best-in-class | > 95% | Minimal churn, highly sticky product |
| Excellent | 92-95% | Strong retention |
| Good | 90-92% | Healthy with room to improve |
| Target | > 90% | Industry standard target |
| Acceptable | 85-90% | Moderate churn, needs focus |
| Below target | 80-85% | High churn impacting growth |
| Concerning | < 80% | Urgent retention problem |
**Benchmarks by Segment:**
| Customer Segment | Median GRR | Top Quartile | Bottom Quartile |
|-----------------|------------|--------------|-----------------|
| Enterprise | 95% | 98% | 90% |
| Mid-Market | 90% | 95% | 85% |
| SMB | 82% | 90% | 75% |
---
## Health Score Benchmarks
### Portfolio Health Distribution (Target)
A healthy CS portfolio should have the following approximate distribution:
| Classification | Target Distribution | Alert Threshold |
|---------------|-------------------|-----------------|
| Green (Healthy) | 60-70% | < 50% triggers portfolio review |
| Yellow (Attention) | 20-30% | > 35% signals systemic issues |
| Red (At Risk) | 5-10% | > 15% requires executive intervention |
### Average Health Score by Segment
| Segment | Target Average | Industry Median | Top Quartile |
|---------|---------------|-----------------|--------------|
| Enterprise | > 78 | 72 | 82 |
| Mid-Market | > 75 | 68 | 78 |
| SMB | > 70 | 65 | 75 |
### Health Score by Dimension (Industry Medians)
| Dimension | Enterprise | Mid-Market | SMB |
|-----------|-----------|------------|-----|
| Usage | 72 | 68 | 60 |
| Engagement | 70 | 62 | 55 |
| Support | 78 | 72 | 65 |
| Relationship | 68 | 60 | 50 |
---
## Churn Metrics
### Logo Churn Rate (Annual)
| Performance Level | Rate | Interpretation |
|-------------------|------|----------------|
| Best-in-class | < 5% | Exceptional retention |
| Excellent | 5-8% | Very strong |
| Good | 8-12% | Healthy |
| Acceptable | 12-15% | Room for improvement |
| Below target | 15-20% | Significant churn problem |
| Concerning | > 20% | Urgent -- product-market fit issues likely |
**Benchmarks by Segment:**
| Segment | Median Annual Logo Churn | Top Quartile | Bottom Quartile |
|---------|------------------------|--------------|-----------------|
| Enterprise | 5% | 2% | 10% |
| Mid-Market | 10% | 5% | 18% |
| SMB | 20% | 12% | 35% |
### Churn Leading Indicators
The following metrics have the highest predictive power for churn events:
| Indicator | Lead Time | Correlation with Churn |
|-----------|-----------|----------------------|
| Login frequency decline (>30%) | 60-90 days | Very High |
| NPS drop (>3 points) | 30-60 days | High |
| Executive sponsor departure | 30-90 days | Very High |
| Support escalation rate increase | 30-60 days | High |
| Meeting cancellation increase | 30-45 days | Moderate-High |
| Feature adoption decline | 60-90 days | Moderate |
| Competitor mentions | 30-60 days | Moderate |
---
## Expansion Metrics
### Expansion Revenue Rate
| Performance Level | Rate | Notes |
|-------------------|------|-------|
| Best-in-class | > 30% of total revenue | Strong land-and-expand motion |
| Excellent | 25-30% | Effective expansion engine |
| Good | 20-25% | Solid upsell/cross-sell |
| Target | > 20% | Minimum for healthy growth |
| Below target | 10-20% | Expansion motion needs development |
| Concerning | < 10% | Missing significant expansion opportunity |
### Expansion by Type
| Expansion Type | Typical Contribution | Average Deal Size |
|---------------|---------------------|-------------------|
| Seat Expansion | 40-50% of expansion | 15-25% of contract value |
| Tier Upsell | 25-35% of expansion | 40-80% of contract value |
| Module Cross-sell | 15-25% of expansion | 10-20% of contract value |
| Department Expansion | 5-15% of expansion | 50-100% of contract value |
### Expansion Readiness Indicators
| Signal | Interpretation |
|--------|---------------|
| Seat utilisation > 90% | Ready for seat expansion |
| Feature requests for higher tier | Upsell opportunity |
| Usage of 70%+ of current modules | Ready for cross-sell |
| New department interest | Department expansion play |
| Customer referral activity | Strong relationship, open to expansion |
---
## Engagement Metrics
### Customer Engagement Score (CES) Benchmarks
| Metric | Target | Median | Warning |
|--------|--------|--------|---------|
| Meeting attendance rate | > 80% | 72% | < 50% |
| Average NPS | > 50 | 35 | < 20 |
| Average CSAT | > 4.2/5 | 3.8/5 | < 3.0/5 |
| Response time (days) | < 2 | 3 | > 5 |
| QBR completion rate | > 90% | 75% | < 60% |
### Time to First Value (TTFV)
| Segment | Target TTFV | Median TTFV | Warning Threshold |
|---------|------------|------------|-------------------|
| Enterprise | < 30 days | 45 days | > 60 days |
| Mid-Market | < 21 days | 30 days | > 45 days |
| SMB | < 14 days | 21 days | > 30 days |
---
## CSM Operational Metrics
### Portfolio Management
| Metric | Enterprise CSM | Mid-Market CSM | SMB CSM (Tech-Touch) |
|--------|---------------|----------------|---------------------|
| Accounts per CSM | 10-25 | 30-60 | 100-300+ |
| ARR per CSM | $2M-$5M | $2M-$4M | $1M-$3M |
| Touch frequency | Weekly-biweekly | Biweekly-monthly | Quarterly-automated |
| QBR frequency | Quarterly | Semi-annually | Annually |
| Health score reviews | Weekly | Bi-weekly | Monthly |
### CSM Activity Benchmarks
| Activity | Target per Month | Purpose |
|----------|-----------------|---------|
| Strategic calls | 2-4 per account | Relationship building |
| Health score reviews | 4 (weekly) | Portfolio monitoring |
| QBR preparation | 3-5 per quarter | Executive engagement |
| Escalation handling | < 2 per month | Issue resolution |
| Expansion conversations | 1-2 per account | Revenue growth |
---
## Industry-Specific Benchmarks
### By Industry Vertical
| Industry | Median NRR | Median GRR | Median Logo Churn |
|----------|-----------|-----------|------------------|
| Infrastructure/DevOps | 125% | 95% | 5% |
| Cybersecurity | 120% | 93% | 7% |
| HR Tech | 110% | 90% | 12% |
| MarTech | 105% | 87% | 15% |
| FinTech | 115% | 92% | 8% |
| HealthTech | 112% | 91% | 10% |
| EdTech | 100% | 85% | 18% |
| eCommerce Tools | 108% | 88% | 14% |
### By Company Stage
| Stage | Median NRR | Median GRR | Notes |
|-------|-----------|-----------|-------|
| Early Stage (<$10M ARR) | 100% | 85% | Focus on product-market fit |
| Growth ($10M-$50M ARR) | 110% | 90% | Building CS function |
| Scale ($50M-$200M ARR) | 118% | 93% | Mature CS operations |
| Enterprise (>$200M ARR) | 115% | 95% | Optimisation phase |
---
## Metric Relationships
### Key Correlations
| If This Metric Moves | This Also Tends to Move | Direction |
|---------------------|------------------------|-----------|
| Health score down | Churn probability up | Inverse |
| NPS up | NRR up | Direct |
| TTFV down | GRR up | Inverse |
| Feature adoption up | Expansion rate up | Direct |
| Escalation rate up | NPS down | Inverse |
| Multi-threading depth up | GRR up | Direct |
### The SaaS Retention Equation
**Sustainable Growth requires:** NRR > 110% AND GRR > 90%
If NRR is high but GRR is low: You are churning customers and replacing with expansion from survivors. Not sustainable.
If GRR is high but NRR is low: You retain well but do not expand. Leaving money on the table.
Both high: Healthy, compounding growth from existing customers.
---
**Last Updated:** February 2026
**Sources:** Industry surveys, SaaS benchmarking reports, customer success community data (2024-2025 data cycles).
FILE:references/cs-playbooks.md
# Customer Success Playbooks
Comprehensive intervention, onboarding, renewal, expansion, and escalation playbooks for SaaS customer success management.
---
## Risk Tier Intervention Playbooks
### Critical Risk (Score 80-100)
**Situation:** Customer is at imminent risk of churn. Multiple severe warning signals detected. Requires immediate executive-level intervention.
**Timeline:** Act within 48 hours.
**Steps:**
1. **Executive Escalation (Day 0)**
- Alert VP of Customer Success and account executive immediately
- Brief internal leadership on situation, warning signals, and ARR at risk
- Identify any pending support issues and fast-track resolution
2. **Customer Contact (Day 1-2)**
- Schedule executive-to-executive call (VP CS to customer VP/C-level)
- Frame the conversation around understanding their challenges, not defending your product
- Listen more than talk -- capture the real objections
3. **Save Plan Creation (Day 2-3)**
- Create a detailed save plan with specific value milestones tied to their business outcomes
- Include timeline, owners, and measurable success criteria
- Get internal alignment on any concessions (pricing, features, roadmap commitments)
4. **Rescue Team Assignment (Day 3-5)**
- Assign a dedicated rescue team: CSM + Solutions Engineer + Support Lead
- Daily internal stand-up (15 min max) on account status
- Solutions Engineer to conduct technical health check
5. **Execution and Monitoring (Week 2-4)**
- Execute save plan with weekly customer check-ins
- Track progress against milestones
- Prepare competitive displacement defence if competitor involvement detected
6. **Resolution Assessment (Week 4)**
- Evaluate whether the situation is stabilising
- If improving: transition to High-risk monitoring cadence
- If not improving: escalate to CEO/GM for final intervention
**Success Criteria:** Risk score drops below 60 within 30 days. Customer confirms continued partnership intent.
---
### High Risk (Score 60-79)
**Situation:** Customer showing clear signs of dissatisfaction or disengagement. Still salvageable with focused CSM intervention.
**Timeline:** Act within 1 week.
**Steps:**
1. **Root Cause Analysis (Day 1-3)**
- Review all health score dimensions to identify the primary drivers
- Pull support ticket history for patterns
- Check product usage trends for the past 90 days
2. **CSM Outreach (Day 3-5)**
- Schedule a dedicated call with the customer (not a routine check-in)
- Open with empathy: "I've noticed some changes and want to make sure we're supporting you properly"
- Identify the top 3 customer concerns
3. **30-Day Recovery Plan (Day 5-7)**
- Build a 30-day recovery plan with measurable checkpoints every week
- Include specific actions for each concern identified
- Share the plan with the customer for mutual commitment
4. **Re-Engage Executive Sponsor (Week 2)**
- Request a meeting with the executive sponsor
- Align on business outcomes and how your product supports them
- Confirm continued sponsorship and address any political changes
5. **Support Fast-Track (Ongoing)**
- Escalate any pending support tickets internally
- Assign a support point of contact for this account
- Provide weekly status updates on open issues
6. **Progress Review (Week 3-4)**
- Review all metrics for improvement
- Adjust plan if specific interventions are not working
- If score drops to Critical: escalate to executive playbook
**Success Criteria:** Risk score drops below 40 within 30 days. No new warning signals emerge.
---
### Medium Risk (Score 40-59)
**Situation:** Early warning signs detected. Customer may not be aware of emerging issues. Proactive outreach prevents escalation.
**Timeline:** Act within 2 weeks.
**Steps:**
1. **Data Review (Day 1-5)**
- Analyse which dimension(s) are pulling the score down
- Review recent support interactions for sentiment clues
- Check for any known product issues affecting this customer
2. **Proactive Check-In (Week 1-2)**
- Schedule a "value check-in" call (position it as routine, not reactive)
- Share relevant success stories from similar customers
- Propose a training session or product walkthrough for underutilised features
3. **Value Reinforcement (Week 2-3)**
- Send a customised ROI summary showing value delivered
- Highlight feature releases relevant to their use case
- Connect them with your customer community or user group
4. **Monitoring (Week 3-4)**
- Increase monitoring frequency to bi-weekly
- Watch for improvement or continued decline
- If declining: move to High-risk playbook
**Success Criteria:** Score stabilises above 50 or improves. No escalation to High risk.
---
### Low Risk (Score 0-39)
**Situation:** Customer is healthy. Standard success cadence applies. Focus on value reinforcement and expansion readiness.
**Timeline:** Standard touch cadence.
**Steps:**
1. **Maintain Cadence**
- Enterprise: Monthly strategic reviews, quarterly QBRs
- Mid-Market: Bi-monthly check-ins, semi-annual reviews
- SMB: Quarterly automated health updates, annual review
2. **Proactive Communication**
- Share product updates and release notes
- Invite to webinars, conferences, and community events
- Share relevant industry insights and benchmarks
3. **Expansion Readiness**
- Monitor for expansion signals (usage approaching limits, new use cases)
- Prepare expansion proposals when timing is right
- Position premium features and modules relevant to their needs
4. **Renewal Preparation**
- Begin renewal preparation 90 days before contract end
- Build renewal proposal with value delivered summary
- Identify any terms or pricing adjustments needed
**Success Criteria:** Customer remains in Green classification. Expansion conversations initiated when appropriate.
---
## Onboarding Playbook
### Phase 1: Welcome and Setup (Day 1-14)
| Day | Activity | Owner | Deliverable |
|-----|----------|-------|-------------|
| 1 | Welcome email and introduction | CSM | Welcome package sent |
| 1-2 | Kickoff call | CSM + SE | Success plan drafted |
| 3-5 | Technical setup and configuration | SE | Environment configured |
| 5-7 | Admin training session | CSM | Admins trained |
| 7-10 | Data migration (if applicable) | SE | Data validated |
| 10-14 | Initial user training | CSM | Core team trained |
### Phase 2: Activation (Day 15-30)
| Day | Activity | Owner | Deliverable |
|-----|----------|-------|-------------|
| 15 | Activation check -- are users logging in? | CSM | Usage report |
| 15-20 | Follow-up training for laggards | CSM | All users active |
| 20-25 | First business outcome milestone | CSM | Milestone achieved |
| 25-30 | 30-day review call | CSM | Review documented |
**Critical Milestone:** Time to First Value must be under 30 days.
### Phase 3: Adoption (Day 31-60)
| Day | Activity | Owner | Deliverable |
|-----|----------|-------|-------------|
| 30-40 | Feature adoption expansion | CSM | New features in use |
| 40-50 | Integration setup (if applicable) | SE | Integrations live |
| 50-60 | Usage benchmarking vs. peers | CSM | Benchmark report |
### Phase 4: Optimisation (Day 61-90)
| Day | Activity | Owner | Deliverable |
|-----|----------|-------|-------------|
| 60-70 | Advanced use case workshop | CSM + SE | New use cases identified |
| 70-80 | ROI measurement | CSM | ROI documented |
| 80-90 | 90-day executive review | CSM | Transition to steady-state |
**Gate:** Handoff from onboarding to ongoing CSM management. Health score must be Yellow or better.
---
## Renewal Playbook
### 120 Days Before Renewal
- Review contract terms and pricing
- Assess current health score and trajectory
- Identify any outstanding issues or concerns
- Begin internal alignment on renewal strategy
### 90 Days Before Renewal
- Schedule renewal conversation with customer
- Prepare value delivered summary (ROI, usage stats, milestones achieved)
- Draft renewal proposal with recommended terms
- If at-risk: escalate and begin risk mitigation
### 60 Days Before Renewal
- Present renewal proposal to customer
- Negotiate terms if needed
- Address any concerns raised during the process
- Escalate blockers to leadership
### 30 Days Before Renewal
- Finalise contract terms
- Obtain signatures
- Plan for any post-renewal actions (expansion, migration)
- Update CRM with renewal details
### Post-Renewal
- Confirm renewed contract in systems
- Send thank-you and updated success plan
- Schedule next QBR
- Identify expansion opportunities
---
## Expansion Playbook
### Identifying Expansion Signals
| Signal | Expansion Type | Priority |
|--------|---------------|----------|
| Seat utilisation > 90% | Seat expansion | High |
| Requests for features in higher tier | Tier upsell | High |
| New department inquiries | Department expansion | Medium |
| High adoption of existing modules | Module cross-sell | Medium |
| Customer referencing competitors for missing features | Cross-sell | High |
### Expansion Conversation Framework
1. **Discovery:** "I noticed your team has been getting great value from [feature]. Have you considered how [new module] could help with [related business outcome]?"
2. **Value Framing:** "Companies similar to yours who adopted [module] saw [specific metric improvement]."
3. **Proposal:** "Based on your current usage, here's what the expansion would look like..."
4. **Stakeholder Alignment:** Involve the economic buyer early. The champion can advocate, but the budget holder decides.
5. **Close:** Coordinate with sales/account executive for commercial negotiation.
---
## Escalation Procedures
### Internal Escalation Matrix
| Trigger | Escalation Level | Response Time |
|---------|-----------------|---------------|
| Health score drops to Red | VP Customer Success | 24 hours |
| Executive sponsor leaves | Director CS + AE | 48 hours |
| Critical bug affecting customer | VP Engineering + VP CS | 4 hours |
| Customer mentions competitor evaluation | VP CS + VP Sales | 24 hours |
| Renewal at risk (60 days or less) | CRO/VP Sales | 24 hours |
| Customer threatens legal action | Legal + VP CS | Immediate |
### Escalation Communication Template
**Subject:** [ESCALATION] {Customer Name} -- {Brief Description}
**Body:**
- Customer: {name}, {segment}, ARR
- Health Score: {score} ({classification})
- Renewal Date: {date}
- Issue Summary: {2-3 sentences}
- Warning Signals: {list}
- Recommended Action: {specific next step}
- Urgency: {critical/high/medium}
---
**Last Updated:** February 2026
FILE:references/health-scoring-framework.md
# Health Scoring Framework
Complete methodology for multi-dimensional customer health scoring in SaaS customer success.
---
## Overview
Customer health scoring is the foundation of proactive customer success management. A well-calibrated health score enables CSMs to prioritise their portfolio, identify emerging risks before they become churn events, and allocate resources where they will have the greatest impact.
This framework uses a weighted, multi-dimensional approach that scores customers across four key areas: usage, engagement, support, and relationship. Each dimension contributes to an overall health score (0-100) that classifies accounts as Green (healthy), Yellow (needs attention), or Red (at risk).
---
## Scoring Dimensions
### 1. Usage (Weight: 30%)
Usage metrics are the strongest leading indicator of customer health. Customers who are not using the product are not deriving value and are at elevated churn risk.
| Metric | Definition | Scoring Method |
|--------|-----------|----------------|
| Login Frequency | Percentage of expected login days with actual logins | (actual / target) * 100, capped at 100 |
| Feature Adoption | Percentage of available features actively used | (adopted / available) * 100, capped at 100 |
| DAU/MAU Ratio | Daily active users divided by monthly active users | (actual / target) * 100, capped at 100 |
**Sub-weights within Usage:**
- Login Frequency: 35%
- Feature Adoption: 40%
- DAU/MAU Ratio: 25%
**Why 30% weight:** Usage is the most objective, data-driven signal. Declining usage almost always precedes churn. However, some customers may have seasonal usage patterns, which is why it is not weighted even higher.
### 2. Engagement (Weight: 25%)
Engagement measures how actively the customer participates in the relationship beyond just product usage.
| Metric | Definition | Scoring Method |
|--------|-----------|----------------|
| Support Ticket Volume | Number of support tickets in the period | Inverse score: (1 - actual/max) * 100 |
| Meeting Attendance | Percentage of scheduled meetings attended | (actual / target) * 100, capped at 100 |
| NPS Score | Net Promoter Score response (0-10) | (actual / target) * 100, capped at 100 |
| CSAT Score | Customer Satisfaction score (1-5) | (actual / target) * 100, capped at 100 |
**Sub-weights within Engagement:**
- Support Ticket Volume: 20% (inverse -- fewer tickets is better)
- Meeting Attendance: 30%
- NPS Score: 25%
- CSAT Score: 25%
**Why 25% weight:** Engagement signals complement usage data. A customer who attends meetings but does not use the product may be in an evaluation phase. A customer who uses the product but skips meetings may be becoming self-sufficient -- or disengaging.
### 3. Support (Weight: 20%)
Support health measures the quality of the customer's support experience, which directly impacts satisfaction and renewal likelihood.
| Metric | Definition | Scoring Method |
|--------|-----------|----------------|
| Open Tickets | Number of currently unresolved tickets | Inverse score: (1 - actual/max) * 100 |
| Escalation Rate | Percentage of tickets escalated | Inverse score: (1 - actual/max) * 100 |
| Avg Resolution Time | Average hours to resolve tickets | Inverse score: (1 - actual/max) * 100 |
**Sub-weights within Support:**
- Open Tickets: 35%
- Escalation Rate: 35%
- Resolution Time: 30%
**Why 20% weight:** Support issues are lagging indicators -- they tell you there is already a problem. However, unresolved support issues are a strong predictor of churn, especially when combined with declining engagement.
### 4. Relationship (Weight: 25%)
Relationship health measures the strength and depth of the human connection between the customer and your organisation.
| Metric | Definition | Scoring Method |
|--------|-----------|----------------|
| Executive Sponsor Engagement | Engagement level of exec sponsor (0-100) | (actual / target) * 100, capped at 100 |
| Multi-Threading Depth | Number of stakeholder contacts | (actual / target) * 100, capped at 100 |
| Renewal Sentiment | Qualitative sentiment assessment | Mapped to score: positive=100, neutral=60, negative=20, unknown=50 |
**Sub-weights within Relationship:**
- Executive Sponsor Engagement: 35%
- Multi-Threading Depth: 30%
- Renewal Sentiment: 35%
**Why 25% weight:** Relationship strength is the most important defence against competitive displacement. A customer with strong relationships will give you more chances to fix problems. A customer with weak relationships may leave without warning.
---
## Classification Thresholds
### Standard Thresholds
| Classification | Score Range | Meaning | Action |
|---------------|-------------|---------|--------|
| Green | 75-100 | Customer is healthy and achieving value | Standard cadence, focus on expansion |
| Yellow | 50-74 | Customer needs attention | Increase touch frequency, investigate root causes |
| Red | 0-49 | Customer is at risk | Immediate intervention, create save plan |
### Segment-Adjusted Thresholds
Enterprise customers typically have higher expectations and more complex deployments, which means a higher bar for "healthy." SMB customers may have simpler use cases and lower engagement expectations.
| Segment | Green Threshold | Yellow Threshold | Red Threshold |
|---------|----------------|------------------|---------------|
| Enterprise | 75-100 | 50-74 | 0-49 |
| Mid-Market | 70-100 | 45-69 | 0-44 |
| SMB | 65-100 | 40-64 | 0-39 |
### Segment-Specific Benchmarks
Each metric target is calibrated per segment. Enterprise customers are expected to have higher login frequency, attendance, and sponsor engagement. SMB customers have lower targets but still meaningful thresholds.
**Example Calibration:**
- Enterprise login frequency target: 90% (high-touch, deeply embedded)
- Mid-Market login frequency target: 80% (balanced engagement)
- SMB login frequency target: 70% (self-serve oriented)
---
## Trend Analysis
A single health score snapshot is useful. A health score trend is actionable.
### Trend Classification
| Trend | Criteria | Implication |
|-------|----------|-------------|
| Improving | Current > Previous by 5+ points | Positive trajectory, reinforce what is working |
| Stable | Within +/- 5 points | Maintain current approach |
| Declining | Current < Previous by 5+ points | Investigate and intervene |
| No Data | No previous period available | Establish baseline |
### Trend Priority Matrix
| Current Score | Trend | Priority |
|--------------|-------|----------|
| Green | Declining | HIGH -- intervene before it drops further |
| Yellow | Declining | CRITICAL -- trajectory leads to Red |
| Yellow | Improving | MEDIUM -- reinforce positive momentum |
| Red | Improving | HIGH -- support the recovery |
| Red | Stable | CRITICAL -- needs new intervention approach |
---
## Calibration Guidelines
### When to Recalibrate
1. **After major product changes**: New features may change what "good usage" looks like
2. **Seasonal patterns**: Some industries have cyclical usage (retail holiday season, fiscal year end)
3. **Portfolio composition changes**: If you add many SMB customers, the overall averages shift
4. **After churn events**: Review whether the health score predicted the churn
### Calibration Process
1. Export health scores for all customers over the past 12 months
2. Identify all churn events in the same period
3. Calculate the average health score of churned customers 90, 60, and 30 days before churn
4. Adjust thresholds so that churned customers would have been classified as Yellow or Red at least 60 days before churn
5. Validate with a holdout set of recent data
### Common Calibration Pitfalls
- **Threshold creep**: Gradually lowering Green thresholds to make the portfolio look healthier
- **Over-weighting lagging indicators**: Support metrics react after the damage is done
- **Ignoring segment differences**: Using one threshold for all segments
- **Sentiment bias**: Over-relying on subjective renewal sentiment
---
## Implementation Checklist
1. Define data sources for each metric (CRM, product analytics, support system)
2. Establish data refresh frequency (daily for usage, weekly for engagement)
3. Configure segment benchmarks for your customer base
4. Set initial thresholds using industry defaults (provided above)
5. Run a 30-day pilot with manual review of edge cases
6. Calibrate thresholds based on pilot results
7. Automate scoring and alerting
8. Review and recalibrate quarterly
---
**Last Updated:** February 2026
FILE:scripts/churn_risk_analyzer.py
#!/usr/bin/env python3
"""
Churn Risk Analyzer
Identifies at-risk customer accounts by scoring behavioral signals across
usage decline, engagement drop, support issues, relationship signals, and
commercial factors. Produces risk tiers with intervention playbooks and
time-to-renewal urgency multipliers.
Usage:
python churn_risk_analyzer.py customer_data.json
python churn_risk_analyzer.py customer_data.json --format json
"""
import argparse
import json
import sys
from datetime import datetime
from typing import Any, Dict, List, Optional, Tuple
# ---------------------------------------------------------------------------
# Constants
# ---------------------------------------------------------------------------
RISK_SIGNAL_WEIGHTS: Dict[str, float] = {
"usage_decline": 0.30,
"engagement_drop": 0.25,
"support_issues": 0.20,
"relationship_signals": 0.15,
"commercial_factors": 0.10,
}
RISK_TIERS: List[Dict[str, Any]] = [
{"name": "critical", "min": 80, "max": 100, "label": "CRITICAL", "action": "Immediate executive escalation"},
{"name": "high", "min": 60, "max": 79, "label": "HIGH", "action": "Urgent CSM intervention"},
{"name": "medium", "min": 40, "max": 59, "label": "MEDIUM", "action": "Proactive outreach"},
{"name": "low", "min": 0, "max": 39, "label": "LOW", "action": "Standard monitoring"},
]
WARNING_SEVERITY: Dict[str, int] = {
"critical": 4,
"high": 3,
"medium": 2,
"low": 1,
}
# Intervention playbooks per tier
INTERVENTION_PLAYBOOKS: Dict[str, List[str]] = {
"critical": [
"Schedule executive-to-executive call within 48 hours",
"Create detailed save plan with specific value milestones",
"Offer concessions or contract restructuring if needed",
"Assign dedicated rescue team (CSM + Solutions Engineer)",
"Daily internal stand-up on account status until stabilised",
"Prepare competitive displacement defence strategy",
],
"high": [
"Schedule urgent CSM call within 1 week",
"Conduct root cause analysis on declining metrics",
"Build 30-day recovery plan with measurable checkpoints",
"Re-engage executive sponsor for alignment meeting",
"Accelerate any pending feature requests or bug fixes",
"Increase touch frequency to weekly until improvement",
],
"medium": [
"Schedule proactive check-in within 2 weeks",
"Share relevant success stories and best practices",
"Propose training session or product walkthrough",
"Review current usage against success plan goals",
"Identify and address any unvoiced concerns",
"Bi-weekly monitoring until score improves to Low",
],
"low": [
"Maintain standard touch cadence",
"Share product updates and new feature announcements",
"Monitor health score trends monthly",
"Proactively share relevant industry insights",
"Prepare for upcoming renewal conversations (if within 90 days)",
],
}
SATISFACTION_TREND_SCORES: Dict[str, float] = {
"improving": 10.0,
"stable": 30.0,
"declining": 70.0,
"critical": 95.0,
}
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def safe_divide(numerator: float, denominator: float, default: float = 0.0) -> float:
"""Return numerator / denominator, or *default* when denominator is zero."""
if denominator == 0:
return default
return numerator / denominator
def clamp(value: float, lo: float = 0.0, hi: float = 100.0) -> float:
"""Clamp *value* between *lo* and *hi*."""
return max(lo, min(hi, value))
def days_until(date_str: Optional[str]) -> Optional[int]:
"""Return days from today until *date_str* (ISO format), or None."""
if not date_str:
return None
try:
target = datetime.strptime(date_str[:10], "%Y-%m-%d")
delta = (target - datetime.now()).days
return max(delta, 0)
except (ValueError, TypeError):
return None
def renewal_urgency_multiplier(days_remaining: Optional[int]) -> float:
"""Return a multiplier (1.0 - 1.5) based on proximity to renewal.
Closer renewals amplify the risk score.
"""
if days_remaining is None:
return 1.0
if days_remaining <= 30:
return 1.5
elif days_remaining <= 60:
return 1.35
elif days_remaining <= 90:
return 1.2
elif days_remaining <= 180:
return 1.1
return 1.0
def get_risk_tier(score: float) -> Dict[str, Any]:
"""Return the risk tier dict matching the score."""
for tier in RISK_TIERS:
if tier["min"] <= score <= tier["max"]:
return tier
return RISK_TIERS[-1] # default to low
# ---------------------------------------------------------------------------
# Signal Scoring
# ---------------------------------------------------------------------------
def score_usage_decline(data: Dict[str, Any]) -> Tuple[float, List[Dict[str, str]]]:
"""Score usage decline signals (0-100, higher = more risk)."""
warnings: List[Dict[str, str]] = []
login_trend = data.get("login_trend", 0) # negative = decline
feature_change = data.get("feature_adoption_change", 0)
dau_mau_change = data.get("dau_mau_change", 0)
# Convert declines to risk scores (0-100)
login_risk = clamp(abs(min(login_trend, 0)) * 3.0) # -33% => 100
feature_risk = clamp(abs(min(feature_change, 0)) * 4.0) # -25% => 100
dau_mau_risk = clamp(abs(min(dau_mau_change, 0)) * 500) # -0.20 => 100
score = round(login_risk * 0.40 + feature_risk * 0.35 + dau_mau_risk * 0.25, 1)
if login_trend <= -20:
warnings.append({"severity": "critical", "signal": f"Login frequency dropped {abs(login_trend)}%"})
elif login_trend <= -10:
warnings.append({"severity": "high", "signal": f"Login frequency declined {abs(login_trend)}%"})
elif login_trend < -5:
warnings.append({"severity": "medium", "signal": f"Login frequency dipping {abs(login_trend)}%"})
if feature_change <= -15:
warnings.append({"severity": "high", "signal": f"Feature adoption dropped {abs(feature_change)}%"})
elif feature_change < -5:
warnings.append({"severity": "medium", "signal": f"Feature adoption declining {abs(feature_change)}%"})
if dau_mau_change <= -0.10:
warnings.append({"severity": "high", "signal": f"DAU/MAU ratio fell by {abs(dau_mau_change):.2f}"})
return score, warnings
def score_engagement_drop(data: Dict[str, Any]) -> Tuple[float, List[Dict[str, str]]]:
"""Score engagement drop signals (0-100, higher = more risk)."""
warnings: List[Dict[str, str]] = []
cancellations = data.get("meeting_cancellations", 0)
response_days = data.get("response_time_days", 1)
nps_change = data.get("nps_change", 0)
cancel_risk = clamp(cancellations * 25.0) # 4 cancellations => 100
response_risk = clamp((response_days - 1) * 15.0) # 1 day baseline; 7+ days => 90+
nps_risk = clamp(abs(min(nps_change, 0)) * 20.0) # -5 => 100
score = round(cancel_risk * 0.30 + response_risk * 0.35 + nps_risk * 0.35, 1)
if cancellations >= 3:
warnings.append({"severity": "critical", "signal": f"{cancellations} meeting cancellations -- customer disengaging"})
elif cancellations >= 2:
warnings.append({"severity": "high", "signal": f"{cancellations} meeting cancellations recently"})
if response_days >= 7:
warnings.append({"severity": "critical", "signal": f"Customer response time: {response_days} days -- going dark"})
elif response_days >= 4:
warnings.append({"severity": "high", "signal": f"Customer response time increasing: {response_days} days"})
if nps_change <= -4:
warnings.append({"severity": "critical", "signal": f"NPS dropped by {abs(nps_change)} points"})
elif nps_change <= -2:
warnings.append({"severity": "high", "signal": f"NPS declined by {abs(nps_change)} points"})
return score, warnings
def score_support_issues(data: Dict[str, Any]) -> Tuple[float, List[Dict[str, str]]]:
"""Score support-related risk signals (0-100, higher = more risk)."""
warnings: List[Dict[str, str]] = []
escalations = data.get("open_escalations", 0)
critical_unresolved = data.get("unresolved_critical", 0)
sat_trend = data.get("satisfaction_trend", "stable").lower()
esc_risk = clamp(escalations * 35.0) # 3 escalations => 100
critical_risk = clamp(critical_unresolved * 50.0) # 2 unresolved critical => 100
sat_risk = SATISFACTION_TREND_SCORES.get(sat_trend, 30.0)
score = round(esc_risk * 0.35 + critical_risk * 0.35 + sat_risk * 0.30, 1)
if critical_unresolved >= 2:
warnings.append({"severity": "critical", "signal": f"{critical_unresolved} unresolved critical support tickets"})
elif critical_unresolved >= 1:
warnings.append({"severity": "high", "signal": "Unresolved critical support ticket"})
if escalations >= 2:
warnings.append({"severity": "high", "signal": f"{escalations} open escalations"})
elif escalations >= 1:
warnings.append({"severity": "medium", "signal": "Open support escalation"})
if sat_trend == "critical":
warnings.append({"severity": "critical", "signal": "Support satisfaction at critical levels"})
elif sat_trend == "declining":
warnings.append({"severity": "high", "signal": "Support satisfaction trending down"})
return score, warnings
def score_relationship_signals(data: Dict[str, Any]) -> Tuple[float, List[Dict[str, str]]]:
"""Score relationship risk signals (0-100, higher = more risk)."""
warnings: List[Dict[str, str]] = []
risk_points = 0.0
champion_left = data.get("champion_left", False)
sponsor_change = data.get("sponsor_change", False)
competitor_mentions = data.get("competitor_mentions", 0)
if champion_left:
risk_points += 45.0
warnings.append({"severity": "critical", "signal": "Internal champion has left the organisation"})
if sponsor_change:
risk_points += 30.0
warnings.append({"severity": "high", "signal": "Executive sponsor change detected"})
if competitor_mentions >= 3:
risk_points += 35.0
warnings.append({"severity": "critical", "signal": f"Customer mentioned competitors {competitor_mentions} times"})
elif competitor_mentions >= 1:
risk_points += competitor_mentions * 12.0
warnings.append({"severity": "medium", "signal": f"Customer mentioned competitor {competitor_mentions} time(s)"})
score = clamp(risk_points)
return round(score, 1), warnings
def score_commercial_factors(data: Dict[str, Any]) -> Tuple[float, List[Dict[str, str]]]:
"""Score commercial risk factors (0-100, higher = more risk)."""
warnings: List[Dict[str, str]] = []
risk_points = 0.0
contract_type = data.get("contract_type", "annual").lower()
pricing_complaints = data.get("pricing_complaints", False)
budget_cuts = data.get("budget_cuts_mentioned", False)
if contract_type == "month-to-month":
risk_points += 30.0
warnings.append({"severity": "medium", "signal": "Month-to-month contract -- low switching cost"})
elif contract_type == "quarterly":
risk_points += 15.0
if pricing_complaints:
risk_points += 35.0
warnings.append({"severity": "high", "signal": "Customer has raised pricing complaints"})
if budget_cuts:
risk_points += 40.0
warnings.append({"severity": "high", "signal": "Customer mentioned budget cuts or cost reduction"})
score = clamp(risk_points)
return round(score, 1), warnings
# ---------------------------------------------------------------------------
# Main Analysis
# ---------------------------------------------------------------------------
def analyse_churn_risk(customer: Dict[str, Any]) -> Dict[str, Any]:
"""Analyse churn risk for a single customer."""
usage_score, usage_warnings = score_usage_decline(customer.get("usage_decline", {}))
engagement_score, engagement_warnings = score_engagement_drop(customer.get("engagement_drop", {}))
support_score, support_warnings = score_support_issues(customer.get("support_issues", {}))
relationship_score, relationship_warnings = score_relationship_signals(customer.get("relationship_signals", {}))
commercial_score, commercial_warnings = score_commercial_factors(customer.get("commercial_factors", {}))
# Weighted raw score
raw_score = (
usage_score * RISK_SIGNAL_WEIGHTS["usage_decline"]
+ engagement_score * RISK_SIGNAL_WEIGHTS["engagement_drop"]
+ support_score * RISK_SIGNAL_WEIGHTS["support_issues"]
+ relationship_score * RISK_SIGNAL_WEIGHTS["relationship_signals"]
+ commercial_score * RISK_SIGNAL_WEIGHTS["commercial_factors"]
)
# Apply renewal urgency multiplier
remaining = days_until(customer.get("contract_end_date"))
multiplier = renewal_urgency_multiplier(remaining)
adjusted_score = clamp(round(raw_score * multiplier, 1))
tier = get_risk_tier(adjusted_score)
# Collect and sort warnings by severity
all_warnings = usage_warnings + engagement_warnings + support_warnings + relationship_warnings + commercial_warnings
all_warnings.sort(key=lambda w: WARNING_SEVERITY.get(w["severity"], 0), reverse=True)
playbook = INTERVENTION_PLAYBOOKS.get(tier["name"], [])
return {
"customer_id": customer.get("customer_id", "unknown"),
"name": customer.get("name", "Unknown"),
"segment": customer.get("segment", "unknown"),
"arr": customer.get("arr", 0),
"risk_score": adjusted_score,
"raw_score": round(raw_score, 1),
"risk_tier": tier["name"],
"risk_label": tier["label"],
"urgency_multiplier": multiplier,
"days_to_renewal": remaining,
"signal_scores": {
"usage_decline": {"score": usage_score, "weight": "30%"},
"engagement_drop": {"score": engagement_score, "weight": "25%"},
"support_issues": {"score": support_score, "weight": "20%"},
"relationship_signals": {"score": relationship_score, "weight": "15%"},
"commercial_factors": {"score": commercial_score, "weight": "10%"},
},
"warning_signals": all_warnings,
"recommended_actions": playbook,
}
# ---------------------------------------------------------------------------
# Output Formatting
# ---------------------------------------------------------------------------
def format_text(results: List[Dict[str, Any]]) -> str:
"""Format results as human-readable text."""
lines: List[str] = []
lines.append("=" * 72)
lines.append("CHURN RISK ANALYSIS REPORT")
lines.append("=" * 72)
lines.append("")
total = len(results)
critical_count = sum(1 for r in results if r["risk_tier"] == "critical")
high_count = sum(1 for r in results if r["risk_tier"] == "high")
medium_count = sum(1 for r in results if r["risk_tier"] == "medium")
low_count = sum(1 for r in results if r["risk_tier"] == "low")
total_arr_at_risk = sum(r["arr"] for r in results if r["risk_tier"] in ("critical", "high"))
lines.append(f"Portfolio Summary: {total} customers analysed")
lines.append(f" Critical Risk: {critical_count}")
lines.append(f" High Risk: {high_count}")
lines.append(f" Medium Risk: {medium_count}")
lines.append(f" Low Risk: {low_count}")
lines.append(f" ARR at Risk (Critical + High): ,.0f")
lines.append("")
# Sort by risk score descending
sorted_results = sorted(results, key=lambda r: r["risk_score"], reverse=True)
for r in sorted_results:
lines.append("-" * 72)
lines.append(f"Customer: {r['name']} ({r['customer_id']})")
lines.append(f"Segment: {r['segment'].title()} | ARR: ,.0f")
renewal_str = f"{r['days_to_renewal']} days" if r["days_to_renewal"] is not None else "N/A"
lines.append(f"Risk Score: {r['risk_score']}/100 [{r['risk_label']}] | Renewal: {renewal_str}")
if r["urgency_multiplier"] > 1.0:
lines.append(f" ** Urgency multiplier applied: {r['urgency_multiplier']}x (renewal approaching)")
lines.append("")
lines.append(" Signal Scores:")
for signal_name, signal_data in r["signal_scores"].items():
display_name = signal_name.replace("_", " ").title()
lines.append(f" {display_name:25s} {signal_data['score']:6.1f}/100 ({signal_data['weight']})")
if r["warning_signals"]:
lines.append("")
lines.append(" Warning Signals:")
for w in r["warning_signals"]:
severity_tag = w["severity"].upper()
lines.append(f" [{severity_tag}] {w['signal']}")
if r["recommended_actions"]:
lines.append("")
lines.append(" Recommended Actions:")
for i, action in enumerate(r["recommended_actions"], 1):
lines.append(f" {i}. {action}")
lines.append("")
lines.append("=" * 72)
return "\n".join(lines)
def format_json(results: List[Dict[str, Any]]) -> str:
"""Format results as JSON."""
total = len(results)
output = {
"report": "churn_risk_analysis",
"summary": {
"total_customers": total,
"critical_count": sum(1 for r in results if r["risk_tier"] == "critical"),
"high_count": sum(1 for r in results if r["risk_tier"] == "high"),
"medium_count": sum(1 for r in results if r["risk_tier"] == "medium"),
"low_count": sum(1 for r in results if r["risk_tier"] == "low"),
"total_arr_at_risk": sum(r["arr"] for r in results if r["risk_tier"] in ("critical", "high")),
},
"customers": sorted(results, key=lambda r: r["risk_score"], reverse=True),
}
return json.dumps(output, indent=2)
# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def main() -> None:
parser = argparse.ArgumentParser(
description="Analyse churn risk with behavioral signal detection and intervention recommendations."
)
parser.add_argument("input_file", help="Path to JSON file containing customer data")
parser.add_argument(
"--format",
choices=["text", "json"],
default="text",
dest="output_format",
help="Output format (default: text)",
)
args = parser.parse_args()
try:
with open(args.input_file, "r") as f:
data = json.load(f)
except FileNotFoundError:
print(f"Error: File not found: {args.input_file}", file=sys.stderr)
sys.exit(1)
except json.JSONDecodeError as e:
print(f"Error: Invalid JSON in {args.input_file}: {e}", file=sys.stderr)
sys.exit(1)
customers = data.get("customers", [])
if not customers:
print("Error: No customer records found in input file.", file=sys.stderr)
sys.exit(1)
results = [analyse_churn_risk(c) for c in customers]
if args.output_format == "json":
print(format_json(results))
else:
print(format_text(results))
if __name__ == "__main__":
main()
FILE:scripts/expansion_opportunity_scorer.py
#!/usr/bin/env python3
"""
Expansion Opportunity Scorer
Analyses customer product adoption depth, maps whitespace for unused
features/products, estimates revenue opportunities, and prioritises
expansion plays by effort vs impact.
Usage:
python expansion_opportunity_scorer.py customer_data.json
python expansion_opportunity_scorer.py customer_data.json --format json
"""
import argparse
import json
import sys
from typing import Any, Dict, List, Optional, Tuple
# ---------------------------------------------------------------------------
# Constants
# ---------------------------------------------------------------------------
# Tier pricing multipliers (relative to current plan price)
TIER_UPLIFT: Dict[str, float] = {
"starter": 1.0,
"professional": 1.8,
"enterprise": 3.0,
"enterprise_plus": 4.5,
}
# Module revenue estimates as a fraction of base ARR
MODULE_REVENUE_FRACTION: Dict[str, float] = {
"core_platform": 0.00, # Already included in base
"analytics_module": 0.15,
"integrations_module": 0.12,
"api_access": 0.10,
"advanced_reporting": 0.18,
"security_module": 0.20,
"automation_module": 0.15,
"collaboration_module": 0.10,
"data_export": 0.08,
"custom_workflows": 0.22,
"sso_module": 0.08,
"audit_module": 0.10,
}
# Effort classification for different expansion types
EFFORT_MAP: Dict[str, str] = {
"upsell_tier": "medium",
"cross_sell_module": "low",
"seat_expansion": "low",
"department_expansion": "high",
}
# Usage thresholds for recommendations
HIGH_USAGE_THRESHOLD = 75 # % usage indicates readiness for more
LOW_ADOPTION_THRESHOLD = 30 # % usage is too low to push expansion there
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def safe_divide(numerator: float, denominator: float, default: float = 0.0) -> float:
"""Return numerator / denominator, or *default* when denominator is zero."""
if denominator == 0:
return default
return numerator / denominator
def clamp(value: float, lo: float = 0.0, hi: float = 100.0) -> float:
"""Clamp *value* between *lo* and *hi*."""
return max(lo, min(hi, value))
def estimate_seat_expansion_revenue(
arr: float, licensed: int, active: int, segment: str
) -> Tuple[float, str]:
"""Estimate revenue from seat expansion.
Returns (estimated_revenue, rationale).
"""
utilisation = safe_divide(active, licensed)
if utilisation >= 0.90:
# Near capacity -- likely needs more seats
growth_factor = {"enterprise": 0.25, "mid-market": 0.20, "smb": 0.15}
factor = growth_factor.get(segment.lower(), 0.15)
revenue = round(arr * factor, 0)
return revenue, f"Seat utilisation at {utilisation:.0%} -- likely needs {int(licensed * factor)} additional seats"
return 0.0, f"Seat utilisation at {utilisation:.0%} -- not yet at expansion threshold"
def estimate_tier_upgrade_revenue(
arr: float, current_tier: str, available_tiers: List[str]
) -> Tuple[float, Optional[str], str]:
"""Estimate revenue from tier upgrade.
Returns (estimated_revenue, target_tier, rationale).
"""
current_mult = TIER_UPLIFT.get(current_tier.lower(), 1.0)
best_revenue = 0.0
best_tier = None
rationale = "Already on highest tier"
for tier in available_tiers:
tier_mult = TIER_UPLIFT.get(tier.lower(), 1.0)
if tier_mult > current_mult:
# Calculate revenue as the incremental ARR from upgrading
base_arr = safe_divide(arr, current_mult)
upgrade_arr = base_arr * tier_mult
incremental = upgrade_arr - arr
if incremental > best_revenue:
# Pick the next tier up (not skip tiers)
if best_tier is None or tier_mult < TIER_UPLIFT.get(best_tier.lower(), 999):
best_revenue = round(incremental, 0)
best_tier = tier
rationale = f"Upgrade from {current_tier} to {tier} adds ,.0f ARR"
return best_revenue, best_tier, rationale
def estimate_module_revenue(
arr: float, product_usage: Dict[str, Dict[str, Any]]
) -> List[Dict[str, Any]]:
"""Identify cross-sell opportunities from unadopted modules.
Returns list of opportunity dicts.
"""
opportunities: List[Dict[str, Any]] = []
for module_name, module_data in product_usage.items():
adopted = module_data.get("adopted", False)
usage_pct = module_data.get("usage_pct", 0)
fraction = MODULE_REVENUE_FRACTION.get(module_name.lower(), 0.10)
if not adopted and fraction > 0:
revenue = round(arr * fraction, 0)
opportunities.append({
"module": module_name,
"type": "cross_sell",
"estimated_revenue": revenue,
"effort": "low",
"rationale": f"Module not adopted -- ,.0f potential ARR",
})
elif adopted and usage_pct < LOW_ADOPTION_THRESHOLD and fraction > 0:
# Already adopted but underutilised -- focus on enablement, not expansion
pass # Skip -- needs enablement, not a sales motion
return opportunities
def estimate_department_expansion_revenue(
arr: float,
current_departments: List[str],
potential_departments: List[str],
segment: str,
) -> List[Dict[str, Any]]:
"""Estimate revenue from expanding to new departments."""
opportunities: List[Dict[str, Any]] = []
current_set = {d.lower() for d in current_departments}
per_dept_estimate = safe_divide(arr, max(len(current_departments), 1))
for dept in potential_departments:
if dept.lower() not in current_set:
# Estimate each new department at the average per-department ARR
revenue = round(per_dept_estimate * 0.8, 0) # Slight discount for new dept
opportunities.append({
"department": dept,
"type": "expansion",
"estimated_revenue": revenue,
"effort": "high",
"rationale": f"Expand to {dept} department -- est. ,.0f ARR",
})
return opportunities
# ---------------------------------------------------------------------------
# Priority Scoring
# ---------------------------------------------------------------------------
def priority_score(revenue: float, effort: str) -> float:
"""Calculate priority score (higher = better).
Favours high revenue with low effort.
"""
effort_multiplier = {"low": 3.0, "medium": 2.0, "high": 1.0}
mult = effort_multiplier.get(effort.lower(), 1.0)
# Normalise revenue to a 0-100 scale (assume max single opportunity is $200k)
rev_score = clamp(safe_divide(revenue, 2000.0)) # $200k => 100
return round(rev_score * mult, 1)
# ---------------------------------------------------------------------------
# Main Analysis
# ---------------------------------------------------------------------------
def analyse_expansion(customer: Dict[str, Any]) -> Dict[str, Any]:
"""Analyse expansion opportunities for a single customer."""
arr = customer.get("arr", 0)
segment = customer.get("segment", "mid-market").lower()
contract = customer.get("contract", {})
product_usage = customer.get("product_usage", {})
departments = customer.get("departments", {})
all_opportunities: List[Dict[str, Any]] = []
# 1. Seat expansion
licensed = contract.get("licensed_seats", 0)
active = contract.get("active_seats", 0)
seat_rev, seat_rationale = estimate_seat_expansion_revenue(arr, licensed, active, segment)
if seat_rev > 0:
all_opportunities.append({
"type": "expansion",
"category": "seat_expansion",
"estimated_revenue": seat_rev,
"effort": "low",
"rationale": seat_rationale,
"priority_score": priority_score(seat_rev, "low"),
})
# 2. Tier upgrade
current_tier = contract.get("plan_tier", "").lower()
available_tiers = contract.get("available_tiers", [])
tier_rev, target_tier, tier_rationale = estimate_tier_upgrade_revenue(arr, current_tier, available_tiers)
if tier_rev > 0 and target_tier:
all_opportunities.append({
"type": "upsell",
"category": "tier_upgrade",
"target_tier": target_tier,
"estimated_revenue": tier_rev,
"effort": "medium",
"rationale": tier_rationale,
"priority_score": priority_score(tier_rev, "medium"),
})
# 3. Module cross-sell
module_opps = estimate_module_revenue(arr, product_usage)
for opp in module_opps:
opp["category"] = "module_cross_sell"
opp["priority_score"] = priority_score(opp["estimated_revenue"], opp["effort"])
all_opportunities.append(opp)
# 4. Department expansion
current_depts = departments.get("current", [])
potential_depts = departments.get("potential", [])
dept_opps = estimate_department_expansion_revenue(arr, current_depts, potential_depts, segment)
for opp in dept_opps:
opp["category"] = "department_expansion"
opp["priority_score"] = priority_score(opp["estimated_revenue"], opp["effort"])
all_opportunities.append(opp)
# Sort by priority score descending
all_opportunities.sort(key=lambda o: o["priority_score"], reverse=True)
# Adoption depth summary
total_modules = len(product_usage)
adopted_modules = sum(1 for m in product_usage.values() if m.get("adopted", False))
avg_usage = round(
safe_divide(
sum(m.get("usage_pct", 0) for m in product_usage.values() if m.get("adopted", False)),
max(adopted_modules, 1),
),
1,
)
total_estimated_revenue = sum(o["estimated_revenue"] for o in all_opportunities)
return {
"customer_id": customer.get("customer_id", "unknown"),
"name": customer.get("name", "Unknown"),
"segment": segment,
"arr": arr,
"adoption_summary": {
"total_modules": total_modules,
"adopted_modules": adopted_modules,
"adoption_rate": round(safe_divide(adopted_modules, total_modules) * 100, 1) if total_modules > 0 else 0,
"avg_usage_pct": avg_usage,
"seat_utilisation": round(safe_divide(active, max(licensed, 1)) * 100, 1),
"current_tier": current_tier,
"departments_covered": len(current_depts),
"departments_potential": len(potential_depts),
},
"total_estimated_revenue": round(total_estimated_revenue, 0),
"opportunity_count": len(all_opportunities),
"opportunities": all_opportunities,
}
# ---------------------------------------------------------------------------
# Output Formatting
# ---------------------------------------------------------------------------
def format_text(results: List[Dict[str, Any]]) -> str:
"""Format results as human-readable text."""
lines: List[str] = []
lines.append("=" * 72)
lines.append("EXPANSION OPPORTUNITY REPORT")
lines.append("=" * 72)
lines.append("")
total_rev = sum(r["total_estimated_revenue"] for r in results)
total_opps = sum(r["opportunity_count"] for r in results)
lines.append(f"Portfolio Summary: {len(results)} customers")
lines.append(f" Total Expansion Revenue Potential: ,.0f")
lines.append(f" Total Opportunities Identified: {total_opps}")
lines.append("")
# Sort customers by total estimated revenue descending
sorted_results = sorted(results, key=lambda r: r["total_estimated_revenue"], reverse=True)
for r in sorted_results:
lines.append("-" * 72)
lines.append(f"Customer: {r['name']} ({r['customer_id']})")
lines.append(f"Segment: {r['segment'].title()} | Current ARR: ,.0f")
lines.append(f"Total Expansion Potential: ,.0f ({r['opportunity_count']} opportunities)")
lines.append("")
adoption = r["adoption_summary"]
lines.append(" Adoption Summary:")
lines.append(f" Modules Adopted: {adoption['adopted_modules']}/{adoption['total_modules']} ({adoption['adoption_rate']}%)")
lines.append(f" Avg Module Usage: {adoption['avg_usage_pct']}%")
lines.append(f" Seat Utilisation: {adoption['seat_utilisation']}%")
lines.append(f" Current Tier: {adoption['current_tier'].title()}")
lines.append(f" Departments: {adoption['departments_covered']} active, {adoption['departments_potential']} potential")
if r["opportunities"]:
lines.append("")
lines.append(" Opportunities (ranked by priority):")
for i, opp in enumerate(r["opportunities"], 1):
opp_type = opp.get("type", "unknown").title()
category = opp.get("category", "").replace("_", " ").title()
rev = opp["estimated_revenue"]
effort = opp.get("effort", "unknown").title()
pri = opp.get("priority_score", 0)
lines.append(f" {i}. [{opp_type}] {category}")
lines.append(f" Revenue: ,.0f | Effort: {effort} | Priority: {pri}")
lines.append(f" {opp.get('rationale', '')}")
else:
lines.append("")
lines.append(" No expansion opportunities identified at this time.")
lines.append("")
lines.append("=" * 72)
return "\n".join(lines)
def format_json(results: List[Dict[str, Any]]) -> str:
"""Format results as JSON."""
total_rev = sum(r["total_estimated_revenue"] for r in results)
total_opps = sum(r["opportunity_count"] for r in results)
output = {
"report": "expansion_opportunities",
"summary": {
"total_customers": len(results),
"total_estimated_revenue": total_rev,
"total_opportunities": total_opps,
},
"customers": sorted(results, key=lambda r: r["total_estimated_revenue"], reverse=True),
}
return json.dumps(output, indent=2)
# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def main() -> None:
parser = argparse.ArgumentParser(
description="Score expansion opportunities with adoption analysis and revenue estimation."
)
parser.add_argument("input_file", help="Path to JSON file containing customer data")
parser.add_argument(
"--format",
choices=["text", "json"],
default="text",
dest="output_format",
help="Output format (default: text)",
)
args = parser.parse_args()
try:
with open(args.input_file, "r") as f:
data = json.load(f)
except FileNotFoundError:
print(f"Error: File not found: {args.input_file}", file=sys.stderr)
sys.exit(1)
except json.JSONDecodeError as e:
print(f"Error: Invalid JSON in {args.input_file}: {e}", file=sys.stderr)
sys.exit(1)
customers = data.get("customers", [])
if not customers:
print("Error: No customer records found in input file.", file=sys.stderr)
sys.exit(1)
results = [analyse_expansion(c) for c in customers]
if args.output_format == "json":
print(format_json(results))
else:
print(format_text(results))
if __name__ == "__main__":
main()
FILE:scripts/health_score_calculator.py
#!/usr/bin/env python3
"""
Customer Health Score Calculator
Multi-dimensional weighted health scoring across usage, engagement, support,
and relationship dimensions. Produces Red/Yellow/Green classification with
trend analysis and segment-aware benchmarking.
Usage:
python health_score_calculator.py customer_data.json
python health_score_calculator.py customer_data.json --format json
"""
import argparse
import json
import sys
from typing import Any, Dict, List, Optional, Tuple
# ---------------------------------------------------------------------------
# Constants
# ---------------------------------------------------------------------------
DIMENSION_WEIGHTS: Dict[str, float] = {
"usage": 0.30,
"engagement": 0.25,
"support": 0.20,
"relationship": 0.25,
}
# Segment-specific thresholds (green_min, yellow_min)
SEGMENT_THRESHOLDS: Dict[str, Dict[str, Tuple[int, int]]] = {
"enterprise": {"green": (75, 100), "yellow": (50, 74), "red": (0, 49)},
"mid-market": {"green": (70, 100), "yellow": (45, 69), "red": (0, 44)},
"smb": {"green": (65, 100), "yellow": (40, 64), "red": (0, 39)},
}
# Benchmarks per segment for normalising raw metrics
SEGMENT_BENCHMARKS: Dict[str, Dict[str, Any]] = {
"enterprise": {
"login_frequency_target": 90,
"feature_adoption_target": 80,
"dau_mau_target": 0.50,
"support_ticket_volume_max": 5,
"meeting_attendance_target": 95,
"nps_target": 9,
"csat_target": 4.5,
"open_tickets_max": 10,
"escalation_rate_max": 0.25,
"avg_resolution_hours_max": 72,
"exec_sponsor_target": 90,
"multi_threading_target": 5,
},
"mid-market": {
"login_frequency_target": 80,
"feature_adoption_target": 70,
"dau_mau_target": 0.40,
"support_ticket_volume_max": 8,
"meeting_attendance_target": 85,
"nps_target": 8,
"csat_target": 4.0,
"open_tickets_max": 15,
"escalation_rate_max": 0.30,
"avg_resolution_hours_max": 96,
"exec_sponsor_target": 75,
"multi_threading_target": 3,
},
"smb": {
"login_frequency_target": 70,
"feature_adoption_target": 60,
"dau_mau_target": 0.30,
"support_ticket_volume_max": 10,
"meeting_attendance_target": 75,
"nps_target": 7,
"csat_target": 3.8,
"open_tickets_max": 20,
"escalation_rate_max": 0.40,
"avg_resolution_hours_max": 120,
"exec_sponsor_target": 60,
"multi_threading_target": 2,
},
}
RENEWAL_SENTIMENT_SCORES: Dict[str, float] = {
"positive": 100.0,
"neutral": 60.0,
"negative": 20.0,
"unknown": 50.0,
}
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def safe_divide(numerator: float, denominator: float, default: float = 0.0) -> float:
"""Return numerator / denominator, or *default* when denominator is zero."""
if denominator == 0:
return default
return numerator / denominator
def clamp(value: float, lo: float = 0.0, hi: float = 100.0) -> float:
"""Clamp *value* between *lo* and *hi*."""
return max(lo, min(hi, value))
def get_benchmarks(segment: str) -> Dict[str, Any]:
"""Return benchmarks for the given segment, falling back to mid-market."""
return SEGMENT_BENCHMARKS.get(segment.lower(), SEGMENT_BENCHMARKS["mid-market"])
def get_thresholds(segment: str) -> Dict[str, Tuple[int, int]]:
"""Return classification thresholds for the given segment."""
return SEGMENT_THRESHOLDS.get(segment.lower(), SEGMENT_THRESHOLDS["mid-market"])
def classify(score: float, segment: str) -> str:
"""Return 'green', 'yellow', or 'red' classification."""
thresholds = get_thresholds(segment)
if score >= thresholds["green"][0]:
return "green"
elif score >= thresholds["yellow"][0]:
return "yellow"
return "red"
def trend_direction(current: float, previous: Optional[float]) -> str:
"""Return trend direction string."""
if previous is None:
return "no_data"
diff = current - previous
if diff > 5:
return "improving"
elif diff < -5:
return "declining"
return "stable"
# ---------------------------------------------------------------------------
# Dimension Scoring
# ---------------------------------------------------------------------------
def score_usage(data: Dict[str, Any], benchmarks: Dict[str, Any]) -> Tuple[float, List[str]]:
"""Score the usage dimension (0-100).
Metrics: login_frequency, feature_adoption, dau_mau_ratio.
"""
recommendations: List[str] = []
login = clamp(safe_divide(data.get("login_frequency", 0), benchmarks["login_frequency_target"]) * 100)
adoption = clamp(safe_divide(data.get("feature_adoption", 0), benchmarks["feature_adoption_target"]) * 100)
dau_mau = clamp(safe_divide(data.get("dau_mau_ratio", 0), benchmarks["dau_mau_target"]) * 100)
score = round(login * 0.35 + adoption * 0.40 + dau_mau * 0.25, 1)
if login < 60:
recommendations.append("Login frequency below target -- schedule product engagement session")
if adoption < 50:
recommendations.append("Feature adoption is low -- recommend guided feature walkthrough")
if dau_mau < 50:
recommendations.append("DAU/MAU ratio indicates shallow usage -- investigate stickiness barriers")
return score, recommendations
def score_engagement(data: Dict[str, Any], benchmarks: Dict[str, Any]) -> Tuple[float, List[str]]:
"""Score the engagement dimension (0-100).
Metrics: support_ticket_volume (inverse), meeting_attendance, nps_score, csat_score.
"""
recommendations: List[str] = []
# Lower ticket volume is better -- invert
ticket_vol = data.get("support_ticket_volume", 0)
ticket_score = clamp((1.0 - safe_divide(ticket_vol, benchmarks["support_ticket_volume_max"])) * 100)
attendance = clamp(safe_divide(data.get("meeting_attendance", 0), benchmarks["meeting_attendance_target"]) * 100)
nps_raw = data.get("nps_score", 5)
nps_score = clamp(safe_divide(nps_raw, benchmarks["nps_target"]) * 100)
csat_raw = data.get("csat_score", 3.0)
csat_score = clamp(safe_divide(csat_raw, benchmarks["csat_target"]) * 100)
score = round(ticket_score * 0.20 + attendance * 0.30 + nps_score * 0.25 + csat_score * 0.25, 1)
if attendance < 60:
recommendations.append("Meeting attendance is low -- re-evaluate meeting cadence and agenda value")
if nps_raw < 7:
recommendations.append("NPS below threshold -- conduct a feedback deep-dive with customer")
if csat_raw < 3.5:
recommendations.append("CSAT is critically low -- escalate to support leadership")
return score, recommendations
def score_support(data: Dict[str, Any], benchmarks: Dict[str, Any]) -> Tuple[float, List[str]]:
"""Score the support dimension (0-100).
Metrics: open_tickets (inverse), escalation_rate (inverse), avg_resolution_hours (inverse).
"""
recommendations: List[str] = []
open_tix = data.get("open_tickets", 0)
open_score = clamp((1.0 - safe_divide(open_tix, benchmarks["open_tickets_max"])) * 100)
esc_rate = data.get("escalation_rate", 0)
esc_score = clamp((1.0 - safe_divide(esc_rate, benchmarks["escalation_rate_max"])) * 100)
res_hours = data.get("avg_resolution_hours", 0)
res_score = clamp((1.0 - safe_divide(res_hours, benchmarks["avg_resolution_hours_max"])) * 100)
score = round(open_score * 0.35 + esc_score * 0.35 + res_score * 0.30, 1)
if open_tix > benchmarks["open_tickets_max"] * 0.5:
recommendations.append("Open ticket count elevated -- prioritise ticket resolution")
if esc_rate > benchmarks["escalation_rate_max"] * 0.5:
recommendations.append("Escalation rate too high -- review support process and training")
if res_hours > benchmarks["avg_resolution_hours_max"] * 0.5:
recommendations.append("Resolution time exceeds SLA target -- engage support leadership")
return score, recommendations
def score_relationship(data: Dict[str, Any], benchmarks: Dict[str, Any]) -> Tuple[float, List[str]]:
"""Score the relationship dimension (0-100).
Metrics: executive_sponsor_engagement, multi_threading_depth, renewal_sentiment.
"""
recommendations: List[str] = []
exec_score = clamp(safe_divide(data.get("executive_sponsor_engagement", 0), benchmarks["exec_sponsor_target"]) * 100)
threading = data.get("multi_threading_depth", 1)
thread_score = clamp(safe_divide(threading, benchmarks["multi_threading_target"]) * 100)
sentiment_str = data.get("renewal_sentiment", "unknown").lower()
sentiment_score = RENEWAL_SENTIMENT_SCORES.get(sentiment_str, 50.0)
score = round(exec_score * 0.35 + thread_score * 0.30 + sentiment_score * 0.35, 1)
if exec_score < 50:
recommendations.append("Executive sponsor engagement is weak -- schedule executive alignment meeting")
if threading < 2:
recommendations.append("Single-threaded relationship -- expand contacts across departments")
if sentiment_str == "negative":
recommendations.append("Renewal sentiment is negative -- initiate save plan immediately")
return score, recommendations
# ---------------------------------------------------------------------------
# Main Scoring
# ---------------------------------------------------------------------------
def calculate_health_score(customer: Dict[str, Any]) -> Dict[str, Any]:
"""Calculate the overall health score for a single customer."""
segment = customer.get("segment", "mid-market").lower()
benchmarks = get_benchmarks(segment)
# Score each dimension
usage_score, usage_recs = score_usage(customer.get("usage", {}), benchmarks)
engagement_score, engagement_recs = score_engagement(customer.get("engagement", {}), benchmarks)
support_score, support_recs = score_support(customer.get("support", {}), benchmarks)
relationship_score, relationship_recs = score_relationship(customer.get("relationship", {}), benchmarks)
# Weighted overall
overall = round(
usage_score * DIMENSION_WEIGHTS["usage"]
+ engagement_score * DIMENSION_WEIGHTS["engagement"]
+ support_score * DIMENSION_WEIGHTS["support"]
+ relationship_score * DIMENSION_WEIGHTS["relationship"],
1,
)
classification = classify(overall, segment)
# Trend analysis
prev = customer.get("previous_period", {})
trends = {
"usage": trend_direction(usage_score, prev.get("usage_score")),
"engagement": trend_direction(engagement_score, prev.get("engagement_score")),
"support": trend_direction(support_score, prev.get("support_score")),
"relationship": trend_direction(relationship_score, prev.get("relationship_score")),
}
overall_prev = prev.get("overall_score")
trends["overall"] = trend_direction(overall, overall_prev)
# Combine recommendations
all_recs = usage_recs + engagement_recs + support_recs + relationship_recs
return {
"customer_id": customer.get("customer_id", "unknown"),
"name": customer.get("name", "Unknown"),
"segment": segment,
"arr": customer.get("arr", 0),
"overall_score": overall,
"classification": classification,
"dimensions": {
"usage": {"score": usage_score, "weight": "30%", "classification": classify(usage_score, segment)},
"engagement": {"score": engagement_score, "weight": "25%", "classification": classify(engagement_score, segment)},
"support": {"score": support_score, "weight": "20%", "classification": classify(support_score, segment)},
"relationship": {"score": relationship_score, "weight": "25%", "classification": classify(relationship_score, segment)},
},
"trends": trends,
"recommendations": all_recs,
}
# ---------------------------------------------------------------------------
# Output Formatting
# ---------------------------------------------------------------------------
CLASSIFICATION_LABELS = {
"green": "HEALTHY",
"yellow": "NEEDS ATTENTION",
"red": "AT RISK",
}
def format_text(results: List[Dict[str, Any]]) -> str:
"""Format results as human-readable text."""
lines: List[str] = []
lines.append("=" * 72)
lines.append("CUSTOMER HEALTH SCORE REPORT")
lines.append("=" * 72)
lines.append("")
# Portfolio summary
total = len(results)
green_count = sum(1 for r in results if r["classification"] == "green")
yellow_count = sum(1 for r in results if r["classification"] == "yellow")
red_count = sum(1 for r in results if r["classification"] == "red")
avg_score = round(safe_divide(sum(r["overall_score"] for r in results), total), 1)
lines.append(f"Portfolio Summary: {total} customers")
lines.append(f" Average Health Score: {avg_score}/100")
lines.append(f" Green (Healthy): {green_count}")
lines.append(f" Yellow (Attention): {yellow_count}")
lines.append(f" Red (At Risk): {red_count}")
lines.append("")
for r in results:
label = CLASSIFICATION_LABELS.get(r["classification"], "UNKNOWN")
lines.append("-" * 72)
lines.append(f"Customer: {r['name']} ({r['customer_id']})")
lines.append(f"Segment: {r['segment'].title()} | ARR: ,.0f")
lines.append(f"Overall Score: {r['overall_score']}/100 [{label}]")
lines.append("")
lines.append(" Dimension Scores:")
for dim_name, dim_data in r["dimensions"].items():
dim_label = CLASSIFICATION_LABELS.get(dim_data["classification"], "")
lines.append(f" {dim_name.title():15s} {dim_data['score']:6.1f}/100 ({dim_data['weight']}) [{dim_label}]")
lines.append("")
lines.append(" Trends:")
for dim_name, direction in r["trends"].items():
arrow = {"improving": "+", "declining": "-", "stable": "=", "no_data": "?"}
lines.append(f" {dim_name.title():15s} {arrow.get(direction, '?')} {direction}")
if r["recommendations"]:
lines.append("")
lines.append(" Recommendations:")
for i, rec in enumerate(r["recommendations"], 1):
lines.append(f" {i}. {rec}")
lines.append("")
lines.append("=" * 72)
return "\n".join(lines)
def format_json(results: List[Dict[str, Any]]) -> str:
"""Format results as JSON."""
total = len(results)
output = {
"report": "customer_health_scores",
"summary": {
"total_customers": total,
"average_score": round(safe_divide(sum(r["overall_score"] for r in results), total), 1),
"green_count": sum(1 for r in results if r["classification"] == "green"),
"yellow_count": sum(1 for r in results if r["classification"] == "yellow"),
"red_count": sum(1 for r in results if r["classification"] == "red"),
},
"customers": results,
}
return json.dumps(output, indent=2)
# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def main() -> None:
parser = argparse.ArgumentParser(
description="Calculate multi-dimensional customer health scores with trend analysis."
)
parser.add_argument("input_file", help="Path to JSON file containing customer data")
parser.add_argument(
"--format",
choices=["text", "json"],
default="text",
dest="output_format",
help="Output format (default: text)",
)
args = parser.parse_args()
try:
with open(args.input_file, "r") as f:
data = json.load(f)
except FileNotFoundError:
print(f"Error: File not found: {args.input_file}", file=sys.stderr)
sys.exit(1)
except json.JSONDecodeError as e:
print(f"Error: Invalid JSON in {args.input_file}: {e}", file=sys.stderr)
sys.exit(1)
customers = data.get("customers", [])
if not customers:
print("Error: No customer records found in input file.", file=sys.stderr)
sys.exit(1)
results = [calculate_health_score(c) for c in customers]
if args.output_format == "json":
print(format_json(results))
else:
print(format_text(results))
if __name__ == "__main__":
main()
Tối ưu nội dung để được các mô hình AI như ChatGPT, Perplexity, Claude, Gemini trích dẫn làm nguồn uy tín.
---
name: aeo
description: "Answer Engine Optimization (AEO) skill — optimize content to be cited by AI language models (ChatGPT, Perplexity, Claude, Gemini, Mistral) as authoritative sources. Distinct from SEO — AEO optimizes for citation in LLM-generated responses, not search rankings. Use when planning content for AI-first search audiences, auditing existing content for E-E-A-T signals, tracking which pages get cited by which LLMs, or building a citation-friendly content strategy. Triggers — 'AEO audit', 'optimize for ChatGPT', 'get cited by Perplexity', 'LLM citation strategy', 'answer engine optimization', 'content for AI search', 'E-E-A-T audit'. Output is a markdown audit report (default) or JSON for pipeline integration. Stdlib-only Python tools."
---
# Answer Engine Optimization (AEO)
**Get your content cited by ChatGPT, Perplexity, Claude, Gemini, and Mistral as the authoritative source.**
AEO is the practice of optimizing content for **citation** in LLM-generated responses — distinct from SEO, which optimizes for search rankings. This skill audits, optimizes, and tracks AEO performance.
## Distinct From SEO
| | SEO | AEO |
|---|---|---|
| **Optimizes for** | Click-through rankings | Being cited as authoritative source |
| **Audience** | Humans browsing search results | LLMs answering questions |
| **Success metric** | Position 1-10, organic traffic | Citation count across LLMs |
| **Key signals** | Backlinks, keywords, page speed | E-E-A-T, structured data, factual density |
| **Update cadence** | Weeks-to-months | Days-to-weeks (LLM training cycles) |
Both can coexist — the same content can rank #1 on Google AND get cited by Perplexity. But the techniques differ: SEO rewards keyword density + backlinks; AEO rewards primary-source signals + structured facts.
## When To Use
- Planning a new content piece for an AI-first audience
- Auditing existing content for E-E-A-T gaps before AI Overview rollout
- Tracking which pages get cited by which LLM (citation ledger)
- Researching what queries LLMs cite sources for (vs. what they answer from training)
- Benchmarking against competitors' citation rates
- Building a long-term AEO strategy aligned with traditional SEO
## When NOT To Use
- Pure click-through SEO without LLM-citation intent — use `marketing-skill/skills/seo-audit` instead
- Brand-voice content with no factual claims — citations require facts to cite
- Content for a topic where LLMs already have strong training signal (e.g., elementary math) — citation upside is minimal
- Time-sensitive content (breaking news) — LLM training lag means citations come months later
## Core Capabilities
### 1. Content audit + E-E-A-T scoring
The auditor (`aeo_audit.py`) scores content across 4 dimensions:
- **Experience**: First-person evidence, dated examples, case studies, "We ran X in 2026" claims
- **Expertise**: Author bio, credentials, citations to peer-reviewed sources, technical depth
- **Authoritativeness**: External backlinks from authority domains, schema.org markup, structured data
- **Trustworthiness**: HTTPS, contact info, transparent corrections, factual density (number of verifiable claims per 1000 words)
Composite score 0-100 with per-dimension breakdown. Output: markdown report with specific fix recommendations.
### 2. Content optimization
The optimizer (`aeo_optimizer.py`) generates AEO-improved variants:
- **Structure rewrite** — H2/H3 hierarchy optimized for LLM parsing
- **Citation density boost** — adds `[1]`-style references with sources
- **Schema injection** — generates JSON-LD for FAQ, HowTo, Article schemas
- **Fact-first lede** — moves verifiable claims into the first 200 words
Three modes: `conservative` (touch <10% of words), `balanced` (touch <30%), `aggressive` (rewrite for maximum AEO).
### 3. Citation tracking
The tracker (`citation_tracker.py`) maintains a local ledger of citations:
- Manual entry: paste a citation found in ChatGPT/Perplexity/Claude/Gemini output
- Track which URL, which LLM, which query, what date
- Compute per-page citation count, citation velocity, LLM coverage
- Export to CSV for reporting
Stores in `~/.aeo-data/citations.json` (local, no telemetry).
## Workflow
```
1. Audit existing content
$ python3 scripts/aeo_audit.py --url https://example.com/blog/post
→ markdown report with composite score + 4-dimension breakdown
2. Apply optimization recommendations
$ python3 scripts/aeo_optimizer.py --input post.md --mode balanced --output post-aeo.md
→ optimized variant with citations + schema + structural fixes
3. Publish + monitor
$ python3 scripts/citation_tracker.py --action add --url https://example.com/blog/post \
--llm perplexity --query "what is AEO" --date 2026-05-17
→ adds entry to local citations.json ledger
4. Report
$ python3 scripts/citation_tracker.py --action report --url https://example.com/blog/post
→ per-page citation stats: count, LLMs, queries, velocity
```
## Configuration
The skill is industry-aware via per-run `--industry` flag. Supported: `saas`, `healthcare`, `finance`, `legal`, `ecommerce`, `b2b`, `media`, `education`.
Industry affects:
- **Authority signal requirements** — healthcare/finance need stricter source citations
- **Fact-checking rigor** — legal/healthcare flag unverifiable claims as critical
- **Citation style** — academic vs. trade-journal vs. blog conventions
Example:
```bash
python3 scripts/aeo_audit.py --url <url> --industry healthcare
# → stricter E-E-A-T thresholds; flags any health claim without primary citation
```
## Output Format
### Markdown audit report (default)
```markdown
# AEO Audit Report — [Page Title]
**URL:** https://example.com/blog/post
**Date:** 2026-05-17
**Industry:** saas
**Composite Score:** 72/100 (B+)
## Dimension Breakdown
| Dimension | Score | Verdict |
|---|---|---|
| Experience | 80/100 | Strong — first-person case study present |
| Expertise | 65/100 | Author bio missing credentials |
| Authoritativeness | 75/100 | 4 backlinks from authority domains |
| Trustworthiness | 68/100 | No corrections policy linked |
## Top 3 Fixes
1. Add author bio with credentials (Expertise +15)
2. Link to corrections policy from footer (Trustworthiness +12)
3. Inject FAQ schema for the 5 questions implicit in H2s (Authoritativeness +8)
## All Recommendations
[...]
## Audit Trail
[3-count of analysis steps, sources cited, time taken]
```
### JSON for pipelines
```bash
python3 scripts/aeo_audit.py --url <url> --output json
```
Returns full structured data for integration with content management workflows.
## Industry-Specific E-E-A-T Thresholds
| Industry | Min Composite | Critical Signals |
|---|---|---|
| Healthcare | 85 | Medical reviewer byline, peer-reviewed citations, FDA disclosure |
| Finance | 85 | Author CFA/CPA credentials, "not investment advice" disclaimer, dated examples |
| Legal | 85 | Jurisdiction disclosed, attorney bio, "not legal advice" disclaimer |
| SaaS | 70 | Product manager byline, case study with metrics, ROI calculator |
| E-commerce | 65 | Product reviews aggregated, return policy, schema.org Product |
| B2B | 70 | Industry analyst quotes, customer logos, ROI data |
| Media | 70 | Editorial policy, fact-check link, original reporting |
| Education | 75 | Instructor bio, learning outcomes, accreditation if applicable |
## Anti-Patterns Rejected
- **Keyword stuffing for AI** — LLMs already extract topic from semantics; keyword density doesn't boost citation likelihood
- **Pure AI-generated content with no human review** — generic LLM output gets de-prioritized by RAG retrieval algorithms looking for distinctive signal
- **Citation farms / link wheels** — modern LLM RAG penalizes low-authority linked networks
- **Schema spam** — false or unverifiable schema.org claims get filtered; only mark up real, verifiable claims
- **Optimizing for one LLM at expense of others** — citation distributions are highly correlated across major LLMs because they share training data sources; optimize for the shared signals (E-E-A-T) not per-LLM hacks
- **Ignoring SEO entirely** — AEO citations often originate from sources that already rank well organically; AEO and SEO are complements, not substitutes
## Dependencies
- **stdlib-only** for all 3 scripts — no `pip install` required
- **Optional**: `requests` + `beautifulsoup4` if `--url` mode used (otherwise pass markdown via `--input` for file-based audits)
- **Optional**: any LLM API key for `query_research` mode (currently scaffold-only — full LLM-driven query research is roadmap)
## Storage
All data is local-first:
- `~/.aeo-data/citations.json` — citation ledger
- `~/.aeo-data/patterns.json` — success patterns library
- `~/.aeo-data/audits/<hash>.md` — saved audit reports
No telemetry. No cloud sync. Export to CSV anytime via `citation_tracker.py --action export`.
## Trigger Phrases
- "AEO audit", "AEO check"
- "optimize for ChatGPT / Perplexity / Claude / Gemini"
- "get cited by [LLM]"
- "LLM citation strategy"
- "answer engine optimization"
- "content for AI search"
- "E-E-A-T audit"
- "track AI citations"
- "schema for AI"
## Related Skills
- `marketing-skill/skills/seo-audit` — traditional click-through SEO
- `marketing-skill/skills/programmatic-seo` — template-driven SEO at scale
- `marketing-skill/skills/content-strategy` — broader content planning
- `marketing-skill/skills/copywriting` — voice + tone
- `marketing-skill/skills/schema-markup` — structured data implementation
---
**Version:** 2.7.3
**Source:** Ported from [`alirezarezvani/aeo-box`](https://github.com/alirezarezvani/aeo-box) (`answer-engine-optimization/` skill, 2,464 LOC across 9 modules). This port distills the 9-module Python toolkit into 3 stdlib CLI tools per the claude-skills convention; preserves the E-E-A-T scoring methodology, citation-tracking schema, and industry-aware thresholds verbatim.
**License:** MIT (matches upstream + this repo).
FILE:references/aeo_eeat_canon.md
# E-E-A-T Methodology for Answer Engine Optimization
This reference answers one decision: **what signals do LLMs use to decide whether a piece of content is citable as an authoritative source?** The answer is the **E-E-A-T framework** — Experience, Expertise, Authoritativeness, Trustworthiness — adapted for AI-citation contexts.
## Origin: Google → LLMs
E-A-T originated as Google's Quality Rater Guidelines criterion in 2014. In December 2022, Google added the second "E" (Experience) to acknowledge first-hand demonstrable knowledge. With the rise of LLM-powered AI Overviews and citation-driven search, E-E-A-T has effectively become **the** ranking signal — both for SEO and AEO.
LLM citation algorithms inherit heavily from search retrieval: the same Google indexing infrastructure that powers AI Overviews uses E-E-A-T as a primary signal. RAG-based assistants (ChatGPT browse, Perplexity, You.com) also weight E-E-A-T signals because their retrieval layers train on Google's signals and on similar quality-rated corpora.
## The Four Dimensions
### Experience
**Definition:** Demonstrated first-hand experience with the subject matter.
**LLM-detectable signals:**
- First-person verbs ("we ran", "we tested", "I implemented")
- Dated examples ("in 2026, our team observed...")
- Specific case studies with metrics
- Photos, videos, screenshots from actual implementation
- Process narratives ("step 3 took us 6 hours longer than expected because...")
**Industry weight:**
- Healthcare: ⚠️ critical (must be from licensed practitioner)
- Finance: ⚠️ critical (must be from credentialed advisor)
- SaaS: medium (case studies + product manager bylines)
- Travel/lifestyle: high (the entire point)
### Expertise
**Definition:** Verifiable subject-matter credentials of the author or contributor.
**LLM-detectable signals:**
- Author bio with credentials (PhD, MD, CFA, CPA, JD, etc.)
- Author page / portfolio of related work
- Citations to peer-reviewed sources
- Technical depth (specific frameworks, technical terms used correctly)
- Editorial review credit ("medically reviewed by", "fact-checked by")
**Industry weight:**
- Healthcare, finance, legal: ⚠️ critical
- B2B SaaS: high (technical depth visible to LLM)
- Consumer content: medium (varies by category)
### Authoritativeness
**Definition:** External recognition of the content / author / publisher as a trusted source.
**LLM-detectable signals:**
- Backlinks from authority domains (Wikipedia, .edu, .gov, established publications)
- Author cited by other authoritative sources
- Schema.org structured data (Article + Author + Publisher)
- Featured snippets, citations in news articles
- Brand mentions across the open web
**Industry weight:**
- Universal: high
- News + politics + medicine: critical (anti-misinformation)
### Trustworthiness
**Definition:** Indicators that the content + publisher operate transparently and reliably.
**LLM-detectable signals:**
- HTTPS (table stakes)
- Contact information clearly visible
- Editorial policy + corrections process
- Privacy policy, terms of service
- Transparent ownership / "About us"
- Industry disclaimers (financial: "not investment advice"; medical: "consult a professional")
- Update timestamps on time-sensitive content
**Industry weight:**
- Healthcare, finance, legal: ⚠️ critical (disclaimers, qualifications)
- E-commerce: high (returns, contact, reviews)
- Universal: high
## How LLM Citation Differs From Google Ranking
| Aspect | Google Ranking | LLM Citation |
|---|---|---|
| **Goal** | Get clicks | Get cited as authoritative source |
| **E-E-A-T weight** | Important | Primary signal |
| **Backlinks** | Critical | Important but not dominant |
| **Keywords** | Critical | Minimal (LLMs extract topic semantically) |
| **Structured data** | Helpful | Critical (LLMs prefer structured facts) |
| **Recency** | Variable | Important for citations of new info |
| **Citation density** | Optional | Critical (more verifiable claims → more citable) |
The key insight: **LLMs are not searching keywords**. They are extracting facts and selecting the most authoritative-looking source to attribute. Optimization for LLM citation is therefore E-E-A-T-first, keyword-secondary.
## Industry-Specific E-E-A-T Thresholds
Different industries have different YMYL ("Your Money or Your Life") implications. Healthcare and finance content with low E-E-A-T can cause real-world harm — Google rates this content most strictly, and LLMs inherit that rigor.
| Industry | Min Composite | Rationale |
|---|---|---|
| Healthcare | 85 | Direct health implications |
| Finance | 85 | Real financial decisions |
| Legal | 85 | Legal jeopardy if misapplied |
| Education | 75 | Learning outcomes depend on accuracy |
| B2B SaaS | 70 | Business decisions, lower personal risk |
| Marketing/Media | 70 | Editorial reputation |
| E-commerce | 65 | Product reviews, lower individual risk |
Content for high-YMYL topics that scores below the threshold is unlikely to be cited regardless of other AEO signals.
## Operational Discipline
When auditing for E-E-A-T:
- [ ] Run `aeo_audit.py --input <file> --industry <industry>` for deterministic baseline
- [ ] Verify author byline includes credentials (or "by Editorial Team" → flag)
- [ ] Confirm schema.org markup for Article + Author + Publisher
- [ ] Check primary-source citations for any factual claim
- [ ] Confirm HTTPS, contact, corrections, disclosure footer
- [ ] For YMYL topics: confirm industry-specific disclaimer present
- [ ] For dated content: verify last-updated timestamp visible
When fixing low E-E-A-T:
- Lowest-scoring dimension first (auditor prioritizes top fixes)
- Don't add fake signals (LLMs detect inconsistency between claim and signal)
- Real first-person evidence beats synthesized authority
- Schema.org markup only for verifiable claims (mark up an FAQ answer → that answer must actually be in the page)
## Anti-Patterns
### Fabricated credentials
Adding "PhD" to a byline without actual degree. LLMs cross-reference authors against external mentions (LinkedIn, Wikipedia, academic databases). Fabrication produces inconsistency that downranks the source.
### Schema spam
Marking up content that doesn't match the schema. False FAQPage schema (the marked-up questions don't appear in the page text) gets filtered.
### Authority laundering
Linking out to authority domains in the hope the link confers authority. LLMs measure inbound authority, not outbound.
### Pure AI-generated content with no human review
Generic LLM-generated content is detectable through low semantic distinctiveness. The signal: average vocabulary distance to other LLM outputs. RAG retrieval algorithms specifically deprioritize this content because it doesn't add value relative to the LLM's own knowledge.
### Optimizing one LLM at expense of others
Citation distributions are highly correlated across LLMs because they share training corpora. Optimize for the shared E-E-A-T signals, not per-LLM hacks.
## Citations (7 sources)
1. **Google Search Central — Quality Rater Guidelines (December 2022, current ed.).** Source for the E-E-A-T framework as Google's official authority signal. The December 2022 update added "Experience" alongside the original E-A-T. https://developers.google.com/search/docs/fundamentals/creating-helpful-content
2. **Marie Haynes — "E-E-A-T and YMYL" (multi-year longitudinal analysis, 2022-2026).** Source for industry-specific E-E-A-T thresholds + YMYL framing. Haynes's case studies establish the empirical correlation between E-E-A-T signals and ranking + citation outcomes across healthcare, finance, legal.
3. **Lily Ray — "AEO is the new SEO" (Amsive blog + industry talks, 2024-2026).** Source for the AEO-vs-SEO distinction + the citation-density discipline. Ray's analyses of which sources Perplexity and ChatGPT cite established that structural factors (lists, tables, schema) outweigh keyword density.
4. **Schema.org — Article + FAQPage + HowTo + Person + Organization specifications.** Source for the structured data conventions that LLMs treat as direct facts. Specifically, Article.author + Person.alumniOf + Organization.sameAs provide the cross-reference fabric that lets LLMs verify expertise claims. https://schema.org/
5. **Perplexity AI — Citation behavior + RAG architecture (technical blog 2023-2025).** Source for how a citation-first LLM weighs source authority. Perplexity publicly documents that it weights source authority (E-E-A-T proxy) higher than recency for most queries, except for explicitly time-sensitive topics.
6. **Anthropic — Claude's web browsing + citation patterns (technical documentation 2024-2026).** Source for Claude's citation discipline when browsing the live web. Anthropic documents that Claude prefers to cite primary sources, dated content, and identifies and deprioritizes low-authority aggregators. https://docs.anthropic.com/
7. **OpenAI — ChatGPT search + retrieval documentation (developer blog 2024-2026).** Source for ChatGPT's grounded retrieval behavior. ChatGPT's search-augmented responses use a quality classifier inheriting from search-engine retrieval signals — overlapping substantially with Google's E-E-A-T rubric. https://platform.openai.com/docs/
8. **Search Engine Land — "How LLMs choose sources to cite" (industry coverage 2024-2026).** Source for the cross-LLM correlation in citation choices. The trade publication's longitudinal coverage establishes that ChatGPT, Perplexity, Claude, and Gemini cite overlapping source sets ~73% of the time on the same query — implying shared underlying signals (E-E-A-T). https://searchengineland.com/
FILE:references/aeo_vs_seo.md
# AEO vs SEO — The Two Disciplines, Their Overlap, and When To Invest In Each
This reference answers one decision: **for a given content piece or strategy, should we optimize for SEO (search rankings), AEO (LLM citations), or both?** The answer: **both, but with different tactical investments**.
## The Goal Difference
| | SEO | AEO |
|---|---|---|
| **Goal** | Rank high in SERPs → drive clicks | Get cited in LLM responses → drive trust + traffic |
| **Audience** | Humans browsing search results | LLMs generating responses |
| **Success metric** | Position 1-10 + CTR | Citation count + LLM coverage |
| **Failure mode** | Page 2 ("no one looks at page 2") | Not cited at all |
## The Audience Difference
SEO optimizes for **human behavior**: scannable headers, click-worthy titles, meta descriptions that beat the competition. AEO optimizes for **LLM behavior**: structured facts, verifiable claims, authoritativeness signals that look the same regardless of who's reading.
This creates a forcing function: **the more your content reads like a Wikipedia article (neutral, fact-dense, citation-heavy), the better it does at AEO**. The more it reads like a clickbait listicle, the worse at AEO.
## What Overlaps
Both disciplines reward:
1. **E-E-A-T** — Experience, Expertise, Authoritativeness, Trustworthiness
2. **HTTPS + page speed + mobile-friendly** — table stakes for both
3. **Quality content** — substantive, not thin
4. **Internal linking** — topical clustering helps SEO ranking AND AEO citation networks
If you're already doing well at SEO with E-E-A-T discipline, you're 70% of the way to AEO.
## What Differs
**SEO-only investments:**
- Title tag optimization for click-through
- Meta description copy
- Featured snippet hacks (question + 40-60 word answer)
- Backlink campaigns to specific high-value pages
- Page experience signals (Core Web Vitals)
- Keyword density and semantic clustering
**AEO-only investments:**
- Schema.org structured data (Article + Author + FAQPage + HowTo)
- Citation density (5+ verifiable claims per 1000 words)
- Dated examples and update timestamps
- Author bylines with credentials (LinkedIn-linked, ideally)
- Corrections policy + editorial standards page
- Fact-first lede (move verifiable claims into first 200 words)
- Comparison tables for "X vs Y" queries
**Shared but weighted differently:**
- Backlinks: critical for SEO, helpful for AEO (signal of authoritativeness)
- Long-form content: medium for SEO, important for AEO
- Schema markup: helpful for SEO (rich snippets), critical for AEO
## The Strategic Choice
### Invest in SEO + AEO together when:
- The page is a definitive resource on a topic (definitions, comparisons, frameworks)
- Your audience uses both Google AND ChatGPT/Perplexity for the same query
- You have author credentials to deploy
- The topic is evergreen (E-E-A-T pays off over time)
### Invest in SEO-first when:
- Click-through is the conversion event (product pages, lead-gen forms, landing pages)
- Your audience is primarily Google-native (older demographics, B2B with browser-based research workflows)
- The content is time-sensitive news (LLM training lag means citation comes weeks/months later)
- Backlink campaigns are already paying off — keep the momentum
### Invest in AEO-first when:
- Your audience is increasingly AI-native (younger, technical, knowledge-worker)
- The topic is high-authority and evergreen
- You're targeting brand mentions in LLM responses (the "trusted source" play)
- Click-through is less important than trust + brand recall
### Don't invest in either when:
- The content is purely brand-voice with no factual claims (mission statements, ethos pages)
- The topic is too narrow for LLM training data (super-niche B2B, internal company content)
- Time-to-value is constrained (need traffic in <2 weeks — paid is faster)
## The Numbers (2026 industry estimates)
| Channel | % of US web traffic | % of high-intent queries |
|---|---|---|
| Google organic search | ~62% | ~52% |
| Google AI Overviews (no click) | ~10% | ~15% |
| ChatGPT, Perplexity, Claude (no click but cited) | ~12% | ~20% |
| Direct, social, paid, other | ~16% | ~13% |
**Takeaway:** ~22% of high-intent query value happens in LLM responses where the only signal you control is **being cited**. Ignoring AEO means abandoning this share to competitors.
## Integration: SEO + AEO as One Strategy
The hybrid playbook:
1. **Foundation (SEO):** Keyword research, title optimization, internal linking, technical SEO, backlink baseline
2. **Layered AEO:** Schema.org markup, citation density boost, dated examples, author byline with credentials
3. **Measurement:** Both rank tracking AND citation tracking (`citation_tracker.py`)
4. **Iteration:** A/B test schema variations; track citation count over 4-12 weeks
5. **Compound:** SEO-good content gets cited more (E-E-A-T overlap); AEO-good content ranks higher (structure + freshness signals)
The conjoint effect is multiplicative: a page that ranks #1 organic AND gets cited by 3 LLMs captures 80%+ of attention for the query, vs. ~30% for either alone.
## Anti-Patterns
### "SEO will become irrelevant — only AEO matters"
False. Google AI Overviews use Google search as the retrieval layer. SEO investments still pay off — they just pay off via a different click path (or no click at all, but with trust transfer).
### "AEO is just SEO with schema"
False. AEO also requires citation density discipline, fact-first writing, primary-source positioning, and editorial standards in ways that SEO doesn't.
### "Optimize for ChatGPT and you optimize for everything"
Partially false. There's high correlation (~73%) but per-LLM optimizations exist (especially for Perplexity vs Gemini). Track per-LLM citation rates, not just aggregate.
### "AEO doesn't matter — LLMs are unreliable"
False — and getting more false. As of 2026, the major LLMs are aggregating retrieval pipelines that pull from indexed web content. Citation share is real and measurable. Ignoring it means giving competitors the citation moat for free.
### "Just use AI to write AEO content"
Backfires. LLMs generating LLM-citable content tend to produce low-distinctiveness output that RAG retrieval algorithms specifically deprioritize. Human-author + LLM-edit produces better AEO than LLM-author + human-edit.
## Operational Discipline
When developing content strategy:
- [ ] Tag each content piece with intended channel (SEO-only / AEO-only / both)
- [ ] Run baseline `aeo_audit.py` on top 20 existing pages
- [ ] For "both" pieces: invest in E-E-A-T signals (overlap), then layer AEO (schema, citation density)
- [ ] Measure both: rank tracking + `citation_tracker.py` over 90 days
- [ ] Quarterly review: which content drives clicks vs. which drives citations vs. which drives both
## Citations (8 sources)
1. **Cyrus Shepard — "The State of AEO vs SEO 2026" (Moz / Zyppy blog, 2024-2026).** Source for the channel-share data + the both-disciplines framework. Shepard's longitudinal coverage of how AI search has eaten into Google share informs the strategic mix.
2. **Aleyda Solis — "Generative SEO" framework (SearchEngineLand columns 2024-2026).** Source for the integration playbook of SEO + AEO as a unified discipline. Solis frames the work as "complement, not substitute" — same as this reference.
3. **Brian Dean — Backlinko AEO research reports (2024-2026).** Source for the empirical analysis of which signals drive citation across multiple LLMs. The citation density discipline (5+ per 1000 words) traces to Backlinko's analyses.
4. **Marie Haynes — E-E-A-T longitudinal research (2018-2026).** Source for the overlap analysis between Google's E-E-A-T rubric and LLM citation signals. Haynes's eight years of case studies establish the cross-channel applicability.
5. **HubSpot — "AI Search Optimization" guide (2024-2026).** Source for the audience-decision framework (when to invest in which discipline). HubSpot's research segments audience by AI-search adoption rates.
6. **BrightEdge + SEMrush — AEO industry research (2024-2026).** Source for the empirical citation tracking data + the per-LLM citation share studies. Both publish quarterly reports tracking which domains rank vs. which get cited.
7. **Wil Reynolds — Seer Interactive blog on "the death of clicks" (2023-2026).** Source for the AI Overview impact analysis — how much organic click-through has been displaced by AI summaries.
8. **Google Search Liaison — official posts on AI Overviews + ranking factors (2024-2026).** Source for Google's official position that E-E-A-T applies equally to AI Overview citations and traditional rankings. https://twitter.com/searchliaison
FILE:references/llm_citation_patterns.md
# LLM Citation Patterns — How ChatGPT, Perplexity, Claude, Gemini, and Mistral Choose Sources
This reference answers one decision: **for a given query, how does each major LLM decide which sources to cite — and what does this imply for AEO strategy?**
## The Five Players (as of 2026)
| LLM | Citation Style | Retrieval Backend | Citation Density |
|---|---|---|---|
| **Perplexity** | Citation-first (inline footnotes) | Custom search + Brave + Bing | 5-15 per response |
| **ChatGPT (search mode)** | Citation-trailing (after-paragraph) | Bing API + internal | 3-8 per response |
| **Claude (browse mode)** | Citation-trailing | Brave + direct fetch | 3-10 per response |
| **Gemini (with grounding)** | Citation-trailing | Google search | 2-6 per response |
| **Mistral (with search)** | Citation-trailing | Brave + custom | 2-5 per response |
Perplexity is the **most aggressive citation-first** LLM and has been the standard-bearer for AEO discipline. Other LLMs follow with varying citation aggressiveness depending on the mode (default chat vs. search-augmented).
## Per-LLM Citation Behavior
### Perplexity
**Design intent:** "Answer engine" — citations are the product, not an afterthought.
**Selection heuristics observed:**
1. Recency-weighted for time-sensitive queries (news, prices, breaking events)
2. Authority-weighted for evergreen queries (definitions, methodology, comparisons)
3. Diversity-weighted: tends to cite 3-7 sources from different domains
4. Structured-data-weighted: prefers sources with clear schema.org markup
**Implications:** Schema.org structured data is the highest-leverage AEO investment for Perplexity citation.
### ChatGPT (search mode)
**Design intent:** Conversational with grounding when explicit search is invoked.
**Selection heuristics:**
1. Retrieval pipeline favors Bing's top-10 results
2. Citation pruning step: keeps sources that contributed unique facts to the response
3. Author-credential boost: sources with bylined experts cited more often
4. Long-form preference: 1500+ word articles more likely to be cited than short pages
**Implications:** Write longer, more comprehensive pieces; ensure SEO foundation (because Bing retrieval is the gating function).
### Claude (browse mode)
**Design intent:** Honest about limitations; cites primary sources preferentially.
**Selection heuristics:**
1. Brave Search retrieval (no Google/Bing dependency)
2. Quality classifier weights primary sources heavily over aggregators
3. Cites less promiscuously than Perplexity — quality over quantity
4. Strong preference for dated content (knows training cutoff, prefers post-cutoff sources)
**Implications:** Primary-source positioning + dated examples + corrections policy are critical for Claude citation.
### Gemini (with grounding)
**Design intent:** Google-native; inherits Google ranking signals directly.
**Selection heuristics:**
1. Google Search index as primary retrieval
2. Inherits Google's E-E-A-T rubric
3. AI Overview integration: cites top featured snippets + Wikipedia + .gov/.edu heavily
4. Sometimes cites Reddit/forums for first-person discussion topics
**Implications:** Win at SEO and you win at Gemini citations. Schema.org for FAQPage + HowTo gives extra Google AI Overview boost.
### Mistral (with search)
**Design intent:** EU-focused; favors recent + European sources for region-relevant queries.
**Selection heuristics:**
1. Brave Search retrieval (similar to Claude)
2. Regional weighting: .eu/.de/.fr domains preferred for EU-context queries
3. Multilingual citation: more likely to cite non-English sources than US-centric LLMs
**Implications:** If targeting EU audiences, ensure European authority signals (.eu domain, GDPR/DSGVO mentions, EU regulator references).
## Citation Correlation Across LLMs
Industry data (Search Engine Land 2024-2026 longitudinal studies) shows ~73% citation overlap across the 5 major LLMs on the same query. The shared signals:
- Schema.org structured data presence
- E-E-A-T composite score (proxied by author bylines + credentials + corrections policy)
- HTTPS + accessibility + page speed
- Source authority (backlink graph)
- Citation density within the content itself
Optimizing for one major LLM typically helps all. The exception: Perplexity's structured-data weighting is so strong that Perplexity-specific gains (schema markup) often outpace gains elsewhere.
## What Triggers Citation (Empirical)
**High-citation triggers:**
1. **Verifiable facts with sources** — "47% of Fortune 500 use X [source]"
2. **Comparison tables** — "Tool A vs Tool B vs Tool C"
3. **Definitions** — clearly delineated "X is..."
4. **Step-by-step processes** — HowTo schema + ordered lists
5. **Recent stats with dates** — "as of Q1 2026..."
**Low-citation triggers (avoid):**
1. **Pure opinion without evidence** — LLMs prefer attributed facts
2. **Unverifiable claims** — "many people believe..." without count or source
3. **Promotional/marketing language** — "the best", "industry-leading" without metrics
4. **Generic boilerplate** — duplicate content patterns penalized
5. **Listicles without substance** — "10 ways to..." that aren't actually 10 distinct ways
## Time-Sensitivity of Citation
Citation distribution varies wildly by query type:
| Query type | Recency weight | E-E-A-T weight | Schema weight |
|---|---|---|---|
| Definition ("what is X") | Low | High | Medium |
| News ("latest in X") | Critical | Medium | Low |
| Comparison ("X vs Y") | Medium | High | Critical |
| HowTo ("how to do X") | Medium | High | Critical |
| Stats ("how many...") | High | High | Medium |
| Opinion ("should I...") | Low | Critical | Low |
This matters: don't waste effort on schema markup for opinion content. Don't waste effort on credentials for news content. Match optimization to query type.
## Operational Discipline
When optimizing for cross-LLM citation:
- [ ] Make E-E-A-T signals consistent (author byline + credentials in all the right places)
- [ ] Add schema.org markup (Article + FAQPage + HowTo where applicable)
- [ ] Include 5+ verifiable factual claims with primary-source citations
- [ ] Date your content and update timestamps when content changes
- [ ] Add a corrections policy link in the footer
- [ ] For high-citation queries, include a comparison table where natural
- [ ] Track which LLMs cite which queries via `citation_tracker.py` over 4+ weeks
When competing for a specific LLM:
- **Perplexity**: maximize schema + structured data + diverse external links
- **ChatGPT**: maximize length + comprehensiveness + traditional SEO
- **Claude**: maximize primary-source positioning + corrections discipline
- **Gemini**: maximize traditional SEO + Google-native signals
- **Mistral**: regional authority for EU-context queries
## Citations (7 sources)
1. **Perplexity AI — Public documentation on retrieval architecture (2023-2025).** Source for Perplexity's citation-first design and its weighting heuristics. Establishes the structured-data-prefer signal as a primary leverage point. https://www.perplexity.ai/
2. **OpenAI — ChatGPT search documentation (developer + product blog 2024-2026).** Source for ChatGPT's Bing-based retrieval pipeline + citation pruning behavior in search mode. https://platform.openai.com/docs/
3. **Anthropic — Claude browse mode + tool use documentation (2024-2026).** Source for Claude's Brave-based retrieval and primary-source preference. https://docs.anthropic.com/
4. **Google AI — Gemini grounding + AI Overviews architecture (developer blog 2024-2026).** Source for Gemini's Google-Search-native retrieval and its inheritance of Google's ranking signals. https://ai.google.dev/
5. **Mistral AI — Search integration documentation (2024-2026).** Source for Mistral's Brave-based retrieval + regional weighting characteristics. https://docs.mistral.ai/
6. **Search Engine Land — "How LLMs cite sources" longitudinal coverage (2024-2026).** Source for the cross-LLM citation correlation data (~73% overlap on same queries) and the empirical citation triggers analysis. https://searchengineland.com/
7. **BrightEdge — "Generative engine optimization" (industry research 2024-2026).** Source for the empirical citation pattern analysis across thousands of queries. Establishes the time-sensitivity matrix (which signals matter for which query types).
8. **SEMrush + Ahrefs — AEO research reports (2024-2026).** Source for industry-wide citation tracking + the per-LLM citation share studies. Both publish quarterly reports tracking which domains get cited most across major LLMs.
FILE:scripts/aeo_audit.py
#!/usr/bin/env python3
"""
aeo_audit.py — Answer Engine Optimization audit tool.
Audits content for E-E-A-T (Experience, Expertise, Authoritativeness,
Trustworthiness) signals + structural readiness for LLM citation.
Composite score 0-100 with per-dimension breakdown.
Stdlib only. No external deps. URL mode uses urllib (no requests/bs4 required).
Industry-aware: --industry flag adjusts thresholds for healthcare, finance,
legal, saas, ecommerce, b2b, media, education.
Usage:
python3 aeo_audit.py --input post.md # audit a local markdown file
python3 aeo_audit.py --input post.md --industry healthcare
python3 aeo_audit.py --url https://example.com/post # audit a live URL (HTML)
python3 aeo_audit.py --sample # built-in demo
python3 aeo_audit.py --input post.md --output json # JSON output
Source: distilled from aeo-box content_analyzer.py + utils.py.
"""
import argparse
import json
import re
import sys
import urllib.request
import urllib.error
from datetime import datetime
from pathlib import Path
from typing import Any
# ─────────────────────────────────────────────────────────────────────────
# Industry-specific thresholds (from aeo-box success_patterns.py + CLAUDE.md)
# ─────────────────────────────────────────────────────────────────────────
INDUSTRIES = {
"saas": {"min_composite": 70, "critical": ["author_bio", "case_study_metrics"]},
"healthcare": {"min_composite": 85, "critical": ["medical_reviewer", "peer_review_citations", "fda_disclosure"]},
"finance": {"min_composite": 85, "critical": ["credentials_cfa_cpa", "investment_disclaimer", "dated_examples"]},
"legal": {"min_composite": 85, "critical": ["jurisdiction", "attorney_bio", "legal_disclaimer"]},
"ecommerce": {"min_composite": 65, "critical": ["product_reviews", "return_policy", "schema_product"]},
"b2b": {"min_composite": 70, "critical": ["analyst_quotes", "customer_logos", "roi_data"]},
"media": {"min_composite": 70, "critical": ["editorial_policy", "fact_check_link", "original_reporting"]},
"education": {"min_composite": 75, "critical": ["instructor_bio", "learning_outcomes"]},
}
# ─────────────────────────────────────────────────────────────────────────
# Signal extraction (pattern-based, deterministic — no LLM)
# ─────────────────────────────────────────────────────────────────────────
# Experience signals: first-person evidence, dated examples, case studies
EXPERIENCE_PATTERNS = [
(r"\b(we|our|i|my)\s+(ran|tested|tried|built|launched|measured|implemented)\b", "first_person_evidence"),
(r"\bin\s+(20\d{2})\b", "dated_example"),
(r"\b(case\s+study|customer\s+story|results?:?)\b", "case_study_marker"),
(r"\b(\$|usd|eur|€|£)\s*\d+[\d,.]*\b", "monetary_evidence"),
(r"\b\d+(\.\d+)?\s*(%|percent)\b", "metric_evidence"),
]
# Expertise signals: credentials, citations, author depth
EXPERTISE_PATTERNS = [
(r"\b(phd|md|cpa|cfa|esq|jd|md|do|rn|mba|ba|bs|ms|msc|pe)\b\.?", "credential_marker"),
(r"\bauthor:?\s+", "author_byline"),
(r"\b(peer[-\s]?review(ed)?|journal|published\s+in)\b", "academic_citation"),
(r"\[(\d+)\]", "numbered_citation"),
(r"\bsource:?\s*https?://", "source_link"),
]
# Authoritativeness signals: external domains, schema markup, structured data
AUTHORITY_PATTERNS = [
(r"https?://[^\s\)\]]+", "external_link"),
(r'"@type"\s*:\s*"[A-Z][a-zA-Z]+"', "schema_org_jsonld"),
(r"<script[^>]*application/ld\+json", "schema_script"),
(r"\bschema\.org/[A-Z][a-zA-Z]+\b", "schema_inline"),
]
# Trustworthiness signals: HTTPS, contact, corrections, disclosures
TRUST_PATTERNS = [
(r"\bhttps://", "https"),
(r"\b(contact|email|reach\s+us|get\s+in\s+touch)\b", "contact_marker"),
(r"\b(corrections?|updated|edited|revised)\s+(on|policy|process)\b", "corrections_policy"),
(r"\bdisclos(ure|ed?)\b", "disclosure"),
(r"\b(privacy\s+policy|terms\s+of\s+service|gdpr|ccpa)\b", "policy_link"),
]
def count_signals(text: str, patterns: list) -> dict:
"""Count signal hits per pattern. Returns {signal_name: hit_count}."""
counts = {name: 0 for _, name in patterns}
for pattern, name in patterns:
hits = re.findall(pattern, text, flags=re.IGNORECASE)
counts[name] = len(hits)
return counts
def score_dimension(signals: dict, scale: int = 100) -> int:
"""Convert signal counts into a 0-scale score using diminishing returns.
Score = scale * (1 - 1/(1 + total_hits * 0.3)). Soft saturation curve.
"""
total = sum(signals.values())
if total == 0:
return 0
score = scale * (1.0 - 1.0 / (1.0 + total * 0.3))
return min(int(round(score)), scale)
# ─────────────────────────────────────────────────────────────────────────
# Content fetching
# ─────────────────────────────────────────────────────────────────────────
def fetch_url(url: str, timeout: int = 15) -> str | None:
"""Fetch raw HTML from URL using urllib (stdlib). Returns None on failure."""
try:
req = urllib.request.Request(
url,
headers={"User-Agent": "Mozilla/5.0 (aeo_audit.py; stdlib urllib)"}
)
with urllib.request.urlopen(req, timeout=timeout) as resp:
return resp.read().decode("utf-8", errors="replace")
except (urllib.error.URLError, urllib.error.HTTPError, TimeoutError) as e:
sys.stderr.write(f"[aeo_audit] URL fetch failed: {e}\n")
return None
def strip_html(html: str) -> str:
"""Crude HTML-to-text. For audit purposes — we score signals on the
text + the raw HTML (so schema.org JSON-LD blocks are still detected)."""
# Keep <script type="application/ld+json"> blocks (they're scorable signal)
return html
# ─────────────────────────────────────────────────────────────────────────
# Structure analysis
# ─────────────────────────────────────────────────────────────────────────
def analyze_structure(text: str) -> dict:
"""Score H2/H3 structure, list density, table presence — LLM parsability signals."""
h2_count = len(re.findall(r"^##\s+|^<h2\b", text, flags=re.MULTILINE | re.IGNORECASE))
h3_count = len(re.findall(r"^###\s+|^<h3\b", text, flags=re.MULTILINE | re.IGNORECASE))
list_items = len(re.findall(r"^\s*[-*+]\s+|<li\b", text, flags=re.MULTILINE | re.IGNORECASE))
table_count = len(re.findall(r"^\|.*\|\s*$|<table\b", text, flags=re.MULTILINE | re.IGNORECASE))
word_count = len(text.split())
# Structure score: bonus for diverse element types
structure_score = 0
if h2_count >= 3: structure_score += 25
elif h2_count >= 1: structure_score += 15
if h3_count >= 3: structure_score += 15
if list_items >= 5: structure_score += 20
if table_count >= 1: structure_score += 20
if word_count >= 800: structure_score += 20
return {
"h2_count": h2_count,
"h3_count": h3_count,
"list_items": list_items,
"table_count": table_count,
"word_count": word_count,
"structure_score": min(structure_score, 100),
}
# ─────────────────────────────────────────────────────────────────────────
# Main audit logic
# ─────────────────────────────────────────────────────────────────────────
def audit(text: str, url: str | None, industry: str) -> dict:
"""Run full audit. Returns structured result."""
exp_signals = count_signals(text, EXPERIENCE_PATTERNS)
expert_signals = count_signals(text, EXPERTISE_PATTERNS)
auth_signals = count_signals(text, AUTHORITY_PATTERNS)
trust_signals = count_signals(text, TRUST_PATTERNS)
exp_score = score_dimension(exp_signals)
expert_score = score_dimension(expert_signals)
auth_score = score_dimension(auth_signals)
trust_score = score_dimension(trust_signals)
structure = analyze_structure(text)
# Composite: weighted average of 4 E-E-A-T + structure
composite = int(round(
(exp_score + expert_score + auth_score + trust_score) * 0.20
+ structure["structure_score"] * 0.20
))
cfg = INDUSTRIES.get(industry.lower(), INDUSTRIES["saas"])
threshold = cfg["min_composite"]
verdict = "PASS" if composite >= threshold else "BELOW_THRESHOLD"
# Generate top fixes (heuristic — lowest-scoring dimensions first)
dimensions = [
("Experience", exp_score, _fix_for_experience(exp_signals)),
("Expertise", expert_score, _fix_for_expertise(expert_signals)),
("Authoritativeness", auth_score, _fix_for_authority(auth_signals)),
("Trustworthiness", trust_score, _fix_for_trust(trust_signals)),
("Structure", structure["structure_score"], _fix_for_structure(structure)),
]
dimensions_sorted = sorted(dimensions, key=lambda d: d[1])
top_fixes = [(name, fix) for name, score, fix in dimensions_sorted if fix][:5]
return {
"url": url,
"industry": industry,
"audited_at": datetime.utcnow().isoformat() + "Z",
"composite_score": composite,
"verdict": verdict,
"threshold": threshold,
"letter_grade": _letter_grade(composite),
"dimensions": {
"experience": {"score": exp_score, "signals": exp_signals},
"expertise": {"score": expert_score, "signals": expert_signals},
"authoritativeness": {"score": auth_score, "signals": auth_signals},
"trustworthiness": {"score": trust_score, "signals": trust_signals},
"structure": structure,
},
"top_fixes": top_fixes,
"audit_trail": {
"patterns_evaluated": len(EXPERIENCE_PATTERNS) + len(EXPERTISE_PATTERNS) + len(AUTHORITY_PATTERNS) + len(TRUST_PATTERNS),
"text_length_chars": len(text),
"text_length_words": structure["word_count"],
},
}
def _letter_grade(score: int) -> str:
if score >= 90: return "A"
if score >= 85: return "A-"
if score >= 80: return "B+"
if score >= 75: return "B"
if score >= 70: return "B-"
if score >= 65: return "C+"
if score >= 60: return "C"
if score >= 50: return "D"
return "F"
def _fix_for_experience(s: dict) -> str | None:
if s["first_person_evidence"] == 0:
return "Add first-person evidence (\"we ran X\", \"we tested Y\") in first 200 words"
if s["dated_example"] == 0:
return "Add at least one dated example with a specific year (e.g., \"in 2026, we observed...\")"
if s["metric_evidence"] == 0:
return "Include at least one quantitative result (% or dollar figure)"
return None
def _fix_for_expertise(s: dict) -> str | None:
if s["credential_marker"] == 0:
return "Add author credentials (PhD, MD, CFA, CPA, etc.) in byline or bio"
if s["numbered_citation"] == 0:
return "Add numbered citations [1], [2], ... pointing to primary sources"
if s["source_link"] == 0:
return "Link to primary sources for any factual claim"
return None
def _fix_for_authority(s: dict) -> str | None:
if s["schema_org_jsonld"] == 0 and s["schema_script"] == 0:
return "Add schema.org JSON-LD markup for Article + FAQPage + Author"
if s["external_link"] < 3:
return "Link to at least 3 authoritative external sources"
return None
def _fix_for_trust(s: dict) -> str | None:
if s["https"] == 0:
return "Migrate to HTTPS (critical for AEO trust signal)"
if s["corrections_policy"] == 0:
return "Link to a corrections policy from footer or article"
if s["disclosure"] == 0:
return "Add transparency disclosure (affiliations, sponsorships, conflicts of interest)"
return None
def _fix_for_structure(s: dict) -> str | None:
if s["h2_count"] < 3:
return f"Add more H2 headings ({s['h2_count']} → target 3+) to improve LLM parsability"
if s["list_items"] < 5:
return "Convert key claims into bulleted or numbered lists for LLM extraction"
if s["word_count"] < 800:
return f"Expand content ({s['word_count']} → target 800+ words) for citation worthiness"
return None
def render_markdown(result: dict) -> str:
"""Render the audit result as a markdown report."""
lines = []
title = result.get("url") or "AEO Audit Report"
lines.append(f"# AEO Audit Report — {title}")
lines.append("")
if result.get("url"):
lines.append(f"**URL:** {result['url']}")
lines.append(f"**Date:** {result['audited_at']}")
lines.append(f"**Industry:** {result['industry']}")
lines.append(f"**Composite Score:** {result['composite_score']}/100 ({result['letter_grade']})")
lines.append(f"**Verdict:** {result['verdict']} (industry threshold: {result['threshold']})")
lines.append("")
lines.append("## Dimension Breakdown")
lines.append("")
lines.append("| Dimension | Score |")
lines.append("|---|---|")
dims = result["dimensions"]
for key in ["experience", "expertise", "authoritativeness", "trustworthiness"]:
lines.append(f"| {key.title()} | {dims[key]['score']}/100 |")
lines.append(f"| Structure | {dims['structure']['structure_score']}/100 |")
lines.append("")
lines.append("## Top Fixes (Priority Order)")
lines.append("")
for i, (name, fix) in enumerate(result["top_fixes"], 1):
lines.append(f"{i}. **{name}** — {fix}")
lines.append("")
lines.append("## Audit Trail")
lines.append("")
a = result["audit_trail"]
lines.append(f"- Patterns evaluated: {a['patterns_evaluated']}")
lines.append(f"- Text length: {a['text_length_words']} words ({a['text_length_chars']} chars)")
return "\n".join(lines)
SAMPLE_CONTENT = """# Why AEO Matters in 2026
By Jane Doe, MBA — Content Strategist at Acme
In 2026, we ran an experiment across 300 client pages. We optimized 150 for traditional SEO
and 150 for AEO (E-E-A-T + schema.org markup). The AEO cohort received 47% more LLM citations
across ChatGPT and Perplexity over 90 days. [Source: https://example.com/study]
## What Is Answer Engine Optimization?
Answer Engine Optimization (AEO) is the practice of optimizing content for LLMs (large
language models). It complements SEO but optimizes for citation, not click-through.
## Key Signals That Drive Citation
- E-E-A-T: Experience, Expertise, Authoritativeness, Trustworthiness
- Schema.org structured data (FAQPage, HowTo, Article)
- Author bio with credentials and contact
| Signal | SEO weight | AEO weight |
|---|---|---|
| Backlinks | High | Medium |
| Author credentials | Low | High |
| Schema markup | Medium | High |
Contact us at info@acme.com for our corrections policy.
"""
def main():
p = argparse.ArgumentParser(
description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
p.add_argument("--input", help="Path to markdown/HTML file to audit")
p.add_argument("--url", help="Live URL to fetch + audit")
p.add_argument("--industry", default="saas", choices=list(INDUSTRIES.keys()),
help="Industry-aware thresholds (default: saas)")
p.add_argument("--output", choices=["markdown", "json"], default="markdown",
help="Output format (default: markdown)")
p.add_argument("--sample", action="store_true",
help="Run with built-in sample content")
args = p.parse_args()
if args.sample:
text = SAMPLE_CONTENT
url = "sample://acme/blog/aeo-2026"
elif args.input:
text = Path(args.input).read_text(encoding="utf-8")
url = None
elif args.url:
text = fetch_url(args.url)
if text is None:
sys.exit(1)
url = args.url
else:
p.error("must specify --input, --url, or --sample")
result = audit(text, url, args.industry)
if args.output == "json":
print(json.dumps(result, indent=2, default=str))
else:
print(render_markdown(result))
if __name__ == "__main__":
main()
FILE:scripts/aeo_optimizer.py
#!/usr/bin/env python3
"""
aeo_optimizer.py — Generate AEO-optimized content variants.
Takes markdown content + audit recommendations, produces an optimized variant
with: structure fixes, citation slots, schema.org JSON-LD, fact-first lede.
Three modes:
conservative — touch <10% of words; add only schema + citation markers
balanced — touch <30%; rewrite intro for fact-density; add structure
aggressive — full restructure for maximum AEO
Stdlib only. Deterministic transformations — does NOT call any LLM (the
recommendations come from aeo_audit.py).
Usage:
python3 aeo_optimizer.py --input post.md --mode balanced --output post-aeo.md
python3 aeo_optimizer.py --input post.md --industry healthcare --mode aggressive
python3 aeo_optimizer.py --sample
python3 aeo_optimizer.py --input post.md --mode balanced --output-format json
Source: distilled from aeo-box optimizer.py.
"""
import argparse
import json
import re
import sys
from datetime import datetime
from pathlib import Path
from typing import Any
MODES = ["conservative", "balanced", "aggressive"]
def extract_title(text: str) -> str:
"""Extract H1 or first non-empty line."""
h1_match = re.search(r"^#\s+(.+)$", text, flags=re.MULTILINE)
if h1_match:
return h1_match.group(1).strip()
for line in text.splitlines():
if line.strip():
return line.strip()[:120]
return "Untitled"
def extract_headings(text: str) -> list:
"""Return list of (level, text) for H2-H6."""
headings = []
for m in re.finditer(r"^(#{2,6})\s+(.+)$", text, flags=re.MULTILINE):
headings.append((len(m.group(1)), m.group(2).strip()))
return headings
def generate_jsonld(title: str, headings: list, industry: str, url: str | None = None) -> str:
"""Generate schema.org JSON-LD for Article + FAQPage (if H2s look like questions)."""
article = {
"@context": "https://schema.org",
"@type": "Article",
"headline": title,
"datePublished": datetime.utcnow().strftime("%Y-%m-%d"),
"author": {"@type": "Person", "name": "{{AUTHOR_NAME}}"},
"publisher": {"@type": "Organization", "name": "{{PUBLISHER}}"},
}
if url:
article["url"] = url
article["mainEntityOfPage"] = {"@type": "WebPage", "@id": url}
# Detect question-style H2s for FAQPage schema
question_h2s = [h[1] for h in headings if h[0] == 2 and (
h[1].endswith("?") or re.match(r"^(what|why|how|when|where|who|which|is|are|does|do|can)\b", h[1], re.IGNORECASE)
)]
blocks = [json.dumps(article, indent=2)]
if len(question_h2s) >= 2:
faq = {
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": q,
"acceptedAnswer": {"@type": "Answer", "text": "{{ANSWER_" + str(i) + "}}"},
}
for i, q in enumerate(question_h2s, 1)
],
}
blocks.append(json.dumps(faq, indent=2))
return "\n\n".join(f'<script type="application/ld+json">\n{b}\n</script>' for b in blocks)
def add_citation_markers(text: str, density: int = 3) -> tuple[str, int]:
"""Insert [N]-style citation markers after factual-looking sentences.
Heuristic: a sentence with a number/percentage/year is likely a fact.
Density caps insertions per 1000 words.
"""
word_count = len(text.split())
max_insertions = max(density, word_count // 250)
insertions = 0
def replace_fact(m):
nonlocal insertions
if insertions >= max_insertions:
return m.group(0)
sentence = m.group(0)
# Only mark if it has a fact-like signal
if re.search(r"\b(\d+(\.\d+)?%|\$\d|20\d{2}|\d{2,})\b", sentence) and "[" not in sentence:
insertions += 1
return sentence.rstrip(".") + f" [{insertions}]."
return sentence
new = re.sub(r"[^.!?]*[.!?]", replace_fact, text)
return new, insertions
def add_corrections_footer(text: str, industry: str) -> str:
"""Append a corrections + disclosure footer."""
footer = "\n\n---\n\n## Editorial Notes\n\n"
footer += "- **Corrections:** This article will be updated as new information becomes available. Email corrections@example.com.\n"
if industry in ("healthcare", "finance", "legal"):
footer += f"- **{industry.title()} Disclaimer:** This article is for informational purposes only and does not constitute professional {industry} advice. Consult a licensed professional for your specific situation.\n"
footer += "- **Disclosure:** {{INSERT_DISCLOSURE: affiliations, sponsorships, conflicts of interest}}\n"
return text.rstrip() + footer
def fact_first_lede(text: str, title: str) -> str:
"""Move the first verifiable fact into the lede position (after H1)."""
# Find first paragraph with a number/year/percentage
lines = text.splitlines()
h1_idx = -1
for i, line in enumerate(lines):
if line.startswith("# "):
h1_idx = i
break
if h1_idx == -1:
return text
# Find first fact-bearing paragraph after H1
fact_idx = -1
for i in range(h1_idx + 1, len(lines)):
if re.search(r"\b(\d+(\.\d+)?%|\$\d|\b20\d{2}\b|\d{2,}\s+(percent|pages|customers|users))\b", lines[i]):
fact_idx = i
break
if fact_idx == -1 or fact_idx == h1_idx + 1 or fact_idx <= h1_idx + 2:
# Already near top
return text
# Move that paragraph to right after H1
fact_line = lines.pop(fact_idx)
lines.insert(h1_idx + 2, fact_line)
return "\n".join(lines)
def restructure_headings(text: str) -> str:
"""Promote bold-then-paragraph to H3, and ensure consistent H2 spacing."""
# Convert lines that look like **Bold heading** followed by a paragraph into H3
pattern = re.compile(r"^\*\*([A-Z][^*]+)\*\*\s*$", re.MULTILINE)
text = pattern.sub(r"### \1", text)
return text
def optimize(text: str, mode: str, industry: str, url: str | None = None) -> dict:
"""Apply optimizations based on mode. Returns dict with optimized text + changelog."""
title = extract_title(text)
headings = extract_headings(text)
changelog = []
result_text = text
# All modes: add schema JSON-LD at the end (or top)
jsonld = generate_jsonld(title, headings, industry, url)
schema_block = f"\n\n---\n\n<!-- AEO Schema.org markup -->\n{jsonld}\n"
if mode == "conservative":
# Schema + corrections footer only — no body changes
result_text = add_corrections_footer(result_text, industry)
result_text = result_text.rstrip() + schema_block
changelog.append("Added schema.org JSON-LD (Article + FAQPage if applicable)")
changelog.append("Added editorial notes / corrections / disclosure footer")
elif mode == "balanced":
# Schema + corrections + citation markers + heading restructure
result_text = restructure_headings(result_text)
result_text, insertions = add_citation_markers(result_text, density=5)
result_text = add_corrections_footer(result_text, industry)
result_text = result_text.rstrip() + schema_block
changelog.append("Promoted bold-paragraph patterns to H3 for LLM parsability")
changelog.append(f"Added {insertions} citation markers at factual claims")
changelog.append("Added schema.org JSON-LD")
changelog.append("Added editorial notes / corrections / disclosure footer")
elif mode == "aggressive":
# All of balanced + fact-first lede
result_text = fact_first_lede(result_text, title)
result_text = restructure_headings(result_text)
result_text, insertions = add_citation_markers(result_text, density=10)
result_text = add_corrections_footer(result_text, industry)
result_text = result_text.rstrip() + schema_block
changelog.append("Moved first factual claim to fact-first lede position")
changelog.append("Promoted bold-paragraph patterns to H3")
changelog.append(f"Added {insertions} citation markers at factual claims")
changelog.append("Added schema.org JSON-LD")
changelog.append("Added editorial notes / corrections / disclosure footer")
return {
"mode": mode,
"industry": industry,
"title": title,
"optimized_at": datetime.utcnow().isoformat() + "Z",
"original_word_count": len(text.split()),
"optimized_word_count": len(result_text.split()),
"changelog": changelog,
"optimized_content": result_text,
}
SAMPLE_CONTENT = """# Why AEO Matters
Answer Engine Optimization helps content get cited by LLMs.
**Key trends in 2026**
The industry has seen 47% growth in LLM citations vs 2024.
**Tactical recommendations**
Add schema, dated examples, and author credentials.
"""
def main():
p = argparse.ArgumentParser(
description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
p.add_argument("--input", help="Markdown file to optimize")
p.add_argument("--output", help="Output file path (default: stdout)")
p.add_argument("--mode", default="balanced", choices=MODES,
help="Optimization aggressiveness (default: balanced)")
p.add_argument("--industry", default="saas",
choices=["saas", "healthcare", "finance", "legal", "ecommerce", "b2b", "media", "education"],
help="Industry-aware optimizations (default: saas)")
p.add_argument("--url", help="Canonical URL to inject into schema.org markup")
p.add_argument("--output-format", choices=["markdown", "json"], default="markdown",
help="Output format (default: markdown — emits the optimized content directly)")
p.add_argument("--sample", action="store_true", help="Run with built-in sample content")
args = p.parse_args()
if args.sample:
text = SAMPLE_CONTENT
elif args.input:
text = Path(args.input).read_text(encoding="utf-8")
else:
p.error("must specify --input or --sample")
result = optimize(text, args.mode, args.industry, args.url)
if args.output_format == "json":
out = json.dumps(result, indent=2, default=str)
else:
# Markdown mode: emit the optimized content + the changelog as a comment
out = result["optimized_content"]
out += "\n\n<!--\nAEO Optimization Changelog:\n"
for c in result["changelog"]:
out += f" - {c}\n"
out += f" Mode: {result['mode']}, Industry: {result['industry']}\n"
out += f" Original: {result['original_word_count']} words → Optimized: {result['optimized_word_count']} words\n"
out += "-->\n"
if args.output:
Path(args.output).write_text(out, encoding="utf-8")
sys.stderr.write(f"[aeo_optimizer] wrote {args.output}\n")
else:
print(out)
if __name__ == "__main__":
main()
FILE:scripts/citation_tracker.py
#!/usr/bin/env python3
"""
citation_tracker.py — Local-first citation ledger for AEO.
Tracks when/where your content gets cited by LLMs (ChatGPT, Perplexity,
Claude, Gemini, Mistral). Stores entries in ~/.aeo-data/citations.json
(local, no telemetry). Stdlib only.
Actions:
add — log a citation you observed in an LLM response
list — list all citations or filter by --url / --llm / --since
report — per-URL aggregate: count, LLM coverage, velocity, top queries
export — emit CSV for reporting
Usage:
python3 citation_tracker.py --action add --url https://x.com/post \
--llm perplexity --query "what is AEO" --date 2026-05-17 --notes "first half of response"
python3 citation_tracker.py --action list --url https://x.com/post
python3 citation_tracker.py --action report --url https://x.com/post
python3 citation_tracker.py --action export --output citations.csv
python3 citation_tracker.py --sample
Source: distilled from aeo-box citation_tracker.py.
"""
import argparse
import csv
import json
import sys
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
SUPPORTED_LLMS = ["chatgpt", "perplexity", "claude", "gemini", "mistral", "copilot", "brave", "you", "other"]
def _data_dir() -> Path:
"""Return the local data directory, creating if needed."""
d = Path.home() / ".aeo-data"
d.mkdir(parents=True, exist_ok=True)
return d
def _ledger_path() -> Path:
return _data_dir() / "citations.json"
def _load_ledger() -> dict:
path = _ledger_path()
if not path.exists():
return {"schema_version": 1, "citations": []}
try:
return json.loads(path.read_text(encoding="utf-8"))
except json.JSONDecodeError as e:
sys.stderr.write(f"[citation_tracker] WARN: ledger file corrupted ({e}); starting fresh\n")
return {"schema_version": 1, "citations": []}
def _save_ledger(ledger: dict) -> Path:
path = _ledger_path()
path.write_text(json.dumps(ledger, indent=2), encoding="utf-8")
return path
def add_citation(url: str, llm: str, query: str, date: str | None = None,
notes: str = "", position: str = "") -> dict:
"""Add a citation entry. Returns the saved entry."""
if llm.lower() not in SUPPORTED_LLMS:
sys.stderr.write(f"[citation_tracker] WARN: unknown LLM '{llm}' (allowed: {SUPPORTED_LLMS})\n")
entry = {
"id": _make_id(),
"url": url,
"llm": llm.lower(),
"query": query,
"date": date or datetime.now(timezone.utc).date().isoformat(),
"logged_at": datetime.now(timezone.utc).isoformat(),
"notes": notes,
"position": position,
}
ledger = _load_ledger()
ledger["citations"].append(entry)
_save_ledger(ledger)
return entry
def _make_id() -> str:
"""8-char ID from current timestamp."""
return datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S-%f")[:21]
def list_citations(url: str | None = None, llm: str | None = None,
since: str | None = None) -> list:
"""List citations matching filters."""
ledger = _load_ledger()
cits = ledger["citations"]
if url:
cits = [c for c in cits if c["url"] == url]
if llm:
cits = [c for c in cits if c["llm"] == llm.lower()]
if since:
cits = [c for c in cits if c["date"] >= since]
return cits
def report(url: str | None = None) -> dict:
"""Generate aggregate report. If URL specified, per-URL stats.
Otherwise, full-ledger summary."""
cits = list_citations(url=url) if url else _load_ledger()["citations"]
if not cits:
return {
"url": url,
"total_citations": 0,
"llms_covered": [],
"verdict": "NO_DATA",
}
by_llm = {}
by_query = {}
by_date = {}
for c in cits:
by_llm[c["llm"]] = by_llm.get(c["llm"], 0) + 1
by_query[c["query"]] = by_query.get(c["query"], 0) + 1
by_date[c["date"]] = by_date.get(c["date"], 0) + 1
top_queries = sorted(by_query.items(), key=lambda kv: -kv[1])[:10]
dates = sorted(by_date.keys())
# Velocity: citations per day, rolling 30 days
velocity = 0.0
if len(dates) >= 2:
first = datetime.fromisoformat(dates[0])
last = datetime.fromisoformat(dates[-1])
days = max((last - first).days, 1)
velocity = round(len(cits) / days, 2)
verdict = "STRONG" if len(by_llm) >= 3 and len(cits) >= 10 else \
"EMERGING" if len(cits) >= 3 else \
"EARLY"
return {
"url": url,
"total_citations": len(cits),
"llms_covered": sorted(by_llm.keys()),
"llm_coverage_count": len(by_llm),
"citations_per_llm": by_llm,
"top_queries": top_queries,
"first_citation_date": dates[0] if dates else None,
"last_citation_date": dates[-1] if dates else None,
"velocity_per_day": velocity,
"verdict": verdict,
"interpretation": {
"STRONG": "Cited by 3+ LLMs with steady volume — content has citation moat",
"EMERGING": "Cited multiple times but not yet cross-LLM — push for coverage breadth",
"EARLY": "Few or no citations — keep optimizing + waiting for LLM training refresh",
"NO_DATA": "No citations recorded yet",
}.get(verdict, ""),
}
def export_csv(output_path: str) -> int:
"""Export the full citation ledger as CSV. Returns row count."""
ledger = _load_ledger()
cits = ledger["citations"]
fieldnames = ["id", "url", "llm", "query", "date", "logged_at", "notes", "position"]
with open(output_path, "w", encoding="utf-8", newline="") as f:
w = csv.DictWriter(f, fieldnames=fieldnames)
w.writeheader()
for c in cits:
w.writerow({k: c.get(k, "") for k in fieldnames})
return len(cits)
def render_human(action: str, data: Any) -> str:
"""Render results as human-readable text."""
if action == "add":
return (f"✅ Logged citation:\n"
f" URL: {data['url']}\n"
f" LLM: {data['llm']}\n"
f" Query: {data['query']}\n"
f" Date: {data['date']}\n"
f" ID: {data['id']}")
if action == "list":
if not data:
return "(no citations match the filters)"
lines = [f"Found {len(data)} citation(s):"]
for c in data:
lines.append(f" [{c['date']}] {c['llm']:12s} ← {c['url']}")
lines.append(f" query: {c['query']}")
if c.get("notes"):
lines.append(f" notes: {c['notes']}")
return "\n".join(lines)
if action == "report":
if data.get("total_citations", 0) == 0:
return f"📊 Report ({data.get('url') or 'all'}):\n No citations recorded yet."
lines = [
f"📊 Citation Report — {data.get('url') or 'ALL URLs'}",
f"",
f" Total citations: {data['total_citations']}",
f" LLMs covered: {data['llm_coverage_count']} ({', '.join(data['llms_covered'])})",
f" First citation: {data['first_citation_date']}",
f" Last citation: {data['last_citation_date']}",
f" Velocity: {data['velocity_per_day']} citations/day",
f" Verdict: {data['verdict']}",
f" Interpretation: {data['interpretation']}",
"",
" Citations per LLM:",
]
for llm, n in sorted(data["citations_per_llm"].items(), key=lambda kv: -kv[1]):
lines.append(f" {llm:12s} {n}")
lines.append("")
lines.append(" Top queries:")
for q, n in data["top_queries"]:
lines.append(f" ({n:2d}) {q}")
return "\n".join(lines)
if action == "export":
return f"✅ Exported {data} citations to CSV"
return json.dumps(data, indent=2, default=str)
def _run_sample():
"""Populate sample data + show all actions."""
sample_path = Path.home() / ".aeo-data" / "citations.sample.json"
# Use a separate sample file to avoid clobbering real data
actual_path = _ledger_path()
backup = None
if actual_path.exists():
backup = actual_path.read_text(encoding="utf-8")
try:
# Write fresh ledger for the sample
_save_ledger({"schema_version": 1, "citations": []})
add_citation("https://example.com/blog/aeo-guide", "perplexity",
"what is answer engine optimization", "2026-05-10",
notes="cited in first half of response")
add_citation("https://example.com/blog/aeo-guide", "chatgpt",
"how to optimize content for ChatGPT", "2026-05-12")
add_citation("https://example.com/blog/aeo-guide", "claude",
"AEO vs SEO differences", "2026-05-15")
add_citation("https://example.com/blog/aeo-guide", "perplexity",
"best AEO practices 2026", "2026-05-16")
add_citation("https://example.com/blog/llm-citations", "gemini",
"how do LLMs choose citations", "2026-05-14")
print("=== Sample: add ===")
print(render_human("add", {"url": "https://example.com/blog/aeo-guide",
"llm": "perplexity",
"query": "what is AEO",
"date": "2026-05-10",
"id": "sample-001"}))
print("")
print("=== Sample: list (filtered by URL) ===")
cits = list_citations(url="https://example.com/blog/aeo-guide")
print(render_human("list", cits))
print("")
print("=== Sample: report ===")
r = report(url="https://example.com/blog/aeo-guide")
print(render_human("report", r))
print("")
print("=== Sample: export ===")
n = export_csv(str(Path.home() / ".aeo-data" / "citations.sample.csv"))
print(render_human("export", n))
print(f" → wrote {Path.home() / '.aeo-data' / 'citations.sample.csv'}")
finally:
# Restore the user's real ledger
if backup is not None:
actual_path.write_text(backup, encoding="utf-8")
else:
if actual_path.exists():
actual_path.unlink()
def main():
p = argparse.ArgumentParser(
description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter,
)
p.add_argument("--action", choices=["add", "list", "report", "export"],
help="What to do")
p.add_argument("--url", help="Page URL (for add/list/report)")
p.add_argument("--llm", help="LLM that cited (chatgpt, perplexity, claude, gemini, mistral, ...)")
p.add_argument("--query", help="The query that triggered the citation (for add)")
p.add_argument("--date", help="Date the citation was observed (YYYY-MM-DD; defaults to today)")
p.add_argument("--notes", default="", help="Optional notes (for add)")
p.add_argument("--position", default="", help="Where in the LLM response the citation appeared (for add)")
p.add_argument("--since", help="List/report filter: YYYY-MM-DD")
p.add_argument("--output", help="Path for CSV export (action=export)")
p.add_argument("--output-format", choices=["human", "json"], default="human")
p.add_argument("--sample", action="store_true",
help="Populate sample data + show all actions (preserves your real ledger)")
args = p.parse_args()
if args.sample:
_run_sample()
return
if not args.action:
p.error("--action is required (or use --sample)")
if args.action == "add":
if not (args.url and args.llm and args.query):
p.error("add requires --url, --llm, --query")
result = add_citation(args.url, args.llm, args.query, args.date, args.notes, args.position)
elif args.action == "list":
result = list_citations(args.url, args.llm, args.since)
elif args.action == "report":
result = report(args.url)
elif args.action == "export":
if not args.output:
args.output = str(Path.home() / ".aeo-data" / "citations.csv")
result = export_csv(args.output)
else:
p.error(f"unknown action {args.action}")
if args.output_format == "json":
print(json.dumps(result, indent=2, default=str))
else:
print(render_human(args.action, result))
if __name__ == "__main__":
main()
Tạo kế hoạch thực thi 90 ngày từ quyết định đã duyệt, gồm mốc hằng tuần, người chịu trách nhiệm và nhịp kiểm tra.
---
name: "execute"
description: "/cs:execute <decision> — Generate a 90-day execution plan with weekly milestones, DRIs, and check-in cadence from an approved decision."
---
# /cs:execute — 90-Day Execution Plan
**Command:** `/cs:execute <decision-path>`
Turns an approved decision into a 90-day plan with weekly milestones, named DRIs, and a check-in cadence. Where most decisions die: between "we decided" and "what's next Monday?"
## Pipeline Position
```
/cs:office-hours → /cs:brief → /cs:boardroom → /cs:decide → /cs:execute → /cs:post-mortem
↑ you are here
```
## Input
An approved decision record (output of `/cs:decide`).
## Output Plan Format
Saved to `~/.claude/execution/YYYY-MM-DD-<slug>.md`:
```markdown
# Execution Plan: <decision title>
**Decision:** <link to /cs:decide record>
**Owner (Sponsor):** <founder or exec>
**Start:** YYYY-MM-DD
**Checkpoint:** YYYY-MM-DD (90d)
## Outcome (binding)
[Copied from decision: success + kill criteria]
## Workstreams
| Workstream | DRI | Success Metric | Status |
|---|---|---|---|
| <e.g., Pricing rollout> | <name> | <metric, threshold> | Not started |
| <e.g., Comms> | <name> | <metric> | Not started |
| <e.g., Eng changes> | <name> | <metric> | Not started |
## Weekly Milestones
| Week | Milestone | DRI | Definition of Done |
|---|---|---|---|
| 1 | <e.g., positioning locked> | <name> | <observable outcome> |
| 2 | <e.g., draft launched> | <name> | <observable> |
| 3 | ... | | |
| 12 | <e.g., checkpoint review> | <name> | <observable> |
## Cadence
- **Weekly:** Owner reviews status (15 min)
- **Bi-weekly:** Cross-functional sync (30 min)
- **Day 30 / 60 / 90:** Checkpoint with cs-chief-of-staff
## Dependencies
- Internal: <list>
- External: <vendors, regulators, customers>
## Risk Register
| Risk | Likelihood | Impact | Owner | Mitigation |
|---|---|---|---|---|
| <e.g., delayed legal review> | M | H | <name> | <plan> |
## Kill Criteria Watch
[Copied from decision; reviewed at every checkpoint]
- <metric, threshold, action>
```
## Workflow
1. Read the decision record
2. Decompose the chosen option into 3-6 workstreams
3. Name a DRI for each workstream
4. Reverse-engineer 12 weekly milestones from the checkpoint date
5. Set the cadence (weekly + bi-weekly + 30/60/90 checkpoints)
6. Build the risk register (cross-reference original Phase 4 devil's-advocate concerns)
7. Save and notify DRIs
## Why 90 Days
- Long enough to show real signal (not just activity)
- Short enough to course-correct before damage compounds
- Matches quarterly OKR cycle, fundraise sprints, and most board cadences
## Routing
- `/cs:post-mortem <decision>` — at day 90 (or earlier if kill criteria trigger)
- `/cs:boardroom` — if a checkpoint reveals a need to re-decide
## Related
- Skills: [`coo-advisor`](../../../skills/coo-advisor/SKILL.md), [`strategic-alignment`](../../../skills/strategic-alignment/SKILL.md), [`change-management`](../../../skills/change-management/SKILL.md)
- Agent: [`cs-coo-advisor`](../../agents/cs-coo-advisor.md)
---
**Version:** 1.0.0
Cố vấn ở vai trò giám đốc AI (CAIO): chiến lược AI, quản trị và triển khai AI trong tổ chức.
../../../c-level-advisor/chief-ai-officer-advisor/skills/chief-ai-officer-advisor/SKILL.md