{ "cells": [ { "cell_type": "markdown", "metadata": { }, "source": [ "*Module 12 Self-Driven Lab*\r\n", "\r\n", "# Enterprise Prospecting \u0026 Outreach Orchestrator\r\n", "\r\n", "**Objective:** You are building the **Enterprise Prospecting \u0026 Outreach Orchestrator**: a system that identifies high-value prospects, performs deep research using a multi-agent hub-and-spoke model, generates high-volume outreach campaigns, and maintains strict reliability through deterministic hooks and context compaction. Every concept from Modules 1 through 11 lands in this single project.\r\n", "\r\n", "## Challenge Outline\r\n", "\r\n", "Build a complete notebook that demonstrates the following outcomes:\r\n", "\r\n", "- **Agent Definition \u0026 Data Governance:** define the Coordinator Agent, data residency posture, adaptive thinking settings, and governance assumptions.\r\n", "- **Knowledge Mapping with MCP \u0026 Resources:** connect CRM knowledge through MCP configuration and Resources so the agent can inspect schemas before tool calls.\r\n", "- **Multi-Agent Research Workflow:** implement and verify this capstone phase in the notebook.\r\n", "- **Synthesis \u0026 Scaled Outreach:** synthesize findings and prepare scaled outreach through Advisor-style tools and Message Batches.\r\n", "- **Production Reliability \u0026 Context Hygiene:** add context hygiene, compaction, hooks, and deterministic safety checks for long-running work.\r\n", "- **Claude Code \u0026 Skills:** capture project rules, skills, and independent review automation for generated outreach.\r\n", "- **Evaluation \u0026 Success Metrics:** define quantitative and qualitative success metrics for reliability, recovery, and output quality.\r\n", "\r\n", "Your solution should include enough code, output, or written observations to prove each outcome worked. Keep the notebook focused on final behavior and evidence rather than a guided walkthrough.\n" ] }, { "cell_type": "markdown", "metadata": { }, "source": [ "## Student Workspace\n", "\n", "Use the sections below to build your solution. Each section maps to one required outcome from the challenge outline.\n" ] }, { "cell_type": "markdown", "metadata": { }, "source": [ "### Part 1: Agent Definition \u0026 Data Governance\n", "\n", "Define the Coordinator Agent, data residency posture, adaptive thinking settings, governance assumptions, and the interview pattern. Before generating a research plan, the Coordinator should ask 2-3 clarifying questions about tone constraints, competitor sensitivity, technical depth, and approval-sensitive claims.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { }, "outputs": [ ], "source": [ "# Part 1: Agent Definition \u0026 Data Governance\n", "# Add your implementation, outputs, or notes here.\n" ] }, { "cell_type": "markdown", "metadata": { }, "source": [ "### Part 2: Knowledge Mapping with MCP \u0026 Resources\n", "\n", "Connect CRM knowledge through MCP configuration and Resources so the agent can inspect schemas before tool calls. Engineer CRM tool descriptions so the model prefers MCP CRM tools over built-in text tools like Grep for structured data lookups, and use Resources as catalogs to reduce exploratory discovery calls.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { }, "outputs": [ ], "source": [ "# Part 2: Knowledge Mapping with MCP \u0026 Resources\n", "# Add your implementation, outputs, or notes here.\n" ] }, { "cell_type": "markdown", "metadata": { }, "source": [ "### Part 3: Multi-Agent Research Workflow\n", "\n", "Implement and verify this capstone phase in the notebook.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { }, "outputs": [ ], "source": [ "# Part 3: Multi-Agent Research Workflow\n", "# Add your implementation, outputs, or notes here.\n" ] }, { "cell_type": "markdown", "metadata": { }, "source": [ "### Part 4: Synthesis \u0026 Scaled Outreach\n", "\n", "Synthesize findings and prepare scaled outreach through Advisor-style tools and Message Batches. Put key findings at the beginning of aggregated research input, organize evidence under explicit section headers, and use other-plus-detail schema fields for extensible categories such as industry or reason for outreach.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { }, "outputs": [ ], "source": [ "# Part 4: Synthesis \u0026 Scaled Outreach\n", "# Add your implementation, outputs, or notes here.\n" ] }, { "cell_type": "markdown", "metadata": { }, "source": [ "### Part 5: Production Reliability \u0026 Context Hygiene\n", "\n", "Add context hygiene, compaction, hooks, and deterministic safety checks for long-running work.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { }, "outputs": [ ], "source": [ "# Part 5: Production Reliability \u0026 Context Hygiene\n", "# Add your implementation, outputs, or notes here.\n" ] }, { "cell_type": "markdown", "metadata": { }, "source": [ "### Part 6: Claude Code \u0026 Skills\n", "\n", "Capture project rules, skills, and independent review automation for generated outreach. Implement generate-audit using three-level progressive disclosure: trigger-rich frontmatter, actionable SKILL.md steps and examples, and detailed compliance references loaded only when needed.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { }, "outputs": [ ], "source": [ "# Part 6: Claude Code \u0026 Skills\n", "# Add your implementation, outputs, or notes here.\n" ] }, { "cell_type": "markdown", "metadata": { }, "source": [ "### Part 7: Evaluation \u0026 Success Metrics\n", "\n", "Define quantitative and qualitative success metrics for reliability, recovery, and output quality. Include independent review in a fresh Claude session and stratified random sampling across prospect segments and source types.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { }, "outputs": [ ], "source": [ "# Part 7: Evaluation \u0026 Success Metrics\n", "# Add your implementation, outputs, or notes here.\n" ] }, { "cell_type": "markdown", "metadata": { }, "source": [ "### Verification Notes\n", "\n", "Summarize the evidence that each part worked. Capture API signals, validation outcomes, errors, recovery behavior, cost observations, or comparisons required by this lab.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { }, "outputs": [ ], "source": [ "# Verification notes\n", "# Record the evidence that proves each lab outcome worked.\n" ] }, { "cell_type": "markdown", "metadata": { }, "source": [ "---\n", "\n", "## Answer Key\n", "\n", "The cells below contain the completed reference implementation/content for this module. Use this section only after attempting the self-driven lab." ] }, { "cell_type": "markdown", "metadata": { }, "source": [ "*Final Project*\n", "\n", "# Enterprise Prospecting \u0026 Outreach Orchestrator\n", "\n", "The capstone integrates the Claude API, the Claude Agent SDK, and Claude Code into a single production system. You will build a Coordinator Agent that decomposes prospecting requests into parallel research subagents, synthesizes findings via the Advisor tool, ships outreach at scale through Message Batches, and stays reliable under long-running sessions with hooks, compaction, and case facts.\n", "\n", "\n", "## Project Overview\n", "\n", "You are building the **Enterprise Prospecting \u0026 Outreach Orchestrator**: a system that identifies high-value prospects, performs deep research using a multi-agent hub-and-spoke model, generates high-volume outreach campaigns, and maintains strict reliability through deterministic hooks and context compaction. Every concept from Modules 1 through 11 lands in this single project.\n", "\n", "## Phase 1: Agent Definition \u0026 Data Governance\n", "\n", "Define the Coordinator Agent with the Claude Agent SDK so the runtime owns history, tool dispatch, and termination. Pair it with adaptive thinking, US data residency, and high effort at call time.\n", "\n", "- **Model \u0026 Capabilities:** `claude-opus-4-7` with adaptive thinking so Claude can dynamically determine reasoning depth.\n", "- **Data Residency:** set `inference_geo: \"us\"` on each call. Note the **1.1x pricing multiplier** on Opus 4.6+.\n", "- **Effort Control:** set `effort: \"xhigh\"` for long-horizon prospecting work.\n", "- **Interview Pattern:** before planning, the Coordinator must ask 2-3 clarifying questions about outreach goals, such as tone constraints, competitor sensitivity, required technical depth, and any claims that require human approval." ] }, { "cell_type": "code", "execution_count": null, "metadata": { }, "outputs": [ ], "source": [ "import os\n", "\n", "import anthropic\n", "from dotenv import load_dotenv\n", "\n", "load_dotenv() # reads ANTHROPIC_API_KEY from your .env file\n", "\n", "client = anthropic.Anthropic()\n", "\n", "coordinator_agent = client.beta.agents.create(\n", " name=\"Enterprise Research Coordinator\",\n", " model=\"claude-opus-4-7\",\n", " description=\"Orchestrates prospecting research and outreach strategy.\",\n", " system=(\n", " \"You are the lead orchestrator. Use the Task tool to delegate to \"\n", " \"specialist subagents. Always maintain a \u0027case facts\u0027 block with \"\n", " \"non-negotiable transactional data. Before generating the research plan, ask 2-3 concise clarifying questions about outreach goals, including tone, competitor sensitivity, and technical depth.\"\n", " ),\n", " tools=[\n", " {\n", " \"type\": \"agent_toolset_20260401\",\n", " \"configs\": [\n", " {\"name\": \"web_search\", \"enabled\": True, \"permission_policy\": {\"type\": \"always_allow\"}},\n", " {\"name\": \"web_fetch\", \"enabled\": True, \"permission_policy\": {\"type\": \"always_allow\"}},\n", " ],\n", " }\n", " ],\n", ")" ] }, { "cell_type": "markdown", "metadata": { }, "source": [ "## Phase 2: Knowledge Mapping with MCP \u0026 Resources\n", "\n", "Connect the CRM directly to the Messages API with the MCP Connector instead of paying for exploratory tool calls. Expose Resources as a navigable map of the data so Claude can plan its tool sequence before executing.\n", "\n", "- **Beta Header:** `mcp-client-2025-11-20`.\n", "- **Project Config:** `.mcp.json` with environment variable substitution (`${CRM_TOKEN}`) keeps secrets out of version control.\n", "- **CRM Resource Hierarchy:** expose `crm://docs`, `crm://schemas`, and `crm://workflows` resources so Claude can inspect the CRM\u0027s documentation hierarchy before calling query or write tools.\n", "- **Selection Reliability:** tool descriptions must explain why CRM MCP tools are preferred over built-in text tools such as `Grep` for structured data lookups.\n", "- **Discovery Tax Reduction:** Resources act as catalogs. The agent should inspect `crm://schemas` and `crm://workflows` before exploratory CRM queries instead of discovering tables through repeated tool calls.\n", "- **ZDR note:** the MCP Connector is **not ZDR-eligible** due to server-side routing.\n", "\n", "*.mcp.json*\n", "```json\n", "{\n", " \"mcpServers\": {\n", " \"crm\": {\n", " \"command\": \"node\",\n", " \"args\": [\"./mcp-servers/crm-server.js\"],\n", " \"env\": {\n", " \"CRM_TOKEN\": \"${CRM_TOKEN}\",\n", " \"CRM_BASE_URL\": \"${CRM_BASE_URL}\"\n", " }\n", " }\n", " }\n", "}\n", "```\n", "\n", "*CRM resources/list response*\n", "```json\n", "{\n", " \"resources\": [\n", " {\n", " \"uri\": \"crm://docs/prospecting\",\n", " \"name\": \"CRM prospecting documentation\",\n", " \"description\": \"Use before lead searches. Explains lead stages, required fields, and routing rules.\",\n", " \"mimeType\": \"text/markdown\"\n", " },\n", " {\n", " \"uri\": \"crm://schemas/leads\",\n", " \"name\": \"Leads schema\",\n", " \"description\": \"Use before CRM lead lookups. Lists column names, allowed enum values, and join keys so the agent chooses crm_query_leads instead of built-in Grep for structured account data.\",\n", " \"mimeType\": \"application/json\"\n", " },\n", " {\n", " \"uri\": \"crm://workflows/outreach-approval\",\n", " \"name\": \"Outreach approval workflow\",\n", " \"description\": \"Approval gates required before process_outreach can run.\",\n", " \"mimeType\": \"text/markdown\"\n", " }\n", " ]\n", "}\n", "```\n", "\n", "## Phase 3: Multi-Agent Research Workflow\n", "\n", "Hub-and-Spoke architecture: the Coordinator decomposes a prospecting request and spawns specialist subagents in parallel inside a single turn.\n", "\n", "- **Parallel Execution:** the Coordinator calls the `Task` tool multiple times to spawn subagents for Web Research, Financial Analysis, and Competitor Tracking.\n", "- **Isolated Context:** subagents do not inherit the Coordinator\u0027s history. Pass each subagent only the data it needs.\n", "- **Structured Errors:** subagents return `errorCategory` and `isRetryable` so the Coordinator can recover intelligently rather than failing the workflow." ] }, { "cell_type": "code", "execution_count": null, "metadata": { }, "outputs": [ ], "source": [ "session = client.beta.sessions.create(\n", " agent_id=coordinator_agent.id,\n", " mcp_servers=[\n", " {\n", " \"name\": \"crm-server\",\n", " \"type\": \"url\",\n", " \"url\": \"https://mcp.your-enterprise.com/crm\",\n", " \"authorization_token\": os.environ[\"CRM_TOKEN\"],\n", " }\n", " ],\n", ")\n", "\n", "# Coordinator turn: spawn parallel research subagents in a single message\n", "client.beta.sessions.events.send(\n", " session_id=session.id,\n", " event={\n", " \"type\": \"user.message\",\n", " \"content\": [{\n", " \"type\": \"text\",\n", " \"text\": (\n", " \"Research AI adoption in Fintech and Healthcare simultaneously. \"\n", " \"Spawn three Task subagents: web_research, financial_analysis, \"\n", " \"and competitor_tracking. Pass each only the prospect summary.\"\n", " ),\n", " }],\n", " },\n", ")" ] }, { "cell_type": "markdown", "metadata": { }, "source": [ "When using the Messages API directly, include the MCP Connector beta header on the call:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { }, "outputs": [ ], "source": [ "response = client.beta.messages.create(\n", " model=\"claude-opus-4-7\",\n", " max_tokens=4096,\n", " mcp_servers=[{\n", " \"name\": \"crm-server\",\n", " \"type\": \"url\",\n", " \"url\": \"https://mcp.your-enterprise.com/crm\",\n", " \"authorization_token\": os.environ[\"CRM_TOKEN\"],\n", " }],\n", " tools=[{\n", " \"type\": \"mcp_toolset\",\n", " \"mcp_server_name\": \"crm-server\",\n", " \"configs\": {\n", " \"resources/read\": {\"enabled\": True},\n", " \"crm_query\": {\"enabled\": True},\n", " \"process_outreach\": {\"enabled\": True},\n", " },\n", " }],\n", " betas=[\"mcp-client-2025-11-20\"],\n", " messages=[{\"role\": \"user\", \"content\": \"Map the CRM docs, then find fintech prospects.\"}],\n", ")" ] }, { "cell_type": "markdown", "metadata": { }, "source": [ "*subagent error handler*" ] }, { "cell_type": "code", "execution_count": null, "metadata": { }, "outputs": [ ], "source": [ "def subagent_tool_handler(tool_input):\n", " try:\n", " return {\"data\": run_research(tool_input)}\n", " except TimeoutError:\n", " return {\n", " \"errorCategory\": \"TIMEOUT\",\n", " \"isRetryable\": True,\n", " \"message\": \"Research API timed out. Retry with a narrower query.\"\n", " }\n", " except PermissionError:\n", " return {\n", " \"errorCategory\": \"PERMISSION\",\n", " \"isRetryable\": False,\n", " \"message\": \"Token lacks access. Coordinator should pivot, not retry.\"\n", " }" ] }, { "cell_type": "markdown", "metadata": { }, "source": [ "## Phase 4: Synthesis \u0026 Scaled Outreach\n", "\n", "Synthesize research findings, then ship outreach at volume.\n", "\n", "- **Advisor Tool (`advisor_20260301`):** Sonnet 4.6 executor handles writing while an Opus 4.7 advisor course-corrects mid-generation.\n", "- **Message Batches:** 50% discount on bulk outreach. Each request needs a unique `custom_id`; results return out of order.\n", "- **Extended Output:** `output-300k-2026-03-24` beta header raises `max_tokens` to 300,000 for book-length intelligence reports.\n", "- **Structured Extraction:** JSON-schema outputs with nullable fields for `annual_revenue`, `budget_range`, etc., to prevent hallucination when data is missing.\n", "- **Position-Aware Input Ordering:** the synthesis prompt must put `key_findings` and decision-critical summaries first, then detailed evidence under explicit prospect and source-type section headers.\n", "- **Extensible Categories:** prospect schemas must use an enum with `other` plus a required detail field for extensible fields such as `industry` or `reason_for_outreach`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { }, "outputs": [ ], "source": [ "aggregated_research = {\n", " \"key_findings\": [\n", " \"Fintech CTOs show high interest in agentic workflow governance.\",\n", " \"Healthcare prospects require stricter compliance language and human review.\"\n", " ],\n", " \"sections\": {\n", " \"prospect_startup_segment\": startup_evidence,\n", " \"prospect_fortune_500_segment\": enterprise_evidence,\n", " \"source_crm_notes\": crm_notes,\n", " \"source_web_research\": cited_web_findings,\n", " },\n", "}" ] }, { "cell_type": "code", "execution_count": null, "metadata": { }, "outputs": [ ], "source": [ "message_batch = client.beta.messages.batches.create(\n", " betas=[\"output-300k-2026-03-24\"],\n", " requests=[\n", " {\n", " \"custom_id\": \"outreach-campaign-001\",\n", " \"params\": {\n", " \"model\": \"claude-sonnet-4-6\",\n", " \"max_tokens\": 300000,\n", " \"messages\": [{\n", " \"role\": \"user\",\n", " \"content\": \"Generate a 100-page market intelligence report for Fintech CTOs.\"\n", " }],\n", " \"tools\": [{\n", " \"type\": \"advisor_20260301\",\n", " \"name\": \"advisor\",\n", " \"model\": \"claude-opus-4-7\",\n", " \"effort\": \"xhigh\",\n", " \"caching\": {\"type\": \"ephemeral\", \"ttl\": \"1h\"}\n", " }],\n", " \"output_config\": {\n", " \"format\": {\n", " \"type\": \"json_schema\",\n", " \"schema\": {\n", " \"type\": \"object\",\n", " \"properties\": {\n", " \"company_name\": {\"type\": \"string\"},\n", " \"contact_email\": {\"type\": \"string\"},\n", " \"annual_revenue\": {\"type\": [\"integer\", \"null\"]},\n", " \"budget_range\": {\"type\": [\"string\", \"null\"]},\n", " \"industry\": {\n", " \"type\": \"string\",\n", " \"enum\": [\"fintech\", \"healthcare\", \"manufacturing\", \"retail\", \"other\"]\n", " },\n", " \"industry_detail\": {\"type\": [\"string\", \"null\"]},\n", " \"reason_for_outreach\": {\n", " \"type\": \"string\",\n", " \"enum\": [\"ai_governance\", \"workflow_automation\", \"security_review\", \"other\"]\n", " },\n", " \"reason_detail\": {\"type\": [\"string\", \"null\"]}\n", " },\n", " \"required\": [\n", " \"company_name\", \"contact_email\", \"annual_revenue\", \"budget_range\",\n", " \"industry\", \"industry_detail\", \"reason_for_outreach\", \"reason_detail\"\n", " ],\n", " \"additionalProperties\": False\n", " }\n", " }\n", " }\n", " }\n", " }\n", " ]\n", ")" ] }, { "cell_type": "markdown", "metadata": { }, "source": [ "## Phase 5: Production Reliability \u0026 Context Hygiene\n", "\n", "Long-running research sessions need active context management and deterministic safety rails.\n", "\n", "- **Deterministic Hooks:** a `PreToolUse` hook blocks any `process_outreach` call if the prospect\u0027s `risk_score` has not been verified.\n", "- **Server-Side Compaction:** `compact_20260112` with a 50,000-token trigger summarizes verbose history automatically.\n", "- **Case Facts Pattern:** non-negotiable transactional data (*\"Max Budget: $50k\"*) lives in a persistent block so compaction never paraphrases it away.\n", "- **Human-in-the-Loop:** if a prospect requests \"no AI contact\" or a policy gap is detected, the agent must call `escalate_to_human` immediately.\n", "\n", "*compaction config*" ] }, { "cell_type": "code", "execution_count": null, "metadata": { }, "outputs": [ ], "source": [ "response = client.messages.create(\n", " model=\"claude-opus-4-7\",\n", " max_tokens=16000,\n", " thinking={\"type\": \"adaptive\"},\n", " effort=\"xhigh\",\n", " inference_geo=\"us\", # 1.1x multiplier; ZDR-eligible\n", " context_management={\n", " \"edits\": [\n", " {\n", " \"type\": \"compact_20260112\",\n", " \"trigger\": {\"input_tokens\": 50000},\n", " \"instructions\": \"Summarize research findings but preserve \u0027case facts\u0027 verbatim.\"\n", " }\n", " ]\n", " },\n", " extra_headers={\"anthropic-beta\": \"compact-2026-01-12\"},\n", " messages=conversation,\n", ")" ] }, { "cell_type": "markdown", "metadata": { }, "source": [ "*PreToolUse hook*\n", "```json\n", "{\n", " \"hooks\": {\n", " \"PreToolUse\": [\n", " {\n", " \"matcher\": {\"tool_name\": \"process_outreach\"},\n", " \"hooks\": [\n", " {\n", " \"type\": \"command\",\n", " \"command\": \"python verify_risk_score.py\"\n", " }\n", " ]\n", " }\n", " ]\n", " }\n", "}\n", "```\n", "\n", "## Phase 6: Claude Code \u0026 Skills\n", "\n", "Manage the project\u0027s development standards through Claude Code.\n", "\n", "- **Path-Scoped Rules:** a rule file in `.claude/rules/` with a `globs` frontmatter key enforces the standard outreach template only when Claude is editing files under `outreach/`.\n", "- **Agent Skills:** a `generate-audit` skill with `context: fork` performs deep compliance checks on generated emails without polluting the main development context.\n", "- **Three-Level Progressive Disclosure:** the skill must keep trigger logic in frontmatter, actionable workflow steps in `SKILL.md`, and detailed compliance material in `references/` so the agent only loads extra context when needed.\n", "- **CI Review Output:** use `claude -p` with `--output-format json` and `--json-schema` so automated review findings can be posted as inline PR comments.\n", "\n", "*.claude/rules/outreach-template.md*\n", "```markdown\n", "# Standard Outreach Template Rules\n", "- Open with the prospect\u0027s confirmed pain point from CRM, never a generic hook.\n", "- Include one quantified case study from the same industry vertical.\n", "- Close with a single, specific call to action (no menu of options).\n", "- Never reference compliance regimes the prospect has not opted into.\n", "```\n", "\n", "*.claude/skills/generate-audit/SKILL.md*\n", "```yaml\n", "---\n", "name: generate-audit\n", "description: Audit generated outreach emails when asked to review campaign copy, regulated claims, unsupported personalization, or compliance-sensitive prospect messaging.\n", "context: fork\n", "allowed-tools: [read_file, web_search]\n", "---\n", "# Audit Skill\n", "1. Read the email under review.\n", "2. Cross-check claims against the prospect\u0027s stated regulated regimes.\n", "3. Open `references/compliance-claims.md` only when the email mentions HIPAA, SOC 2, ISO 27001, privacy, safety, or procurement claims.\n", "4. Flag unsupported claims, weak personalization, policy conflicts, and missing escalation triggers.\n", "5. Return a structured JSON report with findings, severity, evidence, and recommended next action.\n", "```\n", "\n", "```bash\n", "claude -p \"Review this capstone PR and return inline findings.\" \\\n", " --output-format json \\\n", " --json-schema .claude/schemas/pr-review-findings.schema.json\n", "```\n", "\n", "## Phase 7: Evaluation \u0026 Success Metrics\n", "\n", "Evaluate the orchestrator as a product, not only as a demo.\n", "\n", "**Quantitative metrics:**\n", "\n", "- Skill trigger reliability: does the skill trigger on at least **90%** of relevant queries?\n", "- Tool efficiency: does the workflow complete in fewer tool calls than a baseline implementation?\n", "- Recovery quality: does the coordinator salvage `partial_results` instead of re-running failed searches?\n", "- Stratified accuracy: manually review 10 high-confidence extractions sampled across prospect segments, such as startups, mid-market accounts, and Fortune 500 firms, and across document/source types, such as annual reports, support tickets, CRM notes, and web research, to ensure aggregate accuracy is not hiding a weak segment.\n", "\n", "**Qualitative metrics:**\n", "\n", "- Can a new user accomplish the task on the first try with minimal guidance?\n", "- Are escalation summaries clear enough for a human operator to act without re-reading the whole transcript?\n", "- Are generated PR review findings specific enough to become inline comments?\n", "- Does a completely independent Claude instance, started in a fresh session with no prior reasoning context, catch unsupported claims or weak outreach that the generator missed?\n", "\n", "**Architect judgment tasks:**\n", "\n", "- **Independent review:** start a fresh Claude session with no prior reasoning context. Provide only the final outreach, source evidence, and review rubric. The reviewer must catch unsupported claims, policy issues, weak personalization, and missing escalations that the generator may have rationalized.\n", "- **Stratified random sampling:** sample extractions across prospect segments and source types rather than relying on aggregate accuracy. Include at least one startup, one mid-market account, one Fortune 500 account, one CRM-sourced record, and one web-researched record.\n", "\n", "## Final Audit Checklist\n", "\n", "\u003e **Tip.** Architect Tip for the Exam\n", "\n", "**ZDR check:** adaptive thinking, citations, structured outputs, and standard web search are **ZDR-eligible**; Message Batches and the MCP Connector are **not**. Batch results are stored server-side for 29 days. \n", "**Cost check:** `inference_geo: \"us\"` applies a **1.1x multiplier** on Opus 4.6+ across input, output, and cache. The same multiplier applies to Priority-Tier burndown. \n", "**Token management:** after compaction blocks fire, call `client.messages.count_tokens` on the outgoing payload to verify the effective context size. \n", "**Citations vs. Structured Outputs:** mutually exclusive. Sending both returns a 400. Choose grounded research *or* CRM-ready extraction per request. \n", "**Schema safety:** PHI must never appear in tool names, property names, or property descriptions because schemas are cached separately from message content. \n", "**Structured output limits:** maximum **20** strict tools, **24** optional parameters, and **16** union-type parameters per request. \n", "**Independent review:** final outreach must be audited by a fresh Claude session, not the generation session, so the reviewer is isolated from the generator\u0027s assumptions. \n", "**Subagents:** isolated context by default. The Coordinator must explicitly forward only the data each subagent needs." ] }, { "cell_type": "markdown", "metadata": { }, "source": [ "## Stratified Random Sampling\n", "\n", "Manually sample 10 high-confidence extractions across document segments so aggregate accuracy does not hide weak performance on one document type." ] }, { "cell_type": "code", "execution_count": null, "metadata": { }, "outputs": [ ], "source": [ "from collections import defaultdict\n", "\n", "records = [\n", " {\u0027id\u0027: f\u0027annual-{i}\u0027, \u0027segment\u0027: \u0027annual_report\u0027, \u0027confidence\u0027: 0.95} for i in range(4)\n", "] + [\n", " {\u0027id\u0027: f\u0027ticket-{i}\u0027, \u0027segment\u0027: \u0027support_ticket\u0027, \u0027confidence\u0027: 0.94} for i in range(3)\n", "] + [\n", " {\u0027id\u0027: f\u0027crm-{i}\u0027, \u0027segment\u0027: \u0027crm_note\u0027, \u0027confidence\u0027: 0.93} for i in range(3)\n", "]\n", "\n", "def stratified_high_confidence_sample(items, total=10):\n", " by_segment = defaultdict(list)\n", " for item in sorted(items, key=lambda row: row[\u0027confidence\u0027], reverse=True):\n", " by_segment[item[\u0027segment\u0027]].append(item)\n", " sample = []\n", " while len(sample) \u003c total and any(by_segment.values()):\n", " for segment in sorted(by_segment):\n", " if by_segment[segment] and len(sample) \u003c total:\n", " sample.append(by_segment[segment].pop(0))\n", " return sample\n", "\n", "sample = stratified_high_confidence_sample(records, total=10)\n", "segments = {item[\u0027segment\u0027] for item in sample}\n", "print([item[\u0027id\u0027] for item in sample])\n", "assert len(sample) == 10\n", "assert segments == {\u0027annual_report\u0027, \u0027support_ticket\u0027, \u0027crm_note\u0027}\n" ] }, { "cell_type": "markdown", "metadata": { }, "source": [ "## Independent Review Instance\n", "\n", "Run the final outreach audit in a fresh Claude session with no prior reasoning context. Do not resume the generator session; give the reviewer only the generated outreach, source evidence, policy, and review schema." ] }, { "cell_type": "code", "execution_count": null, "metadata": { }, "outputs": [ ], "source": [ "independent_review_command = \u0027claude -p \"Review this generated outreach in a fresh session for unsupported claims, policy issues, and weak personalization.\" --output-format json --json-schema .claude/schemas/outreach-review.schema.json\u0027\n", "\n", "assert \u0027--output-format json\u0027 in independent_review_command\n", "assert \u0027--json-schema\u0027 in independent_review_command\n", "assert \u0027--resume\u0027 not in independent_review_command\n", "print(independent_review_command)\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.0" } }, "nbformat": 4, "nbformat_minor": 5 }