Skip to main content

​
Overview

The PagerDuty integration gives Annie two capabilities:
  • API access — Annie reads your incidents, acknowledges them, and leaves comments via a PagerDuty API token.
  • Real-time webhook — When an incident.triggered event fires, Annie immediately starts a root cause analysis, cross-references recent infrastructure changes, and can notify Slack based on your project’s webhook investigation notification destinations.
  • Graph evidence — PagerDuty alerts, response incidents, services, and explicit on-call windows become read-only operational evidence for the Graph API, SDK, and CLI.

​
Step 1: Add an API Token

  1. In PagerDuty, go to Integrations → API Access Keys → Create New API Key.
  2. Give it a descriptive name (e.g. Anyshift-Annie) and copy the key — it is only shown once.
  3. In Anyshift, go to Integrations → PagerDuty → New Token, paste the key, and save.
Annie uses this token to fetch incident metadata, post acknowledgements, and add resolution comments.

​
Step 2: Configure the Webhook

The webhook delivers real-time incident lifecycle events to Anyshift. incident.triggered can start an Annie investigation. Triggered, acknowledged, and resolved lifecycle changes also update stored graph evidence without making the Graph API call PagerDuty directly.

​
Create the webhook in Anyshift

  1. Go to Integrations → PagerDuty → New Webhook.
  2. Enter a name (e.g. prod-webhook) and choose a secret key.
  3. Copy the Webhook URL and the Secret Key — you need both in the next step.

​
Register it in PagerDuty

  1. In PagerDuty, go to Integrations → Generic Webhooks (V3) → New Webhook.
  2. Paste the Anyshift webhook URL.
  3. Under Scope, select the services or the full account you want Annie to monitor.
  4. Enable the incident lifecycle event types you want Anyshift to retain, including triggered, acknowledged, and resolved.
  5. In the Signature Secret field, paste the secret key you set in Anyshift.
Anyshift verifies every incoming request using HMAC-SHA256 against the secret key. Requests with missing or invalid signatures are rejected before any processing occurs.

​
What happens when an incident fires

When Anyshift receives an incident.triggered event:
  1. Signature verification — the request is validated against the shared secret. Invalid requests are dropped immediately.
  2. Deduplication — repeated deliveries are applied idempotently, preventing duplicate graph state and duplicate RCAs.
  3. Context assembly — Annie retrieves the last 30–60 minutes of Slack messages, recent AWS resource changes, and prior incidents with completed analyses to build a context window.
  4. Routing decision — Annie’s router decides whether to run a full RCA, start a conversational thread, or skip the incident (e.g. for known-noisy alert patterns). Automation rules can override this default.
  5. Analysis delivery — if RCA is triggered, Annie delivers a final summary with a timeline, likely root cause, and recommended remediation steps. Slack notifications for start, complete, and fail follow the destinations configured under Integrations → Slack → Webhook investigation notifications. PagerDuty incidents typically have no attached Slack channel, so turn on Default notify channel and pick a mapped channel if you want Slack updates for these webhooks.
See Root Cause Analysis for a full walkthrough of the analysis pipeline, and Slack for destination settings.

​
Query PagerDuty operational evidence

The public Graph surface is provider-neutral, so the same alerting workflows can later include Datadog, New Relic, Grafana, incident.io, and other providers. There is no PagerDuty-specific CLI namespace.
--person accepts an exact display name, canonical person ID, or PagerDuty user ID. --responder accepts an exact display name, canonical person ID or email, or PagerDuty user ID. Display-name matching is case-insensitive and exact, not fuzzy. The Graph API and SDK expose the same normalized alerts, response_incidents, and oncall evidence. Results distinguish canonical Anyshift identities from unresolved PagerDuty users or services instead of guessing a mapping. These read paths do not acknowledge, reassign, or resolve incidents.

​
Grouped incident context

Use incident_context when you need one stored incident assembled as grouped hops. It reads the graph only; it never calls PagerDuty live. Require exactly one of id or target. Optional since bounds similar-incident history. LIMIT caps history rows (OFFSET is not supported). Returned hops: incident, alerts, service (AFFECTS / RESOLVES_TO), onCall, responders, and history. History cites reviewed resolution evidence only (confirmed_fix, explicit_reference, or unknown). Temporal proximity alone is never treated as a confirmed fix. annie graph triage <resource> includes optional incident_context hops for the named resource. Empty hops are omitted from triage findings.
See incident_context and the Annie CLI graph commands.

​
Map PagerDuty services to canonical graph resources

PAGERDUTY_SERVICE -[:RESOLVES_TO]-> edges are owned mappings. Topology polls never create them from display names. Name a mapping owner per project before creating any link; unowned mappings are out of scope.

​
Create, update, or revoke

  1. In PagerDuty, edit the service Description.
  2. Add only standalone directive lines (no prose on the same line):
  1. Save. The next operational graph poll stamps safe properties and enqueues resolution. Unique exact hostnames (when the whole service name or URL is a hostname) may also resolve when unambiguous.
  2. To update, change the directive. To revoke, delete it. Do not hand-edit Neo4j edges.
Fuzzy name matching is never used. Zero or multiple candidates stay unresolved or ambiguous.

​
Demo ownership

  • Owner: Platform Engineering for project Anyshift Demo Environment.
  • Runbook: this page and docs/pagerduty-service-resolves-to.md in anyshift-backend.
  • Demo service PUXIGO8 (Anyshift Demo - Kubernetes) maps via an explicit directive to the checkout workload. PK9AOSO (Default Service) stays unmapped.

​
Verify the connection

Trigger a test incident in PagerDuty. Within a few seconds Annie should start an investigation. The incident also appears in Anyshift under Incidents. If you enabled Slack destinations (especially Default notify channel), you should also see a notification in that channel.
Map the Slack channel you want as the default notify channel to the correct Anyshift project, enable Default notify channel on the Slack integrations page, and register your PagerDuty bot in the Annie On-Call Registry when you also rely on channel bots. See Slack Integration for setup details.

​
Customize Annie’s behavior

By default Annie decides automatically whether to run RCA or start a chat based on the incident context. You can override this per project using Automation:
  • Force RCA for all incidents from a specific service.
  • Suppress noisy or low-signal alert patterns.
  • Trigger a custom report instead of a standard RCA.
  • Silence Annie for specific channels or time windows.