jacobmckenzie.dev
Dallas–Fort Worth, Texas

Major incident management, engineered into the infrastructure itself.

Jacob McKenzie is a Major Incident Manager with seven years of enterprise incident management experience, including three in financial services. He operates a production-grade, self-hosted infrastructure estate and builds the AI-assisted operations tooling that runs it.

Major incident command Self-hosted infrastructure Agentic AI operations Available for incident, SRE and platform roles
Email Forgejo
Incident management

Seven years of incident management: four at 3M, then major incident command at two of the largest financial platforms in the United States.

Incident command, executive and stakeholder communication, and the post-incident discipline that converts outages into lasting improvements.

Senior Incident Management Analyst

Worldpay · payments processingSep 2024 – Jun 2025
  • Owned high-severity incidents end to end across a global payment-processing estate, from declaration and bridge command through engineering and vendor engagement to closure.
  • Authored and delivered stakeholder communications on a defined cadence throughout live incidents, translating technical status into merchant and executive impact.
  • Led post-incident reviews, including timeline reconstruction, contributing-factor analysis and problem records tracked through to prevention of recurrence.

Major Incident Manager

Capital One · bankingAug 2022 – Jul 2024
  • Commanded major incident bridges for customer-facing banking and card systems, coordinating responders across platform, application, network and third-party teams.
  • Enforced severity criteria, escalation paths and communication service levels, and maintained the single authoritative record of incident state as conditions changed.
  • Facilitated post-incident reviews with engineering leadership and converted findings into owned, dated corrective actions.

Incident Manager

3M · manufacturing and technology2018 – 2022
  • Managed the incident process across a global, multi-business enterprise for four years, coordinating resolution between internal IT, application owners and external service providers.
  • Ran incident bridges, maintained the incident record, and issued status communications to business and technology stakeholders throughout the lifecycle.
  • Contributed to problem management and continual improvement, feeding recurring-incident analysis into process and tooling changes.
7 yrsof enterprise incident management, three in major incident command
24 × 7on-call coverage across regulated financial platforms
Detect → Verifythe incident lifecycle, now implemented in software operated daily
Overseer · operations console

An incident management platform, designed and built from operational experience.

Overseer is a self-hosted operations centre for infrastructure that a single operator is accountable for. It inventories every running component, monitors it with its own probe agents, opens and attributes incidents automatically, investigates them with bounded AI agents, and escalates only the decisions that require human judgement.

Every collected fact carries its source, collection time and freshness. Stale and failed data are first-class states, and missing data is never rendered as healthy: an unobserved check reads unknown, an agent that has stopped reporting reads stale, and a passing check with no running process behind it is excluded from the evidence.
01

Detect

Probe agents and read-only connectors report continuously. A condition sustained for five minutes becomes an incident, dated from first observation. Estate-wide faults open a single incident rather than one per service.

02

Attribute

A service that is healthy internally but unreachable externally is classified as a path fault, with the contradiction recorded in the evidence. A flapping subject is consolidated into one incident whose timeline records each transition.

03

Investigate

A pipeline dispatches a bounded, read-only AI agent. Its conclusion and reasoning are posted to the incident's own Discord thread.

04

Decide

Proposals present Accept and Reject controls alongside their reasoning. Acknowledgement is a single action; a pre-declared remediation requires two, with the exact plan displayed before execution.

05

Verify

Recovery closes the incident with an accurate closure code (self-resolved is recorded as such). Remediation is verified against the monitors and never assumed.

TypeScript monorepo (Node, React, SQLite) with more than 850 commits and Playwright end-to-end tests against a seeded fixture estate. Incident milestones (detection, acknowledgement, mitigation, resolution) are stored as first-class fields and reported as medians and p90 by severity. Runbooks, hypotheses and accepted resolutions are written back into the knowledge base so that each incident begins with the outcome of the last.

Self-hosted infrastructure

A multi-host estate running approximately seventy services, with users who depend on it.

All public traffic enters through Cloudflare Tunnels and is governed by a single identity provider. No service is exposed without an explicit decision about who may reach it.

Application tier

Public services · tunnel edge

Source hosting, photo and document management, collaboration, search and media services, published through a single tunnel origin.

Compute and AI

Operations console · local models

Overseer, local model serving, agent runtimes, and a Linux workstation maintained entirely as code.

Network services

High-availability pair

DNS filtering, identity provider, secrets management and TLS termination, replicated across two nodes with UPS monitoring.

Storage and backup

Resilience

Network storage over NFS / SMB, encrypted Borg backups with scheduled restore tests, and the documentation vault for the estate.

Edge & identity

  • Cloudflare Tunnels as the only ingress; no inbound ports, operable behind CGNAT
  • Authentik SSO: OIDC, LDAP and proxy outposts sharing the application's network namespace, so loopback is the trust boundary
  • Caddy as the ingress splitter, stripping client-supplied identity headers before any application receives them

Operations

  • Prometheus + Grafana, Scrutiny (SMART), PatchMon (patch posture), NUT (UPS)
  • Borg backups with restore tests; nightly update windows with agent-verified verdicts
  • Runbooks and service pages generated into an Obsidian vault and indexed for retrieval

Platform

  • Docker and rootless Podman, systemd units and timers, Compose with pinned subnets
  • Self-hosted forges: Forgejo for code, pull requests and CI runners; OneDev for issue tracking
  • MeshCore LoRa mesh nodes for off-grid messaging and telemetry
AI skillset

Agents with defined tools, budgets and blast-radius limits.

Integrated with production infrastructure, constrained by design, and evaluated before being trusted.

Agentic systems

  • Bounded, read-only investigation agents in Overseer with a human decision step in Discord
  • Claude Code as a daily engineering tool, extended with hooks, skills and custom permission handling
  • Hermes Agent on a local 27B model, delegating complex work to Claude Code under a scoped permission mode

MCP & tool integration

  • Shared MCP servers over HTTP (memory recall, agent notes, forge, operations console) used by several agents at once
  • Tool-surface pruning: 29 of 128 forge tools exposed, as schema volume measurably degrades smaller models
  • Secrets held as environment references, never inline; write-scoping rules loaded on every agent turn

Retrieval & local models

  • RAG over a 1,600-note operations vault with bge-m3 embeddings served by Ollama
  • Embedding model selected by measured evaluation (hit@3: bge-m3 10/10, MiniLM 8/10, mxbai 7/10) rather than reputation
  • Prompt-caching and token-cost analysis for long-running agent loops

Applied AI products

  • AI answer engine built on SearXNG with private local sources
  • Adversarial browsing agent that degrades the accuracy of data-broker profiles for a household
  • Spectral audio alignment for synchronised listening rooms
Selected projects

Systems built for sustained operation.

Source is hosted on a self-managed Forgejo instance. Repository access is available on request.

Overseer flagship

Self-hosted operations centre: inventory, probe agents, automatic incident filing and attribution, AI investigation, Discord decision loop, public status page.

TypeScriptNodeReactSQLitePlaywrightDiscord

Leyr

Music requests, discovery and library management for Navidrome and Soulseek. Listening rooms with synchronised playback, party mode, Discogs collection links and share pages.

TypeScriptSSEOIDCDocker

DataChaff

Adversarial browsing agent that degrades the accuracy of data-broker profiles: shared household cookie jar, recurring and in-market behaviour legs, weekly verification probes.

PythonPlaywrightAgents

Anima09

Bot population control plane for a self-hosted RuneScape (2009scape) server: HTTP API, web UI, headless clients on Xvfb, installed as a patch rather than a fork.

JavaHTTP APIXvfb

romm-launch

Send-to-machine launching for the RomM ROM manager: a broker, per-machine polling agents and an injected UI button, with identity taken from the existing session.

PythonnginxAgents

Search answer engine

Answer cards, theme and private engines for a self-hosted SearXNG, plus an AI answer layer with local sources.

PythonSearXNGLLM

VaporOS

An Arch Linux + Hyprland desktop maintained entirely as code: every configuration, helper, unit, Quickshell bar and session-restore tool, documented.

ArchHyprlandsystemdQML

Nirn public

Mod manager, validator and config version control for a hand-built OpenMW install. Source.

PythonOpenMW

Browser tooling

Chromium extensions: automatic tab groups by domain with favicon-derived colours, and a fully local Earth View new-tab page.

JavaScriptChromium
Contact

Get in touch.

Available for incident management, site reliability and platform operations roles, remote or in Dallas–Fort Worth. Email is the preferred channel.

[email protected] git.jacobmckenzie.dev