Manages IT infrastructure, monitoring, incident response, and service reliability. Provides frameworks for ITIL service management, observability strategies, automation, backup/recovery, capacity planning, and operational excellence practices.

IT Operations Expert A comprehensive skill for managing IT infrastructure operations, ensuring service reliability, implementing monitoring and alerting strategies, managing incidents, and maintaining operational excellence through automation and best practices. Core Principles 1. Service Reliability First Proactive Monitoring : Implement comprehensive observability before incidents occur Incident Management : Structured response processes with clear escalation paths SLA/SLO Management : Define and maintain service level objectives aligned with business needs Continuous Improvement : Learn from incidents through blameless post mortems 2. Automation Over Manual Processes Infrastructure as Code : Manage infrastructure configuration through version controlled code Runbook Automation : Convert manual procedures into automated workflows Self Healing Systems : Implement automated remediation for common issues Configuration Management : Maintain consistency across environments 3. ITIL Service Management Service Strategy : Align IT services with business objectives Service Design : Design resilient, scalable services Service Transition : Manage changes with minimal disruption Service Operation : Deliver and support services effectively Continual Service Improvement : Iteratively enhance service quality 4. Operational Excellence Documentation : Maintain current runbooks, procedures, and architecture diagrams Knowledge Management : Build searchable knowledge bases from incident resolutions Capacity Planning : Forecast and provision resources proactively Cost Optimization : Balance performance requirements with infrastructure costs Core Workflow Infrastructure Operations Workflow Decision Frameworks Alert Configuration Decision Matrix Scenario Alert Type Threshold Response Time Escalation Service completely down Page Immediate < 5 min Immediate to on call Service degraded Page 2 3 failures < 15 min After 15 min to on call High resource usage Warning 80% sustained < 1 hour After 2 hours to team lead Approaching capacity Info 70% trend < 24 hours Weekly capacity review Configuration drift Ticket Any deviation < 7 days Monthly review Incident Severity Classification Priority 1 (Critical) Complete service outage affecting all users Data loss or security breach Financial impact $10K/hour Response: Immediate, 24/7, all hands on deck Priority 2 (High) Partial service outage affecting many users Significant performance degradation Financial impact $1K $10K/hour Response: < 30 minutes during business hours Priority 3 (Medium) Service degradation affecting some users Non critical functionality impaired Workaround available Response: < 4 hours during business hours Priority 4 (Low) Minor issues with minimal impact Cosmetic problems Enhancement requests Response: Next business day Change Management Risk Assessment Monitoring Tool Selection Requirement Prometheus + Grafana Datadog New Relic ELK Stack Splunk Cost Free (self hosted) $$$$ $$$$ Free $$ $$$$$ Metrics Excellent Excellent Excellent Good Good Logs Via Loki Excellent Excellent Excellent Excellent Traces Via Tempo Excellent Excellent Limited Good Learning Curve Steep Moderate Moderate Steep Steep Cloud Native Excellent Excellent Excellent Good Good On Premises Excellent Good Good Excellent Excellent APM Via exporters Excellent Excellent Limited Good Common Operational Challenges Challenge 1: Alert Fatigue Problem : Too many false positive alerts causing team burnout Solution : Challenge 2: Incident Documentation During Crisis Problem : Teams skip documentation during high pressure incidents Solution : Assign dedicated scribe role (not the incident commander) Use incident management tools (PagerDuty, Opsgenie) with automatic timeline Template based incident reports with required fields Post incident review scheduled automatically (within 48 hours) Gamify documentation (track and recognize thorough documentation) Challenge 3: Knowledge Silos Problem : Critical knowledge trapped in individual team members' heads Solution : Challenge 4: Balancing Stability vs Innovation Problem : Operations team resists change to maintain stability Solution : Implement change windows (planned maintenance periods) Use blue green or canary deployments for lower risk Establish "innovation time" (Google 20% time model) Create sandbox environments for experimentation Measure and reward both stability AND improvement metrics Include "toil reduction" as OKR target Key Metrics & KPIs Service Reliability Metrics Operational Efficiency Metrics Integration Points With Development Teams Participate in design reviews for operational requirements Provide deployment automation and CI/CD pipeline support Share monitoring and logging requirements Collaborate on incident response and post mortems Joint ownership of SLOs and error budgets With Security Teams Implement security monitoring and alerting Manage access controls and authentication systems Coordinate vulnerability patching and remediation Conduct security incident response Maintain compliance with security policies With Business Stakeholders Report on service availability and performance Communicate planned maintenance windows Provide capacity planning forecasts Translate technical metrics to business impact Participate in business continuity planning Best Practices 1. Blameless Post Mortems 2. Runbook Standards 3. On Call Best Practices 4. Change Management Discipline Reference Files For detailed technical guidance, see: [reference/monitoring.md](reference/monitoring.md) Observability, metrics, alerting, and dashboard design [reference/incident management.md](reference/incident management.md) Incident response, root cause analysis, post mortems [reference/infrastructure.md](reference/infrastructure.md) Server management, network operations, capacity planning [reference/automation.md](reference/automation.md) Scripting, configuration management, orchestration tools [reference/backup recovery.md](reference/backup recovery.md) Backup strategies, disaster recovery, business continuity Getting Started 1. For New Infrastructure : Start with [reference/infrastructure.md](reference/infrastructure.md) for setup guidance 2. For Monitoring Setup : Review [reference/monitoring.md](reference/monitoring.md) for observability strategy 3. For Incident Response : See [reference/incident management.md](reference/incident management.md) for procedures 4. For Automation Projects : Check [reference/automation.md](reference/automation.md) for tooling recommendations 5. For DR Planning : Consult [reference/backup recovery.md](reference/backup recovery.md) for recovery strategies

it-operations