Alerts

6 min

CommandIT Alerts: Overview

1. Purpose of Alerting

The CommandIT Alerting system is designed to provide timely notification of events, potential issues, or critical conditions occurring within managed client environments or the CommandIT platform itself. By monitoring infrastructure, applications, security events, and backups, the alerting system enables proactive response, faster troubleshooting, automated remediation, and ultimately contributes to maintaining service levels and system health.

2. How Alerts Work in CommandIT

The process involves generating alert events, processing them centrally, tracking their state, and triggering appropriate actions or notifications.

2.1. Alert Generation Sources

Alerts can originate from various sources integrated within CommandIT:

Monitoring Rules: Defined checks for performance counters (CPU, RAM, Disk), service states, event logs, ping responses, hardware health, etc., configured via Monitoring Policies (MonitoringRules).
Compliance Rules: Checks against defined compliance policies (ComplianceRules).
Patching Policies: Notifications related to patch failures or pending reboots (PatchPolicies).
Backup Status: Monitoring of backup job success, failure, or missed schedules (BackupJobStatus).
Website & Domain Monitoring: Checks for website uptime, SSL expiry, domain registration expiry (MonitoredWebsites, MonitoredDomains).
Security Monitoring: Findings from password checks, dark web scans, risky sign-ins, or EDR integrations (PasswordSecurityFindings, DarkWebBreachEvents, AzureAdRiskySignInEvents).
Email Triage: Emails identified as alerts processed by the AI Email Triage Agent.
SNMP Traps: Received traps processed via an SNMP listener integration.
API Integrations: Alerts pushed from external monitoring tools or platforms via CommandIT's API.
Manual Creation: Technicians can manually create alerts if necessary.

2.2. Central Processing: Alert Processing Engine

Regardless of the source, incoming alert data is standardized and processed by the Alert Processing Engine. This engine is the core intelligence behind the alerting system. Its key functions include:

Signature Identification: Determining if a new alert notification relates to the same underlying condition as an already active alert.
State Management: Tracking the lifecycle of each unique alert condition (First Seen, Last Seen, Occurrence Count, Current Status) using the Alerts table.
Policy & Rule Evaluation: Applying hierarchical AlertProcessingPolicies and AlertProcessingRules to determine the appropriate response.
Thresholding: Evaluating frequency and duration thresholds before taking action, reducing noise from transient issues.
Flapping Suppression: Detecting rapidly oscillating alert states and temporarily suppressing actions.
Maintenance Window Awareness: Checking applicable MaintenanceWindows and suppressing actions accordingly (unless overridden by a specific rule).
Action Execution: Triggering actions defined in policies, such as creating tickets, running automation scripts, sending notifications, or calling webhooks.

For detailed information on the engine's internal logic, thresholds, state transitions, and configuration, please refer to the separate document: [CommandIT Global Alert Processing Engine: Functional Specification](link-to-engine-spec.md)

2.3. Alert Lifecycle & Status (Alerts.status)

Alerts progress through various statuses visible in the UI:

New: A new alert condition detected, potentially awaiting threshold confirmation.
Acknowledged: A technician has seen the alert and acknowledged responsibility.
TicketCreated: A CommandIT ticket has been automatically created based on this alert via processing rules.
ActionAttempted / ActionSucceeded / ActionFailed: An automated action (like a script) was triggered.
Suppressed_Maint: The alert occurred during a maintenance window and action was suppressed.
Suppressed_Flapping: Actions are temporarily suppressed due to rapid state changes.
Resolved: The underlying condition has cleared (detected automatically or set manually).
Closed: The alert (and potentially associated ticket) has been fully handled and closed by a technician.
(Other internal statuses like ProcessingError, Unknown may exist but are primarily for backend/admin view).

2.4. Automated Actions

Based on configured Alert Processing Policies, the engine can trigger various automated actions, including:

Ignoring noisy or informational alerts.
Creating detailed tickets assigned to the appropriate Service Board/Technician using specific templates.
Running remediation Scripts via the CommandIT Agent.
Sending notifications via Email, SMS, or other channels using NotificationProfiles.
Calling external systems via Webhooks.

3. Working with Alerts: CommandIT UI

Technicians primarily interact with alerts through the following screens within the CommandIT platform:

3.1. Main Alerts Monitoring Screen / Dashboard

Location: Monitoring > Active Alerts / Dashboard (or similar). This is the primary workspace for NOC/Service Desk teams.
Purpose: Provides a consolidated view of alerts requiring attention.
Key Features:
- View: Configurable Dashboard potentially with panels/widgets, and a primary List/Table view of active alerts (e.g., status NOT 'Resolved' or 'Closed'). Should support auto-refreshing.
- Columns (Configurable): Severity (Color-coded), Alert Title/Signature, Affected CI Name (Device/App/Domain etc. - linked), Client Organization, Status, First Seen Timestamp, Last Seen Timestamp, Occurrence Count, Linked Ticket Number (linked).
- Filtering & Grouping: Powerful options to filter by Severity, Status, Organization, Device/CI Type, Time Range, Acknowledgement Status, etc. Ability to group alerts (e.g., by Client, by Device).
- Sorting: Sortable columns, typically defaulting to sort by Severity then Last Seen time (descending).
- Bulk Actions: Ability to select multiple alerts and perform actions like 'Acknowledge'.
- Row Actions: Quick actions available per alert, such as 'Acknowledge', 'View Details', 'Create Ticket' (if none exists), 'View Ticket' (if exists).
- Visualizations (Optional): Charts showing alert trends, top problematic CIs, alerts by severity over time.

3.2. Individual Alert Detail View (Modal/Drill-down Screen)

Trigger: Accessed by clicking on an alert from the Main Monitoring Screen or potentially from related logs/tickets.
Purpose: Provides comprehensive information and actions for a single alert instance.
Content:
- Header: Displays key information prominently: Severity, Title/Signature, Current Status.
- Context: Associated Organization, primary Affected CI (Device, App, etc. with link), originating Source (Monitoring Rule name, Email Source, etc.).
- Timeline: First Occurred At, Last Occurred At, Last Status Change Timestamp. If resolved/closed, shows those timestamps.
- State Details: Occurrence Count, Action Taken Flag (indicates if a threshold action like ticketing/scripting was performed for this instance).
- Alert Payload: Formatted display of the Alerts.details JSONB content, showing specific metrics or messages that triggered the alert.
- Related Ticket: Displays the linked Tickets.ticket_number (if any) with a direct link. Provides a 'Create Ticket' button if no ticket is linked and the alert status warrants it ('New', 'Acknowledged').
- History: A log/timeline of status changes specifically for this alert_signature or alert_id.
- Notes: Section to view and add technician notes specifically related to the investigation/handling of this alert instance (utilizes the generic Notes table linked to the alert_id).
- Actions: Buttons/options to:
  - Acknowledge: Changes status to 'Acknowledged', records user/time.
  - Resolve/Close: Changes status, requires mandatory reason/resolution notes.
  - Create/View Ticket: Creates a new linked ticket or navigates to the existing one.
  - Run Action (Future Enhancement?): Manually trigger associated remediation scripts.
  - Mute/Suppress: Temporarily silence notifications for this alert (future).

4. Related Configuration

The behavior of the alerting system is configured in several areas within CommandIT Settings:

Alert Rules: Define specific conditions and signature logic (AlertProcessingRules).
Alert Policies: Group rules, define actions, thresholds, flapping settings, and overrides (AlertProcessingPolicies). Policy assignment happens on Orgs, Locations, or Alert Endpoints.
Monitoring Rules: Define the actual checks performed by agents/probes (MonitoringRules).
Maintenance Windows: Define periods during which alerts may be suppressed (MaintenanceWindows).
Notification Profiles: Define who gets notified how and when for actions (NotificationProfiles).
Scripts: Define automation scripts that can be triggered by alert actions (Scripts).

5. Further Details

For an in-depth explanation of the alert processing logic, state management, threshold calculations, and technical implementation details, please consult the:

Alert Processing Engine