AI Cost Tracking

6 min

1. Introduction

1.1. Purpose: This defines a standardized approach for tracking the operational costs associated with Artificial Intelligence (AI) features deployed across the CommandIT platform. This includes the Support AI Chat Agent, AI-powered ticket processing, automation assistance, reporting insights, and future AI-driven functionalities.

1.2. Goals: The primary goals of this strategy are to:

Accurately capture the actual operational costs incurred by AI service usage (primarily Large Language Models (LLMs) and potentially other specialized AI APIs).
Enable precise internal cost accounting and profitability analysis for AI features per client, agreement, or feature area, specifically associating costs with relevant CommandIT Tickets where applicable.
Provide reliable data for incorporating AI costs into relevant MSP Key Performance Indicators (KPIs), such as those used by frameworks like TruMethods and Service Leadership (SLIQ).
Inform pricing and packaging decisions for CommandIT features that leverage AI.
Identify high-cost AI operations or features for potential optimization.

1.3. Scope: This strategy applies to all current and future CommandIT features that utilize LLMs or other third-party AI services with usage-based costs. It covers cost tracking associated with specific user interactions, automated processes, and system tasks, with a primary focus on attributing costs back to specific Tickets when an interaction relates to one.

2. Core Concept & Rationale

Track Actual Costs: We will track the actual, variable costs incurred per AI operation (primarily LLM token costs). We will avoid using arbitrary internal hourly rates or fixed costs for AI "labor."
Rationale: Actual cost tracking provides financial accuracy essential for understanding the true cost-to-serve, identifying the real economic impact of AI automation, calculating accurate profit margins (especially at the ticket and agreement level), and making data-driven decisions.
Unit of Association: Costs should be associated primarily with the relevant CommandIT Ticket (Tickets.ticket_id) whenever an AI interaction or process relates directly to a ticket. For AI actions not tied to a ticket, association might be linked to other relevant entities if necessary, but the focus is per-ticket cost attribution.
Internal Operational Expense: These AI costs are treated as internal operational expenses. They will be logged in a way that prevents them from appearing directly on client invoices by default.

3. Data Source & Measurement

Primary Source (LLMs): LangSmith: LangSmith will be integrated with all LLM-based features. It automatically tracks token usage. By configuring model pricing (see Section 4.4), we can derive estimated costs per LLM call or interaction.
Other AI Services: Costs from non-LLM AI services (e.g., transcription APIs) must be captured by the interacting CommandIT backend module and logged using the same mechanism, potentially with a distinct expense type.
Cost Calculation & Currency: Costs are calculated based on provider pricing. The standard currency for cost logging should be defined (recommend USD). Currency conversion for financial reporting in other currencies (e.g., CAD) should be handled by the reporting/accounting system.

4. Technical Implementation Strategy

Centralized Storage: ExpenseEntries Table: We will utilize the existing ExpenseEntries table within the CommandIT database schema to store the aggregated AI operational costs associated with specific tickets.
- Data Recorded per AI Interaction/Task:
  - user_id: The CommandIT Users.user_id representing the AI agent.
  - org_id: The Client Organizations.org_id related to the ticket.
  - ticket_id: The relevant Tickets.ticket_id.
  - expense_date: Date the cost was incurred/logged.
  - expense_type: A specific identifier, e.g., 'AI_Interaction_Cost', 'AI_Summarization_Cost', 'AI_Transcription_Cost'.
  - description: Standardized text, e.g., "AI operational cost for chat interaction [InteractionID]" or "AI cost for ticket summarization".
  - quantity: 1.
  - unit_amount: NULL.
  - total_amount: The calculated actual cost (e.g., accumulated_llm_cost) for the interaction/task.
  - currency_code: The currency of total_amount (e.g., 'USD').
  - billing_option: 'DoNotBill' (Crucial: Ensures it's treated as an internal cost, not billed to the client via standard invoicing logic).
  - payment_method: 'SystemInternal' or similar indicator.
  - requires_reimbursement: false.
Cost Accumulation:
- Stateful Interactions (e.g., Support Chat AI): Maintain a running total (e.g., accumulated_llm_cost) within the session's state (e.g., LangGraph state). At the end of the interaction (Section 9 of Functional Spec), log a single ExpenseEntries record for the total accumulated cost, linked to the ticket_id.
- Discrete AI Tasks (e.g., Ticket Summarization): After the task completes, capture the cost and log a single ExpenseEntries record immediately, linked to the relevant ticket_id.
Capturing Cost Data:
- Backend services implementing AI features must integrate with LangSmith (for LLMs) or other mechanisms to capture usage/cost data per operation.
- Use LangChain callbacks or inspect run metadata to get token counts/costs.
- Call an internal CommandIT API endpoint (wrapping the addExpenseEntry tool logic) to write the finalized cost record to ExpenseEntries.
AI User Record Configuration: The "AI User" record (Users where is_ai=true) must have its UserCostRates.hourly_cost_rate set to 0.00.
Model Pricing Configuration:
- Clarification Needed: The strategy for maintaining the LLM pricing data (cost per token for different models) needs definition. Will this be configured within CommandIT settings, pulled dynamically from LangSmith, or hardcoded (less flexible)? This needs to be defined and kept up-to-date.
Error Handling:
- Clarification Needed: Define the system behavior if logging to ExpenseEntries fails. Should the primary AI operation still be considered successful? Should errors be logged separately for reconciliation? Recommend logging the error and proceeding, with a mechanism for later review/retry of cost logging.

5. API Impact

Internal Logging API: The internal API endpoint used by the addExpenseEntry tool handles writing cost data to the ExpenseEntries table. Backend AI services call this endpoint.
Reporting APIs: Internal CommandIT APIs for reporting must query ExpenseEntries (filtering by relevant expense_type values) to fetch AI costs associated with tickets, clients, or agreements.
External APIs: No direct impact on client-facing APIs is expected.

6. KPI Integration (TruMethods/SLIQ etc.)

Data Source: AI operational costs associated with tickets are available in the ExpenseEntries table (e.g., WHERE expense_type = 'AI_Interaction_Cost'). Human labor costs are derived from TimeEntries and UserCostRates.
Calculation Logic: Reporting systems calculating KPIs like Cost Per Ticket, Agreement Profitability, Gross Margin, etc., where labor cost is a factor, MUST calculate the total service delivery cost as:
- Total Cost = SUM(Human Costs) + SUM(AI Costs) + SUM(Other Direct Costs)
  - Human Costs = Sum of (TimeEntries.duration_seconds / 3600 * UserCostRates.hourly_cost_rate) for human users.
  - AI Costs = Sum of (ExpenseEntries.total_amount) where expense_type indicates an AI cost related to the scope (ticket, agreement, period).
Rationale: This incorporates the actual variable cost of AI automation into profitability calculations accurately.

7. Granularity Perspective

While this approach stores the aggregated cost per interaction/task in ExpenseEntries for accounting/KPI purposes, the underlying granular data (cost per specific LLM call, token counts, model used, latency) remains available within LangSmith traces. LangSmith should be used for detailed AI performance analysis, cost optimization efforts, and debugging specific high-cost interactions.

8. Conclusion

Using the ExpenseEntries table to log aggregated, non-billable AI operational costs per ticket provides a pragmatic and effective strategy for CommandIT. It enables accurate internal cost accounting and integrates AI costs correctly into established MSP KPI frameworks like TruMethods and SLIQ, using actual incurred costs rather than arbitrary rates. Consistent implementation across all AI features, clear definition of pricing sources, and robust error handling for logging are key to the success of this strategy.