Analytics
Introduction
DIAL Core collects system logs and chat completion logs:
- System logs do not include any user data and contain logs of all requests from system components to AI DIAL Core (using the ELK stack (Elasticsearch, Logstash, Kibana) or other log collection system). Refer to Observability to learn about logs, metrics and traces supported in AI DIAL.
- Chat completion requests logs include information that users send in their requests to LLMs and the information they get in responses.
AI DIAL setup can include a special service called DIAL Analytics Realtime, which uses diverse techniques such as embedding algorithms, clustering algorithms, frameworks, light-weight self-hosted language models, to analyze chat completion logs and extract the needed information, which can be presented in tools such as Grafana for visualization and analytics.
Refer to Analytics Realtime repository to learn more and view the project source code.
Analytics Realtime does not retain any private information, such as user prompts or conversations, beyond the system. Instead, only the computed artifacts are collected and stored in time-series databases like InfluxDB or any scalable database capable of handling voluminous, constantly changing information.
Examples of the computed artifacts:
- Information about user(user hash, title, and never personal data such as names).
- What areas have people asked questions about?
- Are there any recurring patterns?
- Topics of conversations.
- Unique users.
- Sentiments.
- Cost analysis of the communication.
- Language of conversations.
- Any other calculated statistics based on conversations.
Disable Logging of User Information
AI DIAL Core will log all user requests and deployment responses authorized with an access token (like JWT) or unsecured API keys. To avoid this, create and use a secured API key - in this case the system will not collect logs of user requests and related responses from applications, assistants and models.
Refer to DIAL Core configuration to learn how to create secured and other types of API keys.
Configuration
This section outlines the required steps for configuring Analytics Realtime service and other necessary components:
- Step 1: Configure DIAL Core
- Step 2: Install Influx DB
- Step 3: Configure DIAL Analytics Realtime
- Step 4: Configure Log Collector
- Step 5: Configure Grafana
Flow:
AI DIAL Core generates a .log
file containing chat completion logs. A log collector tool then transfers this file to AI DIAL Analytics Realtime for analysis. The insights derived from the analysis are stored in InfluxDB and can subsequently be visualized using Grafana.
Step 1: AI DIAL Core
AI DIAL Core can be configured to write chat completion logs into a specific .log
file.
Use the default AI DIAL Core Configuration as reference. Refer to GFLog to learn more about the logging framework used in DIAL Core.