Skip to main content

Analytics

Introduction​

DIAL Core collects system logs and chat completion logs:

  • System logs do not include any user data and contain logs of all requests from system components to AI DIAL Core (using the ELK stack (Elasticsearch, Logstash, Kibana) or other log collection system). Note: this document does not cover system logs.
  • Chat completion requests logs include information that users send in their requests to LLMs and the information they get in responses.

Important: AI DIAL Core will log user requests and deployment responses unless the requests are authorized with a secured API key. This means that user requests and deployment responses will be recorded in logs for all requests authorized with an access token (like JWT) or unsecured API keys. Refer to configuration to learn how to create secured and other types of API keys.

AI DIAL setup can include a special service called DIAL Analytics Realtime, which uses diverse techniques such as embedding algorithms, clustering algorithms, frameworks, light-weight self-hosted language models, to analyze chat completion logs and extract the needed information, which can be presented in tools such as Grafana for visualization and analytics.

Refer to Analytics Realtime repository to learn more and view the project source code.

Analytics Realtime does not retain any private information, such as user prompts or conversations, beyond the system. Instead, only the computed artifacts are collected and stored in time-series databases like InfluxDB or any scalable database capable of handling voluminous, constantly changing information.

Examples of the computed artifacts:

  • Who has used the AI? – user hash, title, and never personal data such as names.
  • What areas have people asked questions about?
  • Are there any recurring patterns?
  • Topics of conversations.
  • Unique users.
  • Sentiments.
  • Cost analysis of the communication.
  • Language of conversations.
  • Any other calculated statistics based on conversations.

Configuration​

This section outlines the required steps for configuring Analytics Realtime service and other necessary components:

Flow:

AI DIAL Core generates a .log file containing chat completion logs. A log collector tool then transfers this file to AI DIAL Analytics Realtime for analysis. The insights derived from the analysis are stored in InfluxDB and can subsequently be visualized using Grafana.

Step 1: AI DIAL Core​

AI DIAL Core can be configured to write chat completion logs into a specific .log file.

Use the default AI DIAL Core Configuration as reference. Refer to GFLog to learn more about the logging framework used in DIAL Core.

Step 2: Influx DB​

Analytics Realtime uses InfluxDB to store the analytics of chat completion logs. Refer to InfluxDB documentation to learn how to install it and how to create tokens to read from a bucket.

Refer to Configuration to view how to configure InfluxDB for Analytics Realtime service.

Step 3: AI DIAL Analytics Realtime​

Follow the instructions to setup AI DIAL Analytics Realtime service.

Step 4: Log Collector​

AI DIAL uses the external open-source solution Vector as a log collector to transfer a .log file with chat completion logs to AI DIAL Analytics Realtime service via HTTP. Analytics Realtime functions as a "sink" for Vector, providing an endpoint designed to receive logs from it. To connect, you only need to know the hostname and port, such as http://localhost:5001/data - see the example below.

You can find an additional information on delivering observability data to an HTTP server in the Vector documentation.

This is an example of Vector configuration:

sources:
  aidial_logs:
    type: "file"
    max_line_bytes: 100000000
    oldest_first: true
    include:
      - /app/log/*.log # file with chat completion logs
  http_analytics_opensource:
    inputs:
      - aidial_logs
    type: http
    uri: http://dial-analytics.dial:80/data # Analytics Realtime URI
    request:
      timeout_secs: 300
    batch:
      max_bytes: 1049000
      timeout_secs: 60
    encoding:
      codec: "json"

Step 5: Grafana​

Grafana can be configured to use InfluxDB with analytics of DIAL logs as a data source. You can use pre-configured samples of dashboards to visualize data in Grafana.

Refer to Grafana documentation to learn how to install it and import dashboards.