# Dataset description

GAIA contains the data from MicroSS (in MicroSS repository in Github link) and metrics from companions (in Companion Data repository in Github link). Statistically, the data from MicroSS contains more than 6,500 metrics, 7,000,000 log items and detailed trace data continuously collected for two weeks. In this scenario, we also simulate the anomalies that may happen in real systems and provide the record for all anomaly injections for fair evaluation of root cause analysis algorithms. This is achieved by controlling the users' behaviors and mimicking the wrong manipulations to the systems.

The data files are listed below.

Git repository Relevant repository Download
MicroSS metric | trace | business | run https://github.com/CloudWise-OpenSource/GAIA-DataSet (opens new window)
Companion Data metric_detection | metric_forecast | log https://github.com/CloudWise-OpenSource/GAIA-DataSet (opens new window)

# MicroSS

MicroSS rpeository contains all data in different types, selected from the business simulation system MicroSS. It comes from a scenario of logging-in with QR Code. The description of this scenario is also included in MicroSS.

# metric

In "metric" folder, each csv filename contains the node to which the file belongs, ip, and the corresponding indicator name and time period, reformulated from the raw data collected by Metricbeat. The data includes fields as follows.

timestamp value
1625133601000 34201179
  • timestamp: the time of data collection: 13-bit time stamp
  • value: value of metric at the timestamp

# trace

In "trace" folder, each file contains the trace record, reformulated from the raw data collected by OpenTracing. The data includes fields as follows.

timestamp host_ip service_name trace_id span_id parent_id start_time end_time url status_code message
2021-07-01 10:54:23 0.0.0.4 dbservice1 c124e30fb40651dc 58ac80ceea500f66 8b3e4a4003c5119c 2021-07-01 10:54:22.632751 2021-07-01 10:54:22.632751 http://0.0.0.4:9388/db_login_methods?uuid=a3036736-da17-11eb-9811-0242ac110003&user_id=ToeLCkHR (opens new window) 200 request call function 1 dbservice1.db_login_methods
  • timestamp: string of time record with the form YYYY-MM-DD hh:mm:ss
  • host_ip: the IP of the host running the service named service_name
  • service_name: name of service or host
  • trace_id: UUID of the business trace
  • span_id: UUID of the node in current trace
  • parent_id: UUID of the parent node in current trace
  • start_time:the time this call is created
  • end_time: the time this call is closed
  • url: the RPC url
  • status_code: 200 for normal, and others for anomalies.
  • message: the out-band message for this call

# business

In "business" folder, each file contains the business log of a node, reformulated from the raw data. The data includes fields as follows.

datetime service message
2021-07-01 00:00:00 dbservice2 2021-07-01 14:11:54,950 | INFO | 0.0.0.2 | 172.17.0.2 | dbservice2 | 12ef1025e43ec0ef | 3b12f3fa-da33-11eb-875f-0242ac110003-JKrdHZDV-END!RH0>_qOJ token generate success
token=MTYyNTExOTkxNC45NTA0Njk1OjNiMTJmM2ZhLWRhMzMtMTFlYi04NzVmLTAyNDJhYzExMDAwM0pLcmRIWkRWRU5EIVJIMD5fcU9KOjE2MjUxMTk5NzQuOTUwNDc5NTpkZjk2YmIyOThmN2M4ZDg3N2NiYmY2MWZkYWM4ZjBlYw==
  • datetime: string of time record with the form YYYY-MM-DD hh:mm:ss
  • service: the relevant node ID
  • message: extra information in this log.

# run

In "run" folder, we provide system log and anomaly injection records. The data includes fields as follows, with the same meaning to files in "business" folder.

datetime service message
2021-07-01 dbservice1 2021-07-01 22:33:05,033 | WARNING | 0.0.0.4 | 172.17.0.3 | dbservice1 | [memory_anomalies] trigger a high memory program, start at 2021-07-01 22:23:04.230332 and lasts 600 seconds and use 1g memory

# Companion Data

Companion Data contains metric and log data provided by the companions of Cloudwise. All the data in Companion Data has achieved strict hyposensitization to protect users and companies' privacy. It contains a total of 406 anomaly detection and metric prediction data, including 279 label data, and covers the following types of time series data:

  • Changepoint data
  • Concept_drift_data
  • Linear_data
  • Low_signal-to-noise_ratio_data
  • Partially_stationary_data
  • Periodic_data
  • Staircase_data

In terms of logs, the Companion Data contains log parsing, log semantics anomaly detection, and named entity recognition (NER) data. About 218,736 pieces of log data. Please refer to Companion Data for data description.

# metrc_detection

"metrc_detection" folder records the corresponding type of time series data under each subfolder. Notice that all metrics here are labeled, so that metric anomaly detection can be tackled with fair evaluation. The data includes fields as follows.

timestamp value label
1546272000000 168899765 0
1546272300000 168900938.6 0
1546272600000 168902112.2 0
1546272900000 168896334 0
1546273200000 168880129 0
1546273500000 168863924 0
  • timestamp: the time of data collection:13-bit time stamp.
  • value: metric value at the time.
  • label: anomaly label. 0 for normal, and 1 for anomaly.

# metrc_forecast

"metrc_detection" folder records the corresponding type of time series data under each subfolder. Time series prediction algorithms can be trained on this data set. The data includes fields as follows.

timestamp value
1546272000000 168899765
1546272300000 168900938.6
1546272600000 168902112.2
1546272900000 168896334
1546273200000 168880129
1546273500000 168863924
  • timestamp: the time of data collection.
  • value: metric value at the time.

# log

In "log" folder, three sub-folders are included, "log parsing", "log semantics anomaly detection", and "named entity recognition (NER)", serving for the tasks with the same names. Detailed descriptions of the files within can be found in each sub-folder.