Collecting logs by using Fluentd - Documentation for BMC Helix Log Analytics 22.1

FormatDescription

Apache, Apache Error, Nginx, and Regexp

For these formats, expression and supported date format are displayed in the Expression and Time Format fields. Update the expression or date format based on the expression and date format present in your log files. 

 Example

Sample log:

[Mon Jan 10 02:13:55 2022] [necessitatibus:notice] [pid 5441:tid 6660] [client 11.111.111.111:2222] The TCP bus is down, override the wireless capacitor so we can connect the XML interface!
[Mon Jan 10 02:13:55 2022] [necessitatibus:info] [pid 9948:tid 2588] [client 22.222.222.22:3333] You can't bypass the program without programming the bluetooth HDD sensor!
[Mon Jan 10 02:13:55 2022] [et:notice] [pid 4498:tid 4891] [client 111.111.111.1:4444] Programming the alarm won't do anything, we need to hack the 1080p EXE protocol!

Default expression: /^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])? \[client (?<client>[^\]]*)\] (?<message>.*)$/

Updated expression to parse logs without port number: /^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])? \[client (?<client>\d+\.\d+\.\d+\.\d+):\d+\].(?<message>.*)$/

 To filter logs

1. From the Log Filter list, select Grep.
2. From the Directive field, select Regex (to include logs) or Exclude.
3. In the Key field, enter the key from the log expression.
   Get the keys from the log expression. For example, in the Apache expression, host, user, time, method, path, code, size, refer, and agent are keys.
4. In the Pattern field, enter the value to be included or excluded, enclosed within forward slashes (//).
5. Click + to add another grep expression.
   Here is an example:
   
   Sample logs:
   The value of the message field contains cool.
The value of the hostname field matches web<INTEGER>.example.com.
The value of the message field does NOT contain uncool.
   The following logs are collected:
   {"message":"It's cool outside today", "hostname":"web001.example.com"}
{"message":"That's not cool", "hostname":"web1337.example.com"}
   The following logs are excluded:
   {"message":"I am cool but you are uncool", "hostname":"db001.example.com"}
{"hostname":"web001.example.com"}
{"message":"It's cool outside today"}

Java multiline

Date format and firstline format expressions are displayed in the Format Firstline and Format 1 fields.

To parse the following sample logs:

2021-09-07 14:19:17 INFO [main] Generating some log messages 0
2021-09-07 14:19:17 INFO [main] Sleeping for 1 second.
2021-09-07 14:19:17 INFO [main] Generating some log messages 1

Modify the default expression for multiline. Here is how you can modify the out-of-the-box expression (note the square brackets location in the expressions):
Default: /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/

Updated: /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) (?<thread>.*) \[(?<level>[^\s]+)\](?<message>.*)/


To verify the expression, visit  rubular or  fluentular .

 To filter logs

1. From the Log Filter list, select Grep.
2. From the Directive field, select Regex (to include logs) or Exclude.
3. In the Key field, enter the key from the log expression.
   You can get the keys from the log expression. For example, in the Java multiline expression, time, thread, level, and message are keys.
4. In the Pattern field, enter the value to be included or excluded, enclosesd within forward slashes (//).
5. Click + to add another grep expression.
   Here is an example:
   
   Sample logs:
   The value of the message field contains cool.
The value of the message field does NOT contain uncool.
   The following logs are collected:
   {"message":"It's cool outside today"}
   The following logs are excluded:
   {"message":"I am cool but you are uncool"}
   

Json

 To filter logs

1. From the Log Filter list, select Grep.
2. From the Directive field, select Regex (to include logs) or Exclude.
3. In the Key field, enter the key from the log expression.
   Get the keys from logs. For example, you have the following log entry: {"time":1362020400,"host":"111.111.0.1","size":777,"method":"PUT"}. Here, you have the following keys: time, host, size, and method.
4. In the Pattern field, enter the value to be included or excluded, enclosesd within forward slashes (//).
5. Click + to add another grep expression.
   Here is an example:
   
   Sample log:
   {"time":1362020400,"host":"111.111.0.1","size":777,"method":"PUT"}
{"time":1362020400,"host":"111.111.0.1","size":777,"method":"POST"}
{"time":1362020400,"host":"111.111.0.1","size":777,"method":"GET"}
   
   The following logs are collected:
   {"time":1362020400,"host":"111.111.0.1","size":777,"method":"GET"}
   The following logs are excluded:
   {"time":1362020400,"host":"111.111.0.1","size":777,"method":"PUT"}
{"time":1362020400,"host":"111.111.0.1","size":777,"method":"POST"}

CSV

Enter the field names (separated by comma) that you want to provide to the values in the CSV file in the order they appear in the file.

For example, a CSV contains the following values:

2013/02/28 12:00:00,192.168.0.1,111,user1

2013/02/28 12:00:00,192.168.0.1,112,user2

2013/02/28 12:00:00,192.168.0.1,113,user3

For this example, enter time,host,req_ID,user.

The CSV is parsed as:

 To filter logs

1. From the Log Filter list, select Grep.
2. From the Directive field, select Regex (to include logs) or Exclude.
3. In the Key field, enter the key from the log expression.
   Keys are the field names that you entered for the columns in the CSV file.
4. In the Pattern field, enter the value to be included or excluded, enclosesd within forward slashes (//).
5. Click + to add another grep expression.
   Here is an example:
   
   Sample CSV format:

   2013/02/28 12:00:00,111.111.0.1,111,user1
2013/02/28 12:00:00,111.111.0.1,111,user2
2013/02/28 12:00:00,111.111.0.1,111,user3

   
   The following logs are collected:
   2013/02/28 12:00:00,111.111.0.1,111,user2
2013/02/28 12:00:00,111.111.0.1,111,user3
   The following logs are excluded:
   2013/02/28 12:00:00,111.111.0.1,111,user1

Custom

Use the Custom option in the following scenarios:

• The out-of-the-box expression is different from the logs expression in your files.
• The expression in your log files contains multiple key-value pairs.
• The required log format is not listed in the field.

To use the Custom option, enter the format in the Type field and configure the expression in the form of parameter name and parameter value. 
The following expressions are supported:

• regexp
• apache2
• apache_error
• nginx
• csv
• json
• multiline

Example:
For multiline format type, enter the following values:

Parameter Name: format_firstline; Parameter Value: /\d{4}-\d{1,2}-\d{1,2}/

Parameter Name: format1; Parameter Value: /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/

 To filter logs

1. From the Log Filter list, select Grep.
2. From the Directive list, select Regexp or Exclude.
3. Enter the key and pattern.
   Get the keys from the log expression.
4. Click + to add another grep expression.