Software Development

Splunk Basic Search Example

1. Introduction

Splunk Web Interface provides typeahead, context-aware, time range picker, and search history to assist searching. In this example, I will show three types of basic searches.

  • via the Web Interface
  • with Search Processing Language
  • from Extracted Fields

2. Technologies Used

The example in this article was built and run using:

  • Docker 19.03.8
  • Splunk 8.1.1
  • Google Chrome 87.0.4280.88

Please reference my other article to install Splunk.

3. Search via Web Interface

3.1 Data Summary

The “Data Summary” link displays the data available to search in the Splunk server. It has three tabs:

  • Hosts – shows a list of predefined host field.
  • Sources – show a list of predefined source field.
  • SourceTypes – shows a list of predefined sourceType field.
Splunk Basic Search  - data summary
Figure 1 Data Summary

If you click on “maryzhengHost” then it will auto-populate “host=maryzhengHost” in the search text field.

3.2 Search History

Splunk saves the search commands in the “Search History” section. “Add to Search” button will copy and paste the search command to the search text field.

Splunk Basic Search - search history
Figure 2 Search history

3.3 Context Menu

Splunk searching text field is enabled with typeahead or auto-completion feature. It displays the most relative text as a reference as user types.

Splunk Basic Search - autocomplete
Figure 3 Auto-complete Search Command

3.4 Time Range Picker

Splunk UI includes a time range picker to filter the events based on the time period. Time range picker provides various predefined time ranges.

Figure 4 Time Range Picker

Filtering the events based on the time range is a great way to narrow down the searching results.

3.5 Searching By Interesting Fields

Splunk server creates interesting fields during the indexing process. The following screenshot shows the interesting fields. It pops up a quick report for the clicked field.

Here is the screenshot of field: RESPONSE_SUCCESSFUL.

Figure 5 Interesting Fields

4 Search Processing Language

4.1 Search Modes

Splunk has three search modes:

  • Fast Mode – limits the result data based on the fields in the searching condition to gain better performance.
  • Verbose Mode – returns as much event information as possible.
  • Smart Mode – is the default mode. If the searching commands contain transforming commands, such as chart, timechart, stats, top, rare, etc , then it behaves like Fast mode; otherwise, it behaves like Verbose mode.

Splunk automatically switches to the appropriated mode based on the commands.

4.2 UNIX awk Command

Splunk search processing language (SPL) syntax was originally based on UNIX pipeline and SQL. In this step, I will use UNIX awk command to search for long duration events.

Here are events from the server.log file:

Server.log

12:10:08,774 INFO  [com.ciit.PERFORMANCE] (http--0.0.0.0-9090-6) User: unknown  IP Address: 1.1.1.3   Service: CircuitSearchSoapService      Operation: getFooSucceeded: true Duration: 1002ms
12:11:08,774 INFO  [com.ciit.PERFORMANCE] (http--0.0.0.0-9090-6) User: unknown  IP Address: 1.1.1.3   Service: CircuitSearchSoapService      Operation: getFooSucceeded: true Duration: 10002ms
12:12:08,774 INFO  [com.ciit.PERFORMANCE] (http--0.0.0.0-9090-6) User: unknown  IP Address: 1.1.1.3   Service: CircuitSearchSoapService      Operation: getFooSucceeded: true Duration: 100002ms
12:13:08,774 INFO  [com.ciit.PERFORMANCE] (http--0.0.0.0-9090-6) User: unknown  IP Address: 1.1.1.3   Service: CircuitSearchSoapService      Operation: getFooSucceeded: true Duration: 1000002ms
12:14:08,774 INFO  [com.ciit.PERFORMANCE] (http--0.0.0.0-9090-6) User: unknown  IP Address: 1.1.1.3   Service: CircuitSearchSoapService      Operation: getFooSucceeded: true Duration: 1999992ms

The following awk command finds long duration events and prints out fields in position 1, 2, 6, 13, and the last field.

awk command

awk '/Duration/ && length($NF) > 7 {print $1 " " $2 " " $6 " " $13 " " $NF}' server.log

Here is the output from the above awk command:

awk output

# awk '/Duration/ && length($NF) > 7 {print $1 " " $2 " " $6 " " $13 " " $NF}' server.log
12:12:08,774 INFO unknown getFooSucceeded: 100002ms
12:13:08,774 INFO unknown getFooSucceeded: 1000002ms
12:14:08,774 INFO unknown getFooSucceeded: 1999992ms

Here is the screenshot for above awk command running at a docker Linux container:

Figure 6 Unix awk Command

Note: Here is the docker command to start a Linux container.

docker run -it ubuntu

PS C:\MaryZheng\DockerImages> docker run -it ubuntu

4.3 SPL rex Command

I upload the server.log into Splunk and search the long duration events with the Splunk rex command:

rex

index=* sourcetype=log4j | rex "Duration:\s(?<tt>\d+)" | where tt > 199999

Above command creates a new field called “tt“, its value is from regular expression: “Duration:\s(?\d+)“, it searches events whose tt's value is greater than 199999.

Found two events:

Figure 7 Splunk rex Command

SPL rex command is easier to understand comparing to the UNIX awk command.

5. Extract Fields

Splunk field is a searchable name-value pair. Splunk creates default fields during the indexing process. It also provides a “Extract New Field” link to extract fields based on the data. It is helpful as it auto-generates the regular expression based on the selected text.

In this step, I will show how to extract a field – durationInMs from the searching result.

Click “Extract New Field” from the searching result’s Event Actions.

Figure 8 Extract Fields Link

It pops up a page with self-explanation steps. Fill the information and click the Next button.

Figure 9 Add Extraction

After click “Add Extraction“, it shows the details on generated regular expression.

Figure 10 Extract Fields

Note: the generated regular expression is formatted with pcre group. Click Next to validate and save the field.

Figure 11 Save Extracted Field

5.1 Extracted Field Report

After the field is extracted, then it is displayed at the Interesting Fields section. User can click to open the quick report as the following screenshot.

Figure 12 Extract Fields

5.2 Search the Extracted Field

The newly extracted field – durationInMs can be used in the searching command:

search with field

index=* sourcetype=log4j durationInMs >= 1500000

Here is the screenshot with the searching command and results:

Figure 13 Search with Extracted Field – durationInMs

Using the extracted fields is easier comparing to the rex command.

6. Summary

In this example, I showed how to do a basic search of the data from default fields, with Splunk SPL commands, and from an extracted field.

7. Download the Source Code

Download
You can download the source scripts of this example here: Splunk Basic Search Example

Mary Zheng

Mary has graduated from Mechanical Engineering department at ShangHai JiaoTong University. She also holds a Master degree in Computer Science from Webster University. During her studies she has been involved with a large number of projects ranging from programming and software engineering. She works as a senior Software Engineer in the telecommunications sector where she acts as a leader and works with others to design, implement, and monitor the software solution.
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Back to top button