1. Introduction
Splunk Web Interface provides typeahead, context-aware, time range picker, and search history to assist searching. In this example, I will show three types of basic searches.
- via the Web Interface
- with Search Processing Language
- from Extracted Fields
2. Technologies Used
The example in this article was built and run using:
- Docker 19.03.8
- Splunk 8.1.1
- Google Chrome 87.0.4280.88
Please reference my other article to install Splunk.
3. Search via Web Interface
3.1 Data Summary
The “Data Summary” link displays the data available to search in the Splunk server. It has three tabs:
- Hosts – shows a list of predefined host field.
- Sources – show a list of predefined source field.
- SourceTypes – shows a list of predefined sourceType field.
If you click on “maryzhengHost” then it will auto-populate “host=maryzhengHost” in the search text field.
3.2 Search History
Splunk saves the search commands in the “Search History” section. “Add to Search” button will copy and paste the search command to the search text field.
3.3 Context Menu
Splunk searching text field is enabled with typeahead or auto-completion feature. It displays the most relative text as a reference as user types.
3.4 Time Range Picker
Splunk UI includes a time range picker to filter the events based on the time period. Time range picker provides various predefined time ranges.
Filtering the events based on the time range is a great way to narrow down the searching results.
3.5 Searching By Interesting Fields
Splunk server creates interesting fields during the indexing process. The following screenshot shows the interesting fields. It pops up a quick report for the clicked field.
Here is the screenshot of field: RESPONSE_SUCCESSFUL.
4 Search Processing Language
4.1 Search Modes
Splunk has three search modes:
- Fast Mode – limits the result data based on the fields in the searching condition to gain better performance.
- Verbose Mode – returns as much event information as possible.
- Smart Mode – is the default mode. If the searching commands contain transforming commands, such as chart, timechart, stats, top, rare, etc , then it behaves like Fast mode; otherwise, it behaves like Verbose mode.
Splunk automatically switches to the appropriated mode based on the commands.
4.2 UNIX awk Command
Splunk search processing language (SPL) syntax was originally based on UNIX pipeline and SQL. In this step, I will use UNIX awk command to search for long duration events.
Here are events from the server.log file:
Server.log
12:10:08,774 INFO [com.ciit.PERFORMANCE] (http--0.0.0.0-9090-6) User: unknown IP Address: 1.1.1.3 Service: CircuitSearchSoapService Operation: getFooSucceeded: true Duration: 1002ms 12:11:08,774 INFO [com.ciit.PERFORMANCE] (http--0.0.0.0-9090-6) User: unknown IP Address: 1.1.1.3 Service: CircuitSearchSoapService Operation: getFooSucceeded: true Duration: 10002ms 12:12:08,774 INFO [com.ciit.PERFORMANCE] (http--0.0.0.0-9090-6) User: unknown IP Address: 1.1.1.3 Service: CircuitSearchSoapService Operation: getFooSucceeded: true Duration: 100002ms 12:13:08,774 INFO [com.ciit.PERFORMANCE] (http--0.0.0.0-9090-6) User: unknown IP Address: 1.1.1.3 Service: CircuitSearchSoapService Operation: getFooSucceeded: true Duration: 1000002ms 12:14:08,774 INFO [com.ciit.PERFORMANCE] (http--0.0.0.0-9090-6) User: unknown IP Address: 1.1.1.3 Service: CircuitSearchSoapService Operation: getFooSucceeded: true Duration: 1999992ms
The following awk command finds long duration events and prints out fields in position 1, 2, 6, 13, and the last field.
awk command
awk '/Duration/ && length($NF) > 7 {print $1 " " $2 " " $6 " " $13 " " $NF}' server.log
Here is the output from the above awk command:
awk output
# awk '/Duration/ && length($NF) > 7 {print $1 " " $2 " " $6 " " $13 " " $NF}' server.log
12:12:08,774 INFO unknown getFooSucceeded: 100002ms
12:13:08,774 INFO unknown getFooSucceeded: 1000002ms
12:14:08,774 INFO unknown getFooSucceeded: 1999992ms
Here is the screenshot for above awk command running at a docker Linux container:
Note: Here is the docker command to start a Linux container.
docker run -it ubuntu
PS C:\MaryZheng\DockerImages> docker run -it ubuntu
4.3 SPL rex Command
I upload the server.log into Splunk and search the long duration events with the Splunk rex command:
rex
index=* sourcetype=log4j | rex "Duration:\s(?<tt>\d+)" | where tt > 199999
Above command creates a new field called “tt“, its value is from regular expression: “Duration:\s(?\d+)“, it searches events whose tt's value is greater than 199999.
Found two events:
SPL rex command is easier to understand comparing to the UNIX awk command.
5. Extract Fields
Splunk field is a searchable name-value pair. Splunk creates default fields during the indexing process. It also provides a “Extract New Field” link to extract fields based on the data. It is helpful as it auto-generates the regular expression based on the selected text.
In this step, I will show how to extract a field – durationInMs from the searching result.
Click “Extract New Field” from the searching result’s Event Actions.
It pops up a page with self-explanation steps. Fill the information and click the Next button.
After click “Add Extraction“, it shows the details on generated regular expression.
Note: the generated regular expression is formatted with pcre group. Click Next to validate and save the field.
5.1 Extracted Field Report
After the field is extracted, then it is displayed at the Interesting Fields section. User can click to open the quick report as the following screenshot.
5.2 Search the Extracted Field
The newly extracted field – durationInMs can be used in the searching command:
search with field
index=* sourcetype=log4j durationInMs >= 1500000
Here is the screenshot with the searching command and results:
Using the extracted fields is easier comparing to the rex command.
6. Summary
In this example, I showed how to do a basic search of the data from default fields, with Splunk SPL commands, and from an extracted field.
7. Download the Source Code
You can download the source scripts of this example here: Splunk Basic Search Example
