Splunk Enterprise: Uncovering Data-Driven Insights for IT Mastery

Introduction:

With the dynamic software program Splunk Enterprise, designed for IT professionals, you can unleash the power of your data. This book covers the many capabilities that make Splunk a vital tool for data analytics and visualization, from indexing to real-time searches, alarms, and extensive dashboards.

What is Splunk Enterprise?

Splunk Enterprise is a software product that enables you to search, analyze, and visualize the data gathered from the components of your IT infrastructure or business.
Splunk Enterprise takes in data from websites, applications, sensors, devices, and so on. After you define the data source, Splunk Enterprise indexes the data stream and parses it into a series of individual events that you can view and search.

Features of Splunk Enterprise :

Indexing
Search
Alerts
Dashboard
Pivot
Data model
Reports

Indexing :

Splunk Enterprise processes and stores the data that represents your business and its infrastructure.
You can collect data from devices and applications such as websites, servers, databases, operating systems, and more.
Once the data is collected, the index segments, stores, compresses the data and maintains the supporting metadata to accelerate searching.

Search :

Search is the primary way users navigate their data in Splunk Enterprise. You can save a search as a report and use it to power dashboard panels. Searches provide insight from your data, such as:

Retrieving events from an index
Calculating metricsSearching for specific conditions within a rolling time window
Identifying patterns in your data
Predicting future trends

Alerts:

Alerts notify you when search results for both historical and real-time searches meet configured conditions.
You can configure alerts to trigger actions like sending alert information to designated email addresses, posting alert information to an RSS feed, and running a custom script, such as one that posts an alert event to Syslog.

Dashboards:

Dashboards contain panels of modules like search boxes, fields, charts, and so on. Dashboard panels are usually connected to saved searches or pivots.
They display the results of completed searches and data from real-time searches that run in the background.

Pivot :

Pivot refers to the table, chart, or data visualization you create using the Pivot Editor.
Pivots can be saved as reports and added to dashboards.

Data model :

Data models encode specialized domain knowledge about one or more sets of indexed data.
They enable Pivot Editor users to create reports and dashboards without designing the searches that generate them.

Reports :

Data models encode specialized domain knowledge about one or more sets of indexed data.
They enable Pivot Editor users to create reports and dashboards without designing the searches that generate them.

Splunk Forwarder:

Universal Forwarders to provide reliable. secure data collection from remote sources and forward data into the Splunk enterprise server for indexing and consolidation.
They can scale to tens of thousands of remote systems. collecting terabytes of data.

Splunk Installation and Setup

OS – RHEL, minimum 5GB of hard disk.
This product is license-based so a free trial is available for 60 days.
Create the account at https://www.splunk.com/

Navigate the page to products > Free trials and Downloads > Splunk Enterprise (Download free trial 60 dial) > Linux > click on .tgz download now > select the command line > copy the CLI (point to the latest version to download)

On your ec2 instance follow the below steps:

Paste the command line copied from Splunk UI
Example :

wget -----------------------------
$ mv splunk-8.2.6-a6fe1ee8894b-Linux-x86_64.tgz /opt
$ cd /opt
$ tar -xvzf splunk-8.2.6-a6fe1ee8894b-Linux-x86_64.tgz
$ cd splunk/bin

Start the Splunk enterprise service

./splunk start --accept-license
accept the license and set username and password (username : abcd & password : xxxxxx)
Access the UI http://ip:8000

Apps panel

The Apps panel lists the applications that are installed on your Splunk instance. The list shows only the apps that you have permission to view.
When you first open Splunk Web, you see Search & Reporting in the Apps panel.
The Search & Reporting app is sometimes referred to as simply the Search app.
There might be other apps listed on the Apps panel if other applications are installed on your computer.

Account Menu

The Account menu displays Administrator. Use the menu to edit your account settings, set your preferences, and to logout.

How To Install Splunk Universal Forwarder to the remote machine

Download the tar file of the Splunk universal forwarder to the remote machine and use the below link to get the CLI command to install the forwarder.

https://www.splunk.com/en_us/download/universal-forwarder.html?locale=en_us

Extract the tar file to /opt

$ tar -xvzf splunkforwarder-8.2.6-a6fe1ee8894b-Linux-x86_64.tgz -C /opt

Change the permission to 777 of the Splunk forwarder directory

$cd /opt

$ chmod -R 777 splunkforwarder

Start the Splunk service

$ cd /opt/splunkforwarder/bin

$ ./splunk start

Accept the license and provide a new username and password (username: xyz & password: yyyy).

How To Add Remote Hosts to Splunk Server To Push Log For Indexing

Enable the 9997 port on Splunk Enterprise and make sure 9997 is listening

$ netstat -tnlp

$ cd /opt/splunkforwarder/bin
$ ./splunk add forward-server <splunk enterprise server ip address>:9997

The below command will send /var/log/messages data to the Splunk server.

$ ./splunk add monitor /var/log/messgaes

Use Case: Producing error log on the remote server

Install nginx web server

$ yum install nginx

Start the nginx service

$ systemctl start nginx

Add the /var/log/nginx path to the Splunk server

#./splunk add monitor /var/log/nginx

Try to access the UI through public IP http://public-ip

Produce the error from the UI
Example: http://public-ip/efgh this will redirect some error page

Now, check the logs on the remote machine

$ cd /var/log/nginx

$ tail -f error.log

2022/06/07 10:02:59 [error] 1831#0: *5 open() “/usr/share/nginx/html/efgh” failed (2: No such file or directory), client: 122.171.129.72, server: _, request: “GET /efgh HTTP/1.1”, host: “34.228.170.107”

Log in to Splunk Enterprises UI, Navigate to Search and Reporting, and Click on Data Summary where you will able to see all the added hosts.
Search the error generated on nginx through Splunk Enterprise, Navigate search and reporting on Splunk Enterprise.

You can use time ranges to troubleshoot an issue if you know the approximate timeframe when the issue occurred.

Narrow the time range of your search to that timeframe. For example, to investigate an incident that occurred sometime in the last hour, you can use the default time range Last 24 hours, but a better option is the Last 60 minutes.

Example 1: host="ip-10-0-9-60.ec2.internal"

hostname="ip-10-0-9-60.ec2.internal"

Example 2 : host="hostnameA" AND host=”hostnameB”

Example 3 : source = /var/log/nginx/error.log

Preset time ranges

The time range picker has many preset time ranges that you can select from.

Click the time range picker to see a list of the time range options. The Presets option contains Real-time, Relative, and Other time ranges.
Real-time searches display a live, streaming view of events. You can specify a window over which to retrieve events.
Historical searches display events from the past. You can restrict your search by specifying a relative time range or a specific date and time range.
Because the data for the Buttercup Games online store is a snapshot of historical data, you will not use the ”’Real-time”’ preset time ranges in this tutorial.
In the Presets option in the Relative list, click Yesterday.
The number of events returned should be larger. You changed the time range from Last 24 hours to Yesterday.

Custom time ranges

Use a custom time range when one of the preset time ranges is not precise enough for your search.

Specify relative time ranges

You can use the Relative option to specify a custom time range.

Open the time range picker.
To run a search over the last two days, select the Relative time range option.
For Earliest, type 2 in the field, and select Days Ago from the drop-down list.
For the Latest, the default is Now. Select Beginning of Today.
Click Apply.
The timestamps that appear below the radio buttons adjust based on your selections in the Relative list of time ranges.
As mentioned before, if no events are returned, select a different time range, such as 4 Days Ago or 1 Week Ago.

Specify date and time ranges

You can also use the Date Range and Date & Time Range options to specify a custom time range.

Use Between to specify that events must occur between the earliest and latest date.
Use Before to specify that events must occur before a date.
Use Since to specify that events must occur after a date.

You use the Date Range option to specify dates. The following screen image shows the calendar that you can use to select a date.

You use the Date & Time Range option when you want to specify both a date and a time. The following screen image shows the “Between”, “Before”, or “Since” options.

Setting Up Alerts In Splunk

Splunk alerts are actions that get triggered when a specific criterion is met which is defined by the user. The goal of alerts can be logging an action, sending an email, or outputting a result to a lookup file, etc.

Creating an Alert

You create an alert by running a search query and saving its result as an alert. In the below screenshot, we take the search for day-wise file count and save the result as an alert by choosing the Save As option.

In the next screenshot, we configure the alert properties. The below image shows the configuration screen.

The purpose and choices of each of these options are explained below −

Title − It is the name of the alert.
Description − It is a detailed description of what the alert does.
Permission − Its value decides who can access, run, or edit the alert. If declared private, then only the creator of the alert has all the permissions.
To be accessed by others the option should be changed to Shared in App.
In this case, everyone has read access but only the power user has edit access for the alert.
Alert Type − A scheduled alert runs at a pre-defined interval whose run time is defined by the day and time chosen from the drop downs.
But the other option on real-time alert causes the search to run continuously in the background.
Whenever the condition is met, the alert action is executed.
Trigger condition − The trigger condition checks for the criteria mentioned in the trigger and sets off the alert only when the alert criteria are met.
You can define the number of results number of sources or several hosts in the search result to trigger the alert.
If it is set for once, it will execute only once when the result condition is met but if it is set to reach Result, then it will run for every row in the result set where the trigger condition is met.
Trigger Actions − The trigger actions can give a desired output or send an email when the trigger condition is met.
The below image shows some of the important trigger actions available in Splunk.

How to upload the local data logs file to Splunk server https://docs.splunk.com/Documentation/Splunk/8.2.6/SearchTutorial/GetthetutorialdataintoSplunk

How to generate the Report https://docs.splunk.com/Documentation/Splunk/8.2.6/SearchTutorial/GetthetutorialdataintoSplunk

How to create the Dashboard https://docs.splunk.com/Documentation/Splunk/8.2.6/SearchTutorial/Createnewdashboard

Conclusion:

Explore the world of Splunk Enterprise to transform data management and analysis. IT pros can fully use their infrastructure using Splunk, whether it’s for providing comprehensive reports, deploying universal forwarders, or setting up alarms. Accept the challenge of configuring, utilizing, and mastering Splunk to gain unmatched insights and IT proficiency. Cheers to your successful exploration!