Build the Splunk Enterprise Servers for the initial setup

Terminology

License Master (LM)

The License Master is an important component in a Splunk deployment, responsible for managing license allocations and usage across the entire environment. It allows administrators to control and monitor license usage for all Splunk components, including indexers, search heads, and forwarders. The License Master can be deployed as a standalone instance or as part of a distributed environment.

By using the License Master, administrators can prevent over-licensing and ensure compliance with Splunk licensing agreements. The License Master can also help optimize the usage of available licenses by re-allocating unused licenses to other components that require additional capacity. In addition, it provides detailed reporting on license usage, which can be used to identify trends, track usage patterns, and forecast future license requirements.

Monitor Console (MC)

Cluster Manager (CM)

Search Head Cluster (SHC)

Search Peers

The Deployment Server

Splunk Components

Universal Forwarder

Splunk forwarders are agents that collect data from remote sources and forward it to a central Splunk indexer for indexing and analysis. They are lightweight and designed for minimal resource usage. Heavy forwarders are a more powerful version of the forwarder, with additional capabilities such as data transformation, filtering, and routing. They are used in complex data collection scenarios, where the data needs to be preprocessed before indexing. Both forwarders and heavy forwarders play a crucial role in enabling centralized data collection and analysis, making it easier to monitor and troubleshoot complex IT environments

The Splunk Universal Forwarder, which has its own license, is used to collect and forward data to a Splunk deployment. When ingesting data from syslog on port 514, the best practice is to configure syslog to write logs and use a Splunk forwarder to collect them. Using TCP syslog and one or more syslog servers with a Universal Forwarder can improve the reliability of syslog delivery to Splunk.

Integrating third-party systems with Splunk allows for searching alerts to provision actions and using Splunk alerts to provision actions on a third-party system. Data can also be forwarded from Splunk forwarders to a third-party system without indexing it first.

If an update is made to an attribute in inputs.conf on a universal forwarder, the forwarder's fishbucket needs to be reset in order to reindex the data. The Monitoring Console monitors forwarders by forwarding internal logs from the forwarder.

The deployment server feature of Splunk is used for updating configuration and distributing apps to processing components, primarily forwarders. The Universal Forwarder has the capability of indexer acknowledgement and compressing data when sending it.

A remote monitor input is distributed to forwarders as an app. When a file is manually created on a universal forwarder, such as /opt/splunkforwarder/etc/apps/my_TA/local/inputs.conf, and a new app with a new inputs.conf file is deployed via the deployment server, the newly monitored file is /var/log/maillog.

After configuring a universal forwarder to communicate with an indexer, the successful connection to an index can be checked via the Splunk Web UI by searching for Index=_internal.

Using SSL to secure the feed from a forwarder does not automatically compress the feed by default.

Heavy Forwarder

Indexer

Preconfigured Indexes

Difference between Indexer and Forwarder

Directory Structure and Config files

Splunk Directory Structure

Config Files

---

Now lets build

Install the OS and configure network connections

1. If you haven't already install linux/windows OS on the server of choice

2. Add the IP address to the Splunk server

For windows

1. Click Start >Settings >Control Panel.

2. On the control panel, double-click Network Connections.

3. Right-click Local Area Connection.

4. Click Properties. ...

5. Select Internet Protocol (TCP/IP), and then click Properties.

6. Select Use the Following IP Address.

Note: the IP information in the image is only for informative purposes. Select an IP that is reachable for your environment

For linux

Building the Deployment Server:

1. Install Splunk on a server that will serve as the Deployment Server.

2. Once Splunk is installed, navigate to the Splunk Web interface by entering the IP address of the server followed by the port number (usually 8000) in a web browser.

3. Log in to Splunk Web with admin credentials.

4. Navigate to the Settings menu and select "Forwarder Management".

5. Select "Deployment Server" from the list of options.

6. Click on the "Set Up Deployment Server" button.

7. Enter the IP address of the server in the "Deployment Server URL" field.

8. Select the Splunk instance that will be the Deployment Server.

9. Configure any necessary settings, such as authentication and SSL.

10. Save the changes and restart Splunk.

Building Forwarders

1. Get Splunk Universal Forwarder by downloading it from the official website.

2. Extract the installation files using the following command after navigating to the directory where the file is downloaded:

[root@ABS]# tar -xvf splunkforwarder-8.0.0-8c86330ac18-Linux-x86_64.tgz 

Note: In the above command, replace the file name mentioned with the name of the downloaded file.

1. Start Splunk Universal Forwarder using the command:

[root@ABS]# cd splunkforwarder/bin [root@ABS]# ./splunk start --accept-license

1. Configure the forward server details (receiver host and port) in Splunk using the following command:

[root@dashboard]# ./splunk add forward-server ip:port 

Note: Replace 'ip:port' with the IP address and port number of your forward server.

1. Ensure that the receiving port in Splunk is enabled. For example, configure port number 9997 in your Splunk deployment.

2. Edit the 'inputs.conf' file on the Splunk Forwarder as follows:

[root@ABS]# ./splunk add monitor /opt/pingidentity/splunk/data/ [root@dashboard]# cat /opt/splunkforwarder/etc/apps/search/local/inputs.conf [monitor:///opt/pingidentity/pingidentity/dashboard/logs/attack.log/] index = pi_events sourcetype=pi_events_source_type disabled = false

1. Restart the Splunk Universal Forwarder: [root@ABS]# ./splunk restart

2. Verify that data is flowing into Splunk. Check the data flow in the Splunk snapshot.

---

Note:

If you can't see any data in Splunk, check the firewall settings.

---

Building Indexers

1. Ensure that the host you are preparing meets or exceeds the Splunk Enterprise system requirements. Note down the host name and IP address of the host.

2. Confirm that no firewalls are blocking any network traffic into or out of the host.

3. Download the Splunk Enterprise software onto the host.

4. Install the appropriate version of the Splunk software for the host's operating system.

5. Verify that the Splunk Enterprise software starts without any errors. You should also be able to perform a basic search using the Search app.

6. Download the latest version of the Splunk Add-on for Microsoft Exchange Indexes.

7. Extract the package content to $SPLUNK_HOME/etc/apps directory.

8. Restart Splunk Enterprise using PowerShell: > cd \Program Files\Splunk\bin > .\splunk restart

Configuration Steps:

1. Log into Splunk Enterprise on the indexer.

2. Click Settings > Forwarding and Receiving in the system bar. The "Forwarding and Receiving" page will load.

3. Under "Receive Data," click Configure Receiving.

4. Click New and enter the port number that you want Splunk Enterprise to listen on for incoming data from other Splunk instances in the "Listen on this port" field. The conventional port number is 9997.

5. Click Save to enable receiving on the indexer.

Next Steps:

1. Write down the host name or IP address and port number of the indexer.

2. Read Splunk's documentation on apps before proceeding.

3. Create the send to indexer app to continue building out a Splunk App for Microsoft Exchange deployment.

---

Building the Search Head

1. Download the Splunk software package for your operating system from the official Splunk website.

2. Install the software package by following the prompts in the installation wizard.

3. Once installed, launch Splunk and log in to the web interface.

4. Obtain the IP address of the instance you wish to configure as a search head, assuming that Splunk has already been installed. Then, navigate to the SEARCHHEAD UI at https://:8000.

5. Go to Settings, then Distributed Search, and then Search Peer. Click Add New.

6. Enter the indexer URI, as well as the remote username and password. This step involves adding the indexer's URI, username, and password. Click Save after completing this step.

7. Return to the Search Head UI, and navigate to Settings, then Distributed Search, and then Search Peer. You should now see your indexer configured as a search peer.

Congratulations! Your Splunk instance is now configured as a search head, allowing you to return queries and create knowledge objects as needed, provided that you have installed a heavy forwarder/universal forwarder and an indexer.

Once the search head is configured, you can start using it to search your data. To do this, you will need to create a search query in the search bar of the Splunk web interface. The search head will then send the query to the indexers in your environment to retrieve the data, which it will then display in the search results page. You can also create dashboards and alerts on the search head to help you monitor your data over time.