Friday, January 5, 2024

01_PL_Splunk Fundamentals 1 ( SPLUNK #1)


1 Discussion on different Splunk Certification Paths 1

2 Splunk Basic : Architecture of Splunk 1

3 Splunk Basic : install splunk 2

4 Splunk Enterprise: How to get free developer license 2

5 Splunk UF : How to Install Universal Forwarder In Google Cloud Ubuntu Instance 2

6 Splunk Admin : How to install full stack Splunk in Google Cloud Ubuntu Instance 3

7 Splunk Basic: How to install Splunk in Google Cloud Windows instance 3

8 Creating our first app : "The Movie database(TMDB)" 3

9 Splunk Getting the data In : Adding scripted input(Simple Streaming) to "TMDB" app 3

10 Adding custom checkpointing to "TMDB" app scripted input 3

11 Splunk Getting the data In : How HTTP Event Collector works 4

12 Splunk Basic: Know Your Search Interface 4

13 Splunk Basic: Configuration Files Basics 5

13.1 Folders 6

13.1.1 System 6

13.1.2 Users 6

13.1.3 Users 6

14 Splunk Configuration files : Fundamentals about props.conf and transforms.conf 7

14.1 props.conf 7

14.2 transforms.conf 7

15 Splunk Configuration Files : Event line breaking using props.conf 7

16 Splunk Configuration Files : Timestamp extraction using props.conf 10

17 A 10

18 A 10


  1. Discussion on different Splunk Certification Paths

https://github.com/siddharthajuprod07/youtube/tree/master/splunk_certification_path

  1. Splunk Basic : Architecture of Splunk

  • 4 Stages of splunk

    • Input

    • Parse the data into events

    • Store events in Indexs

    • Searches and reports

  • SPL = Splunk Processing Language

  • Components

    • Forwarder

    • Indexer

    • Search Head

  • Deployment Server

    • All APP level configuration and code  in one place 

  • Search Prompt 

    • Index=main


  1. Splunk Basic : install splunk

A screenshot of a computer error Description automatically generated

  • Splunkd and Universal Forwarder– port 8089

  • Splunk web 8000

  • Splunk home = splunk installed directory


  1. Splunk Enterprise: How to get free developer license

  • Valid for 6 months then we can extend

  • 10 GB data

  • Search splunk dev license  and request by submitting the agreement


  1. Splunk UF : How to Install Universal Forwarder In Google Cloud Ubuntu Instance

  • Cloud.google.com

    • Create a project in Google Cloud

    • Compute Engine – VM Instances - US East is the most cheaper one 

    • Download splunk in VM – Install – start Splunk  (Default it will not start) 

      • Provide admin credentials

  1. Splunk Admin : How to install full stack Splunk in Google Cloud Ubuntu Instance

  • breadcrumb menu or trail



  1. Splunk Basic: How to install Splunk in Google Cloud Windows instance

  • Mstsc – remote desktop connection command


  1. Creating our first app : "The Movie database(TMDB)"

  1. Splunk Getting the data In : Adding scripted input(Simple Streaming) to "TMDB" app

  • Tmdb site api (if we registered then we can go here) – GET Upcoming 

  • Create script in python to get the data from TMDP 

  • Check Pointing concept

  • Simple python code provided and loaded in splunk bin folder – we can define the index where to be indexed

  • Once added the source by code, inputs.conf file created

  • Source=”C:\Program Files\Splunk\etc\apps\tmdb\bin\tmdb_upcoming.py”

  • sourcetype=_json

  • In search results. Fields > Selected fields, Interesting fields

  • Checkpoint – to avoid the same data indexing again – this will give duplicate

  1. Adding custom checkpointing to "TMDB" app scripted input

  • Index=”tmdb_index“  

  • Create checpoint folder in bin, Keep one empty text file name as “checkpoint.txt” 

  1. Splunk Getting the data In : How HTTP Event Collector works

https://github.com/siddharthajuprod07/youtube/tree/master/http_event_collector

https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector

https://www.guidgenerator.com/online-guid-generator.aspx


We are exposing data to the outside world for indexing the data

Settings > Data Inputs > HTTP Event Collector > Click Global Settings  --All Tokens = Enabled, Data Source Type = _json, Default index , HTTP Port 8088 (we can change the port)

Create a Token

Authentication  -= basic and tokenized 

Index=main  | delete – this will delete the indexed data


Debugging: splunk/var/log/introspection folder contains all log file

  1. Splunk Basic: Know Your Search Interface

SPL= Splunk Processing Language

Time Picker

Documentation Link

Tutorial Link

Data Summary

Search History


Search a query  index=main – click job – inspect job – It shows Execution Costs, Search Job Properties – This will be useful for performance improvement of query / search

  • We can export the result in csv or json or xml

  • Mode

    • Fast mode (default)

    • Smart 

    • Verbose 

  • Selected Fields

    • Host – which server

    • Source – the data file

    • Sourcetype = what type of files csv or json or etc

  • Interesting Fields

    • Splunk creates these fields based on the mode

    • Punct = overview of data structure

    • We can move the interesting fields to selected fields by clicking “YES”

    • Coverage = if the particular fields appears in all the events that means 100%

    • Extract new fields – 

  • Verbos mode is iom,portant to run statistyics

  • Index=main | stats count – we can not see the events except verbose mode

  • Patterns, statistics, visualization save as dashboard, report, alert

  • Index=main | table columns








  1. Splunk Basic: Configuration Files Basics

A screenshot of a computer Description automatically generated


Search config files 

A screenshot of a computer Description automatically generated

Commands.conf

Contains all the configuration files details

Important folders are > System, users, apps

All configuration stored in either default folder or local folder

  1. Folders 

    1. System

  • All the configuration related to system level – comes from splunk

  1. Users

  • All the different user level configurations 

  1. Users

  • All apps listed here – configurations files will be available in apps folder

  • Users folder will have only local folder – not default folder

Configuration

  • Global

  • Search Time Context / User Context


Same configuration under system and user folder –  based on the task this will be consider 

Index is global – but search is user context

A screenshot of a computer Description automatically generated


Localhost:8000/en-US/debug/refresh – to refresh all the configuration files

https://docs.splunk.com/Documentation/Splunk/9.1.2/Admin/Listofconfigurationfiles

  1. Splunk Configuration files : Fundamentals about props.conf and transforms.conf

    1. props.conf 

  • Data processing stage 

  • Parsing pace we can mask the data like PHI or PI

  • Props.conf we can place based on our settings  generally 

    • SPLUNK_HOME\etc\system\default\props.conf

    • SPLUNK_HOME\etc\apps\appname\default\props.conf

  • Stanza = group name

  • The last occurrence will take preference in props.conf stanza

  • Stanza based on  (we can use regex also) – we can set priority in Stanza

    • sourcetype,  [sourcetype]

    • source,  [source :: <sourcename>]

    • host [host :: <>]

    • [rule :: <>]


  1. transforms.conf


  1. Splunk Configuration Files : Event line breaking using props.conf

  • When we add a new file we can set event breakers 

A diagram of a software development Description automatically generated

  • Event line breaking happens in Parsing phase

A screenshot of a computer program Description automatically generated

Event Breaks

If we want to use the event breaking we can save it – that info will save it in props.conf


  1. Splunk Configuration Files : Timestamp extraction using props.conf

A screenshot of a computer Description automatically generated

_TIME is important 

  • All date time logics handled by splunk store din datetime.xml available in etc folder

  1. Splunk Configuration Files : Search time field extraction



  1. search_time_operation_sequence.PNG

  1. Splunk Configuration Files : Index time field extraction

A screenshot of a computer Description automatically generated

  • Mv_field

  • Props.conf have TRANSFORM_Stanza > transforms.conf will have the xmlExtraction.   ??? we can get sql server userid by using this

  • Seems we need to get the windowslogevent file index set in props.conf and transforms.conf

  • Mv_extraction + REPEAT_MATCH = multivalue key field we can use this to add multiple values to one

  • SEDCMS-<class>  – for data masking

  1. Splunk Lookups : Lookups fundamentals & detail discussion on KV Store Lookups

https://github.com/siddharthajuprod07/youtube/tree/master/kv_store_lookups

1. Lookups Intro

2. Different types of lookup

3. What is KV store lookup

4. Comparison with CSV lookup

5. How to create KV store lookup from Splunk UI

6. How to populate KV store lookups using query and rest API

7. How to update KV store lookups

8. Talk about background conf files 

9. How to create Automatic KV store lookups

  1. A

  2. A

at No comments:
Labels:
Subscribe to: Comments (Atom)