splunk / utbox

URL Toolbox

Building blocks for URL Manipulation in Splunk Enterprise

Documentation
·
Report Bug
·
Splunkbase

About The Project

URL Toolbox (UTBox) is a set of building blocks for Splunk specially created for URL manipulation. UTBox has been created to be modular, easy to use and easy to deploy in any Splunk environments.

One of the core feature of UTBox is to correctly parse URLs and complicated TLDs (Top Level Domain) using the Mozilla Suffix List. Other functions like shannon entropy, counting, suites, meaning ratio, bayesian analysis, etc, are also available.

UTBox has firstly be created for security analysts but may fit other needs as it’s a set of building blocks. UTBox only needs to be deployed on Splunk Search Heads (the bundles will automatically be sent to your Splunk Indexers). Finally, each lookups is shipped with a macro to make it easier to use.

Read about this app on Splunk Blogs!

Getting Started

This section outlines the steps required to use the app on a Splunk Enterprise environment. If you want to develop the code base further, refer to the Development section of this README.

Prerequisites

• Splunk Enterprise / Cloud: Refer to Splunkbase for supported versions.

Installation

The easiest way to install the URL Toolbox is via the in-product app browser (Manage Apps -> Browse More Apps). It will allow you to install the latest version from Splunkbase.

If you need a specific version of the app or you want to inspect the app bundle before installation, please refer to the Splunk Documentation for your respective platform.
This app needs to be installed on the Search tier of your deployment.

Usage

This app provides a set of macros that simplify the interaction with the bundled lookups.

Please find below some selected samples of commands and their respective output. Please find more in-depth examples and explanation in the docs.

ut_parse_simple

SPL

|makeresults count=1 
| eval url="https://splunk.com" 
| `ut_parse_simple(url)`

Output
|_time |url |ut_fragment|ut_netloc |ut_params|ut_path|ut_query|ut_scheme|
|----------------------------|------------------|-----------|----------|---------|-------|--------|---------|
|2021-12-16T10:29:07.000+0000|https://www.splunk.com/en_us/blog/security/ut-parsing-domains-like-house-slytherin.html|None |www.splunk.com|None |/en_us/blog/security/ut-parsing-domains-like-house-slytherin.html|None |https |

ut_parse

SPL

|makeresults count=1 
| eval url="https://www.splunk.com/en_us/blog/security/ut-parsing-domains-like-house-slytherin.html", list="*" 
| `ut_parse(url, list)`

Output
|_time |list |url |ut_domain |ut_domain_without_tld|ut_fragment|ut_netloc|ut_params|ut_path |ut_port|ut_query|ut_scheme|ut_subdomain|ut_subdomain_count|ut_subdomain_level_1|ut_tld|
|----------------------------|------------------|----|----------|---------------------|-----------|---------|---------|-----------------------------------------------------------------|-------|--------|---------|------------|------------------|--------------------|------|
|2021-12-16T10:30:00.000+0000|* |https://www.splunk.com/en_us/blog/security/ut-parsing-domains-like-house-slytherin.html|splunk.com|splunk |None |www.splunk.com|None |/en_us/blog/security/ut-parsing-domains-like-house-slytherin.html|80 |None |https |www |1 |www |com |

ut_shannon

SPL

|makeresults count=1 
| eval url="buttercup" 
| `ut_shannon(url)`

Output
|_time |url |ut_shannon|
|----------------------------|------------------|----------|
|2021-12-16T10:32:19.000+0000|buttercup |2.725480556997868|

ut_countset

SPL

|makeresults count=1 
| eval url="buttercup"
| eval set="tu" 
| `ut_countset(url,set)`

Output
|_time |set |url |ut_countset |
|----------------------------|------------------|----|---------------------------------------------|
|2021-12-16T10:34:17.000+0000|tu |buttercup|{"ut_countset": {"sum": 4, "74": 2, "75": 2}}|

Development

Development Setup

1. Clone this repository
2. Symlink the utbox directory to your $SPLUNK_HOME/etc/apps/ directory.

If you want to use docker compose, you can run

docker compose up

to get a local Splunk environment with the app mounted into `$SPLUNK/etc/apps/utbox.

Running Tests

make test

Creating a release bundle

Run the package target in the provided Makefile.

make package

This will create an app package in dist/utbox.tar.gz

License

Please refer to the License on Splunkbase.

Bug Fixes / Enhancement Requests

🐞 ✍🏼 💡 Create issue with appropriate label to submit the request.

Troubleshooting

SPL: index=_internal ERROR *bin/ut_*py

Acknowledgments

• Cedric Le Roux
• Ian Forrest
• Daniel Federschmidt
• Mayur Pipaliya
• Gurkan Gokdemir