The LQMToolset configuration file contains the local configuration options, including data source definitions, tool instances, and tool chains.
Data Source
This setting specifies root source directories and what to do with the files once they've been processed.
[[Source.Directory]]
dirs = ["/tmp", "/var/spool"]
post_process = "delete"
post_process_location = "/home/lqmt/post_process/"
Setting | Explanation |
---|---|
dirs |
A list of directory paths, whose contents will be scanned for input files to process. |
post_process |
An action for LQMToolset to perform after processing a file. Allowed values include nothing , delete , move , and track . nothing will simply leave the files untouched after processing, and delete will remove them. move will mark the input files to be moved after processing to another directory. track will mark the input files to be tracked in another text file. move is the default value when nothing is set in the user configuration file. |
post_process_location |
Used in conjunction with the track post process option. Used to specify a custom location for where your track file will be placed. |
Source Filters
This setting specifies meta-data file based filtering of files before being processed. This allows a user to include or exclude files from being parsed and utilized in tools. The parameters are not required and do not need to be present in the configuration file.
[[Source.Filters]]
site_includes = [ 'SITE1' ]
site_excludes = [ 'SITE2' ]
payload_types = [ 'Alert' ]
payload_formats = [ 'STIX' ]
sensitivities = [ 'noSensitivity' ]
restrictions = [ 'WHITE' ]
reconnaissance = [ 'Touch' ]
max_file_age = "2 mon"
Setting | Explanation |
---|---|
site_includes |
(Optional) A list of SendingSite entries that are allowed. An empty list or no entry results in True to not filter for this setting. |
site_excludes |
(Optional) A list of SendingSite entries that are to be filtered out. An empty list or no entry results in True to not filter for this setting. |
payload_types |
(Optional) A list of PayloadType entries that are allowed (e.g. Alert , Report , etc). An empty list or no entry results in True to not filter for this setting. |
payload_formats |
(Optional) A list of PayloadFormat entries that are allowed (e.g. STIX , Cfm13Alert , etc). An empty list or no entry results in True to not filter for this setting. |
sensitivities |
(Optional) A list of DataSensitivity entries that are allowed (e.g. ouo , noSensitivity , etc). An empty list or no entry results in True to not filter for this setting. |
restrictions |
(Optional) A list of SharingRestrictions entries that are allowed (e.g. WHITE , AMBER , etc). An empty list or no entry results in True to not filter for this setting. |
reconnaissance |
(Optional) A list of ReconPolicy entries that are allowed (e.g. Touch , NoTouch , etc). An empty list or no entry results in True to not filter for this setting. |
max_file_age |
(Optional) A string formatted %d %s for the maximum age file to be processed. This supports an offset for seconds 's', 'sec', 'secs', 'second', 'seconds' , minutes 'm', 'min', 'minute', 'minutes' , hours 'h', 'hr', 'hrs', 'hour', 'hours' , days 'd', 'day', 'days' , weeks 'w', 'week', 'weeks' , months 'mon', 'month', 'months' , years 'y', 'yr', 'yrs', 'year', 'years' . |
Parsers
LQMT uses special parsers for each type of alert file to get all the data into one common format. By default, the majority of the parsers are enabled, but some parsers are disabled by default. Detailed below are the types of parsers LQMT uses, which are enabled by default, and how you can configure LQMT to disable or enable them.
Parsers | Enabled by Default |
---|---|
Cfm13Alert |
True |
Cfm20Alert |
True |
STIX |
True |
stix-tlp |
True |
IIDactiveBadHosts |
False |
IIDcombinedUrl |
False |
IIDdynamicBadHosts |
False |
IIDrecentBadIP |
False |
STIXParser |
False |
RuleParser |
False |
Setting | Explanation |
---|---|
enable |
A list of parsers to enable. |
disable |
A list of parsers to disable. |
Configuration
[Parsers]
# The enabled and disabled params use lists to track which parsers to override.
enable = [ 'IIDcombinedUrl', 'IIDdynamicBadHosts', 'IIDrecentBadIP' ]
disable = [ 'STIX', 'Cfm20Alert' ]
Individual Parser Configurations
As necessary, each parser can take in a configuration file to control the Parser behavior. Refer to they systemconfig.py to view the default configuration file associations. Additionally, sample configurations are contained at lqmt/resources/parser_configs.
STIX Parser
The STIX parser allows for certain controlling behaviors. An example is as follows.
[[Filters]]
sources = [ 'US-CERT' ]
elements = [ 'indicators', 'incidents' ]
rules = [ 'snort' ]
Setting | Explanation |
---|---|
sources |
(Optional) A list containing case-insensitive names to match in the Information_Source element of the STIX_Header. |
elements |
(Optional) A list of the case-insensitive XML element titles to specifically extract from the STIX file $.<string> . At this time it must be one of the top level values STIX_Header , Observables , Indicators , TTPs , Exploit_Targets , Incidents , Courses_Of_Action , Campaigns , Threat_Actors , Related_Packages . |
rules |
(Optional) A list of case-insensitive Rule types to extract, currently only supports entries for snort , yara to automatically identify. |
Rule Parser
The Rule parser allows for parsing Snort and Yara rules from an input file. An example is as follows.
[[Filters]]
sources = [ 'US-CERT' ]
rules = [ 'snort', 'yara' ]
start_offset = 0
Setting | Explanation |
---|---|
sources |
(Optional) A list containing case-insensitive names to match in the SendingSite, Originator, Custodian elements of the CFM Meta Data File. |
rules |
(Optional) A list of the case-insensitive rule titles to specifically extract. At this time it must be one of the values snort , yara . |
start_offset |
(Optional) An integer represents the line offset inside a rules file. This is intended to be used in scenarios a file may have a header to skip before rules. |
Tool Chains
Tool Chains are built from tool instances, which are created from the available tools in LQMToolset. Chaining these tool instances together creates Tool Chains. To configure a new device, an entry must be added in the configuration file.
[[ToolChains]]
name = "arcsight-1"
chain = [ "cef-mapping", "arcsight-1" ]
Setting | Explanation |
---|---|
name |
A unique name identifying the Tool Chain. |
chain |
A list of tools to chain together to perform processing on each record. |
active |
Enables this Tool Chain if set to true . Otherwise, it will disable this Tool Chain if set to false . active is optional, and will default to true if not specified. |
NOTE: There can be many tools of a specific type--one for each physical device--but they must have distinct names.
Tools
Palo Alto
The Palo Alto tool places or removes blocks on Palo Alto Networks firewall devices.
[[Tools.PaloAlto]]
name = "my-pa200"
api_key = "API KEY OBTAINED FROM DEVICE"
api_username = "username"
api_password = "password"
hostname = "palo-alto.yourdomain.com"
badIPFiles = [ "BlockedIPs-01.txt", "BlockedIPs-02.txt" ]
block_lists = [ "CFM_EBL-01", "CFM_EBL-02" ]
db_location = "paloalto"
cafile = "pa-ames/pa.crt"
prune_method = "Expiration"
default_duration = 259200
unprocessed_file = "unprocessed/pa-ames.csv"
Setting | Explanation |
---|---|
name |
A unique name identifying this device. |
api_key |
The API key retrieved from the Palo Alto device for remote access. Note: You can either use an API key, or a username/password. An API key is recommended. |
api_username |
A username with with API access/privileges to the Palo Alto device. Note: You can either use an API key, or a username/password. An API key is recommended. |
api_password |
The corresponding password to the username listed above. Note: You can either use an API key, or a username/password. An API key is recommended. |
hostname |
The hostname or IP address of the Palo Alto device. |
badIPFiles |
A list of dynamic block-lists files to use. Each file can hold 300 less than the maximum number of IP addresses that the Palo Alto device supports. |
block_lists |
The named block lists configured on the Palo Alto device. |
db_location |
The path to the directory that will hold the local database of blocked IP addresses. |
cafile |
The path to your CA certificate file for the Palo Alto device. |
prune_method |
The method used to prune IP addresses from the database if there are more IP addresses than the Palo Alto device supports. IP addresses are pruned after removing all expired IP blocks. prune_method accepts Expiration , for removing blocks with the earliest expiration time; Added , for removing blocks with the earlist added time; and Detected , for removing blocks with the earliest detection time. |
default_duration |
The default time limit that a block should be held if the duration is not specified in the alert. |
unprocessed_file |
The path to the CSV file that will hold all unprocessed blocks. The contents will consist of filenames, whose names have been altered by adding a timestamp (YYMMDD-HHMMSS) between the name and the extension. (If the file lacks an extension, the timestamp will be appended to the filename.) |
actions_to_process |
A list of actions for this tool to process: Block , Revoke , Notify , Watch , SendReport , OtherAction , All . |
Device Setup & Configuration
LQMT uses Palo Alto's Dynamic Block Lists (also called External Block Lists or EBLs) to block IPs. There is a limit to the number of IP addresses that can be blocked by a Palo Alto device. See your devices documentation for these limits. Each Palo Alto device can have up to 10 block lists and each block list is limited to 300 less than the device's limit. The Palo Alto accesses the EBLs via http request to a web server. Configuring one is beyond the scope of this document.
Requirements:
- Web server to host the Dynamic Block List files
- Only needs to be accessible from the machine LQMT is installed on
- The LQMT machine also needs to have write access to the location it reads the EBLs from as it will write the files to that location
- The LQMT machine needs to trust the root CA for the Palo Alto web server
- One will need to be created if one hasn't already been imported
- The default certificate created by the device will not work.
- To create one,
- Log in to the web UI as an administrator
- Select the Device tab
- Select Certificates under the Certificated management menu
- Click on Generate at the bottom of the window
- Fill in the fields as appropriate
- Common name should be the machine name
- Ensure Certificate Authority is check
- Add any Certificate Attributes you may want
- Click Generate
- The click on the newly generated certificate and check the following: - Forward Trust Certificate - Forward Untrust Certificate - Certificate for Secure Web GUI
- After creating the certificate, you will need to export it
- Click the export button at the bottom of the window and accept the defaults
- Store it in a location accessible to LQMT
- Specify the location in the cafile configuration parameter for the specific Palo Alto device
- One will need to be created if one hasn't already been imported
- Create the block list objects
- Log into the Web GUI as an administrator
- Select the Objects tab
- Select the Dynamic Block Lists item from the menu
- Click the Add button at the bottom of the window
- Enter a name
- This name will be in the block_lists configuration parameter
- Optionally enter a description
- Enter the source location of the file backing this block list
- The physical location of the file will be added to the badIPFiles configuration parameter
- Set the repeat to Monthly at 00:00
- This is sufficient because LQMT will perform a refresh as necessary and doesn't need to rely on a regularly scheduled refresh
- Create the policies that use the block lists to restrict network traffic
- Due to the unique nature of each installation, this is beyond the scope of this document
- Some useful links related to using EBLs
Limitations
The Palo Alto module currently only blocks IP addresses and network ranges (CIDR). Any other block that are not supported will be output to the field specified in the unhandled_blocks configuration parameter.
Checkpoint
Place or remove blocks on Checkpoint devices. The connection to Checkpoint devices is done through a shared SSL key which needs to be generated on the machine LQMTools is installed on and copied to the Checkpoint device.
[[Tools.Checkpoint]]
name = "checkpoint"
hostname = "checkpoint.yourdomain.com"
port = 22
username = "foo"
originator = "bar"
default_duration = 259200
unprocessed_file = "unprocessed.txt"
actions_to_process = "All"
Setting | Explanation |
---|---|
name |
A unique name identifying this tool instance. |
hostname |
The Checkpoint device's hostname or IP address. |
port |
The port number through which to connect via SSH. |
username |
The username with which to login to the Checkpoint device. |
originator |
The originator that will be stored with the blocks put into the Checkpoint device. |
default_duration |
The default time a block should be in place for if the duration is not specified in the alert. |
unprocessed_file |
A file that will hold all unprocessed blocks. This file will be a CSV file and will have the creation timestamp (YYMMDD-HHMMSS) embedded in the filename before the extension, or at the end if no extension is specified. For example, if the filename is dir/file.txt, the file created, if nessecary, would be dir/file.20150401-113524.txt. |
actions_to_process |
Specify the list of actions this cool can/will process. Valid values: Block , Revoke , Notify , Watch , SentReport , OtherAction , All . |
Device Setup & Configuration
The LQMTool module for checkpoint devices uses the checkpoint firewall's Suspicious Activities Monitoring Protocol (samp) to block IP addresses via the command line interface and ssh from the LQMT machine. The following outlines the steps necessary to configure the device for use with LQMT. The following assumes the computer that is running the LQMT software is named lqmt.domain.com, the checkpoint computer is named cp.domain.com
- Use the web UI to create a new user with an adminRole, set the shell to /bin/bash, and ensure that command line access is granted. For purposes of this document, the username cfm will be used.
- Set up password-less ssh access for the cfm user just created. The exact steps will depend on the machine running LQMT, but the basics:
- Create an RSA key pair if not already done. This can be done by running the command:
ssh-keygen -t rsa
on lqmt.domain.com and accepting all the defaults. For these purposes, it is best to not have a passphrase for this key pair. - Copy the public key to the admin user's account on the checkpoint device. Depending on the open-ssl version on the LQMT computer, you could use the command:
ssh -copy-id cfm@cp.domain.com
. If that command isn't available, you can run the following on lqmt.domain.com as the user that will be running LQMT:cat ~/.ssh/id_rsa.pub | ssh user@123.45.56.78 "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys
- Verify password-less access by trying to ssh to the checkpoint machine:
ssh cfm@cp.domain.com
- In addition, the checkpoint device needs to be configured as a firewall in order for the LQMT to be effective.
- Create an RSA key pair if not already done. This can be done by running the command:
Limitations
The Checkpoint module currently only blocks IP addresses and network ranges(CIDR). Any other blocks that are not supported will be output to the file specified in the unhandled_blocks configuration parameter.
ArcSight
The ArcSight tool places or removes blocks on ArcSight devices.
[[Tools.ArcSight]]
name = "arcsight-1"
host = "arcsight1.yourdomain.com"
port = 598
protocol = "tcp"
actions_to_process = [ "All" ]
Setting | Explanation |
---|---|
name |
A unique name identifying this tool instance. |
host |
The hostname or IP address of the ArcSight device. |
port |
The port number on which the ArcSight device is listening. |
protocol |
The IP protocol to use: tcp or udp . |
actions_to_process |
A list of actions for this tool to process: Block , Revoke , Notify , Watch , SendReport , OtherAction , All . |
Device Setup & Configuration
To configure the ArcSight logger to receive data from LQMToolset, a new Receiver needs to be configured. To do so, login through the web interface of the ArcSight device, and navigate to Configuration > Receivers. Click the Add button, enter a name for the Receiver, select CEF TCP Receiver, and click next. At this time, modify any necessary parameters. Do not change the source type. Note the port number! (You'll need that port number for the LQMToolset configuration file). By default, new Receivers aren't enabled. Enable the new Receiver by clicking the box on the far right of the Receiver list for the new Receiver that was just added.
CEF
The CEF tool converts data from the intermediate format to the CEF format.
[[Tools.CEF]]
name = "cef-mapping"
Setting | Explanation |
---|---|
name |
A unique name identifying this tool instance. |
Syslog
The Syslog tool is used to log information to remote Syslog servers.
[[Tools.SysLog]]
name = "syslog-1"
host = "syslog1.yourdomain.com"
port = 514
protocol = "tcp"
messageHead = "WY rsyslogd: Message from LQMT - "
messageFields = ["indicatorType", "indicator", "action1"]
Example output in syslog:
Aug 26 14:23:01 WY rsyslogd: Message from LQMT - indicatorType="IPv4Address" indicator="192.168.1.1" action1="Block"
Aug 26 14:23:01 WY rsyslogd: Message from LQMT - indicatorType="IPv4Address" indicator="192.168.1.2" action1="Block"
Aug 26 14:24:05 WY rsyslogd: Message from LQMT - indicatorType="IPv4Address" indicator="192.168.1.5" action1="Block"
Setting | Explanation |
---|---|
name |
A unique name identifying this tool instance. |
host |
The hostname or IP address of the remote syslog server. |
port |
The port number the Syslog server is listening on. Note: Syslog defaults to 514, so if left blank LQMT will also default to 514 for communication. |
protocol |
The IP protocol to use: tcp or udp . |
messageHead |
Used at the beginning of every message sent to Syslog. |
messageFields |
Used to specify what fields you want extracted from the alerts and sent in the message to Syslog. |
FlexText
FlexText is a tool used to output parsed alert data in a user-defined, character delimited, format.
[[Tools.FlexText]]
name = "flextext-tool"
fileParser = "CSV"
fields = ['action1', 'indicator', 'reportedTime']
delimiter = ","
quoteChar = '"'
escapeChar = '\\'
headerLine = true
doubleQuote = false
quoteStyle = "Minimal"
primarySchemaConfig = "resources/schemaDefinitions/lqmtools.json"
fileDestination = "/home/output/test.csv"
incrementFile = true
Setting | Explanation |
---|---|
name |
A unique name identifying this tool instance. |
fileParser |
Select parser type. Currently only supports and defaults to CSV . |
fields |
Fields, identified from the intermediate format, to be extracted. The order of the fields here determines the order of the output. |
delimiter |
A single character delimiter used to separate fields. Default value is , . |
quoteChar |
Character used to quote respective values. |
escapeChar |
Character used to escape other characters. |
headerLine |
Boolean value used to set if a header line detailing the extracted values should be included in the output. Defaults to False . |
doubleQuote |
Determines how the quoteChar itself is quoted. If True then the character is doubled. If False , the character is prefixed to the quoteChar. |
quoteStyle |
Sets the style of the quoting. Can be one of four values. Minimal : only quotes fields that contain special characters. NonNumeric : only quotes non-numeric fields. All : quotes all fields. None : No fields are quoted |
primarySchemaConfig |
Defines the path to the primary schema configuration. Most users won't need to change the default setting; if you do, then some understanding of FlexTransform is suggested. |
fileDestination |
Sets the destination of the output file. |
incrementFile |
Used to increment the output file. When set to True , the output file name will be incremented with a timestamp. When set to False the output will be appended to any prexisting file of the same name. Defaults to False |
Splunk
Ingest CTI data into your Splunk instance in a keyword value format.
[[Tools.Splunk]]
name = "splunk-tool"
host = "https://test-splunk.dev.yourdomain.gov"
port = 8089
username = "serviceuser"
password = "greatpassword123"
cert_check = true
source = "lqmt-splunk-tool"
sourcetype = "lqmt-test"
index = "main"
fields = ['action1', 'indicator', 'reportedTime']
Setting | Explanation |
---|---|
name |
A unique name identifying this tool instance. |
host |
Host address of your Splunk instance. |
port |
Port used to communicate with your Splunk instances REST API interface. Defaults to 8089 . |
username |
Username that you want to authenticate with. |
password |
Password that you want to authenticate with. |
cert_check |
Used to disable the the certificate check. This is helpful for testing on a machine that you haven't imported your Splunk SSL cert on yet. Defaults to false . |
source |
Name of the source you want the data to be identified by. Defaults to lqmt . |
sourcetype |
The sourcetype that you want to ingest the data into. |
index |
The index that you want to ingest data into. |
fields |
Fields, identified from the intermediate format, to be extracted. If fields are not provided, then it defaults to a value of ['all'], which will automatically extract all supported field types. |
Device Setup and Configuration
LQMT authenticates using Splunk's token-based authentication endpoint. This requires you to provide a username and password. It is recommended that you create a new role specifically for REST API usage as well as a new user that belongs to this role. Having a specific user and role just for REST API access will make auditing your REST API usage much easier.
Role Creation
The following steps will walk you through creating a role with enough permissions to permit usage of the REST API.
- Log into the web interface of your Splunk instance and go to
Settings
and thenAccess Controls
. - From the
Access Control
screen, click theAdd New
action in theRoles
row. - Assign this role any name you see fit. We will use
splunk_rest_api
in this example. - Everything will be left default with the exception of the
Capabilities
section. Go to theCapabilities
section and add theedit_tcp
capability from theAvailable capabilities
section to theSelected capabilities
section. - By default, Splunk should allow this role to access the
main
index under theIndexes searched by default
section and theAll non-internal indexes
index under theIndexes
section. Verify with your Splunk admin that you won't need access to any other indexes. - Once completed, hit
Save
You should now have a custom role created for using the Splunk REST API.
User Creation
The following steps will walk you through creating a new Splunk user and assigning them to a role.
- Log into the web interface of your Splunk instance and go to
Settings
and thenAccess Controls
. - From the
Access Control
screen, click theAdd New
action in theUsers
row. - Name your user. In our case, we will name our user
splunk_rest_api_user
- Fill in the
Full name
,Email address
, andTime zone
fields as you see fit. They are not needed, but can still used if you desire. - Under the
Assign to roles
section, select the role that you previously created for REST API usage. Our role was namedsplunk_rest_api
, so we will select that. - By default, splunk will have the
user
role pre-selected in theSelected roles
section. We recommend removing this role if you are only using this account for user Splunk REST API. - Set a password for your user.
- One completed, hit
Save
You should now have a user created specifically for using Splunk's REST API. This user can now be used in your LQMT instance for pushing data to your Splunk instance.
Implementation Details
LQMT supports Splunk using Splunk's REST API. LQMT authenticates against Splunks /services/auth/login endpoint. Once authenticated, LQMT receives a token from Splunk and uses this for future interactions. When the token exprires, LQMT just reauthenticates and receives a new token. The token's lifetime is determined by how you configure your Splunk instance - more information can be found here
Bro
Convert CTI data into Bro's Intelligence Framework format. Data in this format can be easily read by your Bro instance.
[[Tools.Bro]]
name = "bro-tool"
file = "/home/bro/lqmt-feed.txt"
header_fields = ['indicator', 'reportedTime', 'directSource']
increment_file = true
Setting | Explanation |
---|---|
name |
A unique name identifying this tool instance. |
header_fields |
Specify the fields you want converted and included in the output. Valid fields are extrated from the Intermediate Data Format. By default, LQMT will extract all valid fields from the intermediate data foramt. |
file |
Path and name of where you want the file output(Ex: /home/bro/lqmt-bro-feed.txt). By default, LQMT will output the file in the directory where LQMT as a file named lqmt-bro-feed.txt . |
increment_file |
Used to increment the output file. When set to True , the output file name will be incremented with a timestamp. When set to False the output will be appended to any prexisting file of the same name. Defaults to False |
null_value |
Value to be used to represent that a field is empty. Bro's default value is a hyphen ('-'), so by default LQMT uses a hyphen. |
Snort
Extract Snort rules from a STIX CTI datafile and append to a Snort rules definition file. The ingested CTI datafile must contain Test Mechanism entries that contain Snort rules - STIX Documentation. An example of the STIX data model is in this Implementation section.
<indicator:Test_Mechanisms>
<indicator:Test_Mechanism id="example:testmechanism-a1475567-50f7-4dae-b0d0-47c7ea8e79e1" xmlns:snortTM='http://stix.mitre.org/extensions/TestMechanism#Snort-1' xsi:type='snortTM:SnortTestMechanismType'>
<indicator:Efficacy timestamp="2014-06-20T15:16:56.987966+00:00">
<stixCommon:Value xsi:type="stixVocabs:HighMediumLowVocab-1.0">Low</stixCommon:Value>
</indicator:Efficacy>
<indicator:Producer>
<stixCommon:Identity id="example:Identity-a0740d84-9fcd-44af-9033-94e76a53201e">
<stixCommon:Name>FOX IT</stixCommon:Name>
</stixCommon:Identity>
<stixCommon:References>
<stixCommon:Reference>http://blog.fox-it.com/2014/04/08/openssl-heartbleed-bug-live-blog/</stixCommon:Reference>
</stixCommon:References>
</indicator:Producer>
<snortTM:Rule><![CDATA[alert tcp any any -> any any (msg:"FOX-SRT - Flowbit - TLS-SSL Client Hello"; flow:established; dsize:< 500; content:"|16 03|"; depth:2; byte_test:1, <=, 2, 3; byte_test:1, !=, 2, 1; content:"|01|"; offset:5; depth:1; content:"|03|"; offset:9; byte_test:1, <=, 3, 10; byte_test:1, !=, 2, 9; content:"|00 0f 00|"; flowbits:set,foxsslsession; flowbits:noalert; threshold:type limit, track by_src, count 1, seconds 60; reference:cve,2014-0160; classtype:bad-unknown; sid: 21001130; rev:9;)]]></snortTM:Rule>
<snortTM:Rule><![CDATA[alert tcp any any -> any any (msg:"FOX-SRT - Suspicious - TLS-SSL Large Heartbeat Response"; flow:established; flowbits:isset,foxsslsession; content:"|18 03|"; depth: 2; byte_test:1, <=, 3, 2; byte_test:1, !=, 2, 1; byte_test:2, >, 200, 3; threshold:type limit, track by_src, count 1, seconds 600; reference:cve,2014-0160; classtype:bad-unknown; sid: 21001131; rev:5;)]]></snortTM:Rule>
</indicator:Test_Mechanism>
</indicator:Test_Mechanisms>
[[Tools.Snort]]
name = "snort-tool"
config_paths = ['/etc/nsm/<interface0>', '/etc/nsm/<interface1>']
config_filename = 'snort.conf'
rule_paths = ['/etc/nsm/rules']
rule_filename = 'example.rules'
max_rules_count = 50
mode = "append"
Setting | Explanation |
---|---|
name |
(Required) A unique name identifying this tool instance. |
config_paths |
(Required) List of paths to Snort config file - example is based on Security Onion with one configuration per interface. |
config_filename |
(Optional) String filename for the config file (at each path), defaults to 'snort.conf' if not configured. |
rule_paths |
(Required) List of paths for where to insert Rules files. |
rule_filename |
(Optional) String filename for the rule file (at each path), defaults to TBD if not configured. |
max_rules_count |
(Optional) Integer representing the maximum number of rules to have in a rules file. Set to or allow to default to -1 to turn off management. |
mode |
(Optional) String defining rule writing mode full vs. append . Full overwrites the full file, append smartly enters only unique new rules. |
From Snort
This tool is capable of searching Snort alert and capture logs to zip a data for transmission. It currently only supports Full logging mode for Snort. It is advised that review be performed on alert matching and pcap matching code to ensure it meets the users needs.
[[Tools.From_Snort]]
name = "snort-pull-tool"
alert_paths = ['/var/log/snort/log']
packet_paths = ['/var/log/snort/log']
mode = 'match'
rules = ['/snort/rules/<name>.rules']
max_file_age = '2 weeks'
result_path = '/lqmt/results'
Setting | Explanation |
---|---|
name |
(Required) A unique name identifying this tool instance. |
alert_paths |
(Required) List of paths to Snort alert log files. |
packet_paths |
(Required) List of paths to Snort pcap capture log files. |
mode |
(Optional) String defining the alert and capture mode - full vs. match . In match mode, the tool attempts to grab only alert entries and PCAP matches to the alert IP addresses. |
rules |
(Required) List of full paths and file names to the Snort rules files to extract Snort IDs for matching in alerts. |
max_file_age |
(Required) String defining the maximum file modification age to look for matches. Time can be defined similar to Splunk entries sec , min , hr , day , week , mon , yr . |
result_path |
(Required) Folder path to write the final package, allows for providing to a transmission tool (CFM client, NiFi, etc.) |
Setup and Configuration
The Tool will automatically attempt to add "include
Implementation Details
The Tool automatically managements the Rules file to ensure each line is a unique entry.
The method for performing this action utilizes Python Sets which are unordered. Therefore, the rule entries may change order after updates.
When configured to control the total number of rules in the file, the Tool decides the number of new rules to add and removes existing rules to accomodate.
It is important to note that this removal is performed in an unordered manner, therefore, it does not occur in a FIFO manner.
Limitations
STIX allows for Event Filters, Rate Filters, and Event Suppressions to be expressed within the SnortTestMechanismType. These are currently not utilized in the Tool. If modifying the Tool to use, the Parser does pass them in the Full Rules object that contains the full context as a dictionary.
Logging
[Logging]
logfilebase = "/var/log/lqmt"
debug = true
dailyrotation = true
Setting | Explanation |
---|---|
logfilebase |
The path and filename prefix to the LQMToolset log file. Multiple log files will be created based on the filename prefix specified in this setting. For example, if LogFileBase is "/var/log/lqmt", "/var/log/lqmt.err.log" and "/var/log/lqmt.info.log" will be created. |
debug |
Enable debug-level logging, which will create an additional log file, .debug.log. Debug is optional, and it accepts either true or false . |
dailyrotation |
Enable daily log rotation. When enabled, log file names will be appended with the current date (ex: lqmt_09-01-2017.debug). This makes it easier to parse logs and allows users to create rotation processes. |
Whitelists
LQMToolset allows indicators to be whitelisted. When you define a path to a text file containing the whitelisted indicators, LQMToolset will check the file for modifications and then update the internal database.
[Whitelist]
whitelist = "/path/to/whitelist/file.txt"
dbfile = "/path/to/whitelist/database.db"
Setting | Explanation |
---|---|
whitelist |
The full path to the text file containing whitelisted indicators, adhering to the whitelist format below. |
dbfile |
The location of the SQLite database that holds the whitelist information. |
Whitelist Format
The following sections specify the format of the whitelist file mentioned above.
IPv4 Addresses
[IPv4Address]
192.168.1.1
IPv4 Subnet Ranges
[IPv4Subnet]
192.168.1.0/24
192.168.2.0/24
192.168.3.0/24
IPv6 Addresses
[IPv6Address]
ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff
IPv6 Subnet Ranges
[IPv6Subnet]
ffff:ffff:ffff:ffff/90
Domain Name
[Domain]
example.com
example2.com
Host
[Host]
badguy.example.com
URL
[URL]
http://www.somesite.com/blah
Intermediate Data Format
The intermediate data format is a subset/simplification of the complete threat data that can be directly used by a variety of systems.
Field Name | Description |
---|---|
dataItemID |
UUID for data item |
fieldID |
UUID for file |
detectedTime |
UTC epoch of time in the alert itself (if present, filled with reportedTime) |
reportedTime |
UTC epoch of time reported to CFM (or other system) |
processedTime |
UTC epoch of time processed locally on client (i.e. when this parsed record was created) |
indicator |
The value to be acted up (e.g. ip, domain name, URL) |
indicatorType |
A type name that informs how to interpret the indicator (e.g. ipv4, emailAddress) (enum) |
indicatorDirection |
enum {source, destination} |
secondaryIndicator |
A secondary indicator that restricts (logical AND) the indicator (e.g. a port number) (when appropriate) |
secondaryIndicatorType |
A type name that informs how to interpret the secondaryIndicator (e.g. tcpport, udport) (enum) |
secondaryIndicatorDirection |
enum {source, destination} |
directSource |
The CFM site abbr (or other applicable indentifier) that uploaded the data |
secondarySource |
String representing where the CFM site got it from (when appropriate) |
action1 |
An action to be performed when the indicator is seen (semi-enum) |
duration1 |
How long the action is to be performed |
action2 |
An action to be performed when the indicator is seen (semi-enum) |
duration2 |
How long the action is to be performed |
reason1 |
First reason for this alert |
reference1 |
Reference for info for reason |
reason2 |
Additional reason for this alert |
reference2 |
Reference for info for reason |
majorTags |
A string containing a comma separated list of tags. These are high-level concepts that rend to remain "tag worthy" over time (e.g. Ransomware) |
minorTags |
A string containing a comma separated list of tags. These are highly detailed and/or only important for a short time (e.g. Cryptolocker) |
restriction |
TLP level |
sensitivity |
OUO marking |
reconAllowed |
Boolean: true (default) = recon |
priors |
Number of previous (known) reports |
confidence |
Confidence in accuracy: 0 = none 100 = full |
severity |
How significant is the impact (assuming relevant): 0 = no impact, 100 = catastrophic |
relevancy |
How relevant is this alert: 0 = not applicable, 100 = perfectly targeted |
relatedID |
UUID of a related indicator |
relationType |
enum (supersedes, extends, superseded by, extended by) |
comment |
A string that represents a comment that was included with the alert. |
fileHasMore |
Boolean: false (default) = data has been translated, true = More details on the data item are present in file. |
Actions
The most common action seen for the fields action1
, and action2
is block
, but occasionaly a revoke
action will be put out for alerts that a no longer needed or were sent out by mistake. revoke
actions are taken care of automatically with the supported firewall tools, but you should take special note of them when using a custom outputs like FlexText. Not all data processed should be assumed to be malicious or worthy of a block action, and the action
fields are meant to be better help determine what actions should be taken on data.