Collect IBM Mainframe Storage logs

Supported in:

Google secops SIEM

This document explains how to collect IBM Mainframe Storage logs to Google Security Operations using Bindplane.

IBM Mainframe Storage encompasses the IBM DS8000 series (including DS8900F and DS8880) and IBM FlashSystem families (including FlashSystem 5000, 5200, 5300, 7200, 7300, 9200, and 9500). These enterprise storage systems provide high-performance block storage for mainframe and open systems environments, with built-in syslog forwarding capabilities for audit logs, system events, authentication events, and operational messages.

Before you begin

Make sure you have the following prerequisites:

A Google SecOps instance
Windows Server 2016 or later, or Linux host with systemd
Network connectivity between the Bindplane agent and the IBM storage system management IP address
If running behind a proxy, ensure firewall ports are open per the Bindplane agent requirements
Administrator-level access to the IBM DS8000 Storage Management GUI or IBM FlashSystem management GUI
For IBM FlashSystem, SSH access to the CLI (optional, for CLI-based configuration)

Get Google SecOps ingestion authentication file

Sign in to the Google SecOps console.
Go to SIEM Settings > Collection Agents.
Download the Ingestion Authentication File. Save the file securely on the system where Bindplane will be installed.

Get Google SecOps customer ID

Sign in to the Google SecOps console.
Go to SIEM Settings > Profile.
Copy and save the Customer ID from the Organization Details section.

Install the Bindplane agent

Install the Bindplane agent on your Windows or Linux operating system according to the following instructions.

Windows installation

Open Command Prompt or PowerShell as an administrator.

Run the following command:

msiexec /i "https://github.com/observIQ/bindplane-agent/releases/latest/download/observiq-otel-collector.msi" /quiet

Wait for the installation to complete.
Verify the installation by running:
```
sc query observiq-otel-collector
```

The service should show as RUNNING.

Linux installation

Open a terminal with root or sudo privileges.

Run the following command:

sudo sh -c "$(curl -fsSlL https://github.com/observiq/bindplane-agent/releases/latest/download/install_unix.sh)" install_unix.sh

Wait for the installation to complete.
Verify the installation by running:
```
sudo systemctl status observiq-otel-collector
```
The service should show as active (running).

Additional installation resources

For additional installation options and troubleshooting, see Bindplane agent installation guide.

Configure Bindplane agent to ingest syslog and send to Google SecOps

Locate the configuration file

Linux:

sudo nano /etc/bindplane-agent/config.yaml

Windows:

notepad "C:\Program Files\observIQ OpenTelemetry Collector\config.yaml"

Edit the configuration file

Replace the entire contents of config.yaml with the following configuration:

receivers:
    udplog:
        listen_address: "0.0.0.0:514"

exporters:
    chronicle/ibm_storage:
        compression: gzip
        creds_file_path: '<CREDS_FILE_PATH>'
        customer_id: '<CUSTOMER_ID>'
        endpoint: <REGION_ENDPOINT>
        log_type: IBM_MAINFRAME_STORAGE
        raw_log_field: body
        ingestion_labels:
            log_source: ibm_storage

service:
    pipelines:
        logs/ibm_storage_to_chronicle:
            receivers:
                - udplog
            exporters:
                - chronicle/ibm_storage

Configuration parameters

Replace the following placeholders.

Exporter configuration:

<CREDS_FILE_PATH>: Full path to ingestion authentication file: - Linux: /etc/bindplane-agent/ingestion-auth.json - Windows: C:\Program Files\observIQ OpenTelemetry Collector\ingestion-auth.json
<CUSTOMER_ID>: Customer ID copied from the Google SecOps console
<REGION_ENDPOINT>: Regional endpoint URL: - US: malachiteingestion-pa.googleapis.com - Europe: europe-malachiteingestion-pa.googleapis.com - Asia: asia-southeast1-malachiteingestion-pa.googleapis.com - See Regional Endpoints for complete list

Note: If your IBM storage system is configured to send syslog over TCP (the default for DS8000 systems), change the receiver from udplog to tcplog and update the port accordingly. The default syslog port for IBM DS8000 over TCP is 514, and for IBM FlashSystem over UDP is 514 (or 6514 for TCP).

Example configuration (IBM FlashSystem via UDP syslog)

receivers:
    udplog:
        listen_address: "0.0.0.0:514"

exporters:
    chronicle/ibm_storage:
        compression: gzip
        creds_file_path: '/etc/bindplane-agent/ingestion-auth.json'
        customer_id: 'a1b2c3d4-e5f6-g7h8-i9j0-k1l2m3n4o5p6'
        endpoint: malachiteingestion-pa.googleapis.com
        log_type: IBM_MAINFRAME_STORAGE
        raw_log_field: body
        ingestion_labels:
            log_source: ibm_storage

service:
    pipelines:
        logs/ibm_storage_to_chronicle:
            receivers:
                - udplog
            exporters:
                - chronicle/ibm_storage

Example configuration (IBM DS8000 via TCP syslog)

receivers:
    tcplog:
        listen_address: "0.0.0.0:514"

exporters:
    chronicle/ibm_storage:
        compression: gzip
        creds_file_path: '/etc/bindplane-agent/ingestion-auth.json'
        customer_id: 'a1b2c3d4-e5f6-g7h8-i9j0-k1l2m3n4o5p6'
        endpoint: malachiteingestion-pa.googleapis.com
        log_type: IBM_MAINFRAME_STORAGE
        raw_log_field: body
        ingestion_labels:
            log_source: ibm_ds8000

service:
    pipelines:
        logs/ibm_storage_to_chronicle:
            receivers:
                - tcplog
            exporters:
                - chronicle/ibm_storage

Save the configuration file

After editing, save the file:

Linux: Press Ctrl+O, then Enter, then Ctrl+X
Windows: Click File > Save

Restart the Bindplane agent to apply the changes

To restart the Bindplane agent in Linux:

Run the following command:

sudo systemctl restart observiq-otel-collector

Verify the service is running:

sudo systemctl status observiq-otel-collector

Check logs for errors:

sudo journalctl -u observiq-otel-collector -f

To restart the Bindplane agent in Windows:

Choose one of the following options:
- Command Prompt or PowerShell as administrator:
  
  net stop observiq-otel-collector && net start observiq-otel-collector
- Services console:
  1. Press Win+R, type services.msc, and press Enter.
  2. Locate observIQ OpenTelemetry Collector.
  3. Right-click and select Restart.
Verify the service is running:

sc query observiq-otel-collector
Check logs for errors:

type "C:\Program Files\observIQ OpenTelemetry Collector\log\collector.log"

Configure IBM Mainframe Storage syslog forwarding

Configure syslog forwarding on your specific IBM storage system. Follow the instructions for your product family.

Option A: Configure syslog on IBM DS8000 series (DS8900F, DS8880)

The IBM DS8000 Storage Management GUI supports forwarding audit logs, GUI event information, login/logout events, and command audit trails to a remote syslog server. The DS8000 supports up to eight syslog target servers. By default, DS8000 syslog uses the TCP protocol on port 514.

Sign in to the DS8000 Storage Management GUI using an account with the Administrator role.
Go to Settings > Notifications.
On the Syslog page, click Add Syslog Server.
In the Add Syslog Server window, provide the following configuration details:
- IP Address: Enter the IP address of the Bindplane agent host (for example, 192.168.1.100).
- Port: Enter 514.
Click OK.

Note: The DS8000 uses TCP as the default syslog protocol. Ensure the Bindplane agent is configured with the tcplog receiver to match. If you require TLS encryption, you must enable TLS before adding any syslog servers. To enable TLS, click Enable TLS in the Certificate area of the Syslog page and provide the CA Certificate, HMC Certificate, and HMC Private Key files.

Option B: Configure syslog on IBM FlashSystem (5000, 5200, 5300, 7200, 7300, 9200, 9500)

IBM FlashSystem systems based on IBM Storage Virtualize support syslog forwarding through both the management GUI and the CLI. The system supports up to six syslog servers and can transmit messages using UDP, TCP, or TLS protocols.

Using the management GUI

Sign in to the IBM FlashSystem management GUI.
Go to Settings > Notifications > Syslog.
Click Add Syslog Server to open the Add Syslog Server window.
Select the protocol:
- UDP: Reduces system and network overhead. Default port is 514.
- TCP: Provides reliable delivery. Default port is 6514.
- TLS: Uses TLS to send encrypted syslog messages.
Enter the IP address of the Bindplane agent host and the port number.
Click Add.

Using the CLI (SSH)

Connect to the IBM FlashSystem system via SSH using an administrator account.
Run the following command to create a syslog server with all log types enabled:
```
mksyslogserver -name chronicle -ip <BINDPLANE_IP> -protocol udp -port 514 -facility 4 -error on -warning on -info on -audit on -login on
```
Replace <BINDPLANE_IP> with the IP address of the Bindplane agent host.

Note: Setting -facility to a value between 4 and 7 sends syslog messages in fully expanded format, which provides more detailed log data. Setting -facility to a value between 0 and 3 sends messages in concise format.
Verify the syslog server was created by running:
```
lssyslogserver
```
The output displays the configured syslog servers with their ID, name, IP address, port, protocol, and notification settings.

CLI examples for common configurations

Create a syslog server with UDP on port 514 in expanded format with audit and authentication logging:

mksyslogserver -name chronicle -ip 192.168.1.100 -facility 4 -protocol udp -port 514 -error on -warning on -info on -audit on -login on

Create a syslog server with TCP on port 6514:

mksyslogserver -name chronicle -ip 192.168.1.100 -facility 4 -protocol tcp -port 6514 -error on -warning on -info on -audit on -login on

Modify an existing syslog server to enable audit logging:
```
chsyslogserver -audit on <syslog_server_id>
```
Remove a syslog server:
```
rmsyslogserver <syslog_server_id>
```

FlashSystem syslog notification types

The following table describes the available syslog notification types on IBM FlashSystem:

Notification type	CLI parameter	Default	Description
Error	`-error on\|off`	on	Serious problems with the system
Warning	`-warning on\|off`	on	Problems or unexpected conditions
Information	`-info on\|off`	on	Expected operations completed
Audit Log	`-audit on\|off`	off	CLI or management GUI operations
Authentication Log	`-login on\|off`	off	Successful and failed authentication attempts

UDM mapping table

Log field	UDM mapping	Logic
device_vendor	about.asset.asset_id	Concatenated from device_vendor, device_product, and deviceExternalId as "%{device_vendor}.%{device_product}:%{deviceExternalId}"
device_product	about.asset.asset_id
deviceExternalId	about.asset.asset_id
IPv6_Address	about.ip	Value copied directly
dvcmac	about.mac	Value from dvcmac if valid MAC, else from mac_address if extracted
mac_address	about.mac
fname	about.file.full_path	Value copied directly if not "N/A"
fileHash	about.file.full_path	Value copied directly if not a hash
fileHash	about.file.sha256	Value copied directly if matches hash pattern
_hash	about.file.sha256	Value copied directly
fsize	about.file.size	Converted to uinteger
File_name	about.file.full_path	Value from File_name if not empty, else Object, else Objekt, else Infected_Resource
Object	about.file.full_path
Objekt	about.file.full_path
Infected_Resource	about.file.full_path
about	about	Merged from about
additional_cs1	additional.fields	Merged from various additional_* fields like additional_cs1, additional_cs2, etc.
additional_cs2	additional.fields
cs1	additional.fields	Key from cs1Label, value from cs1
cs1Label	additional.fields
cs2	additional.fields	Key from cs2Label, value from cs2
cs2Label	additional.fields
cs3	additional.fields	Key from cs3Label, value from cs3 if not empty
cs3Label	additional.fields
cs4	additional.fields	Key from cs4Label, value from cs4 after gsub
cs4Label	additional.fields
cs5	additional.fields	Key from cs5Label, value from cs5 if not "NA"
cs5Label	additional.fields
cs6	additional.fields	Key from cs6Label, value from cs6 if not empty
cs6Label	additional.fields
cs7	additional.fields	Key from cs7Label, value from cs7 if not empty
cs7Label	additional.fields
flexString1	additional.fields	Key from flexString1Label, value from flexString1
flexString1Label	additional.fields
cn1	additional.fields	Key from cn1Label, value from cn1
cn1Label	additional.fields
cn2	additional.fields	Key from cn2Label, value from cn2
cn2Label	additional.fields
cn3	additional.fields	Key from cn3Label, value from cn3 if not empty
cn3Label	additional.fields
cfp1	additional.fields	Key from cfp1Label, value from cfp1
cfp1Label	additional.fields
cfp2	additional.fields	Key from cfp2Label, value from cfp2
cfp2Label	additional.fields
cfp3	additional.fields	Key from cfp3Label, value from cfp3
cfp3Label	additional.fields
cfp4	additional.fields	Key from cfp4Label, value from cfp4
cfp4Label	additional.fields
eventId	additional.fields	Key "eventId", value from eventId
devicePayloadId	additional.fields	Key "devicePayloadId", value from devicePayloadId
fname	additional.fields	Key "fname", value from fname if not "N/A"
cs3Label	additional.fields	Key "cs3Label", value from cs3Label if cs3 empty
cs4Label	additional.fields	Key "cs4Label", value from cs4Label if cs4 empty
cs5	additional.fields	Key "cs5 Label", value from cs5 if cs5Label empty
principal_asset_hostname	additional.fields	Key "asset_host_name", value from principal_asset_hostname if differs
target_asset_hostname	additional.fields	Key "device_host_name", value from target_asset_hostname if differs
ssh_key	additional.fields	Key "ssh_key", value from ssh_key if not hash
Comment	additional.fields	Key "Comment", value from try_message
Action Name	additional.fields	Key "Action Name", value from action_name
Error Code	additional.fields	Key "Error Code", value from error_code
rsyslog Module	additional.fields	Key "rsyslog Module", value from module or module_name
More Info	additional.fields	Key "More Info", value from try_message
Action Type	additional.fields	Key "Action Type", value from action_type
Received	metadata.collected_timestamp	Parsed from Received or Mottatt using date match
Mottatt	metadata.collected_timestamp
rt	metadata.event_timestamp	Parsed from rt using date match with timezone
file_full_path	metadata.event_type	Set to "PROCESS_UNCATEGORIZED" if file_full_path not empty; "SCAN_UNCATEGORIZED" if event_name in specific values; "USER_UNCATEGORIZED" if duser not empty; "NETWORK_CONNECTION" if principal_hostname_found and target_ip_found; "STATUS_UPDATE" if principal_hostname_found or src not empty; else "GENERIC_EVENT"
event_name	metadata.event_type
duser	metadata.event_type
principal_hostname_found	metadata.event_type
device_product	metadata.product_name	Value copied directly
device_vendor	metadata.vendor_name	Value copied directly
device_version	metadata.product_version	Value copied directly
externalId	metadata.product_log_id	Value copied directly
event_name	metadata.product_event_type	Concatenated as "[%{device_event_class_id}] - %{event_name}" if both present, else device_event_class_id or event_name
device_event_class_id	metadata.product_event_type
app_protocol_src	network.application_protocol	Set based on app_protocol_src matching predefined values like "22" to "SSH", "25" to "SMTP", etc.
deviceDirection	network.direction	Set to "INBOUND" if deviceDirection == "0", "OUTBOUND" if "1"
in	network.received_bytes	Converted to uinteger if >0
out	network.sent_bytes	Converted to uinteger if >0
proto	network.ip_protocol	Set based on protocol_number_src matching numbers like 1 to "ICMP", 6 to "TCP", etc.
destinationTranslatedPort	network.target.nat_port	Converted to integer if not 0
sourceTranslatedPort	network.principal.nat_port	Converted to integer if not 0
dpt	network.target.port	Converted to integer if not 0 and valid
spt	network.principal.port	Converted to integer if not 0 and not "{srcPort}"
request	network.target.url	Value copied directly
requestMethod	network.http.method	Uppercased
requestClientApplication	network.http.user_agent	Value copied directly
principal_hostname	principal.asset.hostname	Value from principal_hostname if not empty and differs from principal_asset_hostname, else set to principal_hostname
principal_asset_hostname	principal.asset.hostname
Device_name	principal.hostname	Value from Device_name if not empty, else Enhetsnavn
Enhetsnavn	principal.hostname
shost	principal.hostname	Value copied directly if not IP
shost	principal.ip	Extracted as IP if matches IP pattern
sourceTranslatedAddress	principal.nat_ip	Value copied directly if starts with digit
smac	principal.mac	Extracted as MAC after gsub
spid	principal.process.pid	Value copied directly
sproc	principal.process.command_line	Value copied directly
suser	principal.user.user_display_name	Value copied directly if not starts with "{"
suid	principal.user.userid	Value copied directly
spriv	principal.user.attribute.roles.name	Value copied directly
sntdom	principal.administrative_domain	Value copied directly
sourceServiceName	principal.application	Value copied directly
Group_name	principal.group.group_display_name	Value from Group_name if not empty, else Gruppenavn
Gruppenavn	principal.group.group_display_name
action	principal.user.user_authentication_status	Set to "ACTIVE" if action in accepted values, "NO_ACTIVE_CREDENTIALS" if disconnect, "SUSPENDED" if disconnected
act	security_result.action	Set to "ALLOW" if act in allow conditions or outcome matches, "BLOCK" if deny or blocked, "FAIL" if outcome failure
outcome	security_result.action
act	security_result.action_details	Value copied directly
Action_Taken	security_result.action_details	Value copied directly
categoryOutcome	security_result.category_details	Value copied directly
severity	security_result.severity	Set to "LOW" if in [0,1,2,3,"LOW"], "MEDIUM" if [4,5,6,"MEDIUM","SUBSTANTIAL","INFO"], "HIGH" if [7,8,"HIGH","SEVERE"], "CRITICAL" if [9,10,"VERY-HIGH","CRITICAL"]
Spyware	security_result.threat_name	Value from Spyware if not empty, else Virus_Malware_Name, else Unknown_Threat
Virus_Malware_Name	security_result.threat_name
Unknown_Threat	security_result.threat_name
Type	security_result.description	Value from Type if not empty, else Scan_Type
Scan_Type	security_result.description
msg_data_2	security_result.description	Value from msg_data_2 if not empty, else msg
msg	security_result.description
reason	security_result.summary	Value copied directly if not empty or space
appcategory	security_result.summary	Value copied directly
Result	security_result.summary	Value copied directly
mwProfile	security_result.rule_name	Value copied directly
Operation	security_result.detection_fields	Key "Operation", value from Operation or Operasjon
Operasjon	security_result.detection_fields
Permission	security_result.detection_fields	Key "Permission", value from Permission or Tillatelse
Tillatelse	security_result.detection_fields
Infection_Channel	security_result.detection_fields	Key "Infection Channel", value from Infection_Channel
Spyware_Grayware_Type	security_result.detection_fields	Key "Spyware/Grayware_Type", value from Spyware_Grayware_Type
Threat_Probability	security_result.detection_fields	Key "Threat_Probability", value from Threat_Probability
Resource_Type	security_result.detection_fields	Key "Resource_Type", value from Resource_Type
security_result	security_result	Merged from security_result
target_hostname	target.asset.hostname	Value from target_hostname if not empty and differs from target_asset_hostname, else set to target_hostname
target_asset_hostname	target.asset.hostname
temp_dhost	target.hostname	Value copied directly if not IP
temp_dhost_ip	target.hostname	Set to empty if IP extracted
temp_dhost_ip	target.ip	Value copied directly if extracted
temp_dhost_ip	target.ip	Value from temp_dhost_ip or IPv6_Address
IPv6_Address	target.ip
temp_dhost_ip	target.asset.ip	Value copied directly if extracted
destinationTranslatedAddress	target.nat_ip	Extracted as IP
dmac	target.mac	Extracted as MAC after gsub
dpid	target.process.pid	Value copied directly
dproc	target.process.command_line	Value copied directly
temp_duser	target.user.user_display_name	Value copied directly
temp_duid	target.user.userid	Extracted using grok, value copied
User	target.user.userid	Value from User if not empty, else Bruker
Bruker	target.user.userid
CustomerName	target.user.user_display_name	Value copied directly after gsub
dpriv	target.user.attribute.roles.name	Value copied directly
dntdom	target.administrative_domain	Value copied directly
destinationServiceName	target.application	Value copied directly
File_name	target.process.file.full_path	Value from File_name if not empty, else Object, else Objekt, else Infected_Resource
Object	target.process.file.full_path
Objekt	target.process.file.full_path
Infected_Resource	target.process.file.full_path
oldFileSize	target.src.file.size	Converted to uinteger if not 0
oldFilePath	target.src.file.full_path	Value copied directly
oldFilePermission	target.src.resource.attribute.permissions.name	Value copied directly
filePermission	target.resource.attribute.permissions.name	Value copied directly
resource_Type_label	target.resource.attribute.labels	Merged with key "Resource_Type", value from Resource_Type
metadata.vendor_name	metadata.vendor_name	Set to "IBM"
metadata.product_name	metadata.product_name	Set to "MAINFRAME STORAGE"

Need more help? Get answers from Community members and Google SecOps professionals.