Execute a CLI script based on CPU and memory thresholds in Fortigate

abril 16, 2024

Thank to: https://docs.fortinet.com

Automation stitches can be created to run a CLI script and send an email message when CPU or memory usage exceeds specified thresholds.

In this example, two automation stitches are created that run a CLI script to collect debug information, and then email the results of the script to a specified email address when CPU usage threshold is exceeded or memory usage causes the FortiGate to enter conserve mode.

Automation stitches that use High CPU and Conserve Mode triggers can only be created in the CLI. Once create, they can be edited in the GUI.

To define CPU and memory usage thresholds:

config system global
    set cpu-use-threshold <percent>
    set memory-use-threshold-extreme <percent>
    set memory-use-threshold-green <percent>
    set memory-use-threshold-red <percent>
end

Where:

cpu-use-threshold	Threshold at which CPU usage is reported, in percent of total possible CPU utilization (default = 90).
memory-use-threshold-extreme	Threshold at which memory usage is considered extreme, and new sessions are dropped, in percent of total RAM (default = 95).
memory-use-threshold-green	Threshold at which memory usage forces the FortiGate to exit conserve mode, in percent of total RAM (default = 82).
memory-use-threshold-red	Threshold at which memory usage forces the FortiGate to enter conserve mode, in percent of total RAM (default = 88).

Configure the automation stitches

High CPU usage stitch

To create an automation stitch for high CPU usage:

Create an automation action to run a CLI script:

config system automation-action
    edit "high_cpu_debug"
        set action-type cli-script
        set required enable
        set script "diagnose debug cli 8
diagnose debug console timestamp enable
diagnose debug enable
diagnose debug crashlog read
get system performance status
get system session status
diagnose sys session full-stat
diagnose firewall iprope state
diagnose sys flash list
diagnose hardware sysinfo memory
diagnose hardware sysinfo slab
diagnose hardware sysinfo shm
diagnose hardware deviceinfo disk
get system arp
diagnose ip arp list
diagnose ip address list
get router info routing-table all
get router info kernel
diagnose ip rtcache list
diagnose sys top-summary
diagnose sys top 9 99"
    next
end

Create an automation action to send an email:

config system automation-action
    edit "auto_high_cpu_email"
        set action-type email
        set email-to "person@fortinet.com"
        set email-subject "CSF stitch alert: high_cpu"
        set message "%%results%%"
    next
end

Create an automation trigger:

config system automation-trigger
    edit "auto_high_cpu"
        set event-type high-cpu
    next
end

Create an automation stitch:

config system automation-stitch
    edit "auto_high_cpu"
        set trigger "auto_high_cpu"
        set action "high_cpu_debug" "auto_high_cpu_email"
    next
end

To edit the automation stitch in the GUI:

Go to Security Fabric > Automation.
Double click the auto_high_cpu stitch.
Edit the stitch as required, then click OK.

High memory usage stitch

To create an automation stitch for high memory usage:

Create an automation action to run a CLI script:

config system automation-action
    edit "high_memory_debug"
        set action-type cli-script
        set required enable
        set script "diagnose debug cli 8
diagnose debug console timestamp enable
diagnose debug enable
diagnose debug crashlog read
get system performance status
get system session status
diagnose sys session full-stat
diagnose firewall iprope state
diagnose sys flash list
diagnose hardware sysinfo memory
diagnose hardware sysinfo slab
diagnose hardware sysinfo shm
diagnose hardware deviceinfo disk
get system arp
diagnose ip arp list
diagnose ip address list
get router info routing-table all
get router info kernel
diagnose ip rtcache list
diagnose sys top-summary
diagnose sys top 9 99"
    next
end

Create an automation action to send an email:

config system automation-action
    edit "auto_high_memory_email"
        set action-type email
        set email-to "person@fortinet.com"
        set email-subject "CSF stitch alert: high_memory"
        set message "%%results%%"
    next
end

Create an automation trigger:

config system automation-trigger
    edit "auto_high_memory"
        set event-type low-memory
    next
end

Create an automation stitch:

config system automation-stitch
    edit "auto_high_memory"
        set trigger "auto_high_memory"
        set action "high_memory_debug" "auto_high_memory_email"
    next
end

To edit the automation stitch in the GUI:

Go to Security Fabric > Automation.
Double click the auto_high_memory stitch.
Edit the stitch as required, then click OK.

Results

When FortiGate enters conserve mode due to the memory-use-threshold-red being exceeded, the GUI displays a notice, and the auto_high_memory automation stitch is triggered, causing the CLI script to run and the results of the script to be emailed to the specified address.

Here is an example of the email message:

CSF stitch alert: high_memory
noreply@notification.fortinet.net
Thu 11/21/2019 11:06 AM
James Li
FGT[FGVM16TM19000026] Automation Stitch:auto_high_memory is triggered.
########## script name: autod.47 ##########
========== #1, 2019-11-21 11:07:24 ==========
FGVM16TM19000026 $  diag deb cli 8
Debug messages will be on for 25 minutes.
FGVM16TM19000026 $  diag deb console timestamp enable
FGVM16TM19000026 $  diag deb enable
FGVM16TM19000026 $  diag deb crashlog read
1: 2019-08-08 11:35:25 the killed daemon is /bin/dhcpcd: status=0x0
2: 2019-08-08 17:52:47 the killed daemon is /bin/pyfcgid: status=0x0
3: 2019-08-23 11:32:31 from=license status=INVALID
4: 2019-08-23 11:32:32 from=license status=INVALID
5: 2019-11-21 09:53:31 from=license status=VALID

Buscar este blog

El conocimiento al alcance de todos.