Performing a flow basic on a PanOS firewall is a great, detailed troubleshooting technique for when viewing live sessions and traffic logs just isn’t cutting it. The flow basic shows the entire “life of a packet,” so to speak, and can offer insight into why traffic isn’t behaving as expected.
Fair warning: Running packet captures on any PanOS device can increase CPU load and fill up system disk space, flow basic even more so. Take care to only enable the capture for as long as necessary, and ensure concise filters are applied.
Start from scratch
I find it best to leave my previous filter in the packet capture settings as a failsafe. That way, if someone were to inadvertently enable the capture, the firewall won’t start dumping all traffic into a pcap file.
Clear all settings, logs
First, we’ll clear out any existing logs or filter settings with the first two commands. Then, we’ll confirm everything has been cleared with the last.
admin@fw01> debug dataplane packet-diag clear all
Packet diagnosis setting set to default.
admin@fw01> debug dataplane packet-diag clear log log
dataplane debug logs cleared
Confirm settings, logs are cleared out
If all the settings have been cleared out, the output should look like this:
admin@fw01> debug dataplane packet-diag show setting
--------------------------------------------------------------------------------
Packet diagnosis setting:
--------------------------------------------------------------------------------
Packet filter
Enabled: yes
Match pre-parsed packet: no
Filter offload: yes
--------------------------------------------------------------------------------
Logging
Enabled: no
Log-throttle: no
Sync-log-by-ticks: yes
Features:
Counters:
Timeout duration: 60 seconds
Buffer threshold: 80%
CPU threshold: 80%
--------------------------------------------------------------------------------
Packet capture
Enabled: no
Snaplen: 0
Username:
--------------------------------------------------------------------------------
Note: for firewalls with separate mp/dp, the output above will differ, showing the same settings for each dataplane.
Enable and create traffic filters
Now that we have a clean slate, we’ll enable and create our filters. Up to four (4) filters can be defined. Typically, we’ll have at least two: one for ingress, one for egress.
Enable filtering
admin@fw01> debug dataplane packet-diag set filter on
debug packet filter: on
Set a simple source/destination filter
This will enable a simple bidirectional capture for all traffic between hosts 10.10.10.10/32 and 10.20.20.20/32 and exclude any non-IP traffic.
admin@fw01> debug dataplane packet-diag set filter index 1 match source 10.10.10.10 destination 10.20.20.20 non-ip exclude
admin@fw01> debug dataplane packet-diag set filter index 2 match source 10.20.20.20 destination 10.10.10.10 non-ip exclude
Additionally, we can also specify a specific protocol and port. In the below example, we’re looking to capture standard web traffic (ssl/https) on tcp/443.
admin@fw01> debug dataplane packet-diag set filter index 1 match source 10.10.10.10 destination 10.20.20.20 non-ip exclude protocol 6 destination-port 443
admin@fw01> debug dataplane packet-diag set filter index 2 match source 10.20.20.20 destination 10.10.10.10 non-ip exclude protocol 6 source-port 443
Define capture stages
Defining the capture stages is what actually generates the pcap files. There are four stages; receive, transmit, firewall, and drop. The receive stage will capture traffic pre-NAT, whereas transmit will be post-NAT (for more see Getting Started: Packet Capture)
admin@fw01> debug dataplane packet-diag set capture stage receive file rx
admin@fw01> debug dataplane packet-diag set capture stage transmit file tx
admin@fw01> debug dataplane packet-diag set capture stage firewall file fw
admin@fw01> debug dataplane packet-diag set capture stage drop file dr
Enable features (flow basic)
The last configuration step is to enable log features. In this case, we’ll enable the flow basic feature.
admin@fw01> debug dataplane packet-diag set log feature flow basic
Confirm settings and check for existing sessions
At this stage, it’s a good idea to check that all the filters and log settings are configured correctly before we enable any capturing.
admin@fw01> debug dataplane packet-diag show setting
--------------------------------------------------------------------------------
Packet diagnosis setting:
--------------------------------------------------------------------------------
Packet filter
Enabled: yes
Match pre-parsed packet: no
Filter offload: yes
Index 1: 10.10.10.10/32[0]->10.20.20.20/32[443], proto 6
ingress-interface any, egress-interface any, exclude non-IP
Index 2: 10.20.20.20/32[443]->10.10.10.10/32[0], proto 6
ingress-interface any, egress-interface any, exclude non-IP
--------------------------------------------------------------------------------
Logging
Enabled: no
Log-throttle: no
Sync-log-by-ticks: yes
Features:
flow : basic
Counters:
Timeout duration: 60 seconds
Buffer threshold: 80%
CPU threshold: 80%
--------------------------------------------------------------------------------
Packet capture
Enabled: no
Snaplen: 0
Username:
Stage receive : file rx
Captured: packets - 0 bytes - 0
Maximum: packets - 0 bytes - 0
Stage firewall : file fw
Captured: packets - 0 bytes - 0
Maximum: packets - 0 bytes - 0
Stage transmit : file tx
Captured: packets - 0 bytes - 0
Maximum: packets - 0 bytes - 0
Stage drop : file dr
Captured: packets - 0 bytes - 0
Maximum: packets - 0 bytes - 0
--------------------------------------------------------------------------------
Check for existing sessions
We can also check to see if any current traffic flows match our filters.
admin@fw01> debug dataplane packet-diag show filter-marked-session
No Active Marked Sessions
If there are active sessions that we want to exclude from the log once we enable the capture, we can clear them now.
admin@fw01> debug dataplane packet-diag clear filter-marked-session all
Unmark All sessions in packet debug
Tip: Enabling just the filters above without doing any capturing can be a useful troubleshooting tool. You can quickly display sessions that match the filter, or utilize the capture filter to view the global counters.
Turn on the capture, enable logging
At this point we’re ready to start the capture. Recall that depending on how much traffic matches this filter (and it’s important to have a filter enabled for this reason), this can have a performance impact on the firewall.
admin@fw01> debug dataplane packet-diag set capture on
Packet capture is enabled
admin@fw01> debug dataplane packet-diag set log on
Packet log is enabled. WARNING: Enabling of debug commands could result in network outage. Not recommended if dataplane CPU is above 60%.
Monitor the capture
While the capture is on and running, we can check for marked-sessions (sessions matching our filter that are part of the running log file), view packet and byte numbers increase in each of the stages configured (receive, transmit, firewall, and drop), and check for changes in the global counters.
Filter-marked sessions
To view a list of which sessions the filter has currently “marked” for logging, run the command below. You can then run a standard show session id $SESSION_ID to view individual session information.
admin@fw01> debug dataplane packet-diag show filter-marked-session
No Active Marked Sessions
Packet and byte count counters
The standard show setting command will show an incrementing packet and byte counters in each of the four (4) capture stages.
admin@fw01> debug dataplane packet-diag show setting
--------------------------------------------------------------------------------
Packet diagnosis setting:
--------------------------------------------------------------------------------
Packet filter
Enabled: yes
Match pre-parsed packet: no
Filter offload: yes
Index 1: 10.10.10.10/32[0]->10.20.20.20/32[443], proto 6
ingress-interface any, egress-interface any, exclude non-IP
Index 2: 10.20.20.20/32[443]->10.10.10.10/32[0], proto 6
ingress-interface any, egress-interface any, exclude non-IP
--------------------------------------------------------------------------------
Logging
Enabled: yes
Log-throttle: no
Sync-log-by-ticks: yes
Features:
flow : basic
Counters:
Timeout duration: 60 seconds
Buffer threshold: 80%
CPU threshold: 80%
--------------------------------------------------------------------------------
Packet capture
Enabled: yes
Snaplen: 0
Username:
Stage receive : file rx
Captured: packets - 0 bytes - 0
Maximum: packets - 0 bytes - 0
Stage firewall : file fw
Captured: packets - 0 bytes - 0
Maximum: packets - 0 bytes - 0
Stage transmit : file tx
Captured: packets - 0 bytes - 0
Maximum: packets - 0 bytes - 0
Stage drop : file dr
Captured: packets - 0 bytes - 0
Maximum: packets - 0 bytes - 0
--------------------------------------------------------------------------------
Check for changes in global counters
The global counters command can be appended to filter on packets matching the packet capture filter we defined earlier.
admin@fw01> show counter global filter packet-filter yes delta yes
Global counters:
Elapsed time since last sampling: 45.927 seconds
--------------------------------------------------------------------------------
Total counters shown: 0
--------------------------------------------------------------------------------
Disable the capture and logs
Once we have enough traffic captured and logged, we can disable the capture and logging. This will ensure our pcap and log files don’t balloon out of control. I like to turn of logging first, to avoid the extra noise in the already terse log file.
admin@fw01> debug dataplane packet-diag set log off
Packet log is disabled
admin@fw01> debug dataplane packet-diag set capture off
Packet capture is disabled
Collect and view logs
Before attempting to view the flow basic, we need to “aggregate” the logs using the command below. I’ve found, and PAN engineers have recommended to me, that it’s best to run this command twice.
admin@fw01> debug dataplane packet-diag aggregate-logs
pan_packet_diag.log is aggregated
admin@fw01> debug dataplane packet-diag aggregate-logs
pan_packet_diag.log is aggregated
Once complete, we can view the logs using less or tail. Depending on the firewall model, the logs may be in a specific data plane (dp0, 1, 2…) or the management plane.
admin@fw01> less mp-log pan_packet_diag.log
…or…
admin@fw01> less dp-log pan_packet_diag.log
Logs can also be exported as you normally would for any other log file, or included in a TSF.
View pcaps
While you can download and/or scp packet captures, they can also be viewed directly on the firewall.
admin@fw01> view-pcap no-dns-lookup yes filter-pcap rx
admin@fw01> view-pcap no-dns-lookup yes filter-pcap tx
admin@fw01> view-pcap no-dns-lookup yes filter-pcap fw
admin@fw01> view-pcap no-dns-lookup yes filter-pcap dr
At some point, I’ll come back and update this with some examples of an actual problematic traffic flow, and walk through how I analyze the flow basic. For now, it’s a simple set of commands for me to reference in one place.
Cheers.