How Port Scanning Works: SYN, FIN, UDP, and OS Detection
Port scanning is the process of systematically probing a host's TCP or UDP ports to determine which services are listening. Network administrators use it to audit their own infrastructure; attackers use it to map targets before exploitation. Understanding exactly what scanning reveals — and how it works at the packet level — is essential for both hardening systems and tuning intrusion detection.
Authorization matters: scanning systems you do not own or have explicit written permission to test is illegal in most jurisdictions under computer fraud statutes. The techniques below are described for defensive understanding and authorized security assessments.
What Scanning Reveals
A port scan returns, for each probed port, one of three states:
| State | Meaning | Typical cause |
|---|---|---|
| open | A service is actively listening and accepted the probe | Application bound to port |
| closed | Host is reachable but nothing is listening | OS returned RST or ICMP port unreachable |
| filtered | Probe was dropped with no response | Firewall silently discarding packets |
From these states, a scanner can infer what software is running, what version it may be, and — via OS fingerprinting — the operating system and its kernel version. Combined with banner grabbing, a scan often produces an actionable target profile within seconds.
TCP Connect Scan
The simplest scan: the scanner completes a full TCP three-way handshake. It calls connect() on the target port. If the handshake succeeds (SYN → SYN/ACK → ACK), the port is open. If the target replies with RST/ACK, it is closed. If there is no reply within the timeout, it is filtered.
This requires no special privileges — any process can call connect(). The tradeoff is that the full connection is visible in the server's logs (the connection reaches the application layer) and is easily detected.
SYN (Half-Open) Scan
The most common technique, requiring raw socket privileges (root or CAP_NET_RAW). Instead of completing the handshake, the scanner:
- Sends a SYN packet
- Receives SYN/ACK → port is open; immediately sends RST to tear down without completing the connection
- Receives RST/ACK → port is closed
- No reply (or ICMP unreachable) → port is filtered
Because the handshake is never completed, many applications never log the probe — it never reaches the accept queue. The RST is sent by the scanner's kernel in response to the unsolicited SYN/ACK (or explicitly by the scan tool). This is faster and less visible than a connect scan, though the SYN packets themselves are still visible to a stateful firewall or IDS.
Stealth Scans: FIN, NULL, and Xmas
RFC 793, the original TCP specification, defines behavior for receiving unexpected packets. On a closed port, a compliant TCP stack must respond to any segment that does not carry SYN with an RST. On an open port, the stack should silently discard packets that are not part of an established connection and carry no SYN.
This asymmetry enables three scan variants:
- FIN scan: sends a packet with only the FIN flag set. Closed port → RST. Open port → silence.
- NULL scan: sends a packet with no flags set. Same logic: closed → RST, open → silence.
- Xmas scan: sends a packet with FIN, PSH, and URG flags set simultaneously (the name comes from the "lit-up" flag field). Same behavior.
These scans are invisible to simple packet filters that only track SYN packets — a stateless ACL blocking inbound SYN will pass FIN or NULL packets through untouched. However, they are unreliable against Windows and many other non-Unix stacks, which do not follow RFC 793 strictly and may send RST for open ports as well, making open and closed indistinguishable.
UDP Scanning
UDP has no handshake, which makes scanning fundamentally different. The scanner sends a UDP datagram to each port. Possible outcomes:
- A response UDP packet arrives → port is open
- An ICMP "port unreachable" message arrives → port is closed
- No response → port is open|filtered (ambiguous; either a service silently accepted it, or a firewall dropped it)
The ambiguity of no-response makes UDP scanning slow and noisy. Most OS kernels rate-limit outbound ICMP unreachable messages (Linux defaults to 1 per second), so a UDP scan of all 65,535 ports may take 18+ hours without tuning. Nmap sends protocol-specific payloads (a DNS query to port 53, an NTP request to port 123, etc.) to elicit application-layer responses from known services, narrowing the open|filtered ambiguity.
Banner Grabbing and Version Detection
Once open ports are identified, version detection connects and reads the service banner — the first few bytes sent by the server. Many services announce themselves immediately:
- SSH:
SSH-2.0-OpenSSH_9.3p2 Ubuntu-1ubuntu3.6 - SMTP:
220 mail.example.com ESMTP Postfix - FTP:
220 ProFTPD 1.3.8 Server
Nmap's -sV flag sends a library of probes and matches responses against a database of service fingerprints, often identifying not just the software but its version, which can be cross-referenced against public vulnerability databases.
OS Fingerprinting
TCP/IP implementations differ subtly across operating systems. Nmap's -O flag analyzes a combination of fields in probe responses:
- TTL: Linux typically sends TTL=64, Windows TTL=128, Cisco routers TTL=255 (these are starting values; each hop decrements by 1)
- TCP window size: initial window size varies by OS and kernel version
- TCP options order: which options are present (MSS, SACK, timestamps, window scaling) and in what order
- Don't Fragment bit, IP ID sequence: how the OS generates IP identifiers
- Response to malformed packets: behavior varies in ways that strongly identify the stack
Against a known target with a SYN scan plus OS detection, Nmap can often identify the OS family and approximate kernel version with high confidence.
Common Nmap Invocations
| Command | What it does |
|---|---|
nmap -sS 192.0.2.1 | SYN scan of 1,000 most common ports |
nmap -sS -p- 192.0.2.1 | SYN scan all 65,535 ports |
nmap -sV -O 192.0.2.1 | Version detection + OS fingerprint |
nmap -sU -p 53,123,161 192.0.2.1 | UDP scan of specific ports |
nmap -sS --min-rate 5000 192.0.2.0/24 | Fast subnet sweep |
nmap -sn 192.0.2.0/24 | Ping sweep only (no port scan) |
nmap -sS -T4 -A 192.0.2.1 | Aggressive: version, OS, scripts, traceroute |
The -T timing flag (0=paranoid to 5=insane) controls inter-probe delays. Slower timing is harder to detect; faster timing risks missed responses from rate-limited hosts.
Internet-Wide Scanning
Tools like Zmap and Masscan are purpose-built for scanning the entire IPv4 address space. They bypass the OS network stack entirely, generate packets directly at line rate, and can scan all 3.7 billion routable IPv4 addresses on a single port in under 5 minutes from a 10 Gbps link. These tools power services like Shodan and Censys, which continuously scan the internet and index service banners, TLS certificate metadata, and protocol-specific responses, making it trivial to find all exposed instances of a specific software version globally.
For defenders, this means any externally reachable port is effectively scanned continuously. The question is not whether attackers will find your exposed services, but when.
What Defenders See
Modern IDS/IPS systems recognize scan patterns by tracking:
- A single source IP touching many ports on one host (vertical scan)
- A single source IP touching the same port across many hosts (horizontal scan)
- High rates of RST packets in response to the scanner's SYNs
- Packets with unusual flag combinations (NULL, Xmas)
- SYN packets without corresponding established connections
Snort/Suricata ship with rules for all of these patterns. Many cloud providers and CDNs also rate-limit or block known scanner IP ranges.
Hardening: Reducing What Scans Reveal
The primary defense is minimizing attack surface: every open port is a potential vulnerability. Close or firewall any service not needed externally. For the ports that must remain open:
- Suppress banners: configure SSH, SMTP, HTTP, and FTP servers to omit version strings from banners. An attacker who can't distinguish OpenSSH 8.4 from 9.6 cannot trivially grep CVE databases for unpatched versions.
- Rate limiting: firewall rules can limit new connection attempts per source IP per second, throttling scan speed without blocking legitimate traffic.
- DROP vs REJECT: a firewall can respond to closed ports with ICMP unreachable (REJECT) or silently drop packets (DROP). DROP makes all closed ports look filtered — it slightly slows scans but also hides your network topology. REJECT gives faster feedback to legitimate clients but reveals which ports exist. Neither prevents a patient scanner; choose based on your threat model.
- Port knocking or single-packet authorization: services like SSH can be hidden behind a firewall that only opens the port after a specific sequence of connection attempts to decoy ports — or after a cryptographically signed UDP packet.
- Network segmentation: limit which hosts can reach which other hosts at the network layer, so a compromised host cannot pivot and scan the entire internal network.