How IDS and IPS Work: Intrusion Detection and Prevention Systems Explained

Intrusion Detection Systems (IDS) and Intrusion Prevention Systems (IPS) are network security technologies that monitor traffic for malicious activity, policy violations, and known attack patterns. An IDS passively observes traffic and generates alerts when it detects suspicious behavior. An IPS goes further: deployed inline in the traffic path, it actively blocks or drops malicious packets in real time. Together, they form a critical layer in network defense, sitting between the perimeter firewall and the applications it protects.

The distinction between IDS and IPS is operational, not technological. The same detection engine -- whether Snort, Suricata, or a commercial appliance -- can run in either mode. In IDS mode, it mirrors traffic from a SPAN port or network TAP and analyzes it out-of-band, alerting a security team when something suspicious appears. In IPS mode, it sits inline (between the router/switch and the protected network), inspecting every packet in real time and dropping those that match malicious signatures or anomalous patterns. The trade-off is clear: IDS has zero impact on network performance but cannot stop an attack in progress; IPS can block attacks instantly but introduces latency and becomes a potential point of failure.

This article covers the architecture, detection methods, rule syntax, deployment patterns, evasion techniques, and performance considerations of modern IDS/IPS systems.

Architecture: NIDS vs HIDS

Intrusion detection systems come in two fundamental architectures:

Network-based IDS/IPS (NIDS/NIPS)

A network-based IDS monitors traffic on a network segment by capturing packets at a strategic point -- typically at the network perimeter, at data center boundaries, or at segment choke points. The sensor receives a copy of all traffic (via SPAN/mirror port, network TAP, or inline placement) and applies its detection engine to every packet.

Key characteristics of NIDS:

• Visibility -- a single sensor can monitor an entire network segment, seeing traffic between all hosts on that segment without requiring software installation on individual endpoints.
• Protocol awareness -- NIDS engines reconstruct TCP streams, parse application-layer protocols (HTTP, DNS, TLS, SMB, etc.), and detect attacks that span multiple packets.
• Encrypted traffic blind spot -- NIDS cannot inspect the contents of TLS-encrypted traffic without decryption. It can analyze TLS metadata (SNI, JA3/JA4 fingerprints, certificate details) but not payload contents. This is an increasingly significant limitation as TLS adoption approaches 100% of web traffic.
• Performance demands -- a NIDS monitoring a 10 Gbps link must process up to 14.8 million packets per second (minimum-size frames). At 100 Gbps, this rises to 148 million pps. High-performance NIDS use kernel bypass techniques (DPDK, AF_PACKET, PF_RING) and multi-threaded architectures to keep up.

Host-based IDS/IPS (HIDS/HIPS)

A host-based IDS runs on individual endpoints (servers, workstations) and monitors activity at the host level: file system changes, process execution, system calls, log entries, registry modifications (Windows), and local network connections. HIDS has access to decrypted traffic (since it operates after TLS termination) and can detect attacks that are invisible to network-based sensors, such as local privilege escalation, rootkits, and file integrity violations.

Well-known HIDS implementations include:

• OSSEC/Wazuh -- open-source HIDS with file integrity monitoring, log analysis, rootkit detection, and active response capabilities. Wazuh is the actively maintained fork.
• osquery -- Facebook's open-source endpoint visibility tool that exposes operating system state as a relational database, queryable with SQL. Not a traditional IDS but widely used for endpoint detection.
• Linux Audit (auditd) -- the kernel-level audit framework that logs system calls and file accesses based on configurable rules.
• Commercial EDR platforms -- CrowdStrike Falcon, SentinelOne, Microsoft Defender for Endpoint. These are modern successors to HIDS, combining host-level monitoring with cloud-based analytics and response automation.

Detection Methods

Signature-Based Detection

Signature-based detection is the oldest and most widely used IDS/IPS method. It works by comparing network traffic against a database of known attack patterns -- signatures (also called rules). Each signature describes the specific bytes, patterns, or protocol behaviors that characterize a known attack.

For example, a signature might match the specific byte sequence in a buffer overflow exploit for a particular CVE, the SQL injection string ' OR 1=1 -- in an HTTP request, or the DNS query pattern used by a specific malware family's command-and-control beacon.

Strengths of signature-based detection:

• Low false positive rate -- well-written signatures produce very few false positives because they match specific, known attack patterns.
• Fast -- pattern matching can be highly optimized using multi-pattern search algorithms (Aho-Corasick, Hyperscan) that check thousands of signatures simultaneously in a single pass through the packet.
• Forensically useful -- when a signature triggers, it identifies the exact attack, often down to the specific CVE or malware variant.

Weaknesses:

• Cannot detect unknown attacks -- a signature must exist for every attack it detects. Zero-day exploits, novel attack techniques, and new malware variants will not match any signature until one is written.
• Evasion through mutation -- attackers can modify the byte-level representation of an attack to avoid matching a specific signature while preserving the attack's functionality (polymorphic shellcode, obfuscation, encoding variations).
• Signature maintenance burden -- large rulesets (Snort's community ruleset contains 30,000+ rules) must be regularly updated. Each rule update can introduce new false positives that require tuning.

Anomaly-Based Detection

Anomaly-based detection establishes a baseline model of "normal" network behavior and flags traffic that deviates significantly from that baseline. Instead of looking for known attack patterns, it looks for unusual patterns that might indicate an attack.

Anomaly detection can flag behaviors such as:

• A server that normally generates 100 DNS queries per hour suddenly sending 10,000 queries per hour (possible DNS tunneling or exfiltration).
• An internal host communicating with an IP in a country where the organization has no business presence.
• A sudden spike in ICMP traffic volume (possible DDoS attack or network scanning).
• An SSH session at 3 AM from an account that has never logged in outside business hours.
• Network traffic with unusual protocol distributions -- a host sending large volumes of traffic on uncommon ports.

Strengths: can detect novel attacks and zero-day exploits. Weaknesses: high false positive rate (legitimate behavioral changes trigger alerts), requires a training period to establish baselines, and can be slowly normalized by an attacker who gradually shifts baseline behavior over time ("slow and low" attacks).

Stateful Protocol Analysis

Stateful protocol analysis combines elements of both signature and anomaly detection. The IDS/IPS maintains full protocol state machines for the protocols it monitors (HTTP, DNS, SMTP, SMB, etc.) and validates that traffic conforms to the protocol specification. It detects attacks that violate protocol semantics, even if the specific byte pattern does not match a signature.

For example, a stateful HTTP analyzer would detect:

• HTTP request smuggling where Content-Length and Transfer-Encoding headers conflict.
• HTTP responses containing executable content served with an incorrect Content-Type header.
• DNS responses that contain more answer records than the question asked for (possible DNS cache poisoning).
• SMB protocol violations characteristic of the EternalBlue exploit chain.

Snort and Suricata Rule Syntax

Snort and Suricata are the two dominant open-source IDS/IPS engines. They share a common rule format (Suricata extends it with additional keywords), making it easy to discuss rule writing in a unified way.

Rule Structure

A Snort/Suricata rule consists of a header and an options section:

action protocol src_ip src_port -> dst_ip dst_port (options;)

A concrete example -- detecting a SQL injection attempt in an HTTP request:

alert http $HOME_NET any -> $EXTERNAL_NET any (
  msg:"SQL Injection attempt - UNION SELECT";
  flow:established,to_server;
  http.uri;
  content:"UNION"; nocase;
  content:"SELECT"; nocase; distance:0; within:20;
  pcre:"/UNION\s+(ALL\s+)?SELECT/Ui";
  classtype:web-application-attack;
  sid:1000001; rev:3;
  metadata:created_at 2024-01-15, updated_at 2024-06-01;
)

Breaking down the rule components:

• alert -- the action. Options: alert (log and alert), drop (block in IPS mode), pass (allow), reject (block and send TCP RST or ICMP unreachable).
• http -- the protocol. Suricata supports protocol-specific rules for HTTP, DNS, TLS, SSH, SMB, SMTP, and more. Snort uses tcp/udp/icmp/ip with application-layer keywords.
• $HOME_NET / $EXTERNAL_NET -- variables defined in the configuration. $HOME_NET is typically your internal network ranges; $EXTERNAL_NET is everything else.
• -> -- the direction operator. -> means source to destination; <> means bidirectional.
• msg -- human-readable description of the alert.
• flow -- match only on established TCP connections (established), in a specific direction (to_server or to_client).
• http.uri -- (Suricata) inspect only the HTTP URI, not the entire packet. This is called a sticky buffer and is critical for performance -- it narrows the search space.
• content -- match a specific byte string in the buffer. nocase makes it case-insensitive. distance and within constrain where the next content match can appear relative to the previous one.
• pcre -- Perl-compatible regular expression for complex pattern matching. The /U flag indicates URI-decoded input; /i is case-insensitive.
• classtype -- categorizes the alert (web-application-attack, trojan-activity, attempted-recon, etc.).
• sid -- unique signature ID. Custom rules should use SIDs starting at 1,000,000 to avoid conflicts with official rulesets.
• rev -- revision number, incremented when the rule is modified.

Advanced Rule Examples

Detecting DNS exfiltration by entropy analysis (Suricata with Lua scripting):

alert dns $HOME_NET any -> any any (
  msg:"Possible DNS exfiltration - high entropy query";
  dns.query;
  lua:dns_entropy.lua;
  classtype:trojan-activity;
  sid:1000010; rev:1;
)

Detecting TLS connections with suspicious JA3 fingerprints (known malware TLS clients):

alert tls $HOME_NET any -> $EXTERNAL_NET any (
  msg:"Known malware JA3 fingerprint - Cobalt Strike";
  ja3.hash;
  content:"72a589da586844d7f0818ce684948eea";
  classtype:trojan-activity;
  sid:1000020; rev:1;
)

Detecting a lateral movement technique (SMB remote service creation, used by PsExec):

alert smb $HOME_NET any -> $HOME_NET any (
  msg:"SMB - Remote Service Creation (PsExec-like)";
  flow:established,to_server;
  smb.named_pipe;
  content:"svcctl";
  classtype:attempted-admin;
  sid:1000030; rev:1;
)

Inline IPS vs Passive IDS Deployment

Passive (IDS) Mode

In passive mode, the sensor receives a copy of network traffic via:

• SPAN/Mirror port -- the switch copies traffic from one or more ports to a designated monitoring port. Simple to configure but can drop packets under high load (mirroring is lower priority than forwarding).
• Network TAP -- a physical device inserted into the network cable that passively splits the optical or electrical signal, providing a guaranteed copy of all traffic. TAPs do not drop packets and have no impact on the production network, but they require physical installation.
• Packet broker -- an aggregation device that collects traffic from multiple TAPs or SPAN ports, filters and deduplicates it, and distributes it to multiple monitoring tools. Used in large deployments where multiple IDS sensors, NetFlow collectors, and forensic recorders all need traffic visibility.

Passive IDS can detect and alert on attacks but cannot block them. By the time the alert reaches a human analyst (or even an automated response system), the malicious packets have already been delivered to their target. Passive IDS is primarily used for:

• Detection and forensic analysis in environments where inline deployment is too risky.
• Monitoring internal (east-west) traffic where an inline device would add unacceptable latency.
• Supplementing an inline IPS with broader visibility -- e.g., monitoring segments that the IPS does not cover.

Inline (IPS) Mode

In inline mode, the sensor sits directly in the traffic path. Every packet passes through the IPS engine, and packets matching drop rules are silently discarded (never forwarded to the destination). The IPS can also:

• Reject -- send a TCP RST to both sides to tear down the connection, or an ICMP unreachable for UDP traffic.
• Rate-limit -- throttle traffic matching certain patterns rather than dropping it entirely.
• Replace -- modify packet contents in transit (e.g., stripping a malicious JavaScript payload from an HTTP response). This is rarely used due to the risk of corrupting legitimate traffic.

Inline IPS introduces operational risks: if the IPS fails (hardware failure, software crash, overload), it can black-hole all traffic. To mitigate this, inline IPS deployments use:

• Fail-open bypass -- hardware bypass modules that short-circuit the IPS, passing traffic directly through if the IPS engine fails. This maintains network availability at the cost of losing inspection.
• High availability -- redundant IPS sensors in active/standby or active/active configuration.
• Detection-only mode -- running the IPS in "detect-only" mode initially (alerting but not blocking) to tune rules and eliminate false positives before enabling blocking.

Evasion Techniques

Sophisticated attackers use a variety of techniques to evade IDS/IPS detection. Understanding these evasion methods is essential for tuning IDS rules and validating that the IDS deployment is effective.

Fragmentation and Reassembly Attacks

IP fragmentation splits a single IP packet into multiple fragments. If an attack payload is split across fragments, a signature that matches on the complete payload will not trigger unless the IDS reassembles the fragments before inspection. Attackers exploit this in several ways:

• Overlapping fragments -- fragments with overlapping offset values. Different operating systems handle overlapping fragments differently (some take the first data, some take the last). If the IDS reassembles differently from the target, the IDS sees a benign payload while the target reconstructs the malicious one.
• Tiny fragments -- splitting the TCP header across multiple IP fragments so that the first fragment does not contain enough TCP header to identify the destination port. The IDS cannot match port-based rules on the first fragment.
• Fragment floods -- sending a high volume of fragments to exhaust the IDS's reassembly buffers, causing it to drop fragments and miss the attack hidden among them.

TCP Segmentation Attacks

Similar to IP fragmentation but at the TCP layer. The attacker sends the attack payload in very small TCP segments (e.g., 1 byte per segment). If the IDS does not perform full TCP stream reassembly, it will not see the complete attack string. More sophisticated variants include:

• Out-of-order segments -- sending TCP segments in a scrambled order.
• Overlapping segments -- sending segments with overlapping sequence numbers, exploiting OS-specific behavior.
• TTL-based evasion -- sending a benign TCP segment with a TTL high enough to reach the IDS but too low to reach the target. The IDS sees the benign data; the target never receives it. The attacker then sends the real (malicious) segment with a normal TTL. The IDS thinks the benign segment occupies that sequence space; the target sees only the malicious one.

Application-Layer Evasion

• Encoding variations -- HTTP requests can encode characters using URL encoding (%27 instead of '), double encoding (%2527), Unicode encoding, or UTF-8 overlong sequences. An IDS rule matching the literal character ' will miss the URL-encoded variant unless the IDS normalizes the input before matching.
• Protocol-level obfuscation -- HTTP allows many syntactic variations: chunked transfer encoding, HTTP/1.1 pipelining, unusual header ordering, or mixing absolute and relative URIs. Each variation can defeat rules that assume a specific request format.
• Encryption -- the most effective evasion is simply using TLS. If the IDS cannot decrypt the traffic, it cannot inspect it. TLS 1.3 with encrypted SNI (ECH) eliminates even metadata that the IDS could previously use for detection.

Performance Considerations

IDS/IPS performance is a constant engineering challenge. The sensor must inspect every packet at line rate while maintaining complex state for thousands of concurrent connections and matching against tens of thousands of signatures.

Packet Capture and Processing

Standard kernel networking (the AF_PACKET socket interface) can handle up to approximately 1-3 Gbps of traffic on modern hardware before packet drops become unacceptable. For higher speeds, kernel bypass techniques are essential:

• DPDK (Data Plane Development Kit) -- bypasses the kernel entirely, polling the NIC directly from userspace. Suricata supports DPDK capture, enabling 40+ Gbps inspection on appropriate hardware.
• AF_XDP -- Linux kernel 4.18+ provides a high-performance socket type that combines some benefits of kernel bypass with the safety of in-kernel networking. Suricata added AF_XDP support for efficient capture.
• PF_RING -- a Linux kernel module that provides zero-copy packet capture with hardware-level filtering. Used by Suricata and other network tools for 10 Gbps+ capture.
• eBPF -- eBPF programs can pre-filter traffic in the kernel before it reaches the IDS, reducing the volume of packets the IDS engine must process.

Multi-Pattern Matching

The core of IDS performance is the multi-pattern matching engine -- the algorithm that simultaneously checks a packet against thousands of content rules. The dominant algorithms include:

• Aho-Corasick -- builds a finite state machine from all content strings, enabling simultaneous matching in a single pass through the packet. O(n) in the length of the input, regardless of the number of patterns. Memory-intensive for large pattern sets.
• Hyperscan (Intel) -- a high-performance regex matching library that compiles patterns into optimized SIMD (AVX2/AVX-512) machine code. Suricata supports Hyperscan as an alternative to the built-in pattern matcher, providing 2-5x performance improvement on Intel hardware.
• Hardware offload -- some network cards (NVIDIA/Mellanox ConnectX, Intel IPU) can perform simple pattern matching in hardware, pre-filtering packets before they reach the CPU.

Suricata Threading Model

Suricata uses a multi-threaded architecture where each thread handles a subset of traffic flows independently. The threading model includes:

• Capture threads -- read packets from the NIC.
• Worker threads -- perform protocol parsing, stream reassembly, and rule matching. Each worker thread handles a set of flows (determined by RSS hash from the NIC), avoiding lock contention.
• Output threads -- write alerts, logs, and extracted files.

On a 16-core server, Suricata can scale to inspect 40+ Gbps when combined with DPDK capture and Hyperscan matching. However, the threading model means that a single flow (TCP connection) is always handled by one worker thread -- a single very-high-bandwidth flow can saturate one thread even if other threads are idle.

Tuning and False Positive Management

A production IDS deployment generates a staggering volume of alerts. An untuned Suricata sensor on a busy corporate network might produce millions of alerts per day. The vast majority are false positives -- legitimate traffic that happens to match a signature intended for a different context.

Effective IDS management requires systematic tuning:

• Suppress rules -- disable specific signatures that produce excessive false positives in your environment. Suricata's suppress directive can disable a rule entirely or only for specific source/destination addresses.
• Threshold rules -- require a signature to match N times within a time window before alerting. This suppresses sporadic false positives while still detecting sustained attacks.
• Rule categories -- enable only the rule categories relevant to your environment. If you do not run Windows servers, disable Windows exploit rules. If you do not have publicly accessible database servers, disable database attack rules.
• Custom whitelists -- define IP addresses, domains, or traffic patterns that should never trigger alerts (known-good vulnerability scanners, penetration testing tools, internal monitoring systems).
• Regular rule review -- periodically review enabled rules, remove obsolete signatures (for vulnerabilities in software you do not run), and update custom rules to reflect infrastructure changes.

Integration with SIEM and SOAR

IDS/IPS alerts are most valuable when correlated with other security data in a SIEM (Security Information and Event Management) platform. A single IDS alert might be noise, but the same alert combined with a failed login attempt, a firewall deny log, and a DNS query to a known-malicious domain becomes a high-confidence indicator of compromise.

Suricata outputs alerts in EVE JSON format, which is easily ingested by SIEM platforms (Splunk, Elastic Security, QRadar, Sentinel). The EVE log includes not just the alert metadata but also protocol-level details (HTTP headers, DNS queries, TLS certificates, file hashes) that enrich the SIEM's correlation capabilities.

SOAR (Security Orchestration, Automation and Response) platforms can automatically respond to IDS alerts: blocking the attacking IP on the firewall, isolating the targeted host, enriching the alert with threat intelligence, and creating tickets for analyst review.

Relevant RFCs and Standards

• RFC 2722 -- Traffic Flow Measurement: Architecture (RTFM)
• RFC 4765 -- Intrusion Detection Message Exchange Format (IDMEF)
• RFC 4766 -- Intrusion Detection Message Exchange Requirements
• RFC 4767 -- Intrusion Detection Exchange Protocol (IDXP)
• CVE Database -- the Common Vulnerabilities and Exposures system provides standardized identifiers for known vulnerabilities, used extensively in IDS signature metadata.
• STIX/TAXII -- Structured Threat Information eXpression / Trusted Automated eXchange of Indicator Information -- standards for sharing threat intelligence that IDS/IPS systems consume.

See It in Action

IDS/IPS systems protect networks at the border where BGP-routed traffic enters an organization's infrastructure. The DDoS attacks and intrusion attempts they detect arrive via the same routing infrastructure visible in the global BGP table. Use the god.ad BGP Looking Glass to explore the networks behind major IDS/IPS vendors and security organizations:

• AS109 -- Cisco, maintainer of Snort and Talos threat intelligence
• AS13335 -- Cloudflare, which runs one of the largest distributed IDS/IPS deployments via their WAF
• AS16509 -- AWS, provider of GuardDuty and Network Firewall (powered by Suricata)
• AS15169 -- Google, operator of Cloud IDS (also Suricata-based)
• AS5765 -- Palo Alto Networks, a major commercial IPS vendor

Every packet reaching this page was inspected by multiple IDS/IPS sensors along the path -- at your ISP's border, at Cloudflare's edge, and possibly at intermediate transit networks. Look up any IP address to see its route origin and the networks it traverses, each of which deploys its own intrusion detection infrastructure.