Practical Guide to Nmap and iptables on Linux

Imagine a company server connected to the internet, hosting a website and allowing employees to access it remotely via SSH. Every day, thousands of unknown systems try to connect to this server—some are legitimate users, but many are attackers scanning for open ports, vulnerable services, or misconfigured systems.

Without protection, a single exposed service could allow an attacker to gain unauthorized access.

This is where network security mechanisms become essential.

What Is a Firewall?

A firewall is a security system that monitors and controls network traffic based on predefined rules. It acts as a gatekeeper between a trusted internal network and untrusted external networks (such as the internet).

In simple terms:

• Allowed traffic is permitted
• Unauthorized or suspicious traffic is blocked

Firewalls help:

• Prevent unauthorized access — Block malicious connections
• Reduce attack surfaces — Close unnecessary ports
• Enforce network security policies — Control traffic flow

On Linux systems, tools like iptables implement firewall rules, while tools like Nmap are often used to test and analyze those rules from an attacker's or auditor's perspective.

Nmap: Network Scanning and Enumeration

Nmap (Network Mapper) is used to discover hosts, open ports, running services, and operating systems on a network.

Key Capabilities:

• Host discovery — Find active devices on a network
• Port scanning — Identify open ports and services
• Service detection — Determine service versions
• OS detection — Identify operating systems
• Network mapping — Understand network topology

Basic Host Scanning

Scan a single host:

nmap 192.168.1.1

Scans a single host and lists open ports using default settings.

Scan a domain:

nmap example.com

Scans a domain name instead of an IP address.

Port-Specific Scans

Goals of Port Scanning:

• Determine which ports on a target are open, closed, or filtered
• Identify the service running on an open port
• Find attack surface (e.g., services with known vulnerabilities)

Port States Explained:

• Open port: Responds with SYN/ACK to a SYN (meaning a server will accept connections)
• Closed port: Responds with RST (reset) — port reachable but no server listening
• Filtered port: No response, or ICMP unreachable — firewall drops/filters the packet. The scanner cannot determine if the service is open or closed; it is just blocked by a firewall

Port State Response Table:

Scan a single port:

nmap -p 80 192.168.1.1

Scans only port 80 on the target.

Scan a port range:

nmap -p 1-1000 192.168.1.1

Scans ports from 1 to 1000.

Scan all ports:

nmap -p- 192.168.1.1

Scans all 65,535 TCP ports.

Scan Types

TCP SYN Scan:

nmap -sS 192.168.1.1

TCP SYN(Half-Open) Scan – fast and commonly used; does not complete the TCP handshake.

How It Works:

• Sends SYN packet
• Analyzes response
• Does not complete handshake
• Less detectable than full connection

Half-Open Handshake:

TCP Connect Scan:

nmap -sT 192.168.1.1

TCP Connect Scan – completes the handshake; used when SYN scan is not permitted.

When to Use:

• SYN scan blocked by firewall
• User lacks raw socket privileges
• More reliable but slower and more detectable

Full Handshake:

ACK Scan:

nmap -sA 192.168.1.1

TCP ACK Scan – used to map firewall/packet filtering rules, not open ports.

Purpose:

• Map firewall rule sets, not for open/close detection - for firewall rule discovery (which ports are blocked by the firewall)

Example Output:

Starting Nmap 7.94 ( https://nmap.org ) at 2026-01-17 11:00 EST
Nmap scan report for 192.168.1.1
Host is up (0.00042s latency).

PORT     STATE        SERVICE
22/tcp   unfiltered   ssh
80/tcp   filtered     http
443/tcp  unfiltered   https

Nmap done: 1 IP address (1 host up) scanned in 0.38 seconds

Meaning of Each Result:

22/tcp unfiltered:

• Firewall allows ACK packets
• Port is reachable
• Service may be open or closed (ACK scan cannot tell)

80/tcp filtered:

• Firewall blocks ACK packets
• Port is NOT reachable
• Strong indication of firewall rule filtering

443/tcp unfiltered:

• Firewall allows traffic to this port
• Port might still be closed or open (unknown)

Important: ACK Scan States

• open ❌ never appears in ACK scan results
• Only valid states:
  • filtered
  • unfiltered
• Purpose: Map firewall rules
• ACK scan does not detect services or open ports

UDP Scan:

nmap -sU 192.168.1.1

UDP Scan – detects open UDP services (slower than TCP scans).

Purpose:

• Sends UDP packets to the target. If ICMP port-unreachable -> port is closed. If no response -> port is filtered.

Why UDP Is Slower:

• UDP is connectionless
• No acknowledgment packets
• Must wait for timeout
• Slower but necessary for UDP services (DNS, SNMP)

Detection Features

OS Detection:

nmap -O 192.168.1.1

Detects the operating system of the target.

** ICMP ping/host discovery:**

nmap -sn 192.168.1.1

Detects the host alive or not.

Service Version Detection:

nmap -sV 192.168.1.1

Detects service versions running on open ports.

What It Shows:

• Service name (e.g., Apache, OpenSSH)
• Version number (e.g., 2.4.41, 8.2p1)
• Additional information (product name, OS)

Aggressive Scan:

nmap -A 192.168.1.1

Aggressive scan (OS detection, version detection, scripts, traceroute).

Combines:

• -sV — Version detection
• -O — OS detection
• -sC — Default scripts
• --traceroute — Traceroute

Warning: Aggressive scans are more detectable and slower.

iptables: Linux Firewall Management

iptables is used to filter and control incoming and outgoing network traffic.

Key Concepts:

• Chains — INPUT, OUTPUT, FORWARD
• Rules — Match conditions and actions
• Targets — ACCEPT, DROP, REJECT
• Tables — filter, nat, mangle

This guide focuses on the filter table (default).

iptables Chains:

• INPUT — Used for packets coming to this machine
• FORWARD — Used for packets passing through your machine (router)
• OUTPUT — Used for packets generated by this machine

Viewing Rules

List current rules:

iptables -L

Lists current firewall rules.

Default Firewall Policies

Set default policies:

iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT

• Drop all incoming traffic by default
• Drop forwarded packets by default
• Allow outgoing traffic by default

This is a secure baseline configuration.

Policy Options:

• ACCEPT — Allow traffic
• DROP — Silently discard (no response)
• REJECT — Discard with error message

Security Best Practice:

Default DENY — Block everything by default, then allow specific traffic.

Why DROP Instead of REJECT:

• DROP — No response (stealthier)
• REJECT — Sends error (reveals firewall presence)

Allowing Services

Allow SSH access:

iptables -A INPUT -p tcp --dport 22 -j ACCEPT

Allows SSH access.

Breakdown:

• -A INPUT — Append to INPUT chain
• -p tcp — Protocol TCP
• --dport 22 — Destination port 22 (SSH)
• -j ACCEPT — Jump to ACCEPT target

Allow HTTP traffic:

iptables -A INPUT -p tcp --dport 80 -j ACCEPT

Allows HTTP traffic.

Allow HTTPS traffic:

iptables -A INPUT -p tcp --dport 443 -j ACCEPT

Allows HTTPS traffic.

Allow multiple ports:

iptables -A INPUT -p tcp -m multiport --dports 80,443 -j ACCEPT

Allows both HTTP and HTTPS in one rule.

Blocking Traffic

Block specific IP address:

iptables -A INPUT -s 192.168.1.10 -j DROP

Blocks all traffic from a specific IP address.

Block IP range:

iptables -A INPUT -s 192.168.1.0/24 -j DROP

Blocks entire subnet.

Block by protocol:

iptables -A INPUT -p udp -j DROP

Blocks all UDP traffic.

Block by port:

iptables -A INPUT -p tcp --dport 23 -j DROP

Blocks Telnet (port 23).

Rule Order Matters:

iptables processes rules top to bottom. First matching rule wins.

Example:

iptables -A INPUT -s 192.168.1.10 -j DROP
iptables -A INPUT -p tcp --dport 80 -j ACCEPT

The IP block comes first, so that IP cannot access port 80.

Connection Tracking

iptables' connection tracking helps identify a packet's relationship to existing sessions.

Allow established connections:

iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

Allows packets related to existing connections (essential for normal operation).

Connection States:

• NEW — New connection attempt (initial SYN packet)
• ESTABLISHED — Part of existing connection (packets in ongoing session)
• RELATED — Related to existing connection (e.g., FTP data connection, ICMP errors)
• INVALID — Invalid packets (malformed or not matching any state)

Stateless vs Stateful Firewalls:

Stateless Firewall Example:

A stateless firewall makes decisions based only on individual packets, without tracking connection state.

# Stateless: Allow SSH (no connection tracking)
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A OUTPUT -p tcp --sport 22 -j ACCEPT

Problem: Must explicitly allow both directions (incoming and outgoing) for each service.

Stateful Firewall Example:

A stateful firewall tracks connection state and automatically allows related traffic.

# Stateful: Allow established/related connections
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
 
# Allow new SSH connections
iptables -A INPUT -p tcp --dport 22 -j ACCEPT

Advantage: Only need to allow new connections in one direction; responses are automatically allowed via ESTABLISHED state.

Packet flow:

• Client to Your Server:

A  ---- SYN (NEW) ---->  Your Server
 

Firewall sees NEW, allows it because of the second rule

• Server to Client:

Your Server  ---- SYN,ACK (ESTABLISHED) ---->  A

Accepted because by default : iptables -P OUTPUT ACCEPT (allow all outgoing traffic)

• Client to Server:

Client ---- ACK (ESTABLISHED) ---->  Your Server

Firewall sees ESTABLISHED, allows it because of the first rule

Advantage: No need to explicitly allow both directions (incoming and outgoing) for each service.

What happens without the first rule?

Step-by-Step Packet Flow:

Step 1: Client starts SSH

Client ---- SYN (NEW) ----> Server

Matches rule 2 (NEW connection on port 22)

✅ Allowed

Step 2: Client sends next packet

Client ---- ACK / SSH DATA (ESTABLISHED) ----> Server

Now ask:

Does this packet still match --dport 22?

🔴 Not always

Some packets (like pure ACK packets) may not have the destination port in the same way, or the connection may be established and subsequent packets need different handling.

Solution: The first rule (--state ESTABLISHED,RELATED) catches these packets because they're part of an established connection, regardless of port matching.

Why This Is Essential:

Without this rule, responses to outgoing connections would be blocked, breaking normal network operation.

Loopback Interface

Allow local communication:

iptables -A INPUT -i lo -j ACCEPT

Allows local system communication.

Why This Is Needed:

• Many services communicate via loopback
• Localhost (127.0.0.1) must work
• Applications rely on loopback interface

Example local services:

Database servers:

• MySQL / MariaDB → 127.0.0.1:3306
• PostgreSQL → 127.0.0.1:5432

Development tools:

Local test servers:

python -m http.server 8000
node app.js

Browser connects to:

• http://localhost:8000

When to Add:

Always add this rule early to prevent breaking local services.

Port Forwarding

Port forwarding allows you to redirect incoming traffic to a specific port on your server to a different port.

Example:

iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080

Redirects incoming HTTP traffic (port 80) to port 8080 on the server.

ICMP (Ping Control)

Allow ping requests:

iptables -A INPUT -p icmp -j ACCEPT

Allows ping requests.

Block ping requests:

iptables -A INPUT -p icmp -j DROP

Blocks ping requests.

Why Block Ping:

• Security through obscurity — Hide system presence
• Prevent ping floods — Reduce DDoS attack surface
• Reduce information disclosure — Less network information revealed

Why Allow Ping:

• Network troubleshooting — Essential for diagnostics
• Monitoring — Health checks and monitoring systems
• User convenience — Easier network debugging

Recommendation:

Allow ICMP for internal networks, consider blocking for public-facing servers.

Deleting Rules

Delete by line number:

iptables -D INPUT 1

Deletes the first rule in the INPUT chain.

Delete by rule match:

iptables -D INPUT -p tcp --dport 22 -j ACCEPT

Deletes the SSH allow rule.

Then delete using line number.

Flush all rules:

iptables -F

Removes all rules (sets default policy).

Warning: Be careful with deletion—you might lock yourself out of SSH access!

Saving and Restoring Rules

Save rules:

iptables-save > rules.v4

Saves current firewall rules to a file.

Save to default location:

iptables-save > /etc/iptables/rules.v4

Saves to system default location (requires root).

Restore rules:

iptables-restore < rules.v4

Restores firewall rules from a file.

Important Note:

iptables rules are not persistent by default. After reboot, rules are lost unless saved and restored.

Complete Example: Basic Server Firewall

Secure baseline configuration:

# Flush existing rules
iptables -F
 
# Set default policies
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT ACCEPT
 
# Allow loopback
iptables -A INPUT -i lo -j ACCEPT
 
# Allow established connections
iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
 
# Allow SSH (only from specific IP)
iptables -A INPUT -p tcp -s 192.168.1.100 --dport 22 -j ACCEPT
 
# Allow HTTP and HTTPS
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT
 
# Allow ICMP (ping)
iptables -A INPUT -p icmp -j ACCEPT
 
# Save rules
iptables-save > /etc/iptables/rules.v4

This configuration:

• ✅ Blocks all incoming traffic by default
• ✅ Allows essential services (SSH, HTTP, HTTPS)
• ✅ Allows established connections (responses work)
• ✅ Allows loopback (local services work)
• ✅ Allows ping (network diagnostics)

Testing Firewall Rules with Nmap

Test from another machine:

# Scan open ports
nmap 192.168.1.1
 
# Test specific port
nmap -p 22 192.168.1.1
 
# Test if SSH is accessible
nmap -p 22 -sS 192.168.1.1

Test firewall rules:

# See which ports are filtered
nmap -p 1-1000 192.168.1.1
 
# Test with ACK scan (firewall mapping)
nmap -sA 192.168.1.1

Port States:

• open — Port is accessible
• closed — Port is closed (service not running)
• filtered — Port is blocked by firewall
• unfiltered — Port is accessible but state unclear

Exam Question: iptables Connection Tracking

Question:

Explain how connection tracking is achieved in an iptables-based firewall.

Consider the following chain of rules:

num  pkts  bytes  target  prot opt in out  source      destination     options
1    0     0      ACCEPT  icmp --  *  *    0.0.0.0/0   0.0.0.0/0       icmp type 255
2    6     426    ACCEPT  udp  --  *  *    0.0.0.0/0   224.0.0.251     udp dpt:5353
3    228   38248  ACCEPT  all  --  *  *    0.0.0.0/0   0.0.0.0/0       state RELATED,ESTABLISHED

Write the iptables instructions to append the above rules in the INPUT chain of the filter table.

Answer:

# Allow ICMP packets of type 255
iptables -t filter -A INPUT -p icmp --icmp-type 255 -j ACCEPT
 
# Allow UDP multicast traffic to 224.0.0.251 on port 5353 (mDNS)
iptables -t filter -A INPUT -p udp -d 224.0.0.251 --dport 5353 -j ACCEPT
 
# Allow packets belonging to established or related connections
iptables -t filter -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

Explanation:

Rule 1: Allows ICMP packets of type 255 (router discovery/advertisement).

Rule 2: Allows UDP traffic to multicast address 224.0.0.251 on port 5353, which is used for mDNS (multicast DNS) services like Bonjour/ZeroConf.

Rule 3: Uses connection tracking (-m state) to allow packets that are part of established connections or related connections. This is essential for allowing response packets and maintaining bidirectional communication.

Defenses against port scanning and attacks

Port Scanning Defenses:

• Firewall rules: Block unused ports; makes ports appear filtered; the scanner cannot reach the target port.
• IDS/IPS: Detect scanning pattersn and alert/block attackers.
• Rate limiting: Limit the number of packets per second to slow down scans.
• Blacklisting: Block IP addresses that scan frequently
• Stateful firewall: Track TCP states and drop invalid state packets.
• Ingress filtering: Filter packets at the network layer before they reach the application layer. Checks the source IP adress to prevent IP spoofing.

Rate limiting

Rate limiting: Limit the number of packets per second to slow down scans.

Example:

iptables -A INPUT -p tcp --dport 22 -m limit --limit 1/sec -j ACCEPT

Limits the number of packets per second to 1.

Port Scanning Attacks:

• SYN flood: Overwhelming servers with connection requests. The attacker sends a large number of SYN packets to the target port. The target port will be overwhelmed and will not be able to respond to the legitimate requests.
• UDP flood: Overwhelming servers with UDP packets
• ICMP flood: Overwhelming servers with ICMP packets

Common Question: hosts.allow and hosts.deny

Question:

Are hosts.allow and hosts.deny implemented using iptables?

Answer:

No. hosts.allow and hosts.deny are NOT implemented using iptables.

Key Differences:

How They Work:

• hosts.allow / hosts.deny:

  • Work at the application layer
  • Use TCP Wrappers library
  • Applied after packets have been accepted by iptables
  • Service must be linked with libwrap to use TCP Wrappers

• iptables:

  • Works at the network layer
  • Kernel-level packet filtering
  • Applied before application-layer processing
  • Filters packets before they reach applications

In Practice:

1. iptables filters packets at the network layer (first)
2. If packet passes iptables, it reaches the application
3. hosts.allow / hosts.deny (if applicable) filter at application layer (second)

hosts.allow and hosts.deny are applied at the application layer after packets have been accepted by iptables.