UnderFire Evaluation Criteria
UnderFire Introduction | | UnderFire References |

Table Of Contents

Criteria
Rationale
Glossary
References

Criteria

The first stage in the UnderFire development was to create the rating criteria that would be employed. This criteria was designed to be generic enough so that it could be used with the wide variety of firewall solutions on the market today as well as those not yet released. This criteria was also designed to give the reader a good understanding of the specific firewall being examined as well as facilitate at-a-glance comparisons between products. The following is a list of the criteria in outline form.

  1. Cost
    1. Purchase cost of
      1. hardware
      2. software
      3. extra equipment
      4. installation
      5. training (basic, advanced)
      6. ongoing cost for administration
      7. warranty3

  2. Operations and maintenance
    1. For the following items ask:
      1. ease of...
      2. quality of documentation of ...
      3. completeness of documentation
      4. availability of training and service contracts
      5. technical support
    2. Installation requirements
      1. what software is prerequisite?
      2. any third party code required?
      3. what hardware is prerequisite?
        1. router(s): number of units
        2. host(s), (workstation, PC, etc.): number of units
        3. electric power requirements3
        4. specialized network interface requirements3
      4. replacement of router or host
        1. router(s): number of units
        2. host(s), (workstation, PC, etc.): number of units
      5. what administration infrastructure is prerequisite?
        1. serial
        2. ethernet
        3. ATM
        4. seperate, unconnected PC or workstation
    3. Installation of firewall, (hardware and software)
    4. Configuration
      1. initial
      2. how much out of the box
    5. Defaults3
      1. are services by default enabled or disabled?
      2. is logging by default enabled or disabled?
      3. is encryption/authentication by default enabled or disabled?
      4. in general, assumptions of the product out of the box
    6. Reporting
      1. incident reports
        1. how flexible
      2. operations audit
    7. Upgrade schedule
      1. for software
      2. for hardware

  3. Functionality
    1. Integration capability with existing systems
      1. "plug and play"
      2. what software platforms are allowed
      3. what third party software can be integrated
        1. TCP wrapper
        2. socks
      4. what network topologies are possible
        1. internet and/or intranet
        2. demilitarized zoning (DMZ) possible?
        3. virtual private networking (VPN) possible?
        4. address translation available and/or required?
      5. what network technologies are possible and number of interfaces
        1. ethernet
        2. token ring
        3. ATM
    2. Openness
      1. availability of source code
    3. What protocols are covered?
      1. network protocols
        1. IP, IPX, Appletalk, XNS, SNA, X.25, OSI, NFS, RPCs, etc.
      2. management protocols
        1. SNMP, SNMP-II, Bridge, MIB, OOB, IB, Enet MIB, etc.
      3. management agent
        1. HTML, Telnet, SNMP, etc.
    4. Integration with other firewall products
    5. User interaction
      1. transparency for access to the outside
    6. Reporting Mechanisms
      1. extensibility
      2. amount of false positives
      3. configurable detail of audit
      4. paging, e-mail, hardcopy, worm, and/or remote logging capabilities
      5. audit analysis tool available and/or included
      6. report generating or massaging software available and/or included
      7. intrusion detection capabilities3
    7. What attack scenarios does this product protect against?
      1. physical
      2. network based attacks
        1. address spoofing
        2. sequence number prediction
        3. session hijacking
        4. fragmentation
        5. source routing
        6. DNS attacks
        7. RIP attacks
        8. ICMP attacks
        9. port scanning
        10. Xmastree packets
        11. mcast/bcast handling
        12. window of opportunity at boot time because of incomplete filtering tables
        13. heavy load
      3. measures against problems related to software base (e.g. Unix-related vulnerabilities, if it is a Unix-based product)3
    8. Counterattack capability
    9. Fault tolerance of product
      1. behavior under heavy load
      2. congestion conrtol mechanism
      3. congestion control performance and means of measurement
      4. power failure
      5. automated integrity check
    10. Content recognition
      1. virus
      2. executable code
      3. Java object code
      4. mail attachments
    11. Administration
      1. authentication, (e.g. S-key, SecurID, Digital Pathways)
      2. challenge/response
      3. access control mandatory, discretionary, or other3
      4. dial up connection
      5. single sign-on
      6. text only admin interface available, (not X based)
      7. local or remote or both configuration
      8. management tasks separation/role delegation3
    12. Encryption
      1. algorithms used3
      2. firewall to firewall
      3. for administrative dial up connections
      4. user to firewall encryption
      5. frequency of key-exchange
    13. Does the product provide encryption and/or authentication and to what level?
    14. Exportability
    15. Network presence
      1. transparent
      2. addressable
    16. Interaction of multiple firewalls
      1. load balancing
      2. synchronization of actions
    17. Throughput and Bandwidth
      1. packet forwarding rate
      2. benchmarks
      3. number of systems required to handle a saturated T1, T3, and 10BaseT
      4. latency induced by encryption if available and used3
    18. Filtering
      1. separation of input and output
      2. bidirectionality
      3. by protocol
      4. by address
      5. by service
      6. by user-defined patterns
      7. filtering rate
      8. policy easy to specify and read
    19. In case of application level gateways, per-service features3
      1. access control
      2. authentication
      3. logging and auditing

  4. Education and Documentation
    1. Documentation
      1. complete?
      2. comprehensive?
      3. tutorial or manual style?
    2. Technical support
      1. location
      2. availibility
      3. contact methods, (e.g. phone, fax, or E-Mail)
      4. vendor or independent consultancy based
      5. response time commitments3
    3. Training
    4. Service contracting
      1. included
      2. add on
      3. availability

Criteria | | Rationale | | Glossary | | References | | Top Of Page |

Testing Rationale

Sorry, this section is under construction.

Part of the evaluation of a firewall requires testing. In section 7 of the evaluation criteria list there are a list of attacks that can be made against a firewall. These attacks can disable a firewalls or allow dangerous packets to flow through with no restrictions. Testing a firewall for these attack methods are important to the complete evaluation of a firewall. This section will focus on the network based attacks.

A network monitor tool is essential in determining if packets made it through the firewall to the opposite network. Having packet analysis tools in place on both networks can be useful in determining if the firewall has modified the packet once it is passed through. Some useful tools for monitoring are Argus, snoop (which is part of the Solaris operating system), Sniffit, and tcpdump.

Building packets that test for an attack is little more difficult. A few packages exist that can generate packet data and send packets. It is possible to write scripts to automatically generate attacks using these generation tools. Two packages that will generate packets are ipsend and spak.

Below is the list of network based attacks with a simple definition and a way of examining a firewall's susceptibility to such an attack.

Address Spoofing

Sequence Number Prediction

Session Hijacking

Fragmentation

Source Routing is a way of manual routing packets instead of letting routers decide the best method.

DNS Attacks

RIP Attacks

ICMP Attacks

SYN Floods are attacks based on a problem in the TCP/IP protocol. Every time a TCP connection is made to a remote machine a SYN (or "synchronize") packet is sent. The remote machine returns a SYN+ACK (or "synchronize and acknowledge") packet to the original machine to establish a connection. The protocol will wait for another response from the remote machine for a period of time (The protocol specifies 90 seconds, but some implementations wait longer.) There is a finite number of connections that can be in this wait state. When that limit is reached the machine cannot accept any new connections until the timeout period is reached for the current connections. An attacker can generate many SYN packets with a fake host address. The target machine will be unavailable to legitimate users because no new connections can be established. Several SYN Flood exploit programs are available. A firewall can be tested by using a SYN Flood program and trying to make legitimate connections through the firewall.

Port Scanning is a process used by attackers to gain information about the configuration of machines. There are several programs designed to scan machines for problems. SATAN, Strobe, and ISS are a few of them. A firewall should reveal very little information about itself. The firewall should also recognize these attempts at information gathering and create some kind of alert. A firewall can be tested by using one of these programs and checks the results of scan.

Christmas Tree Packets are packets that have all of the option bits set in the header. This condition cannot exist according the specifications of the protocol. Firewalls that do not implement the TCP/IP state machine correctly may allow such packets to pass through to the inside network.

Multicast/Broadcast Packets

Windows of Opportunity occur generally when the firewall machine is booting and the firewall software has not started or is not completely operational. There may a period of time when the machine will route packets as a normal router before the software starts. The operating system's configuration may be source of this problem. Another situation might arise when firewall software has not completed loading the filter rules. It might only filter packets based on the current set of rules it has. This condition is more difficult to test for considering the fact that the vulnerable time may be brief or the condition is difficult to replicate. An examination of the source code may be necessary to determine if this is a problem.

Heavy Load may cause some firewalls to pass packets through due to the tremendous volume of packets in the queue and the load on the firewall machine's resources. This can be simulated by using a lower speed machine and slowing it down by consuming resources (i.e. CPU time, memory, etc.). Then, by saturating the network with valid packets and a few that should be dropped by the filter rules it should be possible to see if all of the packets are allowed through.

Criteria | | Rationale | | Glossary | | References | | Top Of Page |

Glossary

Sorry, this section is under construction.

packet filter
circuit level firewall
application level firewall
hybrid firewall
network interface
router
ethernet
Asynchronous Transfer Mode (ATM)
encryption
authentication
"plug and play"
internet
intranet
demilitarized zoning (DMZ)
virtual private networking (VPN)
address translation
token ring
source code
IP, IPX, Appletalk, XNS, SNA, X.25, OSI, NFS, RPC
SNMP, SNMP-II, Bridge, MIB, OOB, IB, Enet MIB
HTML, Telnet, SNMP
intrusion detection
address spoofing
sequence number prediction
session hijacking
fragmentation
source routing
DNS attacks
RIP attacks
ICMP attacks
port scanning
Xmastree packets
mcast/bcast handling
filtering tables
challenge/response
access control
dial up connection
transparent network presence
T1, T3, 10BaseT
Bidirectional filtering

Criteria | | Rationale | | Glossary | | References | | Top Of Page |

References

[1] Bellovin, Cheswick, Firewalls and Internet Security, 1994
[2] Firewall Evaluation Checklist, Fortified Networks Inc.
[3] Firewall Product Functional Summary template, NCSA.

Criteria | | Rationale | | Glossary | | References | | Top Of Page |