Every layer of communication has its own unique security challenges. The transport Layer (Layer 4 in the OSI model) is especially vulnerable for the Denial of Service (DOS) attack or Distributed Denial of Service (DDOS) attack. Two most popular protocols used in the transport layer are TCP (Transmission Control Protocol) and UDP (User Datagram Protocol). The following are the key security risks at the Transport Layer associated with TCP and UDP:

TCP "SYN" attack is also known as SYN Flooding. It takes advantage of a flaw in how most hosts implement the TCP three-way handshake. When Host B receives the SYN request from A, it must keep track of the partially opened connection in a "listen queue" for at least 75 seconds. Many implementations can only keep track of a very limited number of connections. A malicious host can exploit the small size of the listen queue by sending multiple SYN requests to a host, but never replying to the SYN&ACK the other host sends back. By doing so, the other host's listen queue is quickly filled up, and it will stop accepting new connections, until a partially opened connection in the queue is completed or times out. This ability of removing a host from the network for at least 75 seconds can be used as a denial-of-service attack, or it can be used as a tool to implement other attacks, like IP Spoofing.

SSL Man-in-the-Middle Attacks: SSL/TLS was supposed to mitigate that risk for web transactions by providing endpoint authentication and encryption. However, it is discovered in late 2000 the feasibility of mounting an MITM attack on the protocol. One faulty SSL client mplementation, Microsoft's Internet Explorer, allows for transparent SSL MITM attacks when the attacker has any CA-signed certificate. An even greater risk is posed by unprotected systems where an attacker can preload his/her own trusted root authority certificates. The mitigation for such attack is to properly configure client SSL that would warn the user about problems with the server certificate.

Land Attack: An attacker sends a forged stream of TCP SYN packets with the same source and destination IP address and TCP port numbers. The victim system will be confused and crashed or rebooted. Service providers can block LAND attacks that originate behind aggregation points by installing filters on the ingress ports of their edge routers to check the source IP addresses of all incoming packets. If the address is within the range of advertised prefixes, the packet is forwarded; otherwise it is dropped.

TCP Connecting Hijacking is also known as Man-in-the-Middle attack. With this attack, an attacker can allow normal authentication to proceed between the two hosts, and then seize control of the connection. There are two possible ways to do this: one is during the TCP three-way handshake, and the other is in the middle of an established connection. Connection hijacking exploits a "desynchronized state" in TCP communication. When two hosts are desynchronized enough, they will discard (ignore) packets from each other. An attacker can then inject forged packets with the correct sequence numbers (and potentially modify or add commands to the communication). This requires the attacker to be located on the communication path between the two hosts so that he may eavesdrop, in order to replicate packets being sent.

UDP Flood Attack: UDP is a connectionless protocol and it does not require any connection setup procedure to transfer data. A UDP Flood Attack is possible when an attacker sends a UDP packet to a random port on the victim system. When the victim system receives a UDP packet, it will determine what application is waiting on the destination port. When it realizes that there is no application that is waiting on the port, it will generate an ICMP packet of destination unreachable to the forged source address. If enough UDP packets are delivered to ports on victim, the system will go down.

Port Scan Attack : A Port Scan is one of the most popular reconnaissance techniques attackers use to discover services they can break into. All machines connected to a network run many services that use TCP or UDP ports. A port scan helps the attacker find which ports are available. Essentially, a port scan consists of sending a message to each port, one at a time. The kind of response received indicates whether the port is used and can therefore be probed further for weakness.

Like most of the network security problems, there are no silver bullet solution to FIX the problems, however, there are many technologies and solutions available to mitigate the above security problems and to monitor the network to reduce its damage if attack happens. The above security exposures such as TCP Syn Attack and UDP flood can be effectively reduced (if not completely solved) by deploying Firewalls at critical locations of a network to filter un-wanted traffic and from iffy sources. The specific mitigation for each attacks can be find in article of the topic.

Related Terms: IPsec VPN, Firewall, Port Scan Attack, TCP SYN Attack, UDP Flood, Man in the Middle attack, Denial of Service, DDOS, Connection Hijacking, TCP, UDP