Juniper SRX High Availability (HA) & clustering: Part 1

SRX is an enterprise grade firewall solution build by Juniper, one of the largest network equipment providers. In our organisation we have used SRX extensively as a firewall and IPS device. One of the great features of SRX is the ability to run it in a cluster, working in tandem.

High Availability (HA)

Failure of devices, links getting damaged or bugs crashing the OS are only a tiny fraction of things that terrorize a network/system administrator. Remember- things will fail. That is the mantra that one has to live by. Systems will go down, CPU spikes will crash your aggregate switch. For this reason the concept of HSRP(Cisco), VRRP (Open source)  was introduced in routers. The concept of Virtual chassis is extensively used in Juniper switches to bundle the switches together.

Juniper's SRX chassis cluster is a similar feature. Most of these features described in these articles have been tested on SRX 550 and 650. There are some things which differ from an SRX model to the other and therefore needs to be consulted in its manual. I have posted important links at the bottom.

This particular article will discuss the theory behind clustering and how SRX uses fab and control links to manage a failover.

When can clustering be useful:

Protecting from routing engine failures:

You can configure the routing engine to be shifted from one SRX to another. Routing failure can occur because of a buggy codebase of the JUNOS. Sudden power surges or outages can also cause routing engine failure (like pulling the plug out of SRX). High CPU spikes for a long time can also freeze it.

Upstream and downstream link failures:

Suppose you are using an SRX as an inline firewall and the upstream link to your core router fails or becomes choked? Without clustering all your traffic will start getting dropped.

Internet connectivity failure:

With ip monitoring feature (not supported in every SRX) you can have ICMP probes to multiple ips. If those probes fail the traffic get automatically shifted to the other SRX.

SRX redundancy

In the worst case if the SRX device goes down, the backup SRX will assume the mastership and the traffic will automatically get shifted to the other device. I have tested this and this hardly caused any ping loss.

Before getting into the actual configuration there are a few terms that you absolutely need to know.

Control Links:

The control plane of the two SRX is synced over this link. Different ports are designated as control ports in different SRX models. For example, in SRX 650 and 550 ge-0/0/1 is the control port. Remember this port will behave as a control port only if you enable clustering.

The two SRX's control port have to be physically connected to ge-0/0/1. Logically SRX1 will be connected from ge-0/0/1 to ge-9/0/1. This is how SRX behaves. These are hardcoded values and cannot be changed. Once they become part of the cluster, the primary SRX will continuously synchronize all the control plane information via this link.

Note that only in select models (high end SRX) can you have two links configured as control links in bundle. Otherwise you can have only one link as the control link. More about this in the next article.

FAB links:

Fab links (fabric links) are responsible for syncing the routing engine and passing the transit traffic across the SRX. Refer to the above diagram. If the link from SRX1-VC1 has failed all the traffic coming from RTR1 will reach SRX1, move via the fab link and pass downstream via SRX2.
Fortunately you can configure two data links as a bundle by defining fab0 and fab1 parameters. This will be configured in the next article.

Redundancy groups

This is where Juniper SRX's HA feature scores heavily. Redundancy group is a set of objects and properties which move to the backup SRX in case of a failure. You get the flexibility to define what is meant by a failure. It can be a link failing on either SRX or a ping check failing (known as ip-monitoring). Each redundancy group can specify which SRX node gets to be primary. For example, you can configure traffic for one subnet to go via SRX1 by making it as a primary node in a redundancy group. In another redundancy group you can specify SRX2 to be primary and associate a different subnet to it. This way traffic for subnets can be differentially preferred.

Reth interface

A reth (Redunant ETHernet) interface is a pseduo interface which has "child interface" from both nodes. This is important to maintain high availability. A reth interface belongs to a particular redundant group. All the traffic goes to the primary node of the the redundant group via the configured child interface of the corresponding reth interface. Too hard to digest?

Refer to the network diagram above.

For high availability I need atleast two interfaces, one connecting to each SRX node. Since these two interface serve a specific purpose-redundancy , Juniper classifies them as a virtual interface known as reth interface.

Remember they are not in bonding/lag. In lag all the interfaces are used simultaneously. Not in this case. Traffic will only travel via one particular interface. (We can have reth lags but that is an advanced topic).

Now how does SRX decide which interface to send traffic through? We can specify a primary node (node refers to SRX) and a secondary node. But that is the purpose of a redundancy group. Thus, we can configure a reth interface to be part of a redundancy group and it will inherit all the properties.

For example, in the network diagram if reth1 belongs to redundancy group 1 whose primary node is SRX1 all the traffic coming from RTR1 will travel the left most link and reach SRX1.

If you are still confused the configuration in the next article should solve it.

I will be using the following network topology to set up a redundant SRX cluster which will automatically failover if any of the upstream or downstream link fails. Also some basic troubleshooting steps will be discussed.

SRX provides a robust chassic clustering feature. It is fairly easy to configure once the key concepts are understood.

Important links:

1. Configuring SRX chassis clustering
2. Blog post on SRX clustering
3. Juniper techpub on SRX chassis clustering

Reverse engineering IPmsg: UDP spoof attack to annoy users!

What really caught my attention after reading about Kevin Mitnick was how he conducted the TCP blind spoofing attack. It was an attack which required packet crafting and insertion into an already established TCP connection. What is interesting about this attack is that it does not require knowledge about any specific tools but instead relies on the person's knowledge about the protocol. So I started reading about packet crafting and a protocol or an application that I could play with. The modern OS puts random sequence numbers in the TCP packets making it very difficult to achieve what Kevin Mitnick did.

Thus UDP becomes the obvious choice. In comes IPMsg! It is a UDP based chatting application especially useful in LAN networks. What it suffers from is the spoofing attack. So if you are able to decode the IPmsg packet formats it is upto your imagination how you want to annoy others. From my experience following is an IPMSG packet structure:

|----------------------------------------------------------------|
| 0 | Sequence no | ipmsg-name | hostname | packet-type | data  |                                                                   |                                                                                               |
----------------------------------------------------------------

The 0 is fixed for all packets. Each packet that a user sends is marked by a sequence number, which is increased by one for every new message sent.  The next field is the display name that you have set for Ipmsg. Packet type field defines what kind of packet has been sent. The receiving application processes the data accordingly.

Now by default the sequence number is initialized to the timestamp when the application started.

Let me quickly go through some interesting packet types:

0:startup packet to announce that the host is running ipmsg
288: New packet.
33: acknowledgment for the packet. this is done in the background for all the packets.
1051424: a sealed message. 
1048624: acknowledge a sealed message. Upon receiving such a packet the sender machine will display a pop-up message shown below.

If you start the application, it registers itself on the network by sending a specific message. Following is a sample of the message.

Note that it initializes a counter which increments whenever it sends a new message.

Ipmsg traffic seen on wireshark

The client increases the counter when it sends another message

All the hosts keep track of this counter and ignores any packet with wrong value.

The IPmsg Attack

Ipmsg is a very lightweight and simple application. It has a very basic error detection mechanism at the application layer (verifying the counter value) and it cannot detect a spoofed packet. To slyly insert you can either send an initiation packet and reset the counter value stored by recipient victim. Or you can simply use the values being used by the original sender.

I prefer the former approach since it gives the attacker more control.

Sending the initiation packet:

Send a packet with 0 as the packet type and NULL as data. And set the counter with whichever value you want to. Here is an example.

Note the message type as 0

With this simple packet you have hijacked the connection. The original machine is unaware of this and it will keep using its own counter value which the recipient machine will ignore since the attacker has reset and changed the value.

Sending messages

You need to run Wireshark while you send spoofed packets to know the counter value of the recipient machine. As soon as you send the packet mentioned above the recipient machine will reply with its confirmation message and its own counter value.

A Demo:

Important Note:

The latest version of Ipmsg encrypts all data and it cannot be configured to send in plaintext. Therefore install the older version and change the registry value
HKEY_CURRENT_USER\Software\HSTools\IPMsgEng --> EncryptNum to 0.

This will force the Ipmsg to use plaintext.

I have used colasoft packet crafter which works on windows. it is a little difficult to use and I would highly suggest tools like scapy or hping3.

First send a packet with message type 0 and set a custom counter value.

To send a message use the packet type 288. Construct a packet with appropriate mac entries. If the destination pc does not reside on your own subnet then you will have to put your gateways mac as the destination mac.

Even more fun

There are two ways to send a message in ipmsg. One is the default method and the other is known as a "sealed" message. When you send a sealed message the receiver has to click on the a greyed box to display it. Upon clicking a UDP message is sent to the initial sender of the message and an intrusive pop up is displayed on his screen.

Colasoft packet crafter running 

Now, suppose I start sending these UDP messages to a victim machine acting as someone else?


Moreover you can automate the entire process using hping with Tcl or Scapy.

I hope you get my point!

Some interesting things that you can do-
a) Act as some other device and send packets with someone else's name.
b) Fill the victim machine's screen with confirmation pop ups.

Conclusion

As it is clear by now, there is no way to tell who sent what. Although this post is concentrated on Ipmsg, the subtle point is that it is strongly recommended that such applications are not run in corporate or critical environments.
All important data should be encrypted and only reliable clients should be used.

For a protocol designer, there are techniques to make a protocol robust against such replay or spoofed attacks.

Important Links

1) Wiki on Kevin mitnick
2) TCP/IP spoofing attack
3) Ipmsg older version
4) Replay attacks

IPv6 DoS attack using router flooding

This is one of the simplest DoS attacks I have ever seen. It is very easy to carry out and it's effects are devastating. It is a little different from normal DoS attacks in the sense that even when the attacker stops the attack the victim machines' resources will not free up! This makes it very lethal. The other point of difference is that it can be carried out only in a LAN environment.

Before we delve into the actual attack and its solutions we need to understand a little about ipv6 and its functioning.

Stateless autoconfiguration in IPv6 and Router advertisement packets

When a machine is statically assigned an ip address or it receives one via a dhcp server it is called as a stateful configuration. An ipv6 machine has the ability to generate a unique ipv6 address even if it is not able to communicate to a dhcpv6 server. This is known as stateless autoconfiguration.

It generates the ipv6 address using network blocks advertised in what is known as router advertisement packets (RA). The local router maintains a list of all network blocks which are supposed to be given by a dhcp server and which are supposed to be given as stateless address blocks.

So whenever a router sends a RA packet containing stateless address blocks a hosts accepts the packet and assigns an address to itself belonging to that network.

By default, all the modern OS listen to ipv6 Router advertisement packets without verifying and authenticating the router. Interestingly, OS like Windows (7 and 8) and FreeBSD do not have an upper limit to the number of such stateless network a machine can belong to. ( an ipv6 device can be a part of many ipv6 networks).

Moreover the whole process of listening and generating ipv6 address is somewhat CPU intensive.

These are the key ingredients of an ipv6 attack.

Suppose a malicious attacker rapidly starts sending bogus ipv6 router advertisements. Windows and FreeBSD at some point will get overwhelmed and will start using a lot of memory and CPU. Infact a tool designed to exploit this utility can bring down an entire network of ipv6 hosts within seconds.

The ipv6 router flood attack

All you need is the script which can be found in Kali linux. Connect your machine to a LAN network and simply run this script. Right now, you cannot direct this attack to a particular host and it will bring down the entire network.

It is as easy as running:

flood_router6 [interface]

For demonstration I ran a Kali VM and connected my laptop with another. Both were running windows 7. I ran this command and within seconds the CPU spiked to 100% utilisation. Infact even after stopping the attack the machine remained unresponsive and I had to reboot the machine!

IPv6 addresses bound to the attacked machine

Wireshark capture of the attack.

CPU utilisation after half an hour of stopping the attack

Solutions:

There are many ways to stop this attack.

1) Disabling router discovery

System administrators can disable router discovery. The machine will discard any RA packets.

netsh int ipv6 set int "[int number]" routerdiscovery=disabled

To find your interface number run the command

netsh interface ipv6 show interface

Disabling router discovery

the netsh show interface command

2) RA Guard:

There are switches with RA guard feature. Turning it can mitigate this attack although the script has a few advanced options which can be used to defeat it. More about this in a later post.

3) Using firewall to block such packets. 

Conclusion

This attack is very lethal since it renders the machine totally unusable. Only rebooting solves the problem. Moreover many latest operating systems including Windows 7, 8 , Surface and Android are vulnerable! Since all the modern OS are ipv6 ready and listen for RA packets, this attack can create chaos in a network.

Important links:

1) How IPv6 works
2) IPv6 message formats
3) Kali linux
4) Router flood author
5) Router Guard