Institutional connections to the CUDN

An institution connects to the Cambridge University Data Network (CUDN) physically through one or more interfaces on its PoP switch and logically through one or more VLANs which carry the IP subnets used by the institution, either directly or as a routed link between the CUDN router and the institutional router.

This page describes how these VLANs and subnets are configured, including how redundant routing is configured. The content of this page is of a technical nature and a good understanding of TCP/IP is required to understand some of details; the intended audience is institutional network managers.

Note that configuration described here is currently (as of late 2010) being worked upon. As such, many institutional networks may not be fully compliant. The process for achieving this is described below and is expected to be completed by summer 2011.

Contents

• Connection models
  • Edge connection
    • CUDN edge address organisation
    • Multinetting and proxy ARP optimisation
    • Traffic between public and private addresses on the same VLAN
  • Routed link connection
    • Static routed link
    • Dynamic routed link
• Redundant first hop routing
• Voice client networks
• Hostnames for router addresses
• Migration process

Connection models

The IP configuration on the institutional VLAN presented on the PoP switch typically follows one of two models:

• edge - the VLAN provides a direct connection for edge devices (e.g. computers, printers, etc.), perhaps with a proxy ARP firewall
• routed link - the VLAN provides a routed connection between an institutional router or firewall and the upstream CUDN router

In practice, the configuration can be a hybrid of these models but care must taken by the local network manager to ensure traffic is routed correctly. The two models are described below.

Institutions are entitled to up to two VLANs per PoP (in addition to any service VLANs for VoIP clients, Lapwing access points or Managed Cluster Services, etc.). Whilst most have only one, some may elect to split different parts of their network onto separate VLANs (e.g. colleges - one VLAN for student residential connections and one for staff); each of these are configured independently from one another and can follow a different model.

Edge connections

The most common and simplest type of connection has the IP subnet(s) used by the institution presented directly on the VLAN served by the CUDN router(s). The VLAN is usually taken from the PoP and carried through the institutional network directly to the edge ports serving networked client devices such as computers and printers.

This type of connection is also used with proxy ARP firewalls as the outside or untrusted VLAN. The firewall responds to the CUDN routers on behalf of client devices and selectively permits or denies traffic between this outside VLAN and an internal institutional inside or trusted VLAN where the client devices are connected. To the CUDN routers, it appears that the client devices are directly connected to the outside VLAN and do not need any special configuration to route traffic through the firewall.

CUDN edge address organisation

For subnets routed directly by the CUDN, some of the addresses are reserved for CUDN use. Which addresses are reserved depends on the size of the subnet and when the allocation was made; the table below summarises the different schemes in use (the scheme names are internal terms to refer to the layout). Some of the schemes shown here are used on routed links and not on regular edge subnets, but are shown here for completeness).

The remaining addresses are usually available for institutional use although this should be confirmed in the appropriate IP registry (e.g. IP Register). Note that the CUDN PoP equipment (switch and UPS) no longer reside on institutional subnets but on separate management VLANs and so do not take addresses from the institutional allocations.

The primary router and secondary router addresses should not be configured in institutional devices; they are used internally as part of the redundant routing configuration. The default gateway provides a redundant router address which should be transparent to clients and always be provided, with the same IP address and MAC address, even in the event of a router failure.

Multinetting and proxy ARP optimisation

Address ranges are added to a VLAN using a technique called multinetting; this overlays the new addresses on top of the same VLAN by adding secondary addresses to the router interface serving that VLAN. While this avoids having to renumber existing hosts by retaining the ranges used by existing hosts, there are a number of subtle issues presented.

The most obvious issue is that hosts in different subnets using their natural netmask (the one appropriate to the allocated subnet range for that address) won't be aware that they can actually reach each other directly on the same local network and relay traffic through their default gateway, resulting in the packets ricocheting off the upstream router - this is commonly called tromboning for obvious reasons:

To resolve this, the recommended netmask for subnets on multinetted VLANs routed directly on the CUDN is adjusted from the natural/actual netmask to be 255.255.0.0 (/16) and proxy ARP enabled on the CUDN router interface. Proxy ARP causes the router to respond with its own MAC address to ARP requests for IP addresses which it can reach and are NOT on the VLAN upon which the request was received. The effect of this is to make hosts believe that the whole of the /16 (either the public 131.111.0.0/16 or private 172.x.0.0/16 blocks) is reachable on the local network but have the router respond on behalf of hosts which are not and route them as normal:

This arrangement results in several potential issues:

• If two hosts in different address ranges have mismatching netmasks configured, the traffic flow between them may be asymmetric, resulting in flooding or performance issues. For example, if host B in the above diagram had a 255.255.255.0 netmask, traffic from A to B would go directly, but the returning traffic from B to A would trombone through the router. While this usually isn't immediately a problem, optimal traffic flow will not be achieved (avoiding tromboning) until all hosts are configured with the wider netmask.
• If an institution makes use of institution private addresses (e.g. 10.0.0.0/17, 172.31.0.0/16 or 192.168.0.0/16 - NOT CUDN-wide private addresses), the proxy ARP responses from the CUDN router must be blocked to avoid the router responding on behalf of addresses in these ranges. It may be possible to prevent the router generating responses for these addresses in future, but it is not possible to do this with the current router equipment. However, it is generally more appropriate to create a local institution private VLAN for such devices, which will avoid this issue.
• As the device believes that the whole of the /16 is reachable directly on the local network, it will ARP for any address in this range, potentially building a very large ARP table. For the average client machine, this is generally not an issue as it doesn't talk to a large number of internal hosts; however, for a server presenting content mainly to clients elsewhere inside the CUDN, it may be preferable to configure the natural netmask.

Just because the configured netmask is much larger than the natural netmask of the subnet, local subnet broadcast traffic does not get forwarded by the router to other subnets in the same, wider range. For example, when host B above sends a broadcast to 131.111.255.255, this does not get forwarded to all subnets in 131.111.0.0/16.

If an institution wishes to avoid the use of proxy ARP, it may be possible to allocate a new, larger contiguous block to avoid multiple disparate blocks. However, the institution must manage the renumbering process from the old range(s) into the new range in a timely fashion; this is much easier if DHCP is utilised. Obtaining a large block of public address addresses is often not possible due to the shortage of these; however, large CUDN-wide private blocks are normally not an issue.

Traffic between public and private addresses on the same VLAN

It should be noted that proxy ARP does not optimise the route between public and private IP addresses on the same VLAN (unless the netmask was set to an unworkably large range). As such, traffic between public and private addresses will continue to trombone through the upstream router. If large traffic flows are expected between devices on different types of address, this can be solved in one of three ways:

• renumbering the hosts to all use the same address type
• adding a secondary address of the other type to certain hosts (e.g. adding a secondary, private IP address to a server to optimise communication with local clients on private IP addresses - the public address may need to be retained if the server needs to be accessible from the internet, e.g. it's an institutional webserver)
• using an internal router

Routed link connection

This connection type is used when an institution has a router or firewall which operates at layer 3 (i.e. routes traffic explicitly, rather than relying on techniques such as proxy ARP). A small link subnet is configured on the VLAN between the CUDN router and the institutional router and the IP subnet(s) used by the institution are routed to the IP address(es) used by the institutional router at the end of the link. The institution then takes responsibility for routing the client subnet(s) itself and can configure them however it wishes.

There are two types of routed link; the choice of which to use depends on the complexity of the institution's connection with the CUDN:

• Static routed links are used in the vast majority of cases, with a simple, static configuration for their routing: the institution has a single border router (or multiple border routers which redundantly provide the same IP address to the CUDN routers) and doesn't need to dynamically update the CUDN backbone routing tables, in the event of failover or network reconfiguration
• Dynamic routed links are generally needed in more complex situations, for institutions with multiple links into the CUDN and need to dynamically adjust the routing for their networks, in the event of failover or other reconfiguration

Proxy ARP is usually disabled on such links as routing will be handled within the institution. Whether an institution uses proxy ARP internally, on their own router(s), is a matter of local preference.

Static routed link

Typically, the link subnet is a /29 providing 6 usable addresses which are divided equally between the institution and the CUDN (note that this corresponds to the top scheme used for edge connections, shown above; alternative schemes bottop or bottom may be used for some existing links, to avoid renumbering):

The institution need not configure anything with regards the primary and secondary router physical addresses; they are used internally by the redundant routing configuration and fail over in just the same way as a plain edge connection.

In the subnets routed down to the institutional router, all addresses are typically available to the institution in the IP Register database, with only the base and top addresses reserved for the network and broadcast addresses. If an institution chooses to reorganise the routed space into subnets, there is no requirement to notify the UCS, but IP Register can create the appropriate subnets in the database to better reflect the actual organisation and make the netmasks and router addresses for the internal subnets report correctly.

Dynamic routed link

This connection type involves the institution running a dynamic routing protocol between their border routers and the CUDN routers. The CUDN routers will accept routes to institutional subnets via this protocol, allowing the institution to dynamically adjust the routing to their subnets. Where an institution has multiple connections (for example, through multiple PoP switches), the address range on each will be different and the active path(s) controlled by the institution through their use of routing advertisements.

There is considerable overhead to this arrangement on both the institution and CUDN sides, at the administrative and technical level, so it is permitted only by arrangement and where necessary to support a particular configuration.

Typically no first hop redundancy is implemented on routed links as redundancy and failover will be handled as an intrinsic part of the dynamic routing protocol.

This type of connection is only used in a small number of cases on the CUDN and is in the process of being reorganised during A/Y 2010-11; more information about this will be published in due course. However, it is expected that institutions requiring this service will need to run EBGP (the Exterior Border Gateway Protocol) between their border routers and the CUDN routers.

Redundant first hop routing

To improve resilience against the failure of a single router, the CUDN is undergoing a programme to implement a First Hop Redundancy Protocol (FHRP) on institutional network connections. In contrast to the more complex protocols used on the CUDN backbone, first hop redundancy does not require any special configuration of client devices and the failure of a router should only result in an outage of a few seconds.

The actual protocol used to implement first hop redundancy on the CUDN may change over time, depending on the make, model and configuration of routing equipment used. However, the basic principle of all such protocols is the same: the routers providing the redundancy cooperate to offer a single gateway / default router address with a virtual MAC/ethernet address; in the event of failure of the currently active router, a backup takes over the IP and MAC address within a few seconds meaning the clients do not need to make any adjustments to their configuration.

The introduction of this facility is largely transparent but it should be noted that:

• The routers will frequently emit announcements to indicate they are operating. This is usually limited to the one(s) providing the active default router (if these stop, a backup router will become active). These announcements are typically made using multicast and will be filtered out from most of the network.
• Even a backup router may route traffic onto a subnet as they are still operating on it, just not providing the default router. An example of this would be asymmetric routing where the return path via a backup router is more optimal.

Because of these reasons, care should be taken, particularly when applying filters to inbound traffic from the CUDN. Some firewalls may also have issues if the return path for traffic doesn't exactly match the outbound path.

Voice client networks

At present, all the client networks used by the University Telephone Network Voice-over-IP (VoIP) system are routed directly on the CUDN routers and carried into institutions as layer 2 VLANs, separate from their main data VLAN(s).

In future, however, once the telephone network has stabilised, it is expected that institutions will be able to route their voice subnet(s) themselves. This will typically be done using the routed link method. If this is done, the institution will need to apply and maintain the access control lists and DHCP server addresses (for the IP/UDP helper address relay agents) as advised by the UCS Network Division.

Regardless of the method used, it is not recommended that voice VLANs pass through any stateful firewalls, used within or at the border of an institution, to avoid any delay or jitter being added to the voice traffic. In any case, the protocols used on the telephone system are awkward to firewall correctly.

Hostnames for router addresses

Because the IP addresses used for the routers are held by different routers at different times, depending on failover status, it doesn't make sense for the names of the gateway/router addresses to reflect the actual routers they are assigned to.

As such the router addresses are registered with names such as the following (where <id> is usually a number corresponding to the VLAN ID for this subnet):

• gw-<id>.net[.private].cam.ac.uk - for the default router / gateway
• gw1-<id>.net[.private].cam.ac.uk - for the primary router
• gw2-<id>.net[.private].cam.ac.uk - for the secondary router

Note that these names may resolve into multiple addresses, if the VLAN is multinetted.

Migration process

The above information documents a process which is currently ongoing and expected to be complete by summer 2011. To achieve this, a large amount of work needs to be done to tidy up legacy configurations and standardise edge connections and routed links.

The process for enabling redundant routing is as follows:

1. Examine each subnet connected directly to the CUDN to select an appropriate scheme of reserved addresses.
2. If the reserved addresses are in use by an institution, liaise with the institution to make them available. If this is not possible, a larger block may need allocating and migrating to.
3. Physically connect the new link from the redundant router to the PoP.
4. Prepare the redundant routing configuration on the backup router(s).
5. Liaise with the institution and activate the redundant routing at an agreed time.

A member of the IP Register group from the University Computing Service will contact each institution if the reserved addresses are not available. However, it would be helpful if institutions could confirm that their address ranges do not have hosts in the required addresses and renumber them if necessary. For example, on a /24 subnet, typically .61, .62 and .254 must be made available.

There are a number of other activities going on at present on the CUDN, as part of a general reorganisation of the network. In most cases, these shouldn't result in any obvious difference to the institution and may just happen during a suitable 'at risk' time. The most common of these will be moving the primary routing for an institution to a new CUDN router.

Last updated:>October 2010