Archive for April, 2013

Ok padawans, this is something that I’ve wanted to cover for quite a while, but with the plethora of obligations monopolizing the jedi’s time, I was out of commission for some months. But fear not! Your jedi is back in the saddle and ready to bring it!

COMBAT PREPARATION

Notice to the audience: this article assumes the reader understands basic networking concepts such as CIDR notation, legacy class A/B/C vs VLSM, and how to subnet both class B and Class C networks. If I say to you, “what is the most efficient subnet mask to support 400 hosts and allowing the most possible networks/subnets” you should know 255.255.254.0 (or a /23 if you prefer) off the top of your head will this requirement. In fact you should know it can scale to 510 hosts, and the moment you get that 511th host, you will need to change masks to 255.255.252.0 (or /22) in which case you can scale up to 1022 hosts

I’ve pasted a chart below to act as a refresher:

CIDR Host Addresses Subnet Mask
/19 8192 (8190 usable) 255.255.224.0
/20 4096 (4094 usable) 255.255.240.0
/21 2048 (2046 usable) 255.255.248.0
/22 1024 (1022 usable) 255.255.252.0
/23 512 (510 usable) 255.255.254.0
/24 256 (254 usable) 255.255.255.0
/25 128 (126 usable) 255.255.255.128
/26 64 (62 usable) 255.255.255.192
/27 32 (30 usable) 255.255.255.224
/28 16 (14 usable) 255.255.255.240
/29 8 (6 usable) 255.255.255.248
/30 4 (2 usable) 255.255.255.252
/31 2 (P2P Only) 255.255.255.254

.

/!\ BATTLE TIP: /31 mask can, in fact, be used, as per RFC 3021

This feature has been supported since IOS 12.2T; BUT be aware it is designed to be used on point to point links. Lets think about what you lose going from a /30 to a /31: the network address and the broadcast address. If you’re using a point to point link or non-broadcast media, those addresses are wasted. So /31 will work best on serial links running something like PPP or HDLC, or Frame Relay. They can be used on Ethernet, but since Ethernet is a broadcast-based medium, I don’t recommend it.

So, why do I feel this is important? You’re a Level 5 Network Ninja, CCNA in your hand, burning for the blood of your enemies (or just the AT&T account rep that terminated service due to a small “glitch” in their billing system). You’ve learned every detail of subnetting; you can subdivide a Class C in your sleep, ready to engage…

But, unfortunately, while the various network exams may cover the minute details of protocols and configuration parameters, typically the design aspect…the ~why~ …is left to you to discover through a painful process trial and error (i.e. fix, rinse, repeat). Specifically in this case, the ability to assess an enterprise’s infrastructure and come up with an IP addressing scheme that is easy to manage, easy to route, and consistent across the entire domain.

Think for a minute. You have a college campus or an international retail network, all interconnecting with several global data centers, with multiple classes of traffic, larger sites that contain dozens of IDF’s aggregating via fiber to an MDF with 2 or more service providers, each of which tie into BGP clouds that you control. Oh and let’s not forget…all of this needs to be monitored and secured. How do you tackle this challenge?

Well, first and foremost, you need a consistent method to simplify the administration of the network, and to do that, you need a system that makes all of your network devices as easy as possible to identify, locate, and manage. One of the most critical ways you do this is with your IP addressing scheme:

  • You should be able to look at an IP address and know what it is and where it is.
  • You should use as few lines as possible to control access to and from specific networks
  • You should use as few entries as possible to build a concise, efficient routing table to any destination throughout your enterprise

Sure, if you have a couple branch offices and 100 users …that’s cake. But what about when you have 100 SITES, with voice & video traffic, PCI requirements, multiple 100+TB SAN/NAS devices mirroring across your private WAN that need to be collapsed onto the same core? Or even worse, what if you are the provider with different customer networks that all need to be segregated from each other?

So…rather than repeat the inadequate techniques practiced by all those non-jedi enfeeblings, spewing forth the same generic & over generalized “tips & tricks” …I instead am going to go over a specific case study. One that is based in the real world, and mirrors different aspects of networks I’ve engineered in the past. We will proceed in this exercise making design choices and explaining them as we go.

For that is how you must learn young padawan. You must observe live combat, watching the jedi’s tactics as he battles the forces of darkness, and eventually come to understand the techniques employed, use them, and make them your own. Every thrust and parry; every defensive stance and offensive strike, and above all, to preempt your enemy as he adapts to your fighting style.

To do any less is to overindulge the pedantic at the expense of the practical. And while I strongly believe in knowing your theory, theory alone will not determine the victor in combat.

Balance young Skywalker

CASE STUDY – CISCO JEDI, LLC

The company we will be using is Cisco Jedi, LLC. A US-based retail company, with corporate offices in Los Angeles, Chicago, and New York, as well as 2 co-located data centers (one local to HQ in Los Angeles, and the other, functioning as a DR site in Scottsdale.) Additionally, they have their own retail chain of 400+ stores across US & Canada, with plans to expand another 80 stores, including expanding into EU and South America, by the end of 2015.

Cisco Jedi Network

Cisco Jedi Network

CORE NETWORK

All sites are interconnected by a private MPLS cloud through Verizon, running BGP to redistribute each site’s private networks. They connect over 100mb Ethernet loops that are rate limited down accordingly at each site. Los Angeles branch is using a Cisco 2821 and is rate-limited down to 50mb/s, while the New York and Chicago branch offices are using somewhat newer 1921’s but are rate-limited down to 20mb/s.  Both data centers are running at the full 100mb/s and connect through Cisco 3845.

Corporate HQ in Los Angeles is divided between 2 main sites: Site A and Site B. Both connect to each other by a point to point Cisco 1410 wireless bridge running at 54mb/s over 802.11g. Site B is the warehouse & distribution center which sits approximately 100 meters from Site A, the corporate office containing HR, Operations, Finance, Production, Marketing, and Design departments.

NOTE: Site B (the warehouse) is NOT directly connected to the MPLS network, but rather accesses internal applications & services through its wireless P2P bridge. Contrarily, Site A (HQ) is not directly connected to the internet, but rather connects through the warehouse, traversing the wireless bridge as well.

This is your network.

/!\BATTLE TIP: There are no small amount of considerations as you examine this network. Your mind should look at this topology and attempt to understand design choices & the challenges surrounding them. For instance, the latency between the two main corporate sites over the wireless bridge; esp considering if they have IP phones in the warehouse. You should ask yourself, why was it setup this way? Why not an internet connection and/or MPLS connection at both locations? The two culprits that should immediately come to mind are ISP availability and money. Also, take note the retail sites. These are templatized setups in which the stores internet access all goes through a centralized choke point. Again, analyze why. This being a retail company with PCI requirements, this would allow easy control & restriction of traffic in or out of the retail network. Clearly internet access is needed (otherwise it wouldn’t be provided), more than likely for some type of cloud-hosted application, be it for document collaboration, payroll, or email.

The end goal of this IP scheme is to provide us with a consistent structure that in some way simplifies the massive administrative burden of managing a network. Below I will present the solution, and work backwards to explain these design decisions.

GLOBAL ADDRESSING SCHEME

Because of the large number of retail sites that need to be on the network, I’ve elected take our addressing scheme from the 10.0.0.0/8 supernet addressing space, and will adhere to the following format:

10 . <Site ID> . <VLAN##> . X

 

SITE ID GEOGRAPHIC LOCATIONS
10 Los Angeles Corp & Warehouse
12 Chicago Branch
14 New York Branch
200 Los Angeles Data Center
220 Scottsdale Data Center
100-110 Retail Sites**
255 MPLS/BGP Core

.

Each site can be summarized to a 10.##.0.0/16 address. For example, any device located in the NY branch will be somewhere in the 10.14.0.0/16 network. Anything in the Scottsdale data center will be in 10.220.0.0/16.

Store sites require a little more consideration, especially since there are more than 254, we cannot easily summarize the Site ID to just the second octet. Furthermore, each store will need far fewer devices than any of the corporate locations. My recommendation is to assign each store its own /24 class C subnet. However, in doing so, you still need to be able to associate the Store # (assigned by operations) and correlate to a network address. Speaking from experience, it’s highly desirable for all the stores’ subnets to be adjacent to each other to allow for easier route summarization & access control list management. The list below defines the addressing template we will use for the retail environment:

STORE SUBNET
Store 1-199 10.100.(1-199).x
Store 200-399 10.102.(0-199).x
Store 400-599 10.104.(0-199).x
Store 600-799 10.106.(0-199).x
Store 800-999 10.108.(0-199).x
Store 1000-1199 10.110.(0-199).x

.

Examples)

Store 22        10.100.22.x/24

Store 122       10.100.122.x/24

Store 222       10.102.22.x/24

Store 522       10.104.122.x/24

Store 1222      10.112.22.x/24

LOS ANGELES CORPORATE VLANS

VLAN FUNCTION NETWORK HOSTS
8 Standard Corp Users 10.##.8.0/22 1022
16 Design/Graphics 10.##.16.0/24 254
24 Finance/Credit 10.##.24.0/24 254
32 Voice 10.##.32.0/22 1022
40 Video/Presence 10.##.40.0/22 1022
48 Wireless: Corp 10.##.48.0/22 1022
64 Warehouse User 10.##.64.0/22 1022
72 Wireless: Warehouse 10.##.72.0/22 1022
80 Warehouse  Sorting Systems 10.##.80.0/24 254
88 Guest [Internet Only] 10.##.88.0/22 1022
100 Servers 10.##.100.0/22 1022
104 ESX/vMotion 10.##.104.0/24 254
108 Storage 10.##.108.0/24 254
200 Mgmt/ILO/ Monitoring 10.##.200.0/22 1022
8XX DMZ 172.22.XX.0/16 255 Class C Subnets
999 MPLS/BGP** 10.255.##.0/16 65535 BGP Loopbacks

**See MPLS/Core Section Below

DATACENTER VLANS

VLAN FUNCTION NETWORK HOSTS
100 Servers 10.##.100.0/22 1022
104 ESX/vMotion 10.##.104.0/24 254
108 Storage 10.##.108.0/24 254
200 Mgmt/ILO/ Monitoring 10.##.200.0/22 1022
8XX DMZ 172.22.XX.0/16 255 Class C Subnets
999 MPLS/BGP** 10.255.##.0/16 65535 BGP Loopbacks

**See MPLS/Core Section Below

BRANCH VLANS

VLAN FUNCTION NETWORK HOSTS
8 Standard Corp Users 10.##.8.0/22 1022
16 Design/Graphics 10.##.16.0/24 254
24 Finance/Credit 10.##.24.0/24 254
32 Voice 10.##.32.0/22 1022
40 Video/Presence 10.##.40.0/22 1022
48 Wireless: Corp 10.##.48.0/22 1022
88 Guest [Internet Only] 10.##.88.0/22 1022
100 Servers 10.##.100.0/22 1022
200 Mgmt/ILO/ Monitoring 10.##.200.0/22 1022
999 MPLS/BGP** 10.255.##.0/16 65535 BGP Loopbacks

**See MPLS/Core Section Below

COMPREHENDING THE BATTLE PLAN

By now it should be apparent that this company operates with two main paradigms: the corporate environment and the retail environment. They both have somewhat similar needs, however each has its own challenges and requirements.

RETAIL NETWORK

Let’s start with the Retail network. We need something easy, and something that scales—the company is already at 400 stores, and given their expansion plans, you should be prepared to grow to 1000+ over the next 5 years. Furthermore, to control routing updates, ACLs, NAT statements, etc, its best if these addresses are contiguous so the entire retail space can be easily summarized. Again, each store will be given its own /24 subnet for such things as registers, wireless devices, management stations, printers, IP phones, etc.

/!\A NOTE ON EFFICIENCY

This is why subnetting is so critical. I had given a similar exercise to one of my employees, and below is the scheme he came up with.

10.100.(1-99).x = Store 1-99 data 10.100.(101-199).x = Store 1-99 voice
10.101.(0-99).x = Store 100-199 data 10.101.(100-199).x = Store 100-199 voice
10.102.(0-99).x = Store 200-299 data 10.102.(100-199).x = Store 200-299 voice
10.103.(0-99).x = Store 300-399 data 10.103.(100-199).x = Store 300-399 voice

.

Note it’s not necessarily “wrong.” It certainly takes into account separating voice traffic, and overall not a bad solution. However, using TWO class C’s for one retail location; ask yourself the question, do you really think a store will need even 254 devices (much less 510)?

Also consider this is a retail company, whose network must be governed (partially at least) by PCI compliance. Translated: your POS registers need to be on a separate VLAN. Add to that PCI requirements for quarterly wireless scanning, and the fact that the entire earth is using iPads for everything from credit card scans to open heart surgery, you might as well come to grips each retail site will need to be segmented across several VLANs. In light of these considerations, below is template for subnetting each store’s /24

VLAN # FUNCTION SUBNET GATEWAY HOST RANGE
VLAN 10 POS X.X.X.0/26 X.X.X.1 .2 – .62
VLAN 20 WIRELESS X.X.X.64/26 X.X.X.65 .66 – .126
VLAN 30 VOIP X.X.X.128/26 X.X.X.129 .130 – .190
VLAN 40 CORPORATE X.X.X.192/26 X.X.X.193 .194 – .254

.

This design will accommodate 62 POS registers, 62 wireless devices, 62 devices on the internal corporate network, and 62 devices on the VoIP network, all within a single /24.

So at the end of this setup we’ve set the stage for several things

  1. We can summarize the entire address space with a single ACL statement: 10.100.X.X/12. For instance, if the internet connection at the Scottsdale data center fails, it is a simple matter of modifying a single entry to reroute traffic for all retail locations out of the Los Angeles data center. In fact, a clever admin could have IP SLA setup and/or floating static routes to automatically handle that failover.
  2. We’ve allowed each store to be easily summarized to a /24, and effectively used its space in an easily templatized manner.
  3. Because we have several VLANs at each store, we can treat respective traffic differently. We can prevent the POS subnet from accessing other stores on the MPLS cloud, while allowing the VoIP and Corporate subnets to not suffer this restriction. We can police & prioritize traffic relatively easily based on a device’s VLAN membership.

.

DATA CENTERS

Next, let’s look at infrastructure VLANs: server, vMotion, storage, and management. Again, consistent and easily recognizable. If you see 10.220.100.18 is syslog or event entry somewhere, you instantly know that this IP belongs to a server located in the Scottsdale data center. The separation of these functions allow you to treat each VLAN differently. Specifically

  • Storage & vMotion traffic should never go across the WAN
  • IP based storage traffic can experience increased performance by using 9KB jumbo frames, esp if 10Gb is not yet installed and you must milk every ounce of speed that you can.
  • The management VLAN needs to function as a privileged network for the purpose of troubleshooting & diagnostics, and as such should not be subject to the same access lists and firewall rules as production networks. Furthermore, since this would also be used for monitoring, heavy amounts of SNMP and NetFlow are required. These can be isolated, and if necessary prioritized down so as not to interfere with network traffic that is business critical.

.

CORPORATE & BRANCH OFFICES

Lots of subnets here! As with the retail network, separation of these different VLANs allows different prioritizing and security filtering. And again, the management network is given more privileged access for the purpose of troubleshooting.

You should observe a few deliberate design choices right off the bat:

First, there is a correlation between the VLAN # and the third octet of the subnet. While this is not required, when dealing with a network this large, having consistency greatly simplifies the administration and maintenance.

Second, subnets and VLANs are aligned on the mask boundary: 4, 8, 16, etc. This allows for maximum agility. The design department right now is only using a standard class C mask. What for some insane reason they hire another 100 designers? Or decide that every designer needs an additional workstation? Well, we have the space available. Simply go to your DHCP server and update the mask; you have plenty of room to grow. The same is true if you need to insert or subdivide a larger address space into a group of smaller subnets.

/!\SECRET TACTIC: There is a second reason for doing this, however it is far less known. Modern layer 3 devices are moving away from the classic RIB/FIB framework, and now compile their entries into specialized TCAM tables, each with an associated VMR (Value Mask Result). Thus, rather than robbing CPU cycles to perform a sequential lookup, parsing down multiple tables, one entry at a time, these TCAM tables compare a packet to all its entries in parallel! However, like everything else in networking, they live in a binary universe. As a result, entries that can be easily written on binary borders increases the efficiency of how these entries are compiled in the TCAM tables. Much the same way summarization increases efficiency for the routing engine, TCAM tables aggregate entries for multiple scanning engines (routing, ACL, NAT, QoS, etc). Consequently, having fewer entries means a smaller TCAM, which in turn means faster lookups. Thus it behooves the us to utilize prefix alignment as we design our addressing space.

MPLS/BGP CORE

As I’m sure you remember from your CCNA, best practices recommend using a loopback interface as the source/destination for all your routing updates. I’m not going to elaborate on the pros & cons of doing so in this article, but let’s assume for now you’re going to employ this best practice. That being the case, each site’s MPLS router will have a single /32 address in this range, as per below

SITE LOOPBACK IP
10 10.255.0.10/32
12 10.255.0.12/32
14 10.255.0.14/32
200 10.255.0.200/32
220 10.255.0.220/32
Store 22 10.255.100.22/32
Store 122 10.255.100.122/32
Store 222 10.255.102.22/32
Store 522 10.255.104.122/32

.

DMZ

Anything that is publicly accessible will go in this range. However rather than a single dumping ground, it’s probably more prudent to subdivide it as necessary. With this range we can have 255 distinct class C subnets, which should be more than adequate. To keep management as simple as possible, we can correlate the VLAN with the subnet

Ex)

Email               VLAN 808 – 172.22.8.X/24

eCommerce    VLAN 820 – 172.22.20.X/24

CLAIMING VICTORY

As you can see, having the design laid out before you allows for more thorough comprehension of how to tackle such a daunting task. Not all networks are the same, and there is no one plug & play solution. Take this exercise and experiment with it; modify it to fit your existing network.

The hope is that by understanding the methods you can take this process of analysis, compare it to whatever infrastructure is laid before you, and still meet the objectives that are desirable from all networks:

  • Scalable – a network that can grow with your business without the need to be completely restructured
  • Flexible – a network that is agile enough to adapt to changes in the business, as well as the infrastructure
  • Manageable – a consistent, transparent network that is no more complex than it needs to be

As I said earlier in this article, I’ve wanted to write this topic for some time. I spent a good hour googling for relevant articles; and while I found some decent tutorials on TCP/IP in general, I found nothing that would help an up & coming jr. network administrator to design an address space for large enterprise-class network.

I hope this has been helpful in your fight against the forces of darkness (and incompetence)

Thanks for reading!

Jedi…out