ISPadmin
April, 2002
Network Design and Operation
INTRODUCTION
This installment of ISPadmin examines how service providers large and small
might setup their IP (and associated) networks to provide services to their
customers. After covering some network basics, the article illustrates how
a small dial-up provider might setup their network, and move on to a larger
provider. Issues surrounding the traditional small and large dial-up ISP
are examined. Finally, such topics as staff requirements, service level agreements
(SLAs) and network design considerations are pondered.
ISP NETWORKING BACKGROUND
This section contains basic networking concepts and terms and their meanings.
Paid Egress
Egress is a synonym of "exit" and is how network engineers refer to the points
where traffic leaves the providers network and enters another entity's network.
There are two types of egress: paid and peer. Paid egress is bandwidth that
the provider buys from another provider to deliver traffic that is not destined
for the providers network (or peers). In the greater Boston area, it runs
about $500 per megabyte/second/month without local loop charges.
Peer Egress
The second type of egress is peering, where little or no cost besides hardware
is incurred. Peering is exchanging traffic destined for someone else's network
directly with them, rather than using paid bandwidth. There are two types
of peering arrangements, public and private. Private peering agreements are
connections that take place in common facilities but not in a public peering
exchange like MAE EAST (WorldCom's widely known public peering facility;
there are many such public peering points run by a wide variety of providers).
GlobalNAPS (or GNAPS, a CLEC associated with my employer) allows no cost
peering for its customers. If two customers colocated in GNAPS facilites
would like to peer, and GNAPS doesn't incur any cost, the providers are allowed
to peer without additional cost from GNAPS.
In order for most larger providers to peer, they require a considerable amount
of traffic to be exchanged, as well as the traffic to be "roughly balanced".
For example, WorldCom requires 150 Mbps of traffic from the provider's network
to WorldCom and 150 Mbps from WorldCom's network to the provider's network.
(Even providers who qualify for WorldCom's free peering are required to pay
for a connection into their facility.) Public peering points are facilities
set up for the express purpose of enabling peering relationships (such as
the MAE facilities, WorldCom's widely known public peering points such as
MAE EAST). In the case of public peering points, the host of the exchange
point usually charges for connections into the facility, in order to cover
its costs and make a profit.
Autonomous System Number or ASN
When a provider has multiple egress points in their network, an ASN is used
to identify what network the traffic originated from (in the case of outgoing
packets) or is destined to (in the case of incoming packets). It usually
consists of a unique four digit number (for example, "AS1234") which tells
other devices on the Internet which network a particular packet belongs to,
when a network is multi-homed. An ASN is assigned by the American Registry
for Internet Numbers (ARIN) or other Internet numbering authority.
Table One covers common circuit acronyms and associated speeds for the United
States (from the ISP Glossary listed in the references). Table Two covers
a few common service provider definitions.
TABLE ONE
Dedicated Circuit Acronyms and Speeds
DS0 (Digital Service 0): 64 kilobits per second clear channel (normally provisioned
by the telephone company as 56 kilobits per second)
T1 (DS1): 24 DS0's or 1.544 Megabits per second
PRI (Primary Rate Interface) single ISDN channel normally provisioned on
a T1, supports both ISDN and plain old telephone service (POTS) connections
ISDN (Integrated Services Digital Network, 64 kilobits per second)
T3 (DS3): 672 DS0's or about 43 Megabits per second
OC3 (Optical Carrier): 155.52 Megabits per second
OC12: 622.08 Megabits per second
OC48: 2.488 Gigabits per second
TABLE TWO
Some Common Definitions
ILEC: Incumbent Local Exchange Carrier (i.e., Verizon, Qwest, etc.)
CLEC: Competitive Local Exchange Carrier (i.e., Level3, GlobalNAPS)
DLEC: Data Local Exchange Carrier (i.e., Covad)
POP: Point of Presence
RAS: Remote Access Server
ATM: Asynchronous Transfer Mode
SONET: Synchronous Optical Network
DOCSIS: Data Over Cable Service Interface Specification
SMALL PROVIDER BACKBONE
Figure one illustrates how a small provider might design their network. The
box marked "Central POP" is the central site where the provider has access
to the Internet. The boxes marked "Remote POP" represent off site locations
housing RAS gear or customer dedicated connections.
Figure 1
The goals of a "small" provider are centered around the following:
· One egress point for traffic
· Limited peering
· Small routers, not hierarchical
· No or limited redundancy
Of course, limiting cost usually drives the above goals. While multiple egress
points are desirable from a reliability standpoint, redundant access to the
Internet is simply beyond most small providers. The size of the paid egress
is likely to be measured in T1s, not T3s or DS3s. A small provider may utilize
private peering relationships in common facilities.
Peering is another area that most providers won't be able to afford, or qualify
for, except for very specific situations. A small ISP may utilize private
peering in facilities, but likely won't use public peering.
A small provider probably uses smaller routers, with limited port counts
and functionality. For example, the Cisco 2500/2600 series routers would
be used in most places except for the provider's hub (where servers might
be located, for example) where a larger router like the Cisco 3600 router
could be used.
No redundancy is normally engineered in a small provider's network. The cost
and complexity for such is beyond the small provider (and even can be too
much for larger providers as well).
LARGER PROVIDER BACKBONE
Figure two illustrates how a larger provider might design their network.
The boxes marked "Core" indicate the core routers/nodes of the network. Each
core node usually has two or more connections to other core nodes in the
providers network, forming the backbone of their network. The boxes labeled
"Border" indicate remote POPs that terminate customer connections. In the
case of a "traditional" ISP, customer connections might be leased lines running
at T1 or T3 speeds. In the case of a dialup ISP, the border routers are facilities
with RAS gear serving dial-up customers. In the case of a cable modem ISP,
the border routers are cable head ends where traffic exits the cable provider's
network and enters the Internet. (See the DOCSIS.org web page for more information
on this topic). Egress can take the form of peering points, or paid bandwidth.
Egress points are normally on the providers core network where fast routers
and interconnects are located.
Figure 2
· Multiple egress points
· Multiple peers
· Large routers setup hierarchically
· Some redundancy
Cost is less of an issue for a larger provider. They will likely have multiple
paid egress points for redundancy, at T3 speeds. A big service provider will
have multiple peers at both private and public peering points. Large routers
such as the Cisco 7000 series routers or Juniper Networks M-series will be
used, set up in a border/core arrangement. The provider's backbone network
will likely have some redundancy, so the loss of a single link or POP won't
take down the entire network.
The border and core router design is a common method used by larger providers
to segregate their networks. Slower links terminate at the border routers,
which send traffic to nearby core routers. Core routers have faster links
along with peering and paid bandwidth connections. Border routers are where
customer connections are normally terminated, except when the customer purchases
high speed bandwidth and the provider must terminate the connection at the
faster core routers where higher speed cards are available. This explains
why it is always best to buy the fastest connections possible, as the link
may be terminated on a faster router closer to egress/peers. Ordering the
fastest line possible may give the customer better throughput.
The backbone protocol is likely to be ATM. While ATM is designed for voice
and data networks, it is a mature technology and in wide use. Another option
is IP over SONET, though this is normally utilized in OC12 and faster links.
Redundancy is engineered to the extent possible (within economic reason)
in a larger provider. In some cases, additional markets are justified and
by having additional network paths to certain POPs. Of course, the provider's
service level agreements with its customers often dictate where and how certain
links are provisioned for redundancy.
SMALL PROVIDER DIAL NETWORK
In the case of a dial-up ISP, a smaller provider will usually purchase T1
PRI line(s) in the markets they would like to reach. Often, connections from
the local ILEC are purchased and terminated in the ISP's facilities in the
region served by dial-up. If the ISP chooses to use a CLEC, very often a
large coverage area can be obtained from one POP location. For example, GlobalNAPS
serves a substantial part of Massachusetts, New Hampshire, Vermont and Rhode
Island from its Quincy, MA location. The customer simply purchases
appropriate PRI (and rackspace) and obtains coverage for the entire region
and only has to site equipment in Quincy MA. If the same coverage was desired
from Verizon, numerous PRI would have to be ordered and facilities would
be required in many Verizon POPs to obtain similar coverage.
Often, a small provider will have a small state or regional dial coverage.
National/International coverage might be provided by a contract with a larger
provider, if necessary. For example, both GRiC and IPASS provide national/international
coverage.
LARGE PROVIDER DIAL NETWORK
Large ISPs usually utilize DS3 PRI lines across the country. Using such high
bandwidth lines and associated equipment enables the provider to realize
reduced costs. This is due to the fact DS3s and associated RAS equipment
are cheaper by the port in larger capacities.
Most larger dial-up ISPs have merged with telephone companies at this point.
The only possible exception to this (outside of WorldCom and other really
big providers) is StarNet, who has managed to stay independent. Most other
providers in this space (Concentric, Split Rock, Ziplink) have merged with
CLEC's or gone out of business. This consolidation of the industry is a testament
to the cost of PRI lines, as consolidation reduces the cost of these lines
on the balance sheet.
Larger ISPs often have POPs across the country. If the ISP is associated
with a ILEC/CLEC, coverage outside of the home territory will be through
another ILEC/CLEC. This will ensure that the provider has a wide coverage
base.
MISCELLANEOUS TOPICS
The <insert word here> Service Provider
These days, there is no shortage of differing types of service providers.
Most <insert word here> service providers are a variant of the web
hosting provider. These include the Application- and Storage- Service Providers,
among others. These types of providers typically have a backbone network
as outlined in the "Larger Provider Backbone". An important difference would
be the fact that in a dial or dedicated environment, the direction of traffic
is usually inbound, where in a web hosting environment, data flow will normally
be outbound.
Staff
At Ziplink, four network engineers handled the backbone network which included
approximately 70,000 ports. The RAS engineering staff consisted of approximately
6 full time engineers. Of course, a NOC staff was available to both groups,
in order to troubleshoot and perform simple fixes.
Central vs distributed network design
Some providers may not utilize a backbone network for some or all of their
POPs. This means they simply purchase egress at every location where their
RAS gear is located and forego the costs and headaches associated with running
one's own backbone network. The downside of such a design is that the provider
has little control over these individual connections and is at the mercy
of the egress providers. Costs will be higher when the provider runs their
own backbone network, as cross country network links will usually be more
expensive than buying egress at each POP.
Service Level Agreements
Service Level Agreements (SLA's) are formal definitions of the type of service
the provider will give to the customer. SLA's tend to vary widely from provider
to provider, and customer to customer, depending upon each party's particular
business needs. Of course, a provider wants the most flexible SLA's as possible,
while the customer wants 100% uptime no matter what extenuating circumstances
may exist.
Wholesale Dial Providers
Many substantial end user ISPs (such as MSN and Earthlink) have a small dialup
network or none at all. Instead, they purchase access from a wholesale dial
provider such as Level3 and let them manage the RAS gear, ports and associated
headache. The end user ISP purchases access in the form of ports, time (hours)
and/or users.
Security/DOS attacks
No discussion of this topic would be complete without some mention of the
security issues related to providers. Service providers are often the victims
of attacks, as they lease fast connections to other providers. Many attacks
take the form of Denial of Service (DoS) attacks, where an attacker stops
the ISPs customers from being able to access the services they purchase by
filling up their network connections. The Distributed DoS attacks are a variant
of the DoS attack, except the attacker uses multiple hosts to mount the attacks
from. Detecting and mitigating these sorts of attacks are the topic of much
current research. DoS attacks are stopped by implementing appropriate filters
on egress routers. As for the future, DoS attacks do not appear to be getting
any fewer, at least for the near future.
A good source of information for learning about the service provider business
in general is the ISP Planet home page listed in the references. Next time,
the current state of the art in stopping Unsolicited Commercial Email or
spam at the server side will be covered. In the meantime, please send your
questions and comments to me!
REFERENCES
Avi Freedman's Multi Homing page: http://www.netaxs.com/~freedman/multi.html
MAE Services and Facilities: http://www.mae.net/
WorldCom: http://www.worldcom.com/
WorldCom's Peering Policy: http://www.uu.net/peering/
GlobalNAPS: http://www.gnaps.com/
American Registry for Internet Numbers (ARIN): http://www.arin.net/
ISP Glossary: http://isp.webopedia.com/
DOCSIS starting point: http://www.docsis.org/
Cisco Systems: http://www.cisco.com/
Juniper Networks: http://www.juniper.net/
Verizon: http://www.verizon.com/
Qwest: http://www.qwest.com/
Level3: http://www.level3.com/
GlobalNAPS: http://www.gnaps.com/
GRiC: http://www.gric.com/
IPASS: http://www.ipass.com/
StarNet/MegaPOP: http://www.starnetusa.net/
MSN: http://www.msn.com/
Earthlink: http://www.earthlink.net/
Worldcom Business Internet Dial: http://www1.worldcom.com/us/products/access/dial/
ISP Planet: http://www.isp-planet.com/
