mn :: comp

The hosting market and you...

As of early 2003, the hosting market is not in the best of health, and although signs of recovery are apparent, there is still a significant danger zone to be crossed.

Why hosting?

The first thing that crosses many a technologist's mind is, "why should I host externally at all? I'm perfectly capable of running the site from here." The answers are simple:

1. Technology: The worst data center is going to have far better Internet connectivity, power, cooling, and facility redundancy than the average office building. However, the issue isn't 100% clear-cut, since the data center will also have greater demands placed on this infrastructure. Insist on a tour, and take note of growth capacity, systems redundancy, temperature and humidity.
2. Responsibility: Using a hosting provider represents due diligence on management's part; if there are issues, the only mistake was in choosing the wrong provider, and resolution can be described as "we're working with our provider on this breach of the Service Level Agreement." If on the other hand one chooses self-hosting, all issues become management's direct problem, and resolution cannot be explained to the board with descending into techno-babble. In other words, using a hosting provider insulates middle management from the risk of an angry C*O or board of directors by couching all issues in the terms of partnership and communication that they are usually most comfortable discussing. There is nothing wrong with steering away from the technical details of the implementation in favor of strictly negotiated SLAs. After all, if your partner is promising to provide web, app and database services and has demonstrated competence and willingness to rectify problems, the details are relatively unimportant. Of course, this way of thinking only works at some companies; if the mandate from management is "make it reliable at all costs" instead of "make it reasonably reliable," you have a harder job ahead of you than negotiating SLAs because you are prevented from relying on your partners. Every component, every partnership, every location, every connection is a potential source of failure and Murphy's Law does apply.
3. Finance: When your company buys a thing, it is entered into the books as a capital expenditure which then depreciates at a certain rate (typically a five-year lifetime). When your company buys a service, it is a one time cost per month that has no long-term effect on the taxes paid. Accounting is much simpler in the latter model. Additionally, one doesn't get permanently saddled with the things that one buys under the influence of hockeystick growth charts. Many companies own entirely too much Sun and Cisco horsepower. This gear ticks along at under 10% utilization, sucking money into the lease that originally purchased it and the ongoing support contracts that those vendors force customers to maintain, while its market value has dropped to 1 or 2 percent of its original purchase price. Over-powered gear wastes space, energy, and resources in ways that few companies can afford to ignore -- witness the amount of high-end gear to be had on eBay or DoveBid for pennies on the dollar. Had those companies leased this gear from a managed hosting provider, they would have had the option of downsizing; now, they don't.

These are all compelling reasons for management to consider hosting, and indeed many companies use some form of hosting services for at least their corporate web page, if not for service delivery and customer/partner extranets. However, there are a wide spectrum of offerings in hosting and a wide spectrum of providers, which can be difficult to navigate. The most useful distinction to draw for the purposes of this article is as follows: in colocation, your hosting provider gives you "power, ping, and pipe"; in managed hosting, the same services are supplemented with servers, software licenses, and a variety of professional services (many productized). Managed hosting is the Nirvana of most providers, with its promise of relatively high margins and its high lock-in potential, so many providers offer something called "managed" service. Details of capability and service delivery differ greatly.

Where we were

The IT industry was booming through much of the nineties, driven by wave upon wave of economic and social changes: some highlights include IT's penetration of new internal markets; wholesale adoption of packetized and encrypted networking vs circuit-based; commoditization of desktops and servers coupled with standardization of the operating environment; the multimedia hysteria; the Internet hysteria; the Y2K hysteria. Suddenly businesses that had never considered it before needed data center space and hosting services, and markets sprang up to meet those needs. Analysts produced hockeystick growth charts and predicted a multi-billion dollar managed hosting market by 2003, and the IT industry practically fell over itself throwing cash into this idea. Through the last half of the decade, well over one hundred new data centers were built in the United States alone, most of which were funded on loans from venture capitalists, angels, or parent corporations in other industries (such as telecom). The average utilization of these centers was less than 20% in early 2002, and many older facilities are being shut down as the bubble economy collapses into recession and heads towards depression.

In this time of intense growth, a great deal was learned about the process of building and operating data centers, and most of it was learned the hard way. Large aggregations of computing and networking gear produce challenges in networking, physical security, and facilities (particularly power and HVAC) which very few companies had ever needed to deal with before. The first generation of facilities was practically unusable by modern standards, and the second generation (roughly, construction started during 1998 or later) is generally far more capable. Every data center has its quirks, selling features, and problems behind the paint; if the goal is to achieve reasonable results for the dollar spent, one must set a baseline of requirements and try to avoid becoming emotionally attached to a nifty feature of one or the other.

A side-effect of this construction boom has arguably had a greater effect: a large sales and marketing force aggregated around the new world of hosting, and this force soon found that competition was fierce. Faced with competition, a well-trained salesperson has many options, but the trump card is always the junior salesperson's first reaction: lowering the price. As the market grew fiercer, negiotiated prices began to fall from $85 per square foot to a low in 2001 of $30 per square foot (on average, these prices are twenty to thirty percent below published list). Given that the typical data center of 10-30 thousand square feet costs millions to build and hundreds of thousands per month to operate, it's quite challenging to make a profit at $30 per square foot; estimated operational costs in fact average $35 per square foot in American data centers, not including servicing of the loans used to build the facilities. Colocation is bad business for the service provider, but not so bad that they can dare to turn it down; with operational costs like that, any cash flow is better than none.

Where we are

It is now early 2003, and the crows have come home to roost for IT. After nearly ten years of bubble economy, customers are finding that their IT expenditures are too high, the value returned is sometimes questionable, and the fashion-statement cachet is but a distant memory. The industry as a whole is probably going to benefit from this period of consolidation, and promising things are already happening in terms of support for open source software and new hardware methodologies (blade servers at one spectrum endpoint and and a return to the virtualized mainframe on the other). However in the hosting market, little has changed. Data centers are being closed and consolidated across the country, providers are exiting the business, and very few companies are able to boast of a marginally positive balance sheet (hint: they're buoyed by selling other products and services, usually commoditized). Many hosting companies are in the bankruptcy courts, usually on Chapter 11, and the investment community has been soaked.

Worst of all for the hosting industry, many customers who've tried managed hosting are tending to move to colo or even in-sourced services in order to decrease cash expenditures and increase control. This trend is by no means universal, but it does represent a fundamental failure of managed hosting to move upmarket. The issue is complicated, but the most useful way to tackle it is to look at the wide spectrum of customers who are seeking managed hosting services in the first place:

• Non-technical customers: In today's business world, it is not uncommon for a large company to have divisions with technical needs and no skills for meeting these needs. Internal IT rarely has the bandwidth or inclination to support web-facing properties, or charges rates that make external providers attractive. These customers are seeking developers and service providers who can provide as much service as possible, and colocation is not an option; they are locked into managed hosting and require professional services for nearly every action.
• Technical customers: Technically capable customers may consider outsourcing the management of their systems for a number of reasons: diffusement of capital expenditure costs is typically high on this list, but these customers will also consider their ability to move personnel into new projects and gradually offload responsibility. However, these customers are the first to reclaim responsibility when the managed hosting vendor makes a mistake. Because they view the vendor's management of three thousand servers in terms of their own management of three servers, they tend to be unforgiving of long response times, missed patches, and confused help-desk personnel. Additionally, members of the customer team may feel threatened by the outsource if new projects are not readily forthcoming, which leads to presentation of the vendor's mistakes in the worst possible light. Lastly, because the customer is not locked into managed services by lack of technical skill, the managed service is perceived as a luxury which is readily disposed of in the event of fiscal crisis.

Where we're going

It's a buyer's market in hosting, and it is going to become even more so in the next year as providers exit the bankruptcy courts cleansed of debt by Chapter 11 protection. Because they will be struggling to recover their names and grow customer base again, expect these providers to drop pricing even further than before, using the debt-reduction of bankruptcy as an excuse. There is no incentive for pricing to climb back to a point at which a data center is a profitable exercise because there is such a glut of data center space; rather, as older data centers are sold off at firesale rates by the experienced players, expect an increase of low-end providers competing solely on price.

In the short term this is good news for the hosting customer, but beware additional financial difficulties with vendors that may be selling service at a price which leaves them underwater. Customers of any type of hosting should be very specific about the services to be provided, pricing for custom items, and out-clauses in the contract. If the site is not designed for multi-datacenter distribution or at least a quick weekend move, one has a vested interest in the financial health of one's hosting provider that needs to be considered when negotiating contracts. If expenditures  on new projects are still included in the budget, customers are advised to redesign their sites to allow for multi-datacenter deployment, whether this is a simple DNS-based rollover to a static page or a fully redundant and load-balanced copy of the original system.

Copyright 2003, Jack Coates. This document may be redistributed freely as long as it is not altered.

Last modified: Nov 25, 2005 12:48 pm. Contact me.