Sunday, May 26, 2019

Supporting Ancient Software

With another round of fixes to Windows XP, the time is ripe for bloviating about supporting ancient stuff. Every software vendor has to decide what to do about supporting what they used to ship, as well as the broader ecosystem around them. Operating systems, databases, service providers. Maximize use of your new features, minimize maintenance of your old ones. Maximize the number of potential customers, minimize the amount of development time required.  Keeping support for old systems looks like it’s on the maximizing side at first, but it’s an exponentially scaled problem when combined with your own features. It’s worth considering how the vendors of those ecosystem parts do things.

Why do customers stay on an antiquated platform? Perhaps they can’t afford the upgrade job, or perhaps they’re focused elsewhere and willing to accept the risk. For a software vendor, the former is a questionable customer; landing and keeping them may be profitable, but it won’t be great margin. Ah, but the latter... a vendor can charge the latter appropriately for the work to be done through a special one-off development effort. Welcome to the world of extended support contracts.

“Oh come now”, one might say, “that is not charitable at all!” And it’s true, there are nuances: many customers depend on equipment that cannot be upgraded. It was sold as a unified system, its vendor will not provide an upgrade at all or at an affordable cost, and its vendor will not support updates to the system. This sucks. What is the manufacturer or hospital or university to do, fund a new robot or MRI or TEM vendor? And yet from the vendor’s perspective, the predicament of customers who can’t upgrade is not distinguishable from the customers who won’t. They’re still stuck on the dead branch, forced to pay what the market will bear or take the risk of going unpatched. Once again, we are in the world of extended support contracts.

So there’s patches for the dead and unsupported OS from time to time. Who makes them?

I suppose it’s possible that there’s an XP engineering team at Microsoft sitting around on mothballs waiting for the opportunity to fix this stuff, but I’m guessing that is not the case. I think it’s highly unlikely that these patches ever come entirely from a vendor’s internal development teams, because it would be wasteful to maintain the systems and processes to produce two different levels of supportability for a single product, much less maintain a dead product. It would be doubly expensive to pull developers off of the current Windows line into a one-off effort to fix the dead product. More likely, when it breaks badly enough to need fixing, a new development team parachutes in, figures it out, and posts a patch. I’d bet that development team is outsourced, too, at least to another team within the vendor.

That would mean every patch is a special snowflake, provided by giving source access to a services team that charges to sustain it. The vendor collects extra support contracts from X customers to pay for the super smokejumper team, recognizes that revenue every month, and about once a quarter has a patch built. Not hard to make that into a profitable, high-margin business. In fact, if a vendor kept this in-sourced and gambled on one or two developers to maintain their knowledge, they could even defer the cost of the super smokejumper team for quite some time.

A third party software vendor has the opportunity to make the same decision, of course; should they spend their developer time on extending support to old software, or new? The answer is driven by their customers, in theory — but the vendor must evaluate the value of each decision. For a vendor with a small customer base, each customer demanding an oddity can be a significant percentage of revenue potential. For a vendor with a large customer base, each oddity request can have a significant number of requesters.  What’s not clear is the associated margin opportunity versus opportunity cost. Worse, there won’t be associated requests for the obvious choices, because they’re obvious, and a PM would be mistaken to ignore them until customers have to request.

If the vendor embraces the requested oddity, putting aside the non-requested mainstream, the customer should theoretically pay extra for their decision to stay on the old platform; otherwise the vendor is eating their own opportunity cost. The dollars spent on patching old stuff or extending features to old stuff are taken directly from the budget to do new work with. And since most vendors don’t have internal permission to use external super smokejumpers, they’re pulling developers off of (say) Mongo support to build (say) DB/2 support. This adds context-switch costs to the overall pain load.

Adding salt to this wound, many vendors end up giving the customer a hefty discount while bending over backwards to provide one-off snowflake features, robbing their future Peter to pay the present Paul. It’s an easy decision to make when the company’s leaders allow profit-making to be deferred into the invisible future.

Sunday, May 5, 2019

Land and Expand Packaging Decisions

Subsets of packaged content are needed in different system classes. If you're pursuing a land-and-expand model, then you need to have a way to expand. One way is to ship a static monolith with features turned off. Another is to ship dynamic add-ons to your base product. 

Teams make these dynamic vs static decisions early. If the business went dynamic, then content names and versions that are already deployed must be visible. If on-prem with high availability & fault tolerance goals, this can be remarkably challenging.

During installation, upgrade, or removal operations, the admin must fully understand the infrastructure and know more about the internal workings of the packaged content than anyone desires. Proceeding without understanding produces unpredictable installs and high support burden.

Any enterprise vendor with this problem decides: hide the complexity and offer one big package (fully static linking, or shipping the whole product as a service), or expose the complexity and offer separate packages for every role? Plus regional availability problems in cloud.

Option 0 (do nothing): You might say, "this is a relatively infrequent problem; when a customer goes to distributed component infrastructure, we train heavily and plan for dedicated support allotments."

Option 1 (incremental): design the infrastructure so each component can announce what roles it uses, design the package so each file in it is associated with a role, design the package installer to install files that match roles. User repeats desired action on every component.

Option 2 (radical): As above, but a separate deployer policy enforcement service ensures packages are installed, updated, and removed from all infrastructure. User commits desired action once on the policy tool. This is easiest for Cloud-only organizations.


For a sick sort of fun, look at how many times operating systems and programming languages have recreated this wheel since 2000.