Ultimate Data Visualization: Splunk + Electric Sheep

like inward singing, it makes non-stop rocking possible. 

Some individuals may perceive some slight gaps between "enhanced security" and "electric sheep" but just talk fast and you'll be fine. 

Download here: https://dl.dropboxusercontent.com/u/16649188/electric%20sheep.zip

Valuable Jerks

Once upon a time I worked with a developer who was a jerk. His bosses didn't like him, his colleagues didn't like him, he wasn't a team player. He produced though, and his code worked, and he would work all night to fix field-reported issues. Consequently, the field loved him and had nothing bad to say about him. Then he was fired. His replacement was a moron, who re-architected the feature in ways that set it back by man-years. The moron was not sanctioned for this activity, but rather was actively praised for his efforts, even as those efforts dwindled to half-time and quarter-time. Eventually the moron left to pursue consulting, where his charming personality probably led to great success. I haven't looked in a while, but when I left the company this feature still hadn't recovered to the state in which the jerk had left it. Two observations:

1) If all your defenders are field people and you are not, you may be a jerk. Field sales and sales engineers live on the revenue-generation side of the line, and they are allowed to be jerks as long as they produce. Furthermore, they produce by finding common ground and working with whoever is put in front of them, no matter how difficult the customer may be. So, if your only defenders are people who professionally put up with shit and value blunt behavior, it's pretty likely that you're generating an unhelpful level of shit.

2) Shit-generating jerks are disruptive elements in a team, it's true. That doesn't mean that they're not useful though, and the wise manager will compartmentalize them in a way that allows their work to continue. This is intensely difficult to do, particularly in an organization which is process-heavy and metrics-oriented, because the jerk will not play well with your Agile development practice or team-coding sessions. What they will do is produce interesting and useful things. Protect them.

Leading the Horses

Recovered from October 13, 2006 -- thanks, Wayback Machine! (DVD? Is that like some medieval YouTube?)

Computer Education is one of the most useless endeavors known to man, up there with 17th century tulip speculation. I'm not referring to directed training courses on specific products; some people learn better in a classroom environment, and I've attended some extremely well-structured classes on the practical application of technical theories. Rather, I'm thinking of those "how to use a computer" classes that legions of administrative assistants, teachers, and unemployed construction workers are forced to sit through ever week of the year. "Join the Information Super Highway!" "Be A Part of The Computer Revolution!" "Earn Higher Wages In An Exciting Field!" "Learn Network Security In Ten Days!" "Get Your MCSE In Twelve Seconds Flat!"

Let's get real. Some computers are tougher to use than others, but in general, anything with pretensions to desktop use is fairly straightforward. Honestly, sit a pre-teen geek down with a Windows XP, Mac OSX, or *nix/KDE desktop and they'll figure out how to get to the Internet games and DVD player in ten minutes flat. Whether it's a modern desktop or the antiques that the current generations of IT geeks learned on, computers reward exploration with knowledge; if you want to know how to use it, you will play with it and learn. If you really want to know how to use it, you won't even be deterred by making a few accidents; much like someone learning to ride a bike, you'll dust yourself off, cry over your lost money and data, then keep on going.

If you don't want to learn, you're not going to, and no class, technology, or tool is going to change that until someone's able to mug you with an RNA injection (a la The Matrix or A World Out Of Time). Furthermore, at this point in time I'd say that anyone who is lacking computer basics is either economically unable to participate or is firmly in the camp of people who don't want to know. All the former really need is time, hardware, and bandwidth; all the latter need is something to do for a living that doesn't have anything to do with computers.

Race to the Bottom

Recovered from 9/7/2008... thanks, Wayback Machine!

I'm preparing to give an important presentation to a major financial institution from my home office. I start the demonstration VMWare images, bring up the Powerpoint, sign into WebEx, and call into the conference bridge. I'm using a 1 year old 2.4GHz Uniden cordless phone connected to my Vonage ATA, and the base is maybe four feet away on a bookshelf. A slight hum starts coming out of the phone as the prospect and salesweasel sign into the conference call. We natter on a bit, run through the slides, and the background hum grows. Every time I speak, it modulates into a distant howl behind my words. I try changing channels, I run around the house looking for sources of 2.4 GHz noise I can unplug, but nothing works. Eventually I give up and switch to my Blackberry phone. Yes, a cell phone connection to a tower nearly a mile away is more reliable than a cordless phone connection to a base station four feet away. Should I buy a wired phone instead? If such a beast is still available, it will be hard to find and may not include the crucial mute button. Maybe I should buy a new cordless phone? That depends on whether it would actually solve the problem.... This essay is about why it won't.

Why is that? My first cordless phone was a 900 MHz Sony that chugged along through several battery replacements and finally died last year. I paid about $80 for it in the late 1990's, and I paid about $70 for this unreliable Uniden. My wife uses a five year old 900 MHz Uniden cordless on the landline, which works fine -- except that its range is terrible. My colleague in Portland, OR reports the same buzzing and clicking problems with his 1 year old Panasonic cordless; my mother-in-law has the same problem with a six-month old high-end multiline Sony, and my neighbors have issues with their two year old $30 VTech. Looking at the Internet, I see people complaining about range, battery life, and sound quality across all phone models and designs; no one is really happy with the quality, regardless of what they paid. People have brand new cordless phones interfering with wireless Ethernet systems, alarm systems, and their DSL. The early generation cordless phones are aging out of usefulness, and the new ones, to put it bluntly, suck. The basic functionality grew better and better from 1990 to 2000, and then it stopped improving. Why can't you buy a decent cordless phone any more?

The answer lies in a negative feature of insufficiently restrained capitalistic markets: the inevitable race to the bottom. Free markets allowing free information interchange will drive prices down. This is a factor that is obviously true throughout the consumer electronics and computer industries. However, the phenomenon usually has brakes which prevent it from speeding past a certain floor of quality. You'll have to look hard to buy really lousy fruit and vegetables, because supermarkets know that the end-customers will not buy them. Information is exchanged, and the final customers' goals are considered by the supermarket, the distributor, and the farmer. By contrast, information interchange is not free in commoditized consumer electronics markets because the consumers are unable to directly influence the construction of complex products like cordless phones. In fact, from a free-market perspective you could argue that the consumer isn't even the customer at all.

To illustrate, let's look at buying furniture... you can choose to buy a particle-board bookcase for $25 from Target, or a higher-quality $50 particle board and wire shelving unit from Ikea. You could buy an unfinished pine shelf for $100 from Fenton MacLaren, or a $2000 cherrywood shelf from the same shop. Your dollars directly influence the value of the purchase: material quality is obvious, construction quality is obvious, and you get more or less what you pay for. Within the furniture marketplace, the customer's buying decisions can directly influence product quality because the construction materials are obvious to all concerned and technique is clear to anyone who wants to learn it. Furthermore, the entry cost to the marketplace is relatively low, leading to a wide availability of DIY and customized solutions. If a poor student can't even afford Target or Ikea, they can always use discarded lumber and cinderblocks or abandoned milkcrates draped with nice cloth to produce a bookcase. On the other end of the spectrum, a millionaire desiring built-in bookshelves made of inlaid exotic woods and diamond with a built-in computerized library catalog system can certainly have such a thing made by seeking out a craftsman and a programmer; it's just a matter of cost.

In bookshelves, the race to the bottom produces cheap, simple bookshelves that nearly anyone can afford to buy, but it does not preclude the availability of high-quality bookshelves for the discerning customer. This is free market capitalism working more or less as it should -- the lowest price is determined by exterior boundaries such as transportation cost, shelf-rental in a store, and government regulations protecting the safety of particle board workers. Within these constraints, the customer seeking a low price is served by bookshelf producers struggling to find cheaper, more attractive ways to build and sell a bookshelf, while the customer seeking quality is still able to purchase good furniture. Unfortunately, the same race to the bottom fails in the consumer electronics space, leading to a situation in which it is practically impossible to buy quality goods for any amount of money.

To understand why, we need to look at the construction process for a consumer electronics item. To build a product such as a cordless phone, there are many parties and many products involved. The final product is produced by a company which buys circuit boards, plastic cases, and packing material from other companies: the circuit boards are assembled by companies that buy signal processors, radio transmitters and recievers, capacitors, resistors, and assorted other components from other companies. Those companies in turn buy designs, patents and licenses to produce these components from other companies. In other words, there are marketplaces within marketplaces... marketplaces in which a handful of consumer electronics marketers purchase special-purpose materials to specification from a handful of specialized producers. The consumer seeking a quality cordless phone has no direct influence on these decisions; I have no idea what brand of radio, battery, or signal processor my lousy phone uses, nor do I know these things about the good phones I've had in the past. To find out, I'd need to take them apart and do some significant research, and to have a good reason for doing so, I would need to be able to find out what materials are used to build the phones which are currently available for sale. I don't have any leverage over the quality of the phone: in the handy British parlance, I am just a punter.

By the time that the final product has been put on a shelf or website for the end-customer to get it, all the quality-influencing decisions have already been made in other marketplaces. These marketplaces are insulated from direct contact with end-user customers, and so they operate without the end-user's input. The customer is the company which wishes to sell millions of cordless phones, not the individual who wishes to use one or two. The result is that the customer in this marketplace has no interest in quality whatsoever, so long as the quality/price is not bad enough to justify end-users returning the phone within the post-purchase grace period. Rather, the customer is seeking the cheapest way to meet the bare minimum of functional requirements. Money not spent on the basic functionality is money that can be spent on a more stylish case or a new feature unrelated to the basic functionality. So, one can go to the store right now and buy cordless phones with translucent cases revealing colored LEDs, multiple base stations, built-in answering machines, or even an artificial voice to read off Caller ID data to you. What you can't buy is a phone with better basic quality, because the phone producers have no reason to build one.

The basic functionality of a piece of consumer electronics gear is measured against an acceptable quality threshold rather than an optimal quality threshold, because the quality is just as complex as its construction. A rotten banana or a bad bookshelf are obvious when they fail the acceptable quality threshold. No one will buy a stinking, soft piece of fruit. A bookshelf that breaks during construction will get returned promptly. Similarly, a cordless phone that doesn't even pretend to work or bursts into flame when it's plugged in fails the acceptable quality threshold. Ideally, the average capitalism customer rarely meets a product below this acceptable quality threshold because the producer of goods is going to have significantly poorer sales as a direct result of crossing that line.

By contrast, the producer of goods does not necessarily have a lot of reasons or even ability to go above acceptable quality towards optimal quality, unless the end-user has reasonable access to alernatives that do so. For instance, here in Berkeley the produce is pretty good. You can certainly find inexpensive, waxy fruit and vegetables at a Safeway or Albertson's, but you can also find very good fresh stuff at Andronico's or the Berkeley Bowl for a bit more money. Additionally, extremely high quality can be bought for a fairly high price at two weekly Farmer's Markets, or unpredictable quality can always be bought cheaply at Trader Joe's. Within this marketplace, there are a wide range of choices between acceptable quality and optimal quality, so that I as punter can choose the best quality that my wallet will handle. I would not have the same choice in, say, an Alaskan fishing village, because all my quality choices would be made for me by the distributor who selected material and brought it to the company store. If I don't like their choices, too bad, because I don't have any others. This is exactly the type of relationship that we as consumers have with the consumer electronics makers, because we do not have the ability to recognize gradations in quality between acceptable and optimal. The phone is a sealed unit and does not reveal its quality in the same way that a piece of fruit or a nice bookshelf would do; only using the phone day in and day out will tell you whether it's a good one or not. Furthermore, the phone is not the object of attention, so as long as it is performing above the acceptable threshold, its bad quality may even go unnoticed for a time. Then once it is noticed, perhaps even enough to be galling, what can you do? A cordless phone that doesn't work very well but still works as well as its competitors is going to stay in the customer's home. The poor punter may return one or two models, but once they realize that the basic functionality isn't going to improve, they will have to give up. It's either select the best of the bad choices, or don't buy a cordless phone.

Those who remember Econ 101 will recognize this situation as collusion to limit the market, but I should point out that it's a natural collusion rather than any sort of evil attempt to bilk the consumer. No one wants to know how their phone is constructed, really, any more than most people want to know the construction details of a bookshelf. However, should one want to know how to build a bookshelf, it wouldn't be too tough. One can simply go to the library, check out a book, buy some lumber and tools, and get to work. DIY bookshelves have a relatively lower entry barrier, and a person who enjoys the work even has an opportunity to hang out a shingle as a custom bookshelf craftsman. I do not know of anyone with the skills or material to become a garage cordless phone producer, though, even at the level of selecting, buying and integrating pre-built circuit boards (which would hardly solve the optimal quality problem). Because individuals cannot easily break into this marketplace, the gap between acceptable quality and optimal quality never gets filled.

In fact, can it even get filled at all? After all, one of the basic tenets of the consumer electronics world is that all of this stuff would be ridiculously expensive to produce in low numbers... maybe once the quality of basic functionality has been driven low enough, it is actually impossible to affordably produce high quality any more at all.

There is a ray of light though, which is the highly-trained individual's ability to produce a new product. While breaking into a commodity functionality market like cordless phones at a realistic price point is largely hopeless, the garage tinkerer is doing very well in branches of consuler electronics that attract DIY hobbyists. Audiophile stereo gear, computerized home entertainment convergence systems, and the recent case-modding community are a healthy sign of growth at the edges of consumer electronics. I'd like to hear of any one building a quality phone, too.

Data Replication Time Calculator

I was recently reminded of the Data Replication Time Calculator I wrote with PK... and thankfully, the guy who reminded me of it found a copy on the Internet Archive. When DroppedPackets.org was converted from Zope to Wordpress, the copy I'd posted there was lost. So, thanks to David D it's now back on the interwebs, on my public Dropbox. Note that this spreadsheet predates the existence of Gigabit Ethernet and Jumbo Frames, which will throw it off in a probably major way.

The AppleScript equivalent of rubber sheets for Outlook

Here's an AppleScript to deal with Outlook getting itchy if there's email in the outbox and no Internet connection to send it over. Sucks to have your battery die on a plane over something stupid like that.

-- if I can't get the Internet, and Outlook wants to send email, then it behaves badly.
-- since I'm usually on battery in those conditions, this is not good and must be stopped.
-- of course this is a rare condition, so this script needs to be light on CPU too.

-- is Outlook even running?
tell application "System Events"
 set outlook_running to (name of processes) contains "Microsoft Outlook"
end tell
if (outlook_running) then
 -- do I have a network address?
 set ipaddr to IPv4 address of (get system info)
 if (ipaddr is not "127.0.0.1") then
  -- Am I really on the Internet (evil gateways don't count)
  set is_splunk to (do shell script "curl -s -I www.splunk.com | grep -c splunk")
  if (is_splunk > 0) then
   -- it's safe to work online now
   tell application "Microsoft Outlook"
    set working offline to "false"
    return ((current date) & "Set Outlook to work online.")
   end tell
  end if
 else
  -- if Outlook is running but I don't have an address or I'm not really on the Internet, it might be a good idea to work offline. 
  -- Commented out the bits that think about that decision.
  --if (is_splunk < 1) then
  -- tell application "Microsoft Outlook"
  --  set outbox_count to count every message of mail folder "Outbox"
  -- end tell
  -- if (outbox_count is greater than 0) then
  --  tell application "System Events"
  --   try
  --    set The_app to "Microsoft Outlook"
  --    set pid to the unix id of process The_app as Unicode text
  --    set cpu to (do shell script "ps -o %cpu -p " & quoted form of pid & "| tail -n 1")
  --   end try
  --  end tell
  --  
  --  if (cpu is greater than 50) then
  tell application "Microsoft Outlook"
   set working offline to true
   return ((current date) & "Set Outlook to work offline.")
  end tell
  --  end if
  -- end if
  --end if
 end if
end if

Imperfect Product

Software vendor people have all sorts of different backgrounds... while lawyers and architects might have pretty similar educations, software people could have been educated in liberal arts or pre-med or hard knocks instead of STEM or EECS. Furthermore, they might have more experience in selling or more experience in servicing, or no customer-facing experience at all. Some have worked for lots of vendors, some have worked only at customers, while others stumbled into this opportunity straight out of high school or college and know little else. This leads to a pretty wide variance in risk tolerance, which can have funny effects on planning.

"Product XYZ has issues!" Of course it does. Every product has issues. The question is, what will the team do about them? Is it worth holding up the release to fix it? Possibly, but after some number of flare ups don't be surprised if some of your colleagues want to pull back from doing XYZ at all. This must be stopped, and firmly. Remember all that research on the competitive market and customer demand? Remember the reasons that your company is well-suited to take a piece of this pie? Letting some risk-adverse colleagues block your success is a sad way to fail.

The product may have issues, but if you don't sell it, they will never be fixed. Sales are the oxygen of products, and you might as well not bother if the story isn't good enough to produce a flow of oxygen.