Monday, August 8, 2011

my virtualized server performance stinks!

Another recurring issue from the last ten years or so...

Short answer: You need to work with the SAN admins to get more storage IOPS, and you won't be able to give them reliable numbers from where you're sitting because SANs and VM's will both lie to the guest OS. You will be able to give results like "it takes a long time" and "errors occur" but numbers from things like bonnie++ or perfmon won't be very meaningful.

Long answer: As operating systems developed virtual memory and storage models and learned how to manage multi-processor architectures, they learned to lie to the applications about how much of a given type of resource was available, and the apps became abstracted from the hardware. A given process can look up how much RAM the OS has, but if it asks for twice that, the OS cheerily complies, using virtual pages. This led to an opportunity to drop in a second OS which treats the first OS as an application, which would let you load many light workloads onto fewer pieces of heavier hardware. Once this opportunity had been realized, the next step was to oversubscribe the hardware; virtualization systems tell each OS that it has a reasonable amount of resources, gambling that each virtual machine will not really need the requested resources at the same time. Even more complexity ensues when you realize that each guest OS is doing the same thing to its applications... Much like other oversubscription systems, such as airline ticketing and finance markets, this trick sometimes fails. A sustained heavy user such as you describe is the nightmare client for an oversubscribed market because its resources can't be swapped out to other clients. In order to prevent the whole cluster or SAN going down, the virtual OS (AKA hypervisor) or the SAN OS simply denies the request for resources. It doesn't take the time to tell the guest OS that this is happening though, it just rejects the request for more stuff.

Unpopular answer: datacenter virtualization is sold using oversubscription metrics developed from loads like DHCP/DNS or Intranet servers. When given a data-heavy load, the model breaks and the customer is forced to realize that they're not really going to get a 3:1 ratio. Heavyweight systems don't have to have physical hardware, and they don't have to give up on the other benefits of virtualization, but they really do have to have dedicated hardware resources and they won't get oversubscription from virtualization unless they're willing to put up with this kind of performance problem. Virtualization can provide a number of benefits outside of hardware consolidation, though -- there's huge benefit from cloning, snapshot, monitoring and management tools. Just gotta make sure that you're basing the cost-benefit projections on the right benefits.

Friday, July 15, 2011

The Interview Question

There are many things you're after in an interview, after the all-important chemistry question. However, I've found, forgotten, and remembered a question that works well to uncover a lot.

If you're hiring for a sales engineer, services engineer, senior customer support, or architect level position, there's a cluster of fundamentally critical abilities that we'll call "getting the big picture" for now.

  • They need to understand the product in full as well as in part
  • They need to understand their own team dynamics
  • They need to understand customer personalities
  • They need to be able to hold their own in a meeting, possibly a contentious one with angry customers
  • They need to be able to avoid unnecessary disclosure
  • They need to keep groups of people on track to a common goal
  • They need to be able to keep a goal in mind while communicating about other things
  • They need to be able to gauge their audience's interest and engagement
  • They need to be aware of how much time they're spending
  • They need to see enough context to  keep the goal in mind no matter what they're being asked

The question: "Describe in as much detail as you feel comfortable using what happens when a web browser goes to the Internet and renders a page." The details aren't important -- the important thing is to make them talk about a simple task in broken down terms.

Some grades:

  • A player: Outlines the process, gives a couple of details about each steps, and winds it up in a minute or two with "I could go on for hours, but...". Assuming no other red flags come up, this person's got my recommendation.
  • B player: Starts out with that sort of thing in mind, starts to get detailed, but has the presence of mind to snap out of it. Given an attentive manager, this person could do just fine, but I don't want to put them in solitary command of anything important.
  • C player: Dives into the first rat hole they can and gets stuck, then winds it up with "I'll have to get back to you on that". This speaks to a lack of experience and perspective that is concerning. They'll need more than a manager, they'll need a sidekick to take care of them. Not that there's anything wrong with that per se, in the proper environment they may do just fine. But, they're not suited for talking to customers.
  • F+ player:  Dives into the first rat hole, produces a shovel and starts making it up as they go along. Wonderfully creative and I may look forward to reading their novel, but they'll need to look elsewhere to pay the bills.

Sunday, July 10, 2011

iPhone(TM) vs *droid

Switching environments is always a good time to reflect on their relative merits... I've just left Android for iOS and have some thoughts on the matter.

First, the timeline... a loyal Blackberry user through the 2000s, I was used to a few basic concepts like "email in my pocket" and "talk on the phone". Apps like mapping and chat and location aware searching gradually entered, but every one of them served to highlight the platform's shortcomings in terms of speed and stability. AT&T in the Bay Area wasn't doing me any favors either... it's tough to tease out where network problems end and phone problems begin, but by August of 2009 I couldn't rely on the phone for talking or email. I left LANDesk for BigFix that month, and I happily shipped my Blackberry 8800 back to Utah and went to Verizon for a Blackberry 9000. Network performance was better, but the phone was physically cheap, no faster than any older BBerry, and far less stable. I gave up and replaced it with an original Droid.

The Droid had some rough edges to it, but it was excellent in four respects: I could reliably talk on the phone, read email (with attachments even!),  use the Internet, and install cool apps. When I found Tasker, I was truly in love with the phone. Screen widgets are pretty cool, and I really liked the quality hardware build, but powerful automation was the real hook. There are many examples on Tasker's website, but here's a few things that I learned to take for granted:

  • My phone will not make unnecessary noise during an important meeting, and I don't have to flip a switch to make that so.
  • My phone is loud when I'm outside, and quiet when I'm inside, and I don't have to switch profiles manually.
  • My phone diverts calls when I'm busy or from people I don't know to voicemail and transcribes their message for me to read. All I have to do is leave it on the table in front of me and the screen lights up with the message.
  • My phone turns on, unlocks the screen, and starts playing music when I plug in headphones or a car stereo. It pauses the music, locks the screen, and turns off the display when I disconnect.
  • My phone is automatically silent at night, unless there's an emergency call.
  • Bluetooth is turned off if I'm not in my car, and turned on when I am.
  • Wifi is turned off if I'm not at home or work, and turned on when I am.
However, all was not perfect... the Android OS has its issues, and speed is a big problem. All of these awesome features worked fabulously, one at a time. However, receiving a phone call while it was trying to do something else could really throw it for a loop... nothing says "smart phone" quite like:

  • pressing the camera button and waiting 5 seconds for a picture to happen, and another 10 for control of the phone to return to you
  • struggling to answer a call with a touchscreen that won't respond to your touch
  • killing your battery 3 hours into the day (hello, GPS navigation). 
I learned to avoid the cool features in order to save hardware resources for the basics. One home screen, one widget, minimal apps. OS upgrades came, and while each one brought some nifty new feature or promised a new level of stability, those promises didn't really pan out. By the time I'd carried it for a year, I was ready to root it and install a GPA build of Bugless Beast.

In many ways, this move was analogous to the Blackberry 8800>9000 transition... one last chance to see what the platform could do with the latest OS. Sure enough, performance of single tasks improved, and the phone could now often do two things at the same time. Best of all, it would go for a whole 24 hours between charges. However, it also lost the ability to receive email by push, needed to reboot every couple of days, struggled to maintain network connectivity, and occasionally would just reboot itself, no questions asked. It was a lot like having a Linux laptop again... manually kicking off tasks, troubleshooting bizarre behavior, and learning a lot more about the internals than is maybe necessary. So, last week I went to Verizon to see what there was to see in terms of next generation hardware.

It's not looking good for *droid. I looked at the X, Incredible, Thunderbolt, and Charge. They're all more or less the same phone, with tweaks here and there. It's a flimsy feeling, plastic device with a huge screen and a fast processor. It has a good camera. Some of them have a front-facing camera, maybe even with software to use it. But the real work of differentiation has been devoted to breaking compatibility with the core operating system. Motorola has Blur, HTC has Sense, Samsung has TouchWiz... all of them are fantastically annoying and if you just want Android and Tasker, you'll need to pony up twice the cash and go to a Nexus S... which is the same damned piece of hardware! Or root it again... another weekend gone. To draw an analogy, home market PCs were very much like this in the late 90's, when Compaq, Packard Bell, Dell, and IBM were slugging it out in a race to the bottom that only Dell would walk away from... because only Dell shipped plain old Windows on hardware that wasn't drastically broken by design.

I did some research and ruled out the Motorola phones because of locked bootloaders, then picked a Charge because it had the best screen and didn't have that stupid kickstand. The screen was awesome, the processor was fast. Taking a picture was like using a regular point and shoot camera. The phone was half full of bloatware that I couldn't remove. Let's Golf? Rock Band? An IM client that bridges IM into SMS? WTF? Half of my clever Tasker profiles didn't work any more because Samsung had removed stock Google applications and replaced them with their own versions that didn't work the same way. No more ability to automatically control the display and screen lock state... the forums say this is Gingerbread's fault, but it wasn't a problem with Gingerbread on my original Droid.

Then, music... I really like to listen to music in the car and at work. I don't do much Internet streaming, and I don't do playlists... I just like to load my phone with good music and hit shuffle all. Then I expect to see cover art, Artist/Album/Song tags, and a couple of control buttons. Apparently this is a really high bar to hit, because the default TouchWiz music player had no way to do it. I downloaded half a dozen alternate music players from the app store, and they wouldn't play MP3s at all. Really? Reboot, 1/3 of my music plays. Reboot again, none of it plays. Reboot again, all of it plays... until the phone freezes up a few hours later. It's just like Linux on a laptop... everything you try fixes some of the old problems while introducing some new ones. Did I mention that whenever it reboots the BIOS plays a long Michael Bay sound effect at the top of its very loud volume, and that there's no way to shut that up? That's awesome when it's randomly rebooting itself every couple of days. So, I exchanged. I did my research and found just as many stability and bloatware complaints around the Thunderbolt and Incredible... so I went for an iPhone 4.

I miss the awareness and capability of Android. I miss the ability for a weather icon to tell me something (silly in the Bay Area's eternally balmy climate, but why show me a weather icon at all if it isn't accurate?) I miss the Google traffic widget. I miss the computer-less, entirely OTA capabilities of Android. I miss the huge wad of time that it took to import my music into iTunes and sync it to the phone. However, I really like having a phone that just does what it says on the box, reliably and quietly. Through two Blackberries and two Android devices of mine, my wife has had an iPhone 3GS and an iPhone 4. The 3GS was just fine up until the end of its contract. Both phones have just quietly worked for voice, messaging, Pandora, social networking, news browsing, and baseball game streaming. No fuss, no hassle... there's a lot to say for limiting features to the ones that work with certainty.

Monday, July 4, 2011

User Interface and Experience: How Important Are They?

UI and XP design and development are undeniably a big deal, but they're only part of the problem. What size part? That depends on your sales model. For a consumer product designed to sell itself via the web, it's well over half of the problem. UI and XP are pretty much the only differentiators for smartphone social media clients, for instance. 

For an enterprise product designed to be carried in with a professional sales force, UI and XP are under half of the problem of how to design the stuff. Of course it isn't a black and white world, so let's just call the user story "As a prospective buyer, I want a kickass UI and XP" an even 50% of the product design problem.

The other half of the problem hits three points, because I can only hold three things in my head at once.

  1. "why do I want your product?"
  2. "why will I buy your product?"
  3. "why will I use your product?"

Dealing with "want" separately from "need" or "use" or "buy" is preparing the part of your product that will be used by marketing and sales for positioning. Some examples:
  • Is your product an aspirational product? The Ferrari that someone might buy if they hit it so big that money isn't an object? 
  • Or is your product an exclusive product? You don't need a Hole Hawg until you need one, and when you need one you'll probably need to ask someone older and smarter what you need to buy in order to tackle your problem. 
An interesting thing about these two examples is that the UI is utilitarian at best, and the XP is "pros won't kill themselves with it." There's another post in there... but for now, let's settle on a simple observation: the UI and XP needed for a product are dependent on the market and audience for that product. If you defer thinking about UI and XP until after you've designed and built the product, you could miss the target market and need to go back to the drawing board. If you assume a UI and XP paradigm before thinking about your customer, you could totally miss their expectations. Much better to decide who you're selling to and how you're selling it before you build it.

There are obviously more product position types than these, but Aspirational and Exclusive products are a good goal to design for when first getting started. For one thing, their emphasis on function over form can potentially translate into saved development time. For another, early adopters are more forgiving of usability quirks. Best of all, aspirational and exclusive products practically sell themselves during their heyday. Their customers know that they want them, it's just a matter of helping the customers afford them. The real challenge is prolonging that heyday through product design, marketing maintenance, and ethical behavior. Does the product consistently add value? Are people happy to use it? Are they happy to interact with your company? These are the things that decide how long successful products will last.

Thursday, June 30, 2011

at least once a week

Every internal technical email list I've ever been on has had one of these conversations:

A) module XYZ didn't work by itself, what's up?
B) superstition and voodoo, XYZ will EAT YOUR SOUL.
C) did you remember to click the button?
A) seems to work okay when you click the button, but maybe I'd better have some backup in case it EATS MY SOUL.

It really doesn't matter how the button is documented or positioned, once engineer B has been burned, it's game over for product XYZ. Since B is a well-respected senior staffer, everyone else listens to them, and it's up to C to counter the voodoo.

And that is why C must always monitor internal technical email lists very closely.

A closely related issue is the presence of A, AKA the FNG. If it weren't for new blood giving things a try, XYZ would be dead in the water. Something to think about when evaluating the performance of B.

Friday, May 20, 2011

GTFOffa my lawn has me thinking about a conversation I had with a friend last week... The gist is that easier entry levels mean lamer entrants. Oh, we can spin it so many ways... cheaper entrants! Entrants who aren't wasting their time on unimportant details! Or we can question what they know if they go from throwing luggage to sysadminning.
All together it makes me think I would like all these kids off of my lawn.

Tuesday, February 8, 2011

What Technology Wants

Kevin Kelly's What Technology Wants was a pretty good read, but I found myself disagreeing on a fundamental point... I don't like that it treats technology as something separate from humanity. Perhaps I'd feel differently if I hadn't read The Artificial Ape: How Technology Changed the Course of Human Evolution first, but I did. That book's argument that humanity is ipso facto a product of our technology as well as a creator of it is very strong. Without technology we are a failed experiment, unable to even eat successfully; with it we are the dominant species, and can allow evolutionary flaws to propagate without consequence. Instead of a truly solid book, What Technology Wants ends up as an interesting collection of factoids and anecdotes. Furthermore, it seems conflicted; Kelly has obviously thought a lot about this and seems to have reached similar conclusions; the chapter on the Unabomber is clear, and there are continual references to the willingness of poor people to trade anything for more technology. I suspect that compromises were made for publication.