Friday, November 26, 2021

Total Compensation


 Career decisions are complicated, but here are a few models to work with which might help. A salaried job is more than a simple exchange of your time for their money: you will be giving it some headspace in most of your waking moments. This conversation is not relevant to contract or hourly jobs. Still, the model of exchanging time for money is a useful starting point, so let’s start with the three forms of money that you’ll be offered in a salaried position.

  • Unvested stock options are lottery tickets. 
  • Cash is cash. 
  • Titles are free.

An ownership stake in the company is a common starting point in negotiation. We will all work together to make the company grow larger and therefore every dollar in options is potentially worth thousands in future money. Likewise, every lottery ticket in the gutter was once potentially worth thousands or more. They say lotteries are a tax on people who are bad at math; stock options aren’t quite that bad, but they’re certainly not guaranteed income. The form of the ownership doesn’t really matter: Restricted Stock Units (RSUs) are better than Incentive Stock Options (ISOs) but the whole thing is silly if the company doesn’t grow. You may think the risk of stocks is reduced when a company is past its startup years. I can’t help but note that lots of employees lost out in MCI and Enron’s glorious flameouts, while my own turn of the century Intel ISOs remained essentially worthless for twenty years.

Cash, on the other hand, is immediately liquid and can be exchanged for goods and services from coast to coast. Cash comes in two amounts: More Than Enough, and Less Than Enough. 

  • More Than Enough means that you do not worry about providing your dependents with their needs. Health and life insurance is part of cash. 
  • Less Than Enough means that you worry. If you do not receive enough, then you are going to devote part of your headspace to handling this financial pressure. 
As an employee, you should never accept a salaried job that includes financial pressure. If you’re going to accept financial pressure, then you’d better be holding a double-digit ownership stake because you’re not an involved employee any more, you’re a committed owner. See the story of the Chicken and the Pig for graphic illustration of the difference between employee and owner.

The final form of compensation is title. A salaried position has responsibilities and blast radius. The scope and radius can be commensurate with the size of the organization, but it’s not a hard rule. Individuals may be hired to try new ideas, or to revitalize major functions, or to solve long standing problems. The organization’s easy answer is to fit your title into the existing system and hierarchy as another one of a role that already exists, but it’s all open to negotiation. It costs the organization very little to stick “Senior”, “Chief”, “Principal”, or “Head of” in front of your role. That extra filip of title can help you tell a better story on your resume, or it can be useless. A Vice President in a large software company might report to the CEO and lead a thousand people, or they might fly around to conferences evangelizing vaporware and control nothing more than their own expense account. Titles are free.

Those three things are the tangibles that you as a salaried employee can get, but now let’s talk about your mental and spiritual compensation. Paloma Medina’s excellent BICEPS model is a great resource for thinking about what a role does for your happiness. No matter how much money you’re making or gambling, a salaried role should meet your needs for Belonging, Improvement, Choice, Equality, Predictability, and Significance. Really, just go read the link. A salaried role is at minimum two thirds of your waking hours but realistically something more like four fifths… if you’re not getting what you need, then you’re devoting that time to a thing that leaves you unhappy. That’s not a sustainable thing to do with one's life.

One last thing to consider if you’re looking at your current role or a potential role through this lens of compensation and happiness: 

  • Change is constant, but tempo is not.

All organizations will always be changing and that change will affect your happiness, both positively and negatively. At the very least, this can be a challenge to Predictability, and for some people that’s a very big desire. The next organization you join? They’re going to alter the deal also, because the world changes and we must all respond. What is not the same across all organizations is the pace at which change is accepted and implemented. Some organizations lean into change, while some lean back. Asking how changes have been handled in the past might help you find a team that supports your needs.

Thursday, November 25, 2021

Thoughts on Logging



  1. What should go into a log? 
  2. Enough, but not too much.

Those two phrases are contradictory and context dependent. This is why logging has different levels. They may be expressed as numbers or words. Each more verbose level is inclusive of the less verbose levels. Sematext has a nice in-depth overview

I can’t count the number of times that initial log collection didn’t uncover the root cause of a problem. I’ve also disabled or crashed the target system by turning logging too high. Balance is the key.

At the one extreme, Fatal is “don’t log unless you’re dying” which is awfully presumptive that the application will be able to realize it’s dying. A more common option is Error (only log problems) or Info (log what you’re doing, not how).

At the other extreme, Trace level logging is preemptive print-debugging. Each important function in each routine, when did it start, what was the input, what was the outcome. If issuing a special build for a customer to troubleshoot is a common occurrence, or something that you wish you could do, then Trace logging is for you. If the application still performs acceptably in production with Trace on, you’re probably not logging enough. Consider putting a built-in timer into the application to shut Trace off after a set amount of time.

Each line of a log should start with the date and time. UTC time is best. Logging in the local time of the device is regrettable, but acceptable if you’ll never need to correlate the events from one device with another one in another time zone. Logging in local time will cost unexpected effort and cause problems, but sometimes it can’t be avoided.

The time format should be ISO8601. Unix epoch is not ideal because people can’t read it. The format of your favorite locale is not ideal because people will get confused. ISO8601 or GTFO.

Entities should have identifiers so they can be traced. This is pretty simple if you’re the only application of your type running on a single physical system. The thing writing the log is the entity that matters and the hostname is plenty of ID. It gets deep if you’re an elastically scaling micro service on multiple cloud providers. Is the thing writing the log what matters any more? You might even architecturally be able to say that you don’t care and it’s all going to wash out at the services level… as long as you never have to debug or do forensics.

Tasks (or threads, or forked processes, or containers) should have identifiers so they can be traced. This is a remarkably deep problem; don’t let perfect be the enemy of good. Again, your architecture might suggest this isn’t necessary, and that suggestion may be correct in many circumstances.

Each line of the log should include a single event with time, entities, and tasks all recorded up front. Some of that feels repetitive and like you would only need to emit it once per file. That would be fine if you never need to correlate with the logs from other systems in order to trace a transaction.

It may be tempting to log an entire transaction as a single event, but that presumes your application will stay alive long enough to see the transaction neatly complete. It is better to log the start and end of a transaction as separate events. 

It is also tempting to use Event IDs instead of language to describe what is happening, since “404” takes less space than “Page not found.” A human readable string should be included as well so that humans can learn your IDs by use. This also allows you to potentially translate your event ID into different languages for use by different humans. You should never use translated strings without event IDs though, this is annoying. Why? Because it makes things harder for organizations that operate in many languages. Microsoft did not keep this behavior when upgrading Windows, but customers and log analysis vendors have still needed to build separate Windows log parsers for every language they'll support.

Writing a log to a file is the obvious path, and there’s a whole world of tooling for rerouting that file output into searchable storage. The simplicity of a file or cloud bucket is a huge win; if you’re up at all, you can probably write a file; if you can’t write to a file, the reason is probably obvious and more widespread than a subtle problem in your program. However, there are also reasons to write to a structured data system. Log data is at least semistructured, and using a structured system allows indexing and search. There’s cost down this road though; either the commercial cost of a tool or service, or the performance cost of a database. Furthermore, the log consumer now has to get into the database, or your product has to provide an auditing API. In my opinion databases are rarely a good idea for log storage.

Some other thoughts on logging:

Product Manager to Product Ratio

How many chucks could a woodchuck chuck if a woodchuck could chuck wood? It depends on the structure of the organization's tech stack. A highly structured tech stack provides a format that you can build repetitive products with. Structure makes the design obvious and lets the developers work faster, which means more gets done easier. 

Every problem that you're solving is just a case of “fit this into our model and then solve it”, and you can do a lot with a little. For instance, in many systems management products the problem is to recognize endpoint state and take action, and the interface is to show state and guide to action. In many data analytics products the problem is to collect data and show informative panels like counts of types. If you make sponges, the easy use cases are all some form of "clean something". Given the commonality of these goals, a highly structured tech stack can be produced to make common tasks simple. In my experience this might be an average of three devs per product and a one to three PM to product ratio. I’ve seen a lot higher: averages of one to five and one to six in two of the teams I’ve been on.

On the other hand, a lot of structure can be constrictive, and a more flexible approach has benefits. If you treat every problem as a new solution to discover and build from first principles, maybe you’ll come up with new shortcuts to success. But you’ll also be unable to depend on preexisting structure. In my estimation you'll need at least five or six devs per product and a one to one PM relationship.

I've run over a dozen very similar products by myself and I've been taxed to capacity by a single product… it's all about how much structure the development platform provides. I'm defining product as an installable/removable module that’s complex enough to need semi-dedicated development team. A content pack is not a product.

If you’re not sure how to describe the amount of structure in your organization, you can assess it in reverse by asking about past performance. If getting a new product to usable and salable is a one quarter job for under three people, you're probably very structured. The first task is to learn the structure, and studying the existing products will help. If it's more like a year, plan accordingly and lean into product management process.