Imagine the situation — a group of friends, some programmers, perhaps one salesman and one artist, venture together to found a company for a new great tech product. They work for a bit as equals, without any strict system of conduct, guided by feel or by previous experience, held together by bonds of friendship and common gain. This situation lasts for a few months, up to a couple of years. Eventually they begin hiring new people, old friends start to move on to found new ventures, other novices "graduate" to become more experienced. The team grows and suddenly "going by feel" and being held together by just bonds of friendship becomes insufficient.

A system of governance has to be instituted. In a group of friends, it's a very dreaded proposition, naming the djinn that's got out of the bottle, and the g-word is often avoided. Especially because the g-word is very often followed by a p-word that's got even worse connotations: "politics".

A group of friends of our story is in essence an egalitarian, first-generation, consensus-based commonwealth. Egalitarian, because no one friend ought to think of himself as a better of any other one by the definition of the word "friend". First-generation, because all of the friends will usually remember their previous occupation: unemployment, rigid and inhuman strictness of a corporation, or any other hell that they were in previously — and we're assuming it was not pleasant because otherwise why venture into the stormy waters of entrepreneurship, when the alternative is a safe and relatively pleasant job? — and they are determined not to stray into that territory again. Consensus-based because ideally they are aiming to please each one of the group, or at the very least not dramatically upset any one of the group (in the latter case we would call them consent-based). And a commonwealth because they are primarily working towards making their entire group succeed, and their loyalty lies with each other and nothing else in particular in the outside world.

Unfortunately those who do not study history are doomed to repeat it and we know very well from history that flaws in a system of governance show up most evidently after the first generation of its founders passed away. The office of the Roman emperor, held first by the radiant figure of Augustus Caesar, degenerated into the incompetent and tyrannical reign of Caligula mere two generations later. Likewise, it is very possible that once our commonwealth of self-organising team of specialists will degenerate into something unworkable once new people join — those which were not included in drawing up the initial vision of the project and might not be as invested in it as the rest.

Once the team grows, there are two ways it can go. Either it will degenerate into a egalitarian committee with no accountability or responsibility, or it will develop some kind of hierarchy. If there is no accountability or responsibility, we are past the point of fixing anything and the team will simply dissolve and the business will go bankrupt. Therefore, we must embrace the hierarchy.

Before we go any further, I would like to explain what I mean by hierarchy. Hierarchy in this context should be understood as the structure of accountability and responsibility. Anywhere where there is someone to whom another is accountable or where there is someone for whom another is responsible, there is a hierarchy. It does not have to be explicit and it does not have to be rigid, indeed very often members of a hierarchical group might not notice them and simply think that the group is egalitarian. However, if there is responsibility for work done by someone else, there cannot be equality by definition. Hierarchies are good, because they let you orient yourself. They tell you which way is up, who to follow, respect and emulate; and who to guide, teach and protect.

Once we have a hierarchy, there are further two ways which they can be categorised (or rather, two directions which they can tend towards), and they are based on the flow of initiative. A hierarchical organisation can either be goal-oriented or process-oriented. In a goal-oriented organisation, initiative flows from the top and is amplified as it goes down. In a process-oriented organisation, initiative flows from the bottom and is dampened as it goes up.

A goal-oriented organisation is like a fancy restaurant. You have a head chef, a bunch of cooks, some waiters, maybe a bartender. As a customer visiting such a restaurant, the only thing you are really interested in is receiving a nice meal. The goal of the organisation, its raison d'être is to get that meal to you, preferably in an elegant and quick fashion and with a nice cocktail to go along with it.

The waiter will take your order, along with any potential personal modifications you might add to make it suit your taste. The chef will order her cooks to prepare the meal for you, and any cooks who show initiative that could make the meal better will likely be rewarded. The bartender will personally consider the ingredients for the cocktail and determine whether it should be shaken or stirred. All in order to fulfil the goal: make the best meal and in the process please you, the customer.

A goal-oriented organisation, like the army, has a chain of command — with a simple, singular goal at the top. It allows a large amount of flexibility when it comes to accomplishing that goal. It's been perfected by the military, and our example of a restaurant is really no different than a military general ordering his army to take a castle. The general does not particularly care how the castle should be taken, as long as it is. The subordinate commander has the autonomy to decide upon the means as long as the end is as ordered.

The issue with goal-oriented organisations is that you need leaders at all levels of the hierarchy who can creatively solve problems and who have the charisma required to get others to follow them to accomplish a goal. Also, giving them too much initiative too early might cause analysis paralysis and make them afraid to make decisions of their own.

A process-oriented organisation is, in turn, like McDonald's. Every meal is always made according to the same recipe and there is no room for any variation. The entire chain—from cow to sandwich—is invented, described and followed to the letter. And if the process does not suit the goal, the worse for the goal.

An organisation dominated by process evolves (or devolves, depending on your point of view) from a goal-oriented one as a natural attempt by exceptional commanders of the latter to write down and document the factors that led to the initial success. This is quite natural (that's why we have the Bible and why the final guru of the Sikhs is a book), so where in the olden days exceptional generals would write down memoirs and rules of military doctrine, so too these days in corporations leaders will write down documentations and processes that can ensure proper functioning of a team in the event that they climb the corporate ladder high enough that the team will no longer be under their direct command, or they change jobs, or something else happens. Indeed we see that these days processes proliferate at a much faster rate than military doctrines of old.

If a process gets codified and starts being applied, eventually it reaches a situation that was not thought about in the initial draft. Every process, then, must include a meta-rule that is true for every process-oriented organisation under the Sun: "if you don't know what to do, call your manager". Thus, a process-oriented organisation implements a chain of appeal — with a single decision-making oracle at the top. In an organisation that is perfectly process-oriented, the oracle does not really have any initiative of her own, only deciding how a process should be applied.

Both approaches have their advantages and disadvantages, of course. The goal-based approach is naturally deficient in the absence of exceptional individuals who have the courage and ability to think for themselves and execute their plans effectively. Those qualities are difficult to teach and always in short supply. Not everyone is a born leader. Indeed, most people are not.

Therefore organisations will often resolve to processes that can substitute for good leaders for the purposes of scale. Indeed, after Clausewitz: "principles, rules, prescriptions, and methods are conceptions indispensable to a theory of the conduct of war". It takes a lot less skill and energy to be an oracle than to be a commander.

On the other hand we must exercise caution so as not to be dominated by process. It is vital that the process is continuously examined and refined, because ultimately process is supposed to serve those who use it.

I'm sure that anyone who has ever had to do deal with the bureaucratic machine of whichever state he lives in knows what I'm talking about. Likewise with internal access-control systems and privilege-granting processes of corporations and such — they are required to dampen initiative of those who want to use them for something they're not supposed to do, but they get in the way of those who are qualified and must do something unusual.

Aristotle in Politics claimed that no one form of government is good. Whereas he considered that mixed constitutions should be applied so that no one group abuses its power, in a modern business environment we must pick and choose from both goal- and process-based organisation schemata to ensure efficient allocation of resources. And not only that, we are spoiled for choice when it comes to discriminating between different pre-packaged processes, too. Scrum, Kanban, XP, and a whole bunch of other buzzwords compete for our attention. Some miserably pick one and try to follow it to the letter and suffer through its problems with the hope that once the right rituals are performed, the promise of effective development and efficiency will be fulfilled.

Of course that rarely happens because no one process is tailored to suit every product and every organisation. As said previously, processes have to be continuously examined and refined, mixed and matched, to achieve optimal results. This is not an easy task, though, and in practice it is not immediately obvious why it should be done until it is already too late.

"Too late" is when processes which are not examined and refined become traditions. This is not in itself bad, but it obfuscates the initial reason for why a process was established. Issues start when traditions are called to be rejected. Traditions become bad when the conditions under which they evolved rapidly change, in which case rejecting them is good. However, rejecting traditions is also dangerous, because they might have unforeseen second, third, or n-th order effects. Traditions are an evolutionary mechanism of preserving knowledge, but it should be seen as a last resort in short-term projects.

(As a brief aside: from this follows a controversial statement, that "because we've always done it this way" is, in absence of a good, a valid reason to continue doing it — as in the tale of Chesterton's fence. Said another way, just because one does not see utility in a tradition, it does not mean that the utility is not there.)

As usual, the oft-overlooked medieval ways are best, if not for clear-cut solutions then at least for inspirations on figuring them out on your own and applying them to modern times. The middle ages are characterised by fairly large decentralisation, and it seems like applying such decentralisation to governance is a valuable exercise. And indeed, we find such an example in the structure of medieval guilds.

Medieval craft guilds were strictly hierarchical, but mostly on just three levels. In a way, one can think of these as a large collection of tiny hierarchies, with a number of journeymen and apprentices working under the guidance of a master. The progression upward is, at least in principle, very clear and very effective. Since these days programmers strive for the label of "software craftsmen", it might be productive to consider the application of this model in modern software development cycle.

The progression of a craftsman in a medieval craft guild started at the level of apprentice. An apprentice lived with his master and assisted him with the job, learning in the process. In exchange, the apprentice was provided room and board and, after a certain amount of time, also a salary. The value of good mentorship has not gone away and we still consider it important in modern software development. What we sometimes forget is the personal relationship of an apprentice to his mentor, and through it, the increased effectiveness of learning.

Once an apprentice reached a certain level of experience, he graduated to the level of journeyman. Journeymen were craftsmen with a certain level of skill, who have accomplished themselves under one master, and should now venture out into the world to find out more about other guilds and their processes. They travelled through the land, learning the skills of other guilds (or other chapters of the same guild, in rare cases) and contributed to their own development.

Once a journeyman learned enough, he might return to his own home guild and complete a particular work of his own — their master piece — that would then be judged, along with his previous conduct and experience, by the rest of the masters of that guild to decide whether he should be admitted as a master of the guild.

From this we can see certain patterns for each of the levels. Apprentices studied under the eye of the same master in relatively invariable conditions — with the same mentor and the same material for most of their education. Journeymen moved around quite a lot, learning enough to eventually consider themselves good enough for mastery. Masters, much like novices, settled down and tutored new apprentices while crafting a particular work.

This gives us two axes along which governance of a project or an organisation can be structured. The first axis is one of goal and process, which directs the flow of initiative, the other is the axis of change, which directs the flow of people between small hierarchies dedicated to specific works (in the software world - specific projects).

In business, one cannot build an organisation that is entirely oriented towards process or towards goal, at least not one that is going to turn a profit. The further towards the process end of that axis, the more rigid, apprehensible and inefficient it becomes. In contrast, the further towards the goal end, it is more flexible, opaque and reactive.

The second axis operates in terms of experience in a particular hierarchy. Here, those at the ends - novices and masters - should stick to one project and one work. Novices want to stay in an unchanging environment so they can learn the ropes, whereas masters should remain as they have the most vision, insight and understanding of the quirks of a particular project that seem completely invisible to the less experienced. Journeymen, on the other hand, those aspiring to become masters, should want to move around as much as they can and learn from different places.