Archive for April 2009
Do you now bear responsibility for the architecture of some code? I’d like to talk just with you.
I’d like to say that I think it’s really important to “listen and then lead”. You’re responsible to the customers of your code for its quality, and you’ll get a lot of pressure to “just do this” or “just do that”. The processes you put in place, to control the quality of contributions, are for the benefit of its users and the code as a whole—to maximize usefulness, not necessarily to maximize development.
Sometimes it’s okay, even necessary, to inconvenience a few people (including the contributors) in order that you don’t fail an even larger number of people. You’re not (at least I wasn’t) able to please everyone, and trying to do so can easily cause harm, both short term (immediate bugs) and long term (technical debt and bad architecture). It’s your, just your, job to balance all this and do what’s right overall, and it’s very much a judgement call.
Some of this might be replaced by a change control board’s decisions, but if you go that route, then you must have a change process that everyone follows, regardless of the demands of the moment. Some people are really slick and sophisticated in getting what they want, especially when it doesn’t hurt them if they end up creating a problem. (Heh, the more I reread that the more it sounds like parenting.)
I found I had to be confident enough to implement decisions and defend them, but flexible enough to shift policy and correct errors. On the plus side, I think these skills will help move you past your current position, and help you control the direction in which your career heads.
You’re doing pretty well if you don’t instantly take sides in conversation on the merits, which is better than I sometimes do. Even if you’re slow making decisions—and there is so much to learn—a little slow is ok, since it gives a chance for more reflection and more mature ideas. Don’t be afraid to be wrong: if you never make mistakes, never make a bold architectural decision, then you’re playing it too safe, and you could probably be replaced with a small shell script.
Take care and good luck!
This post quotes, collects, organizes, and remixes several sources to provoke comparison and consideration. There’s not a lot of new theory, beyond my idea that this stuff belongs together in the first place.
A change in status
In 1985 Peter Naur described, in Programming as Theory Building, where we still are as a profession (emphasis mine).
[M]uch current discussion of programming seems to assume that programming is similar to industrial production, the programmer being regarded as a component of that production, a component that has to be controlled by rules of procedure and which can be replaced easily. Another related view is that human beings perform best if they act like machines, by following rules, with a consequent stress on formal modes of expression, which make it possible to formulate certain arguments in terms of rules of formal manipulation. Such views agree well with the notion, seemingly common among persons working with computers, that the human mind works like a computer. At the level of industrial management these views support treating programmers as workers of fairly low responsibility, and only brief education.
Here’s where he suggests we be.
On the Theory Building View the primary result of the programming activity is the theory held by the programmers. Since this theory by its very nature is part of the mental possession of each programmer, it follows that the notion of the programmer as an easily replaceable component in the program production activity has to be abandoned. Instead the programmer must be regarded as a responsible developer and manager of the activity in which the computer is a part. In order to fill this position he or she must be given a permanent position, of a status similar to that of other professionals, such as engineers and lawyers, whose active contributions as employers of enterprises rest on their intellectual proficiency.
Further, he comments on necessary changes to programmer education.
The raising of the status of programmers suggested by the Theory Building View will have to be supported by a corresponding reorientation of the programmer education. While skills such as the mastery of notations, data representations, and data processes, remain important, the primary emphasis would have to turn in the direction of furthering the understanding and talent for theory formation. To what extent this can be taught at all must remain an open question. The most hopeful approach would be to have the student work on concrete problems under guidance, in an active and constructive environment.
An active and constructive environment
It occurs to me that the Master Craftsmanship Teams described by Bob Martin comprise distinct, guiding roles, in a context of “concrete problems”, to help us create this “active and constructive environment”. Bob proposes a static structure (remixed below, original emphasis) though which a person progresses (or not) through a career, but in my experience this classification applies between a person and each of many projects.
Apprentice. The apprentices write a significant amount of code—most of which is discarded by the journeymen.
Journeymen. They can make short term tactical decisions, and direct their apprentices accordingly. At the same time they take technical and architectural direction from the master…. The journeymen write a lot of code – almost as much as the master. They also throw away two thirds of the code written by the apprentices, and drive them to redo it better – always better. They teach the apprentices how to refactor, and often pair with them and sit with them while reviewing their code…. When an apprentice delivers code that the journeyman finds acceptable, the journeyman will commit that code to their main line. Otherwise he sends it back to the apprentice to do over…. The Journeymen see every line of code written by their apprentices, and they write a significant amount themselves. Pairing is prevalent though not universal. Anyone on the team can pair with anyone else. It should not be uncommon for the master to pair with the apprentices from time to time.
Master. The master probably writes more code than any of the team members. This code is often architectural in nature, and helps to establish the seed about which the system will crystalize. The master also sets technical direction and overall architecture, and makes all critical technical decisions. However, the chief job of the master is to relentlessly drive the journeymen towards higher quality and productivity by establishing and enforcing the appropriate disciplines and practices…. By the same token the master accepts subsystems from the journeymen and, if acceptable, commits it to the master main line…. The master sees every line of code in the system.
There’s a lot of code written by this team, but a pair of expert eyes reviews all of it. This is manageable if the scope of the work is not too large: my build systems weigh in at under 10,000 lines of (possibly too) intricate and clever shell scripts, makefiles, and flat data files. As with any software system, technical debt is the biggest hurdle to efficiently adding new code. In my case most of the technical debt was revealed by a new point of view (aspect-oriented and functional programming), and fundamental mechanism of customization (sed-scripts rather than sets of macros with varying parts), added only two years before I left my ten-year project.
Ethics and professional responsibility
If software craftsmen would be professionals (note above how Naur separates us from engineers), we should compare existing computer and information science codes of ethics, for example a 1998 Software Engineering Code of Ethics and Professional Practice (SECEPP), to a draft Software Craftsman’s Ethic (SCE, in bold face). Here I roughly group practices by similar intent: the last two groups catch all unmatched practices. (In the remainder of this post, only the italicized paragraph tags are my words.)
Follow process. We follow our chosen practices deliberately, to ensure quality in our work. We adopt development processes to suit our own unique abilities and talents. Ensure an appropriate method is used for any project on which they work or propose to work. Work to follow professional standards, when available, that are most appropriate for the task at hand, departing from these only when ethically or technically justified. Ensure that clients, employers, and supervisors know of the software engineer’s commitment to this Code of ethics, and the subsequent ramifications of such commitment.
Continue to learn. We are skill-centric rather than process-centric. We build competency in a broad range of languages and technologies to meet the needs of our customers. We achieve technical and design excellence through continuous and deliberate learning and practice. We desire to remain Software Craftsmen and write code for our entire careers. We value long lived technologies and languages and are wary of proprietary tools. We are generalists. Not knowingly use software that is obtained or retained either illegally or unethically. Strive to fully understand the specifications for software on which they work. Ensure adequate documentation, including significant problems discovered and solutions adopted, for any project on which they work. Further their knowledge of developments in the analysis, specification, design, development, maintenance and testing of software and related documents, together with the management of the development process. Improve their ability to create safe, reliable, and useful quality software at reasonable cost and within a reasonable time. Improve their ability to produce accurate, informative, and well-written documentation. Improve their understanding of the software and related documents on which they work and of the environment in which they will be used. Improve their knowledge of relevant standards and the law governing the software and related documents on which they work. Improve their knowledge of this Code, its interpretation, and its application to their work.
Protect reputation by accepting responsibility. We build our reputations on delivered quality. Our livelihoods and reputations are interdependent. We are proud of our work and the manner of our work. We take responsibility for our work by signing it. We are proud of our portfolio of successful projects. Accept full responsibility for their own work. Not promote their own interest at the expense of the profession, client or employer. Recognize that violations of this Code are inconsistent with being a professional software engineer. Recognize that personal violations of this Code are inconsistent with being a professional software engineer.
Network to sow the field. We respect master craftsmen as the key to Software Craftsmanship. We will help other craftsmen in their journey. We can point to the people who influenced us and who we influenced. We recommend other craftsmen, hanging our own reputation on their performance. We work in a community with other craftsmen. Not unjustly prevent someone from taking a position for which that person is suitably qualified. Extend software engineering knowledge by appropriate participation in professional organizations, meetings and publications. Support, as members of a profession, other software engineers striving to follow this Code.
Clean code. We write code that is easy to read, understand and modify. We put predictable and forgiving user interfaces on our software. Be accurate in stating the characteristics of software on which they work, avoiding not only false claims but also claims that might reasonably be supposed to be speculative, vacuous, deceptive, misleading, or doubtful.
Test and maintain. We design software for testing and maintenance. We build long-lived applications and support them throughout their life. We automate every task possible (especially testing). We drive our development with tests. Ensure adequate testing, debugging, and review of software and related documents on which they work. Treat all forms of software maintenance with the same professionalism as new development.
Strive for perfection. We deliver our work to our customer without known defect. We value quality over feature set or delivery data. We consistently exceed and raise our customer’s expectations of quality software. We build in quality to the software we create. Take responsibility for detecting, correcting, and reporting errors in software and associated documents on which they work.
Plan tradeoffs. We say no to prevent doing harm to our craft and our projects. Strive for high quality, acceptable cost and a reasonable schedule, ensuring significant tradeoffs are clear to and accepted by the employer and the client, and are available for consideration by the user and the public. Ensure realistic quantitative estimates of cost, scheduling, personnel, quality and outcomes on any project on which they work or propose to work and provide an uncertainty assessment of these estimates. Ensure realistic quantitative estimates of cost, scheduling, personnel, quality and outcomes on any project on which they work or propose to work, and provide an uncertainty assessment of these estimates.
Respect customers. We deliver what our customers really need, not just what they ask for. We build trust and respect in long term relationships with our customers. We take our customer’s needs as seriously as they do. We desire demanding users. Work to develop software and related documents that respect the privacy of those who will be affected by that software. Ensure that specifications for software on which they work have been well documented, satisfy the users’ requirements and have the appropriate approvals.
Promote coworkers and colleagues. We take on Apprentices and put them in an environment where they can learn our craft for themselves. Ensure that software engineers are informed of standards before being held to them. Attract potential software engineers only by full and accurate description of the conditions of employment. Offer fair and just remuneration. Not ask a software engineer to do anything inconsistent with this Code. Encourage colleagues to adhere to this Code. Assist colleagues in professional development. Credit fully the work of others and refrain from taking undue credit. Review the work of others in an objective, candid, and properly-documented way. Give a fair hearing to the opinions, concerns, or complaints of a colleague. Assist colleagues in being fully aware of current standard work practices including policies and procedures for protecting passwords, files and other confidential information, and security measures in general. Not influence others to undertake any action that involves a breach of this Code.
Demonstrate competence. We accept that different craftsmen are better suited for one kind of job over another. Provide service in their areas of competence, being honest and forthright about any limitations of their experience and education. Ensure that they are qualified for any project on which they work or propose to work by an appropriate combination of education and training, and experience. Only endorse documents either prepared under their supervision or within their areas of competence and with which they are in agreement. Assign work only after taking into account appropriate contributions of education and experience tempered with a desire to further that education and experience. Not unfairly intervene in the career of any colleague; however, concern for the employer, the client or public interest may compel software engineers, in good faith, to question the competence of a colleague. In situations outside of their own areas of competence, call upon the opinions of other professionals who have competence in that area.
Manage projects. We keep projects from spinning in circles. We don’t wait for complete definition before beginning. We never allow ourselves to be blocked. Identify, document, collect evidence and report to the client or the employer promptly if, in their opinion, a project is likely to fail, to prove too expensive, to violate intellectual property law, or otherwise to be problematic. Ensure proper and achievable goals and objectives for any project on which they work or propose. Ensure good management for any project on which they work, including effective procedures for promotion of quality and reduction of risk.
Own code. We believe the code is also an end, not just a means. We treat software like capital. Ensure that there is a fair agreement concerning ownership of any software, processes, research, writing, or other intellectual property to which a software engineer has contributed.
Represent the public interest. Moderate the interests of the software engineer, the employer, the client and the users with the public good. Approve software only if they have a well-founded belief that it is safe, meets specifications, passes appropriate tests, and does not diminish quality of life, diminish privacy or harm the environment. The ultimate effect of the work should be to the public good. Disclose to appropriate persons or authorities any actual or potential danger to the user, the public, or the environment, that they reasonably believe to be associated with software or related documents. Cooperate in efforts to address matters of grave public concern caused by software, its installation, maintenance, support or documentation.Be fair and avoid deception in all statements, particularly public ones, concerning software or related documents, methods and tools. Consider issues of physical disabilities, allocation of resources, economic disadvantage and other factors that can diminish access to the benefits of software. Be encouraged to volunteer professional skills to good causes and contribute to public education concerning the discipline. Identify, document, and report significant issues of social concern, of which they are aware, in software or related documents, to the employer or the client. Identify, define and address ethical, economic, cultural, legal and environmental issues related to work projects. Temper all technical judgments by the need to support and maintain human values. Help develop an organizational environment favorable to acting ethically. Promote public knowledge of software engineering. Obey all laws governing their work, unless, in exceptional circumstances, such compliance is inconsistent with the public interest. Express concerns to the people involved when significant violations of this Code are detected unless this is impossible, counter-productive, or dangerous. Report significant violations of this Code to appropriate authorities when it is clear that consultation with people involved in these significant violations is impossible, counter-productive or dangerous.
Deal fairly. Use the property of a client or employer only in ways properly authorized, and with the client’s or employer’s knowledge and consent. Ensure that any document upon which they rely has been approved, when required, by someone authorized to approve it. Keep private any confidential information gained in their professional work, where such confidentiality is consistent with the public interest and consistent with the law. Accept no outside work detrimental to the work they perform for their primary employer. Promote no interest adverse to their employer or client, unless a higher ethical concern is being compromised; in that case, inform the employer or another appropriate authority of the ethical concern. Be careful to use only accurate data derived by ethical and lawful means, and use it only in ways properly authorized. Maintain the integrity of data, being sensitive to outdated or flawed occurrences. Maintain professional objectivity with respect to any software or related documents they are asked to evaluate. Not engage in deceptive financial practices such as bribery, double billing, or other improper financial practices. Disclose to all concerned parties those conflicts of interest that cannot reasonably be avoided or escaped. Refuse to participate, as members or advisors, in a private, governmental or professional body concerned with software related issues, in which they, their employers or their clients have undisclosed potential conflicts of interest. Ensure that software engineers know the employer’s policies and procedures for protecting passwords, files and information that is confidential to the employer or confidential to others. Provide for due process in hearing charges of violation of an employer’s policy or of this Code. Not punish anyone for expressing ethical concerns about a project. Avoid associations with businesses and organizations which are in conflict with this code. Not give unfair treatment to anyone because of any irrelevant prejudices.
The thing that struck me most about this article is how closely it matched the way my own team of coworkers, who came and went, contributed and moved on, helped me develop two significant software build systems over ten years.
Without a doubt I had the role of “master”, and as Peter Naur would say, had the theory of the program in my head. I developed the build system from empty directories twice, to build products for two different software systems. Each time I had key responsibility to fit the system to customers’ needs, and decide how it would involve internally. (I’ve also been programming for thirty years, since BASIC on my first computers in grade school.)
I delegated what I saw fit to a handful of people who grew in responsibility as they demonstrated they understood and could make creative improvements to the system (journeyman). One of the journeymen became committer when I recently left. Since no-one had the theory as well as I did, architectural responsibility devolved from my fiat to a change control board, so that everyone could bring bits and pieces of their expertise to together make better decisions.
Hundreds of small changes were made over the years by a few dozen people who did not really grasp the principles of the system, so needed a significant amount of guidance. I reviewed every line of code in the system, and know how it all works together. I expect that some of this efficiency will suffer in future, at the advantage—to the organization that owns it—of not having “the keys to the kingdom” in one set of hands.
I’m not sure how well this approach would sit with people as a formalized organization chart and pay scale, but I’ve seen it work very well in practice with a software project I managed. Further, I believe that if you look closely most software projects in practice run this way (eg Linux has Linus, Alan, various subsystem committers, and a hoi polloi of patchers.) Even further, a single person may be an apprentice to one project, a journeyman to another, and a master to a third.
Perhaps the thing to do is to compensate people outside the company by bounties for the code they contribute; set up a contractor arrangement for journeymen; and actually hire only masters. This keeps companies small, focused on their core competencies and most valuable people. Even better for the community of programmers, this provides a structure for programmers to put food on the table while contributing to as many projects, and at the levels of depth, as suits their inclinations and abilities.