Encouragement for the new craftsman

Do you now bear responsibility for the architecture of some code? I’d like to talk just with you.

I’d like to say that I think it’s really important to “listen and then lead”. You’re responsible to the customers of your code for its quality, and you’ll get a lot of pressure to “just do this” or “just do that”. The processes you put in place, to control the quality of contributions, are for the benefit of its users and the code as a whole—to maximize usefulness, not necessarily to maximize development.

Sometimes it’s okay, even necessary, to inconvenience a few people (including the contributors) in order that you don’t fail an even larger number of people. You’re not (at least I wasn’t) able to please everyone, and trying to do so can easily cause harm, both short term (immediate bugs) and long term (technical debt and bad architecture). It’s your, just your, job to balance all this and do what’s right overall, and it’s very much a judgement call.

Some of this might be replaced by a change control board’s decisions, but if you go that route, then you must have a change process that everyone follows, regardless of the demands of the moment. Some people are really slick and sophisticated in getting what they want, especially when it doesn’t hurt them if they end up creating a problem. (Heh, the more I reread that the more it sounds like parenting.)

I found I had to be confident enough to implement decisions and defend them, but flexible enough to shift policy and correct errors. On the plus side, I think these skills will help move you past your current position, and help you control the direction in which your career heads.

You’re doing pretty well if you don’t instantly take sides in conversation on the merits, which is better than I sometimes do. Even if you’re slow making decisions—and there is so much to learn—a little slow is ok, since it gives a chance for more reflection and more mature ideas. Don’t be afraid to be wrong: if you never make mistakes, never make a bold architectural decision, then you’re playing it too safe, and you could probably be replaced with a small shell script.

Take care and good luck!

Software Craftsmanship and Programmers’ Status

This post quotes, collects, organizes, and remixes several sources to provoke comparison and consideration. There’s not a lot of new theory, beyond my idea that this stuff belongs together in the first place.

A change in status

In 1985 Peter Naur described, in Programming as Theory Building, where we still are as a profession (emphasis mine).

[M]uch current discussion of programming seems to assume that programming is similar to industrial production, the programmer being regarded as a component of that production, a component that has to be controlled by rules of procedure and which can be replaced easily. Another related view is that human beings perform best if they act like machines, by following rules, with a consequent stress on formal modes of expression, which make it possible to formulate certain arguments in terms of rules of formal manipulation. Such views agree well with the notion, seemingly common among persons working with computers, that the human mind works like a computer. At the level of industrial management these views support treating programmers as workers of fairly low responsibility, and only brief education.

Here’s where he suggests we be.

On the Theory Building View the primary result of the programming activity is the theory held by the programmers. Since this theory by its very nature is part of the mental possession of each programmer, it follows that the notion of the programmer as an easily replaceable component in the program production activity has to be abandoned. Instead the programmer must be regarded as a responsible developer and manager of the activity in which the computer is a part. In order to fill this position he or she must be given a permanent position, of a status similar to that of other professionals, such as engineers and lawyers, whose active contributions as employers of enterprises rest on their intellectual proficiency.

Further, he comments on necessary changes to programmer education.

The raising of the status of programmers suggested by the Theory Building View will have to be supported by a corresponding reorientation of the programmer education. While skills such as the mastery of notations, data representations, and data processes, remain important, the primary emphasis would have to turn in the direction of furthering the understanding and talent for theory formation. To what extent this can be taught at all must remain an open question. The most hopeful approach would be to have the student work on concrete problems under guidance, in an active and constructive environment.

An active and constructive environment

It occurs to me that the Master Craftsmanship Teams described by Bob Martin comprise distinct, guiding roles, in a context of “concrete problems”, to help us create this “active and constructive environment”. Bob proposes a static structure (remixed below, original emphasis) though which a person progresses (or not) through a career, but in my experience this classification applies between a person and each of many projects.

Apprentice. The apprentices write a significant amount of code—most of which is discarded by the journeymen.

Journeymen. They can make short term tactical decisions, and direct their apprentices accordingly. At the same time they take technical and architectural direction from the master…. The journeymen write a lot of code – almost as much as the master. They also throw away two thirds of the code written by the apprentices, and drive them to redo it better – always better. They teach the apprentices how to refactor, and often pair with them and sit with them while reviewing their code…. When an apprentice delivers code that the journeyman finds acceptable, the journeyman will commit that code to their main line. Otherwise he sends it back to the apprentice to do over…. The Journeymen see every line of code written by their apprentices, and they write a significant amount themselves. Pairing is prevalent though not universal. Anyone on the team can pair with anyone else. It should not be uncommon for the master to pair with the apprentices from time to time.

Master. The master probably writes more code than any of the team members. This code is often architectural in nature, and helps to establish the seed about which the system will crystalize. The master also sets technical direction and overall architecture, and makes all critical technical decisions. However, the chief job of the master is to relentlessly drive the journeymen towards higher quality and productivity by establishing and enforcing the appropriate disciplines and practices…. By the same token the master accepts subsystems from the journeymen and, if acceptable, commits it to the master main line…. The master sees every line of code in the system.

There’s a lot of code written by this team, but a pair of expert eyes reviews all of it. This is manageable if the scope of the work is not too large: my build systems weigh in at under 10,000 lines of (possibly too) intricate and clever shell scripts, makefiles, and flat data files. As with any software system, technical debt is the biggest hurdle to efficiently adding new code. In my case most of the technical debt was revealed by a new point of view (aspect-oriented and functional programming), and fundamental mechanism of customization (sed-scripts rather than sets of macros with varying parts), added only two years before I left my ten-year project.

Ethics and professional responsibility

If software craftsmen would be professionals (note above how Naur separates us from engineers), we should compare existing computer and information science codes of ethics, for example a 1998 Software Engineering Code of Ethics and Professional Practice (SECEPP), to a draft Software Craftsman’s Ethic (SCE, in bold face). Here I roughly group practices by similar intent: the last two groups catch all unmatched practices. (In the remainder of this post, only the italicized paragraph tags are my words.)

Follow process. We follow our chosen practices deliberately, to ensure quality in our work. We adopt development processes to suit our own unique abilities and talents. Ensure an appropriate method is used for any project on which they work or propose to work. Work to follow professional standards, when available, that are most appropriate for the task at hand, departing from these only when ethically or technically justified. Ensure that clients, employers, and supervisors know of the software engineer’s commitment to this Code of ethics, and the subsequent ramifications of such commitment.

Continue to learn. We are skill-centric rather than process-centric. We build competency in a broad range of languages and technologies to meet the needs of our customers. We achieve technical and design excellence through continuous and deliberate learning and practice. We desire to remain Software Craftsmen and write code for our entire careers. We value long lived technologies and languages and are wary of proprietary tools. We are generalists. Not knowingly use software that is obtained or retained either illegally or unethically. Strive to fully understand the specifications for software on which they work. Ensure adequate documentation, including significant problems discovered and solutions adopted, for any project on which they work. Further their knowledge of developments in the analysis, specification, design, development, maintenance and testing of software and related documents, together with the management of the development process. Improve their ability to create safe, reliable, and useful quality software at reasonable cost and within a reasonable time. Improve their ability to produce accurate, informative, and well-written documentation. Improve their understanding of the software and related documents on which they work and of the environment in which they will be used. Improve their knowledge of relevant standards and the law governing the software and related documents on which they work. Improve their knowledge of this Code, its interpretation, and its application to their work.

Protect reputation by accepting responsibility. We build our reputations on delivered quality. Our livelihoods and reputations are interdependent. We are proud of our work and the manner of our work. We take responsibility for our work by signing it. We are proud of our portfolio of successful projects. Accept full responsibility for their own work. Not promote their own interest at the expense of the profession, client or employer. Recognize that violations of this Code are inconsistent with being a professional software engineer. Recognize that personal violations of this Code are inconsistent with being a professional software engineer.

Network to sow the field. We respect master craftsmen as the key to Software Craftsmanship. We will help other craftsmen in their journey. We can point to the people who influenced us and who we influenced. We recommend other craftsmen, hanging our own reputation on their performance. We work in a community with other craftsmen. Not unjustly prevent someone from taking a position for which that person is suitably qualified. Extend software engineering knowledge by appropriate participation in professional organizations, meetings and publications. Support, as members of a profession, other software engineers striving to follow this Code.

Clean code. We write code that is easy to read, understand and modify. We put predictable and forgiving user interfaces on our software. Be accurate in stating the characteristics of software on which they work, avoiding not only false claims but also claims that might reasonably be supposed to be speculative, vacuous, deceptive, misleading, or doubtful.

Test and maintain. We design software for testing and maintenance. We build long-lived applications and support them throughout their life. We automate every task possible (especially testing). We drive our development with tests. Ensure adequate testing, debugging, and review of software and related documents on which they work. Treat all forms of software maintenance with the same professionalism as new development.

Strive for perfection. We deliver our work to our customer without known defect. We value quality over feature set or delivery data. We consistently exceed and raise our customer’s expectations of quality software. We build in quality to the software we create. Take responsibility for detecting, correcting, and reporting errors in software and associated documents on which they work.

Plan tradeoffs. We say no to prevent doing harm to our craft and our projects. Strive for high quality, acceptable cost and a reasonable schedule, ensuring significant tradeoffs are clear to and accepted by the employer and the client, and are available for consideration by the user and the public. Ensure realistic quantitative estimates of cost, scheduling, personnel, quality and outcomes on any project on which they work or propose to work and provide an uncertainty assessment of these estimates. Ensure realistic quantitative estimates of cost, scheduling, personnel, quality and outcomes on any project on which they work or propose to work, and provide an uncertainty assessment of these estimates.

Respect customers. We deliver what our customers really need, not just what they ask for. We build trust and respect in long term relationships with our customers. We take our customer’s needs as seriously as they do. We desire demanding users. Work to develop software and related documents that respect the privacy of those who will be affected by that software. Ensure that specifications for software on which they work have been well documented, satisfy the users’ requirements and have the appropriate approvals.

Promote coworkers and colleagues. We take on Apprentices and put them in an environment where they can learn our craft for themselves. Ensure that software engineers are informed of standards before being held to them. Attract potential software engineers only by full and accurate description of the conditions of employment. Offer fair and just remuneration. Not ask a software engineer to do anything inconsistent with this Code. Encourage colleagues to adhere to this Code. Assist colleagues in professional development. Credit fully the work of others and refrain from taking undue credit. Review the work of others in an objective, candid, and properly-documented way. Give a fair hearing to the opinions, concerns, or complaints of a colleague. Assist colleagues in being fully aware of current standard work practices including policies and procedures for protecting passwords, files and other confidential information, and security measures in general. Not influence others to undertake any action that involves a breach of this Code.

Demonstrate competence. We accept that different craftsmen are better suited for one kind of job over another. Provide service in their areas of competence, being honest and forthright about any limitations of their experience and education. Ensure that they are qualified for any project on which they work or propose to work by an appropriate combination of education and training, and experience. Only endorse documents either prepared under their supervision or within their areas of competence and with which they are in agreement. Assign work only after taking into account appropriate contributions of education and experience tempered with a desire to further that education and experience. Not unfairly intervene in the career of any colleague; however, concern for the employer, the client or public interest may compel software engineers, in good faith, to question the competence of a colleague. In situations outside of their own areas of competence, call upon the opinions of other professionals who have competence in that area.

Manage projects. We keep projects from spinning in circles. We don’t wait for complete definition before beginning. We never allow ourselves to be blocked. Identify, document, collect evidence and report to the client or the employer promptly if, in their opinion, a project is likely to fail, to prove too expensive, to violate intellectual property law, or otherwise to be problematic. Ensure proper and achievable goals and objectives for any project on which they work or propose. Ensure good management for any project on which they work, including effective procedures for promotion of quality and reduction of risk.

Own code. We believe the code is also an end, not just a means. We treat software like capital. Ensure that there is a fair agreement concerning ownership of any software, processes, research, writing, or other intellectual property to which a software engineer has contributed.

Represent the public interest. Moderate the interests of the software engineer, the employer, the client and the users with the public good. Approve software only if they have a well-founded belief that it is safe, meets specifications, passes appropriate tests, and does not diminish quality of life, diminish privacy or harm the environment. The ultimate effect of the work should be to the public good. Disclose to appropriate persons or authorities any actual or potential danger to the user, the public, or the environment, that they reasonably believe to be associated with software or related documents. Cooperate in efforts to address matters of grave public concern caused by software, its installation, maintenance, support or documentation.Be fair and avoid deception in all statements, particularly public ones, concerning software or related documents, methods and tools. Consider issues of physical disabilities, allocation of resources, economic disadvantage and other factors that can diminish access to the benefits of software. Be encouraged to volunteer professional skills to good causes and contribute to public education concerning the discipline. Identify, document, and report significant issues of social concern, of which they are aware, in software or related documents, to the employer or the client. Identify, define and address ethical, economic, cultural, legal and environmental issues related to work projects. Temper all technical judgments by the need to support and maintain human values. Help develop an organizational environment favorable to acting ethically. Promote public knowledge of software engineering. Obey all laws governing their work, unless, in exceptional circumstances, such compliance is inconsistent with the public interest. Express concerns to the people involved when significant violations of this Code are detected unless this is impossible, counter-productive, or dangerous. Report significant violations of this Code to appropriate authorities when it is clear that consultation with people involved in these significant violations is impossible, counter-productive or dangerous.

Deal fairly. Use the property of a client or employer only in ways properly authorized, and with the client’s or employer’s knowledge and consent. Ensure that any document upon which they rely has been approved, when required, by someone authorized to approve it. Keep private any confidential information gained in their professional work, where such confidentiality is consistent with the public interest and consistent with the law. Accept no outside work detrimental to the work they perform for their primary employer. Promote no interest adverse to their employer or client, unless a higher ethical concern is being compromised; in that case, inform the employer or another appropriate authority of the ethical concern. Be careful to use only accurate data derived by ethical and lawful means, and use it only in ways properly authorized. Maintain the integrity of data, being sensitive to outdated or flawed occurrences. Maintain professional objectivity with respect to any software or related documents they are asked to evaluate. Not engage in deceptive financial practices such as bribery, double billing, or other improper financial practices. Disclose to all concerned parties those conflicts of interest that cannot reasonably be avoided or escaped. Refuse to participate, as members or advisors, in a private, governmental or professional body concerned with software related issues, in which they, their employers or their clients have undisclosed potential conflicts of interest. Ensure that software engineers know the employer’s policies and procedures for protecting passwords, files and information that is confidential to the employer or confidential to others. Provide for due process in hearing charges of violation of an employer’s policy or of this Code. Not punish anyone for expressing ethical concerns about a project. Avoid associations with businesses and organizations which are in conflict with this code. Not give unfair treatment to anyone because of any irrelevant prejudices.

My experience with Master Craftsman Teams

“Uncle Bob” wrote yesterday about Master Craftsman Teams.

The thing that struck me most about this article is how closely it matched the way my own team of coworkers, who came and went, contributed and moved on, helped me develop two significant software build systems over ten years.

Without a doubt I had the role of “master”, and as Peter Naur would say, had the theory of the program in my head. I developed the build system from empty directories twice, to build products for two different software systems. Each time I had key responsibility to fit the system to customers’ needs, and decide how it would involve internally. (I’ve also been programming for thirty years, since BASIC on my first computers in grade school.)

I delegated what I saw fit to a handful of people who grew in responsibility as they demonstrated they understood and could make creative improvements to the system (journeyman). One of the journeymen became committer when I recently left. Since no-one had the theory as well as I did, architectural responsibility devolved from my fiat to a change control board, so that everyone could bring bits and pieces of their expertise to together make better decisions.

Hundreds of small changes were made over the years by a few dozen people who did not really grasp the principles of the system, so needed a significant amount of guidance. I reviewed every line of code in the system, and know how it all works together. I expect that some of this efficiency will suffer in future, at the advantage—to the organization that owns it—of not having “the keys to the kingdom” in one set of hands.

I’m not sure how well this approach would sit with people as a formalized organization chart and pay scale, but I’ve seen it work very well in practice with a software project I managed. Further, I believe that if you look closely most software projects in practice run this way (eg Linux has Linus, Alan, various subsystem committers, and a hoi polloi of patchers.) Even further, a single person may be an apprentice to one project, a journeyman to another, and a master to a third.

Perhaps the thing to do is to compensate people outside the company by bounties for the code they contribute; set up a contractor arrangement for journeymen; and actually hire only masters. This keeps companies small, focused on their core competencies and most valuable people. Even better for the community of programmers, this provides a structure for programmers to put food on the table while contributing to as many projects, and at the levels of depth, as suits their inclinations and abilities.

Agile practice one-liners, and Alexander on refactoring

A few one-line descriptions of Agile practices, because I feel they get to the heart of each.

Iterative development with feature teams: a few developers complete the development cycle (refactor, code, integrate, build, and test), for each of a few features, in two weeks to two months.

Refactoring: instead of a static, outdated architecture, rework the codebase with each feature.

Test-driven development: write tests of functionality for user stories, then write code to pass these tests.

Pairing and collective ownership: two developers introduce fewer errors to review and test for, and many developers distribute knowledge of the code.

Frequent integration: merge and build new changes at least daily, test at least weekly.

Automated regression test: collect all test cases into push-button test suites, run shortly after each deliverable build.

I also found a passage on refactoring from an OOPSLA keynote by Christopher Alexander:

It turns out that these living structures can only be produced by an unfolding wholeness. That it, there is a condition in which you have space in a certain state. You operate on it through things that I have come to call “structure-preserving transformations,” maintaining the whole at each step, but gradually introducing differentiations one after the other. And if these transformations are truly structure-preserving and structure-enhancing, then you will come out at the end with living structure. Just think of an acorn becoming an oak. The end result is very different from the start point but happens in a smooth unfolding way in which each step clearly emanates from the previous one.

Alexander, C. 1999. The origins of pattern theory: The future of the theory, and the generation of a living world. IEEE Software 16, 5 (Sept/Oct), 71–82. http://doi.ieeecomputersociety.org/10.1109/52.795104

Blacksmithing a Train Engine

Yet another analogy comparing software development to a hardware activity: in this case, train engines.

(Programs|engines) run, within the bounds of a (computer|track). The strength of the (computer|track) determines the rate at which they can run. The design and internal processes of the (program|engine) determines the rate at which they do run. Certain designs of (programs|engines) make it easier to handle various loads of (data|cars), since (data|cars) have different properties when placed under the stress of running.

If (data|cars) properly interface with the (file|rail) format, then the (program|engine) can handle them. If not, then the (program|engine) will fail to make any progress, or crash. The more subtle the difference between the format expected by the (program|engine) and that implemented by the (data|cars), the longer it will run before the difference creates a problem. Some problems might not be found at all, if no situation arises for which the difference has any noticeable effect. The (program|engine) handles a certain amount of (data|cars) before slowing to a crawl. Users can handle more (data|cars) in the same amount of time by running additional (programs|engines) in parallel, but improvement is limited by the time required to coordinate final and intermediate delivery.

Standard (file|rail) formats are better, because more (programs|engines) and (data|cars) are built to use them, their properties are better known, and users can even reuse them in different systems. However, vendors must differentiate themselves to drive business their way, so they often attempt to lock users into a particular format, in addition to (sometimes instead of) competing on the merits of their (program|engine).

In the early days of (programs|engines), each part was laboriously crafted and fit into place by hand, and often failed spectacularly from the additional pressures of running. Each part had to be crafted with a knowledge of all the stresses under which it would run, from materials suitable for the basic processes which powered the (program|engine). Nonetheless, its appearance made a more significant impact on users and bystanders, unless its performance differed drastically from others’.

The tools with which the craftsmen created (program|engine) parts were themselves created by the same process, so a craftsman’s product quality varied dramatically with his knowledge of toolmaking.

There were common parts, interfaces, and controls expected by users who had used similar (programs|engines). There was a great need to explain to users who had not: how they could expect the (program|engine) to behave with their (data|cars), and to make sure they were compatible with the (file|rail) format; how to control the (program|engine) at a basic level, and how to get a feel for its advanced capabilities; and how to get the (data|cars) to the (program|engine), and how to pass them off to others.

Smithing Tools to Smith Tools

Following text from Wikipedia on blacksmiths.

Over the centuries (blacksmiths|programmers) have taken no little pride in the fact that theirs is one of the few crafts that allows them to make the tools that are used for their craft. Time and tradition have provided some fairly standard basic tools which vary only in detail around the world.

There are many other tools used by smiths, so many that even a brief description of the types is beyond the scope of this article and the task is complicated by a variety of names for the same type of tool. Further complicating the task is that making tools is inherently part of the smith’s craft and many custom tools are made by individual smiths to suit particular tasks and the smith’s inclination.

With that caveat one category of tools should be mentioned. A (jig|hack) is generally a custom built tool, usually made by the smith, to perform a particular operation for a particular task or project.

More on software blacksmithing, apprenticeship, and artistry: Craftsmanship as a Bridge.

Pave the Pathways

Roger von Oech provides an example of backing off the design process, to observe usage patterns.

For example, designer Christopher Williams tells a story about an architect who built a cluster of large office buildings that were set on a central green. When construction was completed, the landscape crew asked him where he wanted the pathways between the buildings.

“Not yet,” the architect said. “Just plant the grass solidly between the buildings.”

This was done, and by late summer pedestrians had worn paths across the lawn, connecting building to building. The paths turned in easy curves rather than right angles, and were sized according to traffic.

In the fall, the architect simply paved the pathways. Not only did the new pathways have a design beauty, they responded directly to user needs.

In software, this argues to release to users in iterations, incorporate their feedback, and simplify their use cases. Common use cases, added late, tend not to elegantly fit the software’s architecture, so increase the maintenance burden.

Kathy Sierra reminds us that providing what’s good for users is good business.

Ask the question we keep bringing up here, “What will this help the user do?” Not, “How can we make a great product?” Nobody cares about your company, and nobody cares about your product. Not really. They care about themselves in relation to your product. What it means to them. What it does for them. What it says about them that they use your product or believe in your company.

And that users don’t want us to pave the entire courtyard.

How much control should users have?

Obviously this is a big “it depends”, but the main point is to focus on the relationship between user control and user capability. As user capability (knowledge, skill, expertise) increases, so should control — at least for a lot of things we make, especially software, and especially when we’re aiming not just for satisfied users but potentially passionate users. The big problem is that we make our beginning users suffer just so our advanced users can tweak and tune their configurations, workflow, and output. [For the record, I'm a big fan of splitting capabilities into different products, or having a really good user-level modes--where you use wizards or simpler interfaces for new users, etc. Yes, they're often done badly, but they don't have to be.]

In the build systems I write, the interface tends to one shell script, whose options behave like make options (passed to the main subsidiary make process), and whose parameters are make targets (with an optional subtarget syntax). So if you know how to ask make to build what you want, then you know how to ask my build system.

There is a lot of room in this interface style for build system developers (eg, you can ask the build system to run any internal target), application developer experts (eg, custom-build individual tasks or libraries, or even single files), and application developer novices (eg: all-encompassing clean, build, and package targets; wrapper scripts which set the shell environment).

Best of all, novice features tend to form the foundation of automation. If a novice developer can easily run a use case, then the process scheduler (cron) probably can also run it.

Notes on “AOP: Radical Research in Modularity”

Notes on Aspect Oriented Programming: Radical Research in Modularity, by Gregor Kiczales, for Google TechTalks on 16 May 2006.

Pointcut Language and Overall Program Structure

Ableson and Sussman: We have a language when we have primitives, a way to compose those primitives to perform useful work, and a mechanism to abstract and refer to compositions elsewhere.

Pointcut syntax is a little language, with syntax to select a set of join points (primitives), construct logical expressions with several sets (composition), and label an expression (abstraction).

This is the sole advantage of modularity: abstract a set of primitives in one distinct place, and combine them with tools in the language. Using the language, instead of hand-weaving, provides a more structured way to consider how changing the base concern may change the pointcuts.

Dominant model applies crosscutting concerns to join points in base concerns structured as block models within a hierarchy, instead of composing orthogonal sets of crosscutting concerns with base concerns structured as crosscutting concerns.

Procedural, Object, and Aspect Programming Methods

Say we describe the ordinary process of functional/procedural programming as a problem expert (eg, the task at hand) collecting knowledge from other disciplines (secure, optimize, distribute, synchronize, persist, debug/log/trace, quality of service) to apply to the problem. Obviously the programmer can integrate all of this knowledge on a deep level, since all the knowledge appears before the analytic capability of the human mind. However, there is a limit to the size and number of disparate disciplines which can be integrated, before the scale and complexity produces an error rate unacceptable to customers.

Object-oriented programming does not alleviate this situation, since each class must still integrate all the crosscutting concerns with methods, or statements within methods, in the class definition.

Aspect-oriented programming allows a team of specialists to each apply expertise to many basic problems. One person could do everything the team does, but as mentioned can only work on a few very large problems at once, before reaching human limits; whereas a specialist can dispense information as needed across many large systems. This optimizes resources at the cost of the mentioned deep integration of specialist knowledge with other specialist knowledge. That integration is left up to the resource manager (the weaver), who schedules and reconciles the contributions of the specialists and the basic problem’s manager. The basic problem’s manager, in aspect-oriented programming, can represent the customer, verify correctness of the integration, and validate the integrated solution.

Consider Pointcuts in Aspects, and Pointcut Specifiers in Base Concerns, When Changing Base Concerns

We must be careful, when editing base concerns, to not break pointcuts; or, breaking them, to remember to update the pointcut. This is less likely to cause error than the original method of weaving crosscutting concerns into the base concerns by hand.

With respect to my weaver for shell-scripts and makefiles, I am hesitant to use explicit comments in the base scripts as search patterns for the sed scripts. However, these make a contract with any transforming program: we can consider these points reliable enough to anchor transforms. Without this, especially if the search patterns refer to code statements, additional pointcuts may not find the points they specify still in the program by the time the weaver gets to them (but at least they will fail-safe and apply no advice).

The base concern’s comments should only refer to the base concern, not any crosscutting concern, and should certainly not say “at this point, insert that statement from the other crosscutting concern.” Those kinds of comments defeat the whole purpose of modularization, and do not recognize that we may eventually weave completely different crosscutting concerns into the base concern.

We should not try to apply a single comment across many base concerns, to reduce the effort to write the join point specifier. This binds, in a single relation, all the base concerns, and all the crosscutting concerns which use the comment. If the advice changes for half the base concerns, then half the instances of the single comment must change. If instead the pointcut is a combination of several comments, then we more easily split off a subset of these into a new pointcut, to receive new advice.

The Weaving-Loom Analogy

weave applies one crosscutting concern to one base concern. The variation lace customizes the base concern for an application (eg, by laying sed changes over the file).

loom weaves many crosscutting concerns into many base concerns, adding each crosscutting concern to the base concern as modified by previous crosscutting concerns. Under user control, the loom orders crosscutting concerns applied to each base concern. User control is essential since the user writes the crosscutting concerns, and knows which crosscutting concerns apply to other crosscutting concerns, so should weave last.

warp finds crosscutting concerns which modify a given base concern. It also finds crosscutting concerns and advice which modify a given base concern’s given code statement.

weft finds base concerns modified by a given crosscutting concern. It also finds base concerns’ code statements modified by a given crosscutting concern’s given advice. woof is already taken as a s(n)ide-acronym for another function in my build system, and is slightly silly anyway, although it does have the advantage of turning loom upside-down and backward.

Weaving New Aspects into Reused Legacy Code

AOP removes all code for crosscutting concerns from base concerns. This is better than macros or libraries, which require at least calls in the base concern. The base concern can be completely oblivious to crosscutting concerns, which allows us to weave unanticipated crosscutting concerns as easily as we add the ones we had in mind when we wrote the base concern.

This is key to reusability, and in fact aspects should unlock the entire previous pre-aspect code base to modification, without editing the current working, tested, and delivered code files. I don’t believe in silver bullets, but the separation of concerns here is the first opportunity I’ve seen to keep the old code separate from the new code, yet use them together in a new program, while easily incorporating any further changes to the old code.

For new code, aspects promote writing code for reuse, since each concern is a separate file, and can be bundled and released separately. Reuse is available for base concerns since they do not refer to crosscutting concerns (low coupling), and completely implement one concern of the program (high cohesion). Reuse is easier for crosscutting concerns because: (1) if the aspect has no join point to match, it fails safe and applies no advice to the base concern; and (2) the advice code refers only to methods and fields exposed by the pointcut, which hides (low coupling) the remainder of the base concern.

Integrating Proprietary and Open-Source Code

This method could be very useful in incorporating open-source code into proprietary codebases. If we can show that we did not directly modify the original open-source code, then we may not need to release our proprietary changes to the open-source code.

Before compilation, the reused open-source code is in its files with its copyrights, in exactly the same state as when downloaded. Proprietary code is in its files with its copyrights. Since we must redistribute the open-source code files with any modifications to those files (the files under copyright), we just redistribute the original, unmodified open-source files, since they were not changed.

I am not a lawyer, so this analysis could be way off: specifically, we could still consider the distributed, weaved version a derived work, which means that all code that went into producing it must be distributed. Definitely must speak to an IP lawyer before basing any decisions on this.

Dynamic Join Points

Since I’m writing a weaver, I should consider join points other than code statements (including comments) in shell scripts and makefiles. These implement a static join point model, which change the code before it runs. Dynamic join point models create advice which executes only upon certain runtime behavior: for example, if a sub-script or particular makefile is called with certain parameters or targets, or a certain shell variable or makefile macro receives a particular value.

We could of course explicitly write more complicated advice to implement this. But it would be simpler to write the advice if the pointcut matched the runtime state of the program. To do this, we could either (1) monitor the execution state at runtime (better suited to modeling languages) or (2) extend the pointcut language to describe a runtime state. With (2), the weaver translates the runtime state description into static code to detect the state and apply the simpler advice.

Prototyping

Since the crosscutting concern in my implementation “owns” the final executable form, I can prototype new changes to the base concern and affect only the product or platform which needs the change. Once it’s proven, I can move any new variables or code in the interface back to the base concern, to become a default part of the interface for other products or platforms. At this point, I may need to write advice to change the interface for products and platforms which must differently define its values.

We can extend this thinking to consider prototyping itself a crosscutting concern, and maintain several prototypes concurrently, weaving them in according to control options. Of course, this begs me to make control options crosscutting concerns.

Effect on Software Configuration Management

AOP introduces the weaving step between code checkin and compilation. For C/C++, this step constructs woven header and source code files by combining (1) base header and source code files with (2) header and source code file content in advice gathered in files which encapsulate each crosscutting concern. The order of application of crosscutting concerns should be specified in the target makefile.

Alternatively, we could make the Aspect C and C++ compilers available as alternative platforms (eg, *_aspect), and let the compilers sort it out. The caveat here is that I haven’t used one of these languages yet. I do know that they would integrate most cleanly if they were actually preprocessors which created new *.c/*.cpp (C/C++ source code) or *.i/*.ii (preprocessed C/C++ source code) files suitable for compilation with the build system’s cross-compilers.

Steve Jobs on Microsoft, 1996

The only problem with Microsoft is they just have no taste, they have absolutely no taste, and what that means is – I don’t mean that in a small way I mean that in a big way. In the sense that they they don’t think of original ideas and they don’t bring much culture into their product ehm and you say why is that important – well you know proportionally spaced fonts come from type setting and beautiful books, that’s where one gets the idea – if it weren’t for the Mac they would never have that in their products and ehm so I guess I am saddened, not by Microsoft’s success – I have no problem with their success, they’ve earned their success for the most part. I have a problem with the fact that they just make really third rate products.

Video – Transcript – ©1995-2006 PBS. All rights reserved.

Baselining from Daily Builds

Branching. Agile asks to trade baseline stability for ease of integration. Take the latest baseline available, make it work with your change, and merge it as soon as it passes your tests. This reduces the number of concurrent branches active at once, and therefore the amount of simultaneous variability in the code base. Developers will not need to rebaseline from the last official build to include changes already merged to the integration branch.

Testing. This does mean that problems are more likely to be merged and affect others, so automated unit testing helps avoid defects not due to the interaction with other components. Merging a change sooner gives it more second-order testing with other components before major builds, which increases the likelihood that interaction problems will be found before they can delay a release.

Quicker feedback about integration problems refocuses a developer’s attention on a change sooner, while the developer is still thinking about the problem, instead of at the next major build after a developer has moved on to other issues.

Software Tools – Getting Started

Excerpts from Software Tools, Kernighan and Plauger, 1976.

File Copying

If we use primitives, we can design and write programs that will not be overly dependent on the idiosyncrasies of any one operating system. The primitives insulate a program from its operating system environment and ensure that the high level task to be performed is clearly expressed in a small well-defined set of basic operations.

It is a good practice to write first in an easy-to-read higher level language, then translate into whatever real-life language you happen to be working with. [eg, lib, script, lex/yacc]

If you adhere to the design principle of pushing details as far down as possible, by writing in terms of basic primitives which read from an arbitrary source and write to an arbitrary destination, your new tools will be compatible with previous ones; you will be building a whole set that work together. [pipe]

Counting Characters

do reasonable things with extreme cases

Counting Lines

When bugs occur, they usually arise at the "boundaries" or extremes of program operation.

Counting Words

We do recommend that you be consistent in applying whatever formatting standards you settle on.

The best place to begin this activity [test] is while the program is being written. The main assurance you have that a program is correct is the intellectual effort you put into getting it right in the first place.

Testing is still necessary, however, to check that the algorithm is valid and that the program implements it correctly.

intuitively a "boundary" is a data value for which the program is forced to react differently from an adjacent value.

Yet it is a fundamental principle of testing that you must know in advance what answer each test case is supposed to produce.

So part of the responsibility of writing a program is to prepare a comprehensive test of test inputs, and outputs against which to compare the result of test runs.

Removing Tabs

it's folly to write a program that blindly assumes that its input is legal.

Most real programs are subjected to a steady flow of changes and improvements over their lifetimes, and many programmers spend most of their time maintaining and modifying existing programs. This is an expensive process, so one of the most important design considerations for a program is that it be easy to change.

The best way we know to achieve this is to write the program so its pieces are as decoupled as possible, so that a change in one does not affect others.

Hand Compiling the Code

The moral should be obvious. By writing a program in a straightforward manner, you get it working correctly and minimize the chance of confusion. You can then measure how it performs to decide whether it works well enough and, if not, where to concentrate your attention. For a given algorithm, gains in speed are almost always obtained at the cost of readability. Sacrifice clarity for speed only when you know that you are solving the correct problem correctly and when you know that the sacrifice is worthwhile.

A Word on Structured Programming

Our approach to structured programming is to stick to the basics. We invest extra effort in the design and coding process (which is fun) to minimize the much costlier testing and debugging phase (which is not). We put strong emphasis on clean, comprehensible code, saving efficiency considerations for the end. We check and test code as we write it, rather than relying on a final debugging binge to fix everything. We make extensive use of subroutine calls and statement grouping to modularize code, and we always write control flow in terms of if-else and the looping constructs.

But our primary tool for writing good programs is to strive to make them readable. In our experience, readability is the single best criterion of program quality: if a program is easy to read, it is probably a good program; if it is hard to read, it probably isn't good.