How to compile and run Inferno on Ubuntu 14.04

Download Inferno from Vita Nuova’s web site.

Create /usr/inferno and make it writable by other.

Expand the inferno tgz file with tar xzf while the current directory is /usr.

Update the file mkconfig to set SYSHOST to Linux, uncomment OBJTYPE=386, and comment out OBJTYPE=$objtype.  Run ./makemk.sh.

export PATH=/usr/inferno/Linux/386/bin:$PATH
mk nuke

mk install will fail to find sys/cdefs.h, so add -I/usr/include/x86_64-linux-gnu\ to CFLAGS in mkfiles/mkfile-Linux-386.

mk install will fail to find gnu/stubs-32.h, so run sudo apt-get install libc6-dev-i386 to install 32-bit libraries.

mk install will produce a floating point exception when limbo tries to compile runt.m.  Commenting out the line that sets NaN will avoid the exception.  Instead of trying to find a solution to this problem in the code from 2010, update it with hg pull -uv and hg update to see whether the problem was resolved. (Install mercurial with the command sudo apt-get install mercurial.)  Accept the changed *.dis files.  Resolve conflicts in mkfiles/mkfile-Linux-386, by allowing the new -I option, and deleting the X11R6 include.

hg resolve –mark
hg resolve –all
hg pull -uv # gives no more output

mk install will fail to find X11/Xlib.h, so run sudo apt-get install libx11-dev and libxext-dev to install X11 headers.

mk install will fail to find X11 and Xext, so run sudo apt-get install libx11-6:i386, libxext-dev:i386 to install 32-bit versions of the same.

So we now get far enough to see some code that needs the NaN we earlier removed.

calc.b:135: ‘NaN’ is not a member of ‘Math’ of type Math
calc.b:352: Nan’s value cannot be determined
calc.b:62: cannot index ‘conw’
calc.b:63: cannot index ‘conw’
calc.b:63: cannot index ‘conw’
mk: limbo -I/usr/inferno/module -gw calc.b : exit status=exit(1)

Let’s put it back in and see what happens: we’re back to a floating-point exception, so updating to the most recent version of code didn’t fix it.

Added the keyword volatile and the “=a” output constraint to fpuctl.h:setfcr, per this discussion.

mk nuke, and then mk install appears to complete successfully.

emu wm/wm starts a window manager session within the emulator.

 

Credo example: from lex and c source to cygwin executable, across directories

This article walks through building a trivial example of lex source
code into C, and then compiling the generated C source code into
object files and an executable in a different directory. The point is
to demonstrate using dodep to generate and customize build scripts
and list dependencies, and credo to execute the build process.

Lines which start with ; are shell commands; lines without are output.
Key commands are displayed in red text.

Starting state

We start with the files lorem, src/dcr.l, src/header.c, src/lex.h,
and an empty obj directory. Each line of lorem ends with a
carriage return (not usually printed), which we want to strip.
header.c is an ordinary C source file to compile and link.
lex.h gives us a local header to look for.

Generate source code

; cd src
; echo flex > dcr.c.lex.env

We first go into the src directory, and set the lex shell variable to flex,
to specify which lexer to use. To demonstrate an unusual feature,
that would be valid across ports to different operating systems,¹
we’ll capture it with this command.
¹ Inferno and Plan 9 store variables by name in the directory /env,
different for each process. This gives it a definite advantage in
capturing dependencies on variables, since we can just list the name
of the shell-variable file as a dependency.

src/dcr.c.lex.env:1: flex

This setting applies (is transformed into a shell variable) when credo
processes the target dcr.c, if dcr.c depends on /env/lex.

; dodep (c lex l c) dcr.c
credo dcr.c

We ask dodep to generate a shell script and list of dependencies with
this command. The content of these files comes from a library,
indexed by the four parameters given before the name of the file we
want to build. In this case, we want to look in the “c” namespace,
at the command “lex”, to translate an “l” file to a “c” file.

The generated dependency list includes the name of the lex source file,
and references to the variables lex and lflags.

src/dcr.c.dep:1: 0 dcr.l
src/dcr.c.dep:2: 0 /env/lex
src/dcr.c.dep:3: 0 /env/lflags

Credo passes the name of its target, the C source file, to the shell script,
which prints the name of the C source file if it was able to create it.

src/dcr.c.do:1: #!/dis/sh
src/dcr.c.do:2: c = $1
src/dcr.c.do:3: (sum l) = `{grep '\.l$' $c^.dep}
src/dcr.c.do:4:
src/dcr.c.do:5: if {no $lex} {lex = lex}
src/dcr.c.do:6:
src/dcr.c.do:7: if {
src/dcr.c.do:8: flag x +
src/dcr.c.do:9: os -T $lex $lflags $l
src/dcr.c.do:10: mv lex.yy.c $c
src/dcr.c.do:11: } {
src/dcr.c.do:12: echo $c
src/dcr.c.do:13: }

The hosted-Inferno command os calls host-OS commands. Under Cygwin,
it calls Cygwin or Windows commands.

Dodep finally prints a credo command to build the requested target.

; credo dcr.c
credo dcr.l
credo dcr.c
os -T flex dcr.l
mv lex.yy.c dcr.c
dcr.c

Credo builds all its non-/env dependencies in parallel. Once these
are complete, it determines whether the checksum of each file on which
it depends is different than the checksum stored for that file the
last time credo built this target.

This credo command creates a build-avoidance marker file dcr.l.redont,
which tells further runs of credo that the file dcr.l does not need
any processing, so exit early. It calls the host’s flex program,
on the path set when the Inferno environment started, to create the
target file dcr.c.

It updates the check sums in the file dcr.c.dep, to reflect content at
the time of compilation. This means that each target has its own view
of its dependencies, and any change to any file on which it depends,
or a change to the list of files on which it depends, prompts the
target to rebuild.

src/dcr.c.dep:1: 79f9925952d9cfb5a58270c0e2b67691 dcr.l
src/dcr.c.dep:2: 897a779351421523cadbafccdce22efe /env/lex
src/dcr.c.dep:3: 0 /env/lflags

It creates these checksum files to detect any changes to the target’s
dependencies or build script.

src/dcr.c.dep.sum:1: c33129421847b7d897d4e244caf1b8c0 dcr.c.dep

src/dcr.c.do.sum:1: 71d94d651922bd5ff5b543bba52f47d0 dcr.c.do

It also checksums the target file itself. If credo finds, the next
time it runs, that the current checksum of the target does not match,
then it assumes that the target was changed by hand, and will not
overwrite it.

src/dcr.c.sum:1: 2fed3667f390fd80993e090a7eb92c75 dcr.c

Compile objects and executable

We now have all the source, so we go to the object-file and executable
directory, and ask dodep to provide do and dep files to build dcr.exe.
From the “c”-language toolset, we use “cc” to compile an “o”bject file
into an “exe”cutable.

; cd ../obj
; dodep (c cc o exe) dcr.exe
credo dcr.exe

This time the dodep command created two *.do files, dcr.exe.dep.do and
dcr.exe.do, instead of a do and dep file for the target dcr.exe.

obj/dcr.exe.dep.do:1: #!/dis/sh
obj/dcr.exe.dep.do:2: dep = $1
obj/dcr.exe.dep.do:3: exe = `{echo $dep | sed 's,\.dep,,'}
obj/dcr.exe.dep.do:4: (stem ext o) = `{crext o $exe}
obj/dcr.exe.dep.do:5:
obj/dcr.exe.dep.do:6: adddep $exe /env/ars /env/cc /env/cflags /env/ldflags /env/objs
obj/dcr.exe.dep.do:7: adddep $exe $o
obj/dcr.exe.dep.do:8: adddep $exe `{/lib/do/c/coneed $o}

dcr.exe.dep.do creates the dependency list dcr.exe.dep for the target
dcr.exe by adding a set of credo-specific (ars and objs) and standard
(cc, cflags, and ldflags) shell variables; the primary object file,
named for the executable; and any other object files which the primary
one needs. The tool coneed does this last bit of analysis by compiling
the primary object file, looking for unresolved symbols in available
source code, compiling the source code to make sure, and printing the
set of matching object files.

obj/dcr.exe.do:1: #!/dis/sh
obj/dcr.exe.do:2: exe = $1
obj/dcr.exe.do:3: if {no $cc} {cc = cc}
obj/dcr.exe.do:4: if {
obj/dcr.exe.do:5: flag x +
obj/dcr.exe.do:6: os -T $cc $cflags -o $exe $objs $ars $ldflags
obj/dcr.exe.do:7: } {
obj/dcr.exe.do:8: echo os -T ./$exe
obj/dcr.exe.do:9: }

To gather enough information to compile the executable, credo relays
information from dependencies back to calling targets through *.relay
files, which are shell scripts that set the environment for calling targets.

Finally, to start the build process, we locate the source code,
record which compilation tools to use, and again call credo.

; srcdir = ../src/
; echo cpp-3 > default.cpp.env
; echo -I../src/ > default.cppflags.env
; echo gcc-3 > default.cc.env
; credo dcr.exe
credo dcr.exe.dep
os -T gcc-3 -I../src/ -c ../src/dcr.c
os -T gcc-3 -I../src/ -c ../src/header.c
credo dcr.o.dep
credo dcr.o
credo header.o.dep
credo header.o
credo dcr.exe
os -T gcc-3 -o dcr.exe dcr.o header.o
os -T ./dcr.exe

Credo first runs dcr.exe.dep.do to find and store the dependencies of
dcr.exe in dcr.exe.dep. dcr.exe.dep is an implicit dependency of dcr.exe,
so credo runs its *.do script automatically.

obj/dcr.exe.dep:1: 0 /env/ars
obj/dcr.exe.dep:2: 131eb9ab2f6a65b34e0158de1b321e3c /env/cc
obj/dcr.exe.dep:3: 0 /env/cflags
obj/dcr.exe.dep:4: 0 /env/ldflags
obj/dcr.exe.dep:5: 84a1a3c306e006dd723bebe9df29ee6c /env/objs
obj/dcr.exe.dep:6: 3f0d90c1c7483ad1805080cf9e48d050 dcr.o
obj/dcr.exe.dep:7: 1db618b791dc87ac0bd5504f69434273 header.o

Coneed uses $srcdir to find dcr.c, and the setting of cppflags in
default.cppflags.env to find lex.h. Once coneed compiles dcr.o,
it finds the unresolved symbol for the header function, finds a
definition of header() in header.c, and compiles header.o to verify
that it defines the symbol. Coneed finds no other unresolved symbols
supplied by other source code in $srcdir (c.f. printf), so it stops.
Coneed uses “dodep (c cc c o)” to compile the source code in $srcdir
into object files in the current directory (obj). This generates
*.o.dep.do and *.o.do for each source file: those for dcr are shown,
the ones for header are identical.

obj/dcr.o.dep.do:1: #!/dis/sh
obj/dcr.o.dep.do:2: dep = $1
obj/dcr.o.dep.do:3: o = `{echo $dep | sed 's,\.dep,,'}
obj/dcr.o.dep.do:4: (stem ext c) = `{crext c $srcdir^$o}
obj/dcr.o.dep.do:5:
obj/dcr.o.dep.do:6: adddep $o /env/cc /env/cflags /env/cppflags
obj/dcr.o.dep.do:7: adddep $o $c
obj/dcr.o.dep.do:8: adddep $o `{/lib/do/c/findh $c}

obj/dcr.o.do:1: #!/dis/sh
obj/dcr.o.do:2: o = $1
obj/dcr.o.do:3: (sum c) = `{grep '\.c$' $o^.dep}
obj/dcr.o.do:4:
obj/dcr.o.do:5: if {no $cc} {cc = cc}
obj/dcr.o.do:6:
obj/dcr.o.do:7: if {
obj/dcr.o.do:8: flag x +
obj/dcr.o.do:9: os -T $cc $cflags $cppflags -c $c
obj/dcr.o.do:10: } {
obj/dcr.o.do:11: echo 'objs = $objs '^$o > $o^.relay
obj/dcr.o.do:12: }

Before credo runs dcr.o.do it runs dcr.o.dep.do to generate dcr.o.dep
(header.o likewise). To find the paths to C header files it calls findh,
which gathers header-file #includes from the C source files,
and searches for them in local and system directories using cppflags
and the search list printed by $cpp -v.

obj/dcr.o.dep:1: 131eb9ab2f6a65b34e0158de1b321e3c /env/cc
obj/dcr.o.dep:2: 0 /env/cflags
obj/dcr.o.dep:3: 9134215aef7b4657beb5c4bb7a20d4a1 /env/cppflags
obj/dcr.o.dep:4: 2fed3667f390fd80993e090a7eb92c75 ../src/dcr.c
obj/dcr.o.dep:5: feea1fa232f248baa7a7d07743ee86c4 ../src/lex.h
obj/dcr.o.dep:6: fb584676de41ee148c938983b2338f5b /n/C/cygwin/usr/include/stdio.h
obj/dcr.o.dep:7: f6409b1008743b1866d4ad8e53f925cc /n/C/cygwin/usr/include/string.h
obj/dcr.o.dep:8: 468b1dd86fef03b044dceea020579940 /n/C/cygwin/usr/include/errno.h
obj/dcr.o.dep:9: 4e6678324ba6b69666eba8376069c950 /n/C/cygwin/usr/include/stdlib.h
obj/dcr.o.dep:10: c7575313e03e7c18f8c84a5e13c01118 /n/C/cygwin/usr/include/inttypes.h
obj/dcr.o.dep:11: a8fd5fa102b8f74d1b96c6c345f0e22d /n/C/cygwin/usr/include/unistd.h

obj/header.o.dep:1: 131eb9ab2f6a65b34e0158de1b321e3c /env/cc
obj/header.o.dep:2: 0 /env/cflags
obj/header.o.dep:3: 9134215aef7b4657beb5c4bb7a20d4a1 /env/cppflags
obj/header.o.dep:4: ecd87752a211e69078e6bc37afbb561b ../src/header.c
obj/header.o.dep:5: feea1fa232f248baa7a7d07743ee86c4 ../src/lex.h
obj/header.o.dep:6: fb584676de41ee148c938983b2338f5b /n/C/cygwin/usr/include/stdio.h

At this point credo has the dependencies for dcr.exe, so it works
through them, calling the *.relay files—generated by the object files’
do scripts—to gather object names in $objs.

obj/dcr.o.relay:1: objs = $objs dcr.o

obj/header.o.relay:1: objs = $objs header.o

There’s not much to do since coneed already compiled the object files,
so it links them, and prints an os command to run the executable,
since Inferno can’t directly run a Cygwin executable.

Clean up

To clean up, we remove generated files in the src and obj directories.

; rm -f *.redoing *.redont *.renew *.reold *.sum

This removes all the temporary state files created by credo.
Once this is done credo will rebuild from scratch, and reconstruct
each target’s view of its dependencies.

; rm -f `{lsdo | sed 's,^c?redo ,,'}

This removes all the targets created by do scripts. The lsdo command
(called by credo with no targets) prints credo commands for all the
credo targets in the current directory. For example:

; cd src; credo
credo dcr.c

; cd obj; credo
credo dcr.exe
credo dcr.exe.dep
credo dcr.o
credo dcr.o.dep
credo header.o
credo header.o.dep

Once the targets are removed, credo will unconditionally generate them.

; rm -f *.do *.dep *.relay

This removes all the instructions credo uses to build files.
These may usually be regenerated by dodep, adddep (which adds
to the given target’s dependency list the given files), and
rmdep (which removes the given files).

A single library script is provided to remove the state, target,
and dodep files.

; rm -f *.env

This removes default and per-target environment settings.

Once we remove these sets of files, the directories contain only the
files present in their starting state.

Age: absolutely minimal version control

The timing of this post was inspired by a rant against git by Mike Taylor.

Here’s a version-control system in two rc (Plan 9 shell) scripts. Age copies files (without certain derived files such as objects and archives) to a backup directory under the current directory, and stores in that directory a tgz for distribution. Aged gives you a contextual difference of files changed since your last save. (There’s also little scripts called stardate and monnum that print YYYYMMDD.)

It’s got about none of git’s features, and is woefully inefficient, but it does one thing really well: have a backup when you make a bad change, if you use it regularly. I’ve used this for a few weeks, and it’s Good Enough.

nb—search and index notes in files by keyword

[Download a UTF-8 version of this file.]

nb name search index keyword
nb—search and index notes in files by keyword


nb keyword


nb description search keyword path file line acme
Nb searches for the given keyword in each nbindex file listed in
$HOME/nbindexes.  If it finds a match, nb prints the path, filename,
and line number of the indexed file in the format acme uses to refer to
lines in a file.


nb file format store index line path file line acme nbindexes
This file is an example.  Nb searches all files in the current directory
for lines which begin “^nb ”, and copies them into the file nbindex.
Each match is listed with its filename, and line number, e.g. for acme
to show the line on a right-click of the mouse.  It then adds the
path and filename of the current directory’s nbindex file to the file
$HOME/nbindexes.

By convention I use 1 blank line to separate paragraphs after an nb
line, and 2 blank lines to separate nb lines, but nb doesn’t care about
this.


nb nbindex example search keyword
This is the nbindex file for this file.

	nb.1:1 name search index keyword
	nb.1:5 keyword
	nb.1:8 description search keyword path file line acme
	nb.1:15 file format store index line path file line acme nbindexes
	nb.1:28 nbindex example search keyword
	nb.1:50 source plan9port rc script
	nb.1:70 port plan9port rc shell script grëp
	nb.1:75 author jason.catena@gmail.com
	nb.1:78 bugs

This is the output of the command “nb keyword”.  It lists the full
path since it may refer to files anywhere in the filesystem.

	/usr/local/plan9/bin/nb.1:1 name search index keyword
	/usr/local/plan9/bin/nb.1:5 keyword
	/usr/local/plan9/bin/nb.1:8 description search keyword path file line acme
	/usr/local/plan9/bin/nb.1:28 nbindex example search keyword


nb source plan9port rc script
#!/usr/local/plan9/bin/rc
flag e +

catalog=$HOME^'/nbindexes'
if(test -r $catalog && ! ~ $1 ''){
	list=`{cat $catalog}
	grëp $* $list | sed 's,nbindex:[0-9]+: ,,'
}

grep -n '^nb ' * >[2]/dev/null | sed 's,: ?nb +, ,' > nbindex
echo `{pwd}^'/nbindex' >> $catalog
if(~ $TMPDIR ''){
	TMPDIR=/var/tmp
}
tmp=$TMPDIR^'/nbindexes.'^$USER^'.'^$pid
sort -u $catalog > $tmp
mv $tmp $catalog


nb port plan9port rc shell script grëp
Nb in written in plan9port’s rc, but should be straightforward to port
to any shell.  Use grep instead of grëp.


nb author jason.catena@gmail.com


nb bugs
Nb generates the index after it searches, to present results immediately,
so it does not show changes since its last run.  If you want to see
the latest changes, then run nb with no parameters before you supply a
keyword (e.g. in acme, middle-click on nb after you edit an nb line or
following text).

Update Go scanner to accept non-ASCII operators

Read the discussion of this change on the golang-nuts mailing list. See this notice in UTF-8 format.

For each line, the scanner accepts, in place of the first operator, any of the remaining operators, and outputs a token whose string (set in $GOROOT/src/go/token/token.go) is the first operator.

! ¬

!= ≠

& ∧

&& ⋀

&= ∧=

&^ ∧¬

&^= ∧¬=

… …

:= ≔

<- ←

<< ≪

<<= ≪=

<= ≤

== =? ≟

>= ≥

>> ≫

>>= ≫=

^ ⊻

^= ⊻=

| ∨

|= ∨=

|| ⋁

To install, copy $GOROOT/src/pkg/go/scanner/scanner.go to another file.

Replace scanner.go with scanner.go

Run $GOROOT/src/all.bash and check for 0 unexpected errors.

Changes to scanner.go update gofmt, which accepts UTF-8 operators and outputs their ASCII equivalents.  This mkfile production rule uses gofmt as a preprocessor to create a sharable and compilable file.

%.go: %.ℊℴ

cat $stem.ℊℴ | gofmt > $stem.go

See these files for an example of each new operator form in the context of a simple Go program.

myscanner.ℊℴ

chan.ℊℴ

Please mail me privately if something doesn’t work with this code, to avoid noise on the golang-nuts list, since we’re no longer discussing officially-released code.

Encouragement for the new craftsman

Do you now bear responsibility for the architecture of some code? I’d like to talk just with you.

I’d like to say that I think it’s really important to “listen and then lead”. You’re responsible to the customers of your code for its quality, and you’ll get a lot of pressure to “just do this” or “just do that”. The processes you put in place, to control the quality of contributions, are for the benefit of its users and the code as a whole—to maximize usefulness, not necessarily to maximize development.

Sometimes it’s okay, even necessary, to inconvenience a few people (including the contributors) in order that you don’t fail an even larger number of people. You’re not (at least I wasn’t) able to please everyone, and trying to do so can easily cause harm, both short term (immediate bugs) and long term (technical debt and bad architecture). It’s your, just your, job to balance all this and do what’s right overall, and it’s very much a judgement call.

Some of this might be replaced by a change control board’s decisions, but if you go that route, then you must have a change process that everyone follows, regardless of the demands of the moment. Some people are really slick and sophisticated in getting what they want, especially when it doesn’t hurt them if they end up creating a problem. (Heh, the more I reread that the more it sounds like parenting.)

I found I had to be confident enough to implement decisions and defend them, but flexible enough to shift policy and correct errors. On the plus side, I think these skills will help move you past your current position, and help you control the direction in which your career heads.

You’re doing pretty well if you don’t instantly take sides in conversation on the merits, which is better than I sometimes do. Even if you’re slow making decisions—and there is so much to learn—a little slow is ok, since it gives a chance for more reflection and more mature ideas. Don’t be afraid to be wrong: if you never make mistakes, never make a bold architectural decision, then you’re playing it too safe, and you could probably be replaced with a small shell script.

Take care and good luck!

Software Craftsmanship and Programmers’ Status

This post quotes, collects, organizes, and remixes several sources to provoke comparison and consideration. There’s not a lot of new theory, beyond my idea that this stuff belongs together in the first place.

A change in status

In 1985 Peter Naur described, in Programming as Theory Building, where we still are as a profession (emphasis mine).

[M]uch current discussion of programming seems to assume that programming is similar to industrial production, the programmer being regarded as a component of that production, a component that has to be controlled by rules of procedure and which can be replaced easily. Another related view is that human beings perform best if they act like machines, by following rules, with a consequent stress on formal modes of expression, which make it possible to formulate certain arguments in terms of rules of formal manipulation. Such views agree well with the notion, seemingly common among persons working with computers, that the human mind works like a computer. At the level of industrial management these views support treating programmers as workers of fairly low responsibility, and only brief education.

Here’s where he suggests we be.

On the Theory Building View the primary result of the programming activity is the theory held by the programmers. Since this theory by its very nature is part of the mental possession of each programmer, it follows that the notion of the programmer as an easily replaceable component in the program production activity has to be abandoned. Instead the programmer must be regarded as a responsible developer and manager of the activity in which the computer is a part. In order to fill this position he or she must be given a permanent position, of a status similar to that of other professionals, such as engineers and lawyers, whose active contributions as employers of enterprises rest on their intellectual proficiency.

Further, he comments on necessary changes to programmer education.

The raising of the status of programmers suggested by the Theory Building View will have to be supported by a corresponding reorientation of the programmer education. While skills such as the mastery of notations, data representations, and data processes, remain important, the primary emphasis would have to turn in the direction of furthering the understanding and talent for theory formation. To what extent this can be taught at all must remain an open question. The most hopeful approach would be to have the student work on concrete problems under guidance, in an active and constructive environment.

An active and constructive environment

It occurs to me that the Master Craftsmanship Teams described by Bob Martin comprise distinct, guiding roles, in a context of “concrete problems”, to help us create this “active and constructive environment”. Bob proposes a static structure (remixed below, original emphasis) though which a person progresses (or not) through a career, but in my experience this classification applies between a person and each of many projects.

Apprentice. The apprentices write a significant amount of code—most of which is discarded by the journeymen.

Journeymen. They can make short term tactical decisions, and direct their apprentices accordingly. At the same time they take technical and architectural direction from the master…. The journeymen write a lot of code – almost as much as the master. They also throw away two thirds of the code written by the apprentices, and drive them to redo it better – always better. They teach the apprentices how to refactor, and often pair with them and sit with them while reviewing their code…. When an apprentice delivers code that the journeyman finds acceptable, the journeyman will commit that code to their main line. Otherwise he sends it back to the apprentice to do over…. The Journeymen see every line of code written by their apprentices, and they write a significant amount themselves. Pairing is prevalent though not universal. Anyone on the team can pair with anyone else. It should not be uncommon for the master to pair with the apprentices from time to time.

Master. The master probably writes more code than any of the team members. This code is often architectural in nature, and helps to establish the seed about which the system will crystalize. The master also sets technical direction and overall architecture, and makes all critical technical decisions. However, the chief job of the master is to relentlessly drive the journeymen towards higher quality and productivity by establishing and enforcing the appropriate disciplines and practices…. By the same token the master accepts subsystems from the journeymen and, if acceptable, commits it to the master main line…. The master sees every line of code in the system.

There’s a lot of code written by this team, but a pair of expert eyes reviews all of it. This is manageable if the scope of the work is not too large: my build systems weigh in at under 10,000 lines of (possibly too) intricate and clever shell scripts, makefiles, and flat data files. As with any software system, technical debt is the biggest hurdle to efficiently adding new code. In my case most of the technical debt was revealed by a new point of view (aspect-oriented and functional programming), and fundamental mechanism of customization (sed-scripts rather than sets of macros with varying parts), added only two years before I left my ten-year project.

Ethics and professional responsibility

If software craftsmen would be professionals (note above how Naur separates us from engineers), we should compare existing computer and information science codes of ethics, for example a 1998 Software Engineering Code of Ethics and Professional Practice (SECEPP), to a draft Software Craftsman’s Ethic (SCE, in bold face). Here I roughly group practices by similar intent: the last two groups catch all unmatched practices. (In the remainder of this post, only the italicized paragraph tags are my words.)

Follow process. We follow our chosen practices deliberately, to ensure quality in our work. We adopt development processes to suit our own unique abilities and talents. Ensure an appropriate method is used for any project on which they work or propose to work. Work to follow professional standards, when available, that are most appropriate for the task at hand, departing from these only when ethically or technically justified. Ensure that clients, employers, and supervisors know of the software engineer’s commitment to this Code of ethics, and the subsequent ramifications of such commitment.

Continue to learn. We are skill-centric rather than process-centric. We build competency in a broad range of languages and technologies to meet the needs of our customers. We achieve technical and design excellence through continuous and deliberate learning and practice. We desire to remain Software Craftsmen and write code for our entire careers. We value long lived technologies and languages and are wary of proprietary tools. We are generalists. Not knowingly use software that is obtained or retained either illegally or unethically. Strive to fully understand the specifications for software on which they work. Ensure adequate documentation, including significant problems discovered and solutions adopted, for any project on which they work. Further their knowledge of developments in the analysis, specification, design, development, maintenance and testing of software and related documents, together with the management of the development process. Improve their ability to create safe, reliable, and useful quality software at reasonable cost and within a reasonable time. Improve their ability to produce accurate, informative, and well-written documentation. Improve their understanding of the software and related documents on which they work and of the environment in which they will be used. Improve their knowledge of relevant standards and the law governing the software and related documents on which they work. Improve their knowledge of this Code, its interpretation, and its application to their work.

Protect reputation by accepting responsibility. We build our reputations on delivered quality. Our livelihoods and reputations are interdependent. We are proud of our work and the manner of our work. We take responsibility for our work by signing it. We are proud of our portfolio of successful projects. Accept full responsibility for their own work. Not promote their own interest at the expense of the profession, client or employer. Recognize that violations of this Code are inconsistent with being a professional software engineer. Recognize that personal violations of this Code are inconsistent with being a professional software engineer.

Network to sow the field. We respect master craftsmen as the key to Software Craftsmanship. We will help other craftsmen in their journey. We can point to the people who influenced us and who we influenced. We recommend other craftsmen, hanging our own reputation on their performance. We work in a community with other craftsmen. Not unjustly prevent someone from taking a position for which that person is suitably qualified. Extend software engineering knowledge by appropriate participation in professional organizations, meetings and publications. Support, as members of a profession, other software engineers striving to follow this Code.

Clean code. We write code that is easy to read, understand and modify. We put predictable and forgiving user interfaces on our software. Be accurate in stating the characteristics of software on which they work, avoiding not only false claims but also claims that might reasonably be supposed to be speculative, vacuous, deceptive, misleading, or doubtful.

Test and maintain. We design software for testing and maintenance. We build long-lived applications and support them throughout their life. We automate every task possible (especially testing). We drive our development with tests. Ensure adequate testing, debugging, and review of software and related documents on which they work. Treat all forms of software maintenance with the same professionalism as new development.

Strive for perfection. We deliver our work to our customer without known defect. We value quality over feature set or delivery data. We consistently exceed and raise our customer’s expectations of quality software. We build in quality to the software we create. Take responsibility for detecting, correcting, and reporting errors in software and associated documents on which they work.

Plan tradeoffs. We say no to prevent doing harm to our craft and our projects. Strive for high quality, acceptable cost and a reasonable schedule, ensuring significant tradeoffs are clear to and accepted by the employer and the client, and are available for consideration by the user and the public. Ensure realistic quantitative estimates of cost, scheduling, personnel, quality and outcomes on any project on which they work or propose to work and provide an uncertainty assessment of these estimates. Ensure realistic quantitative estimates of cost, scheduling, personnel, quality and outcomes on any project on which they work or propose to work, and provide an uncertainty assessment of these estimates.

Respect customers. We deliver what our customers really need, not just what they ask for. We build trust and respect in long term relationships with our customers. We take our customer’s needs as seriously as they do. We desire demanding users. Work to develop software and related documents that respect the privacy of those who will be affected by that software. Ensure that specifications for software on which they work have been well documented, satisfy the users’ requirements and have the appropriate approvals.

Promote coworkers and colleagues. We take on Apprentices and put them in an environment where they can learn our craft for themselves. Ensure that software engineers are informed of standards before being held to them. Attract potential software engineers only by full and accurate description of the conditions of employment. Offer fair and just remuneration. Not ask a software engineer to do anything inconsistent with this Code. Encourage colleagues to adhere to this Code. Assist colleagues in professional development. Credit fully the work of others and refrain from taking undue credit. Review the work of others in an objective, candid, and properly-documented way. Give a fair hearing to the opinions, concerns, or complaints of a colleague. Assist colleagues in being fully aware of current standard work practices including policies and procedures for protecting passwords, files and other confidential information, and security measures in general. Not influence others to undertake any action that involves a breach of this Code.

Demonstrate competence. We accept that different craftsmen are better suited for one kind of job over another. Provide service in their areas of competence, being honest and forthright about any limitations of their experience and education. Ensure that they are qualified for any project on which they work or propose to work by an appropriate combination of education and training, and experience. Only endorse documents either prepared under their supervision or within their areas of competence and with which they are in agreement. Assign work only after taking into account appropriate contributions of education and experience tempered with a desire to further that education and experience. Not unfairly intervene in the career of any colleague; however, concern for the employer, the client or public interest may compel software engineers, in good faith, to question the competence of a colleague. In situations outside of their own areas of competence, call upon the opinions of other professionals who have competence in that area.

Manage projects. We keep projects from spinning in circles. We don’t wait for complete definition before beginning. We never allow ourselves to be blocked. Identify, document, collect evidence and report to the client or the employer promptly if, in their opinion, a project is likely to fail, to prove too expensive, to violate intellectual property law, or otherwise to be problematic. Ensure proper and achievable goals and objectives for any project on which they work or propose. Ensure good management for any project on which they work, including effective procedures for promotion of quality and reduction of risk.

Own code. We believe the code is also an end, not just a means. We treat software like capital. Ensure that there is a fair agreement concerning ownership of any software, processes, research, writing, or other intellectual property to which a software engineer has contributed.

Represent the public interest. Moderate the interests of the software engineer, the employer, the client and the users with the public good. Approve software only if they have a well-founded belief that it is safe, meets specifications, passes appropriate tests, and does not diminish quality of life, diminish privacy or harm the environment. The ultimate effect of the work should be to the public good. Disclose to appropriate persons or authorities any actual or potential danger to the user, the public, or the environment, that they reasonably believe to be associated with software or related documents. Cooperate in efforts to address matters of grave public concern caused by software, its installation, maintenance, support or documentation.Be fair and avoid deception in all statements, particularly public ones, concerning software or related documents, methods and tools. Consider issues of physical disabilities, allocation of resources, economic disadvantage and other factors that can diminish access to the benefits of software. Be encouraged to volunteer professional skills to good causes and contribute to public education concerning the discipline. Identify, document, and report significant issues of social concern, of which they are aware, in software or related documents, to the employer or the client. Identify, define and address ethical, economic, cultural, legal and environmental issues related to work projects. Temper all technical judgments by the need to support and maintain human values. Help develop an organizational environment favorable to acting ethically. Promote public knowledge of software engineering. Obey all laws governing their work, unless, in exceptional circumstances, such compliance is inconsistent with the public interest. Express concerns to the people involved when significant violations of this Code are detected unless this is impossible, counter-productive, or dangerous. Report significant violations of this Code to appropriate authorities when it is clear that consultation with people involved in these significant violations is impossible, counter-productive or dangerous.

Deal fairly. Use the property of a client or employer only in ways properly authorized, and with the client’s or employer’s knowledge and consent. Ensure that any document upon which they rely has been approved, when required, by someone authorized to approve it. Keep private any confidential information gained in their professional work, where such confidentiality is consistent with the public interest and consistent with the law. Accept no outside work detrimental to the work they perform for their primary employer. Promote no interest adverse to their employer or client, unless a higher ethical concern is being compromised; in that case, inform the employer or another appropriate authority of the ethical concern. Be careful to use only accurate data derived by ethical and lawful means, and use it only in ways properly authorized. Maintain the integrity of data, being sensitive to outdated or flawed occurrences. Maintain professional objectivity with respect to any software or related documents they are asked to evaluate. Not engage in deceptive financial practices such as bribery, double billing, or other improper financial practices. Disclose to all concerned parties those conflicts of interest that cannot reasonably be avoided or escaped. Refuse to participate, as members or advisors, in a private, governmental or professional body concerned with software related issues, in which they, their employers or their clients have undisclosed potential conflicts of interest. Ensure that software engineers know the employer’s policies and procedures for protecting passwords, files and information that is confidential to the employer or confidential to others. Provide for due process in hearing charges of violation of an employer’s policy or of this Code. Not punish anyone for expressing ethical concerns about a project. Avoid associations with businesses and organizations which are in conflict with this code. Not give unfair treatment to anyone because of any irrelevant prejudices.

My experience with Master Craftsman Teams

“Uncle Bob” wrote yesterday about Master Craftsman Teams.

The thing that struck me most about this article is how closely it matched the way my own team of coworkers, who came and went, contributed and moved on, helped me develop two significant software build systems over ten years.

Without a doubt I had the role of “master”, and as Peter Naur would say, had the theory of the program in my head. I developed the build system from empty directories twice, to build products for two different software systems. Each time I had key responsibility to fit the system to customers’ needs, and decide how it would involve internally. (I’ve also been programming for thirty years, since BASIC on my first computers in grade school.)

I delegated what I saw fit to a handful of people who grew in responsibility as they demonstrated they understood and could make creative improvements to the system (journeyman). One of the journeymen became committer when I recently left. Since no-one had the theory as well as I did, architectural responsibility devolved from my fiat to a change control board, so that everyone could bring bits and pieces of their expertise to together make better decisions.

Hundreds of small changes were made over the years by a few dozen people who did not really grasp the principles of the system, so needed a significant amount of guidance. I reviewed every line of code in the system, and know how it all works together. I expect that some of this efficiency will suffer in future, at the advantage—to the organization that owns it—of not having “the keys to the kingdom” in one set of hands.

I’m not sure how well this approach would sit with people as a formalized organization chart and pay scale, but I’ve seen it work very well in practice with a software project I managed. Further, I believe that if you look closely most software projects in practice run this way (eg Linux has Linus, Alan, various subsystem committers, and a hoi polloi of patchers.) Even further, a single person may be an apprentice to one project, a journeyman to another, and a master to a third.

Perhaps the thing to do is to compensate people outside the company by bounties for the code they contribute; set up a contractor arrangement for journeymen; and actually hire only masters. This keeps companies small, focused on their core competencies and most valuable people. Even better for the community of programmers, this provides a structure for programmers to put food on the table while contributing to as many projects, and at the levels of depth, as suits their inclinations and abilities.

Agile practice one-liners, and Alexander on refactoring

A few one-line descriptions of Agile practices, because I feel they get to the heart of each.

Iterative development with feature teams: a few developers complete the development cycle (refactor, code, integrate, build, and test), for each of a few features, in two weeks to two months.

Refactoring: instead of a static, outdated architecture, rework the codebase with each feature.

Test-driven development: write tests of functionality for user stories, then write code to pass these tests.

Pairing and collective ownership: two developers introduce fewer errors to review and test for, and many developers distribute knowledge of the code.

Frequent integration: merge and build new changes at least daily, test at least weekly.

Automated regression test: collect all test cases into push-button test suites, run shortly after each deliverable build.

I also found a passage on refactoring from an OOPSLA keynote by Christopher Alexander:

It turns out that these living structures can only be produced by an unfolding wholeness. That it, there is a condition in which you have space in a certain state. You operate on it through things that I have come to call “structure-preserving transformations,” maintaining the whole at each step, but gradually introducing differentiations one after the other. And if these transformations are truly structure-preserving and structure-enhancing, then you will come out at the end with living structure. Just think of an acorn becoming an oak. The end result is very different from the start point but happens in a smooth unfolding way in which each step clearly emanates from the previous one.

Alexander, C. 1999. The origins of pattern theory: The future of the theory, and the generation of a living world. IEEE Software 16, 5 (Sept/Oct), 71—82. http://doi.ieeecomputersociety.org/10.1109/52.795104

Blacksmithing a Train Engine

Yet another analogy comparing software development to a hardware activity: in this case, train engines.

(Programs|engines) run, within the bounds of a (computer|track). The strength of the (computer|track) determines the rate at which they can run. The design and internal processes of the (program|engine) determines the rate at which they do run. Certain designs of (programs|engines) make it easier to handle various loads of (data|cars), since (data|cars) have different properties when placed under the stress of running.

If (data|cars) properly interface with the (file|rail) format, then the (program|engine) can handle them. If not, then the (program|engine) will fail to make any progress, or crash. The more subtle the difference between the format expected by the (program|engine) and that implemented by the (data|cars), the longer it will run before the difference creates a problem. Some problems might not be found at all, if no situation arises for which the difference has any noticeable effect. The (program|engine) handles a certain amount of (data|cars) before slowing to a crawl. Users can handle more (data|cars) in the same amount of time by running additional (programs|engines) in parallel, but improvement is limited by the time required to coordinate final and intermediate delivery.

Standard (file|rail) formats are better, because more (programs|engines) and (data|cars) are built to use them, their properties are better known, and users can even reuse them in different systems. However, vendors must differentiate themselves to drive business their way, so they often attempt to lock users into a particular format, in addition to (sometimes instead of) competing on the merits of their (program|engine).

In the early days of (programs|engines), each part was laboriously crafted and fit into place by hand, and often failed spectacularly from the additional pressures of running. Each part had to be crafted with a knowledge of all the stresses under which it would run, from materials suitable for the basic processes which powered the (program|engine). Nonetheless, its appearance made a more significant impact on users and bystanders, unless its performance differed drastically from others’.

The tools with which the craftsmen created (program|engine) parts were themselves created by the same process, so a craftsman’s product quality varied dramatically with his knowledge of toolmaking.

There were common parts, interfaces, and controls expected by users who had used similar (programs|engines). There was a great need to explain to users who had not: how they could expect the (program|engine) to behave with their (data|cars), and to make sure they were compatible with the (file|rail) format; how to control the (program|engine) at a basic level, and how to get a feel for its advanced capabilities; and how to get the (data|cars) to the (program|engine), and how to pass them off to others.

Smithing Tools to Smith Tools

Following text from Wikipedia on blacksmiths.

Over the centuries (blacksmiths|programmers) have taken no little pride in the fact that theirs is one of the few crafts that allows them to make the tools that are used for their craft. Time and tradition have provided some fairly standard basic tools which vary only in detail around the world.

There are many other tools used by smiths, so many that even a brief description of the types is beyond the scope of this article and the task is complicated by a variety of names for the same type of tool. Further complicating the task is that making tools is inherently part of the smith’s craft and many custom tools are made by individual smiths to suit particular tasks and the smith’s inclination.

With that caveat one category of tools should be mentioned. A (jig|hack) is generally a custom built tool, usually made by the smith, to perform a particular operation for a particular task or project.

More on software blacksmithing, apprenticeship, and artistry: Craftsmanship as a Bridge.