An Obsession with Everything Else

Unit Testing The UI

2008-12-31T07:35:00.001-08:00

The conventional wisdom on unit tests is that they stop at the UI. There are, after all, QA-centric tools that automate the tasks of pressing buttons, choosing menu items, and putting text into forms.

But a lot of what I do at work is writing code that generates web pages. It's been relatively easy to introduce unit tests into utility classes or the service layer, but I wanted to give myself the same stability in the UI universe. Since my team is good about separating models, views, and controllers, I figured out an easy way to bring unit tests up against the UI: Execute methods in the controller, and then inspect the resulting model.

We use the Spring framework, which makes this concept pretty easy. It even includes mock objects for setting up HTTP requests and responses. The generic Spring flow enforces good separation: Your controller's handle method gets called and is expected to return a ModelAndView object. Your unit test can call the same method with appropriately configured mock request objects and then peer into the model object to make sure that values are set properly.

Since our views are usually JSPs, which require a running web container, I don't think there's a good way to write unit tests for that layer. But since the view relies on the model being correct, and the unit tests can look at the model, I think it's a pretty good stopping point. I don't want to rewrite those automated button-pushing tools, after all.

Trying to incorporate this idea into my Mac/iPhone programming, I realized I could go a little bit further than I could in the web container world. I'm working on an iPhone app to enable crazy Derrick meal planning, and one of the controllers is responsible for showing me things I have to do on any one day. I started with that as an experiment. To enable my "UI-ish" unit tests, I have split Apple's default template for this controller into a definite Model object and a definite "ViewLogic" object. The ViewLogic object handles code for formatting the date, determining if the table is present, how many sections it has, and what the title of each section is. All of these vary depending on the state of the model, because there are different categories of things to worry about for any given day, and you could have 0-3 of those categories. My test invokes methods on my controller that update the model object. Then I inspect that model and execute various methods on the ViewLogic object to make sure they're returning correct values based on the model.

Now I can refactor the methods in ViewLogic, some of which have a relatively high cyclomatic complexity, with a high confidence that I won't break the code that relies on it or introduce new bugs.

There are a couple of consequences to this, though. One is a bit of class bloat. I can re-use the model object for at least one other view, but the ViewLogic object is tailored to that one view. The other consequence is that the ViewLogic has become an adapter, converting the methods that the framework expects to find (which have context about views and such that don't exist in the unit test world) into methods I define. Rather than doing the normal thing, then:


- (NSInteger)tableView:(UITableView *)tableView numberOfRowsInSection:(NSInteger)section {
    // do calculations about number of rows
}

I do this:


- (NSInteger)tableView:(UITableView *)tableView numberOfRowsInSection:(NSInteger)section {
    return [self rowsInSection: section];
}

Not necessarily bad, but it does make those methods a little anemic. I could pass nil as the table view, but I could see my code at some point needing to execute a method on the table view. So better to isolate "number of rows" logic in a method where I can call it even if I don't have a runtime UITableView object.

Annotation Processors

2008-12-28T14:24:00.000-08:00

Java 1.5 introduced the concept of annotations, markup that you can attach to classes, fields, and methods. We didn't use them a lot at my last job, but in my last round of interviews the subject came up, and I spent some time researching them. At my new job, we — or, rather, the frameworks we use — use them heavily.

I've come to really like them. Like any technology, they can become a hammer that makes everything look like a nail, but they offer a powerful layer on top of code. In particular, they can be used to add cross-cutting functionality that shouldn't be shoehorned into every object. To take one example, you could set up an "Audited" annotation that would fire every time a method is called and would write out the invoking information to a log file. The audited method wouldn't know anything about it and wouldn't need to support the auditing infrastructure: The framework around the method would see the annotation and do the work.

Because they are treated as code, they serve as more-correct documentation. If you the programmer see an @Audited annotation, you know that the given method gets audited. Contrast that with a normal Javadoc comment, which might reflect the method's behavior when it was first written (not audited) but not the current behavior (audited).

Working with annotations in running code is fairly easy. Java's reflection layer, which lets you inspect and manipulate runtime objects, has simple methods that tell you if a given structural element has a given annotation.

But I was intrigued by the concept of annotation processors, compile-time parsers that work off of the annotations in the source. Sun's primary use case for an annotation processor is generating additional files based on an annotation (classes to support XML marshalling and unmarshalling of a given object, for instance). I wondered if you could write an annotation processor that would inspect the source code looking for issues. The earlier you catch problems, the cheaper they are to fix: If I could use annotations to do extra checks on certain code, I could make them compile-time issues that would thus never get released.

I had a particular scenario in mind. At my work, we have some JavaScript code that invokes some of our system's Java objects through indirection, JSON, and reflection. Java supports method overloading (two methods with the same name but different arguments) but JavaScript does not. I inadvertently discovered how this could be a problem when I added an overloaded method to a Java object used by our JavaScript, and it broke our website (in development) for a couple of hours as we tracked down the problem. The JavaScript layer was invoking the new method, not the old method. Now I know this and avoid method overloading in the relevant classes. But some new programmer some day won't know and may make the same mistake. Or I might forget. Every time you have a process that people need to remember, you guarantee that one day someone will forget.

Unfortunately, documentation and examples for this system are sparse. So in the interest of helping others who have similar goals, I've attached the source code below. (There is a Visitor pattern implementation as well, but that is even more poorly documented. I need to revisit my code and figure that out at some point.) The code could use some touch-up, but it gets the idea across. Note that the code uses the Java 1.6 APIs, not the markedly different and unsupported Java 1.5 APIs.

My custom annotation is called RequireUniqueMethodNames and is defined as a class-level annotation. If present, my annotation processor (which you fire by adding a -processor argument to javac) will inspect the class to ensure that there are no overloaded methods. It is smart enough — and the annotation processing system is powerful enough — to distinguish overridden methods (allowed) from overloaded methods (not allowed).

I also allow you to exempt certain method names and not check private methods. These were to support the reality of the legacy code, in which there are overloaded methods that aren't exposed to JavaScript and private methods (which wouldn't be called by anyone) with duplicate names. A refactoring task in my queue will make these attributes irrelevant, but they're there for the moment.

First the annotation declaration:


@Documented
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.CLASS) // required with @Documented
@Inherited
public @interface RequireUniqueMethodNames {
    /** Whether or not to enforce duplicate names on private methods */
    boolean enforceOnPrivates() default false;
 
   String[] exemptedMethodNames() default {};
}

And now the processor code:


public class UniqueMethodNamesProcessor extends AbstractProcessor {

    // stateful!
    private boolean enforcePrivates = false;

    //stateful!
    private List<String> exemptedMethodNames;

    public boolean process(Set<? extends TypeElement> annotations,
   RoundEnvironment curEnv) {

 // each element in the annotations set is a single Annotation
 // (and since we only support one, it's always RequireUniqueMethodNames)
 for (TypeElement te : annotations) {

     // just in case we're passed a stray
     if (!te.getQualifiedName().toString().equals("com.ea.sp.community.annotation.RequireUniqueMethodNames")) {
  notice("Sent a stray annotation: " + te.getQualifiedName().toString());
  continue;
     }

     for (Element element : curEnv.getElementsAnnotatedWith(te)) {
  enforcePrivates = false; // reset each time
  exemptedMethodNames = new ArrayList<String>();

  // double-check to ensure that each is a class
  if (!(element instanceof TypeElement)) {
      error("Sent the wrong type of element: " + 
     element.getKind().name());
      continue;
  }

  TypeElement clazz = (TypeElement)element;

  // figure out the annotation instance used on this class
  // note that class might have many annotations, but we know ours is
  // one of them
  for(AnnotationMirror am : elemUtils().getAllAnnotationMirrors(clazz)){
   // use elemutils... because we declare this annotation as
   // inherited, so we want to make sure we catch the subclasses
   if (am.getAnnotationType().asElement().getSimpleName()
       .toString().equals("RequireUniqueMethodNames")) {

       // todo-dfs: is there a better way to construct the
       // ExecutableElement to use for direct lookup?
   Map<? extends ExecutableElement,? extends AnnotationValue> 
       methodMap = elemUtils().getElementValuesWithDefaults(am);
   for (ExecutableElement curMethod : methodMap.keySet()) {
       AnnotationValue value = methodMap.get(curMethod);
       if (curMethod.getSimpleName().toString().
    equals("enforceOnPrivates")) {
    enforcePrivates = 
        ((Boolean)value.getValue()).booleanValue();
       }

       if (curMethod.getSimpleName().toString().equals("exemptedMethodNames")) {
    
    List<? extends AnnotationValue> exemptValues =
        (List<? extends AnnotationValue>)value.getValue();
    for (AnnotationValue arrayValue : exemptValues) {
        exemptedMethodNames.add(arrayValue.toString());
        
    }
       }
    
   }

      }
  }
   
  List<TypeElement> classChain = calculateClassHierarchy(clazz);

  // this will be a list of method objects that we maintain for
  // each class in the list. because we want to check uniqueness
  // within the universe of one class, not across the entire
  // source base
  List<ExecutableElement> methods = 
      calculateAllSuperMethods(classChain.subList(0,(classChain.size() - 1)));

  screenDuplicateMethods(methods,clazz);

     }
 }
 return false; 
    }

    /** Collate all the methods for every incoming TypeElement. This does not do
     *  a duplicate check because non-annotated classes are allowed to have
     *  duplicate names.
     */
    private List<ExecutableElement>
 calculateAllSuperMethods(List<TypeElement> superclasses) {

 List<ExecutableElement> retVal = new ArrayList<ExecutableElement>();
 for (TypeElement curClass : superclasses) {

     for (Element classMember : curClass.getEnclosedElements()) {
  if (!classMember.getKind().equals(ElementKind.METHOD)) {
      continue;
  }
  retVal.add((ExecutableElement)classMember);
     }
 }
 return retVal;
    }
     
 
    /** Looks for duplicate methods in the aggregate list of method names that
     *  have been accrued as we go up the superclass chain. 
     *  @param methodNames the list of ExecutableElements from all the superclasses
     *  @param curClass the current TypeElement
     */
    private void screenDuplicateMethods(List<ExecutableElement> methods,
                                        TypeElement curClass) {
              
 for (Element classMember : curClass.getEnclosedElements()) {
     
     
     if (!classMember.getKind().equals(ElementKind.METHOD)) {
  continue;
     }

     ExecutableElement method = (ExecutableElement)classMember;
     if (method.getModifiers().contains(Modifier.PRIVATE) &&
  !this.enforcePrivates) {
  continue;
     }

     // oddly, the method names are recorded as "method name" (with the quotes
     // from getannotationvalue. so look for that
     if (this.exemptedMethodNames.
  contains("\"" +method.getSimpleName().toString()+ "\"")) {
  continue;
     }
                
     if (!methodNameExists(method,methods)) {
  methods.add(method);
  continue;
     }



     //method exists, but is it an override? overrides are okay
     if (methodIsOverride(method,methods)) {
                   continue;
     }
               
     error(method.getSimpleName().toString() + " is a duplicate (not an override) of a method elsewhere in class " + method.getEnclosingElement().getSimpleName().toString() + " or in a superclass.");
 }
    }
    
    /** Recursively constructs a List of TypeElements that form the superclass
     *  chain for the given class.
     *  @param classDec the starting class to declare
     *  @return List of superclasses, with Object at 0 and the passed-in class
     *          at the end.
     */
     private List<TypeElement> calculateClassHierarchy(TypeElement startClass) {
  TypeElement superclass = 
      (TypeElement)typeUtils().asElement(startClass.getSuperclass());
  if (superclass == null ||
      superclass.equals(typeUtils().getNoType(TypeKind.NONE)) ) {
      // we're at Object, so make a list and unrecurse
      List<TypeElement> retVal = new ArrayList<TypeElement>();
      retVal.add(startClass);
             return retVal;
  } else {
      List<TypeElement> retVal = calculateClassHierarchy(superclass);
      retVal.add(startClass);
      return retVal;
  }
     }

     /** Does the given ExecutableElement have the same name as something in list
      */
    private boolean methodNameExists(ExecutableElement method,
                                               List<ExecutableElement> methods) {
 for (ExecutableElement existingMethod : methods) {
     if (method.getSimpleName().equals(existingMethod.getSimpleName())) {
  return true;
     }
 }
 return false;
    }    
      
    /** Determines if the passed in method is an override of a method in the list
     *  Overrides are generally okay even if they have the same name.
     *
     */
    private boolean methodIsOverride(ExecutableElement method,
                                      List<ExecutableElement> methods) {
       for (ExecutableElement superMethod : methods) {
    if (elemUtils().overrides(method,superMethod,
       (TypeElement)method.getEnclosingElement()) ) {
               return true;
    }
       }
       
       return false;
    }

     private void notice(String message) {
  print(Diagnostic.Kind.NOTE,message);
     }

     private void error(String message) {
  print(Diagnostic.Kind.ERROR,message);
     }

     private void print(Diagnostic.Kind kind,String message) {
  processingEnv.getMessager().printMessage(kind,message);
     }

    private Types typeUtils() {
 return processingEnv.getTypeUtils();
    }

    private Elements elemUtils() {
 return processingEnv.getElementUtils();
    }

More Unit Testing Revelations

2008-12-28T09:05:00.000-08:00

Since I started making unit tests an active part of my development tasks at home and at work, I've noticed a benefit beyond the promise of stability and support for refactoring.

Unit testing has made me a better programmer. More precisely, it has made me practice what I preach about good programming practice.

One well-known best practice in the OO world — and probably in the procedural programming world, too — is to keep chunks of code small, focused, and free of side effects. This is partly for code maintenance purposes: A large method is cumbersome for a new programmer to understand. But this kind of modularity also allows other programmers (or yourself) to more easily subclass a class and add new behavior to suit new needs.

But in the heat of coding a new feature, I don't always take the time to think about the best way to break it apart. I figure I'll fix it in a refactoring pass. Fifteen years as a professional programmer, and you'd think I know that that chance will never come. Bugs need fixing, or the powers-that-be move me on to a new feature.

With unit tests as part of my task, I'm forced to think about testable chunks, and those are usually exactly the level of size and functionality that makes sense from an object-oriented perspective. I find myself thinking about the tests I will write as I code the class, and that in turn forces me to realize that a given method is too big, has too many side effects, or is otherwise not testable. This is especially true because I have made the decision that our team shouldn't write unit tests for retrieving objects from a database, since that is all handled by the well-vetted Hibernate library. When I write a method that retrieves objects from the database and does calculations with them, the "you'll have to test this" part of my mind realizes that I need to move the calculations into their own method (or methods) so that I can test them even without a database connection. In other words, I move specific functionality into its own method, exactly where it belongs.

Over and over again, every time I code with unit tests in mind, I see myself doing more on-the-fly refactoring as I write a feature. I don't wait for the magical refactoring time that never comes: I do it as I go.

Testing Testing

2008-09-27T09:57:00.000-07:00

Since I claim to advocate Agile development practices, I recently decided to give unit testing, the writing of simple test suites that exercise an application's basic functionality, a solid try.

I've been in various organizations that have tried to do it, but the effort always founders and fails. Everyone knows unit tests are a good idea; no one ever gets any in place. When the subject comes up at job interviews, other companies seem to have had the same problem. New features always creep ahead in priority, and once the code becomes too big, writing unit tests becomes too daunting a task. Plus, the new feature is working, more or less, so the motivation decreases to write tests for it.

As I started my Mac/iPhone programming, however, I realized I had the perfect environment for unit testing: no deadline and no legacy code. So I wrote new code and the test cases to exercise it at the same time. (I didn't do test-driven development, where you write the tests first and then write the code to make them work.)

I haven't since become a unit testing zealot — at least I don't think so — but I have a renewed enthusiasm. In a normal development scenario, I would have written all my support classes and then a UI so that I could test them. If there was a bug, I'd have to step through all the layers I had written to find it. With unit testing, I found several bugs long before I had a UI. When I did attach a UI, the fact that it worked smoothly out of the gate was anticlimactic: I knew the underlying structure was solid, after all.

Recently, I thought of some code reorganization I could do on the project as well as some significant performance improvements. I spent an hour or so doing the work, shuffling this here and that there. I set up my unit tests to run, and they passed: I could assert that my changes hadn't broken anything major. That's a really good feeling; there was a bit of a rush when I saw "Build Succeeded" flicker by in the log.

Even with my short experiment, I realized the two major benefits of unit testing: catching bugs earlier ("The earlier you catch bugs, the cheaper they are to fix," runs one programmer mantra) and making sure that major code changes don't affect the core functionality.

But how to bring that to work? At Maxis, we have a lot of pressure to deliver features, an always-busy team, and a big ball of legacy code even for the website. Just like every other tech company. One of my co-workers and I have talked about it, and the only answer seems to be: just start writing them, and keep them running. The first few will be tough, because you need a support structure for them (especially in the form of mock objects, layers that conform to the contract of the real layer, but deliver static, simplistic data), but once they get going, hopefully it becomes easier to add new ones. I have a squirrely bit of logic in one piece of code: That's my first candidate.

The problem is selling the concept to the business side. I think in a lot of companies, the extra time spent writing unit tests is seen as time that could be spent writing a new feature. In a way that's true, but I now see that unit tests actually are a feature: stability. No one wants an unstable system, but the investment in stability doesn't obviously pay off for a long time. In fact, if you've done it right, the investment never seems to pay off because the system doesn't have as many problems. You can't measure how many problems there would have been if the investment hadn't been made. But you can measure the number of users using a new feature. So you get instant feedback that it was a good addition, and you provide a metric. How does unit testing compete against that?

A Server Programmer On The Client

2008-09-26T07:53:00.000-07:00

I have been teaching myself Objective-C over the last few months. I'm starting with some Mac programs, but let's be honest: The end goal is iPhone applications. I have some already in mind, and I'm actively working on one right now.

I have been a server-side programmer for the last 10 years. And certainly my current senior-programmerness has been forged in the universe of databases, web servers, and application servers. Not even browsers, though I know enough JavaScript and HTML to get by.

A couple months into my client side hobby, I realize how much my experience influences my thinking. A part of me keeps worrying about scalability and threading issues as I write my first application, and I have to remind myself to not overdesign. These are big issues on web servers: You are almost guaranteed to have multiple threads running through every piece of code in your site, especially on my current project, spore.com, where we average 40,000 people logged in at any given moment using 200 active threads on each of our 20 application servers. HTTP requests, complete with database queries and code-level processing, need to move through the system as fast as possible, and you need to be aware of the pieces that modify state outside of that request so that you can prevent multiple threads from changing the same piece of data at the same time. But for my client app, only one person runs it at a time. And Mac OS X keeps threading issues to a minimum. (Though they can exist.)

On the other hand, there are facets of client-side programming that I've all but forgotten. Memory management? Who still worries about that? It's possible to have memory leaks in Java, but the native garbage collection takes care of all but the most esoteric memory management tasks. (Of course, it can also introduce its own performance problem, as anyone can attest who's watched their web app stop dead for a fraction of a second every few seconds as the garbage collector kicks in.) Sure, Objective-C 2.0 has garbage collection, but that limits your application to Mac OS X 10.5. The primary book I'm using, Cocoa Programming For Mac OS X, still follows the reference counting pattern, a fragile system if ever there was one. I haven't yet looked at the autorelease pool pattern that other tutorials use. (On the other hand, being able to discard the memory for objects whenever you want is a nice side effect. You never know when Java's garbage collection scythe will come a-calling, rendering the language's destructors useless for cleanup.)

And linking befuddles me after 13 years with Java (3 on the client side). In Java, you just put the class file where the virtual machine can find it, and it's available to your code. Application servers introduce a bit more complexity, but not much. For my application, in order to integrate regular expressions (an aside: Really? They're not built in?), I have to pass an argument to the package linker and deal with cryptic error messages when it doesn't work.

But overall I'm enjoying this diversion into fat client programming. I think in the end it's making me a more well-rounded developer. As someone whose resume loudly proclaims "All-Purpose Programmer," this is probably a good thing.

Puzzle Quest

2008-07-05T09:01:00.000-07:00

I shouldn’t like Puzzle Quest: Challenge of the Warlords.

For one thing, it’s a role-playing game, and I tend not to like computerized versions of the genre. I find them boring: full of tedious dialog, boring storylines, random battles, and the inevitable “leveling up” times that require you to fight a lot of small battles just to get enough hit points to take on the next Big Bad.

For another, Puzzle Quest’s mechanic is novel for RPGs but trite in the larger universe: You fight each battle by playing Bejeweled, the ubiquitous puzzle game from PopCap, against your opponent. It’s a ridiculous conceit if you think about it, though no more so than other video game mechanics, I guess. As you clear colored gems off the board, you build up enough mana, or ingredients, to cast offensive and defensive spells. Clear a set of skulls, and you do direct damage to your opponent. Clear gold coins and you earn money. Clear a set of four and you get an extra turn. Rinse and repeat ad infinitum.

So why am I so addicted to an RPG that uses one of the most overexposed casual games as its sole means of battle? An equally addicted friend and I have chatted about this, and we can’t figure it out. I’ve done every little side quest. I’ve collected every possible companion. I’ve trained up my mount. I’ve forged items from runes I have found throughout the kingdom. I have, in short, opted to do all the things that I dislike about RPGs. Melissa has even quipped that she should buy me a d20. (As an aside, this game cost me $15 on XBox Live, and I’ve poured hours into it, giving it the best gameplay-to-price ratio in my collection.)

I think one key factor is that it transforms the solo game of Bejeweled into a two-person strategy game. You can’t clear any row you want: You have to consider how your opponent will be able to use the board. And the different strengths and weaknesses of the various monsters create different strategies. I favored green and red jewels in any fight against trolls, because the combination would let me drain the troll’s blue mana, which he can and does use constantly to regenerate health. I favored green gems in battles against elves just to prevent them from getting them, since they need very few to cast some nasty spells.

There are other facets of the game the make it appealing — the scant dialog you find is often humorous, and the music is at least good enough to not get tiring — but the heart of the game is the “strategic Bejeweled” component.

There was one thing about the game I didn’t like. By the time I got to the final boss, I had hit the top level for my character, I had forged some impressive items, and I had done virtually every side quest. But with him I was massively outclassed. Within minutes, he had obliterated me, and I had barely made a dent in his health. I expected an epic match, but this felt way out of balance with respect to the rest of the game. It felt disheartening, not challenging. On the other hand, I did beat him after a dozen or so attempts, so maybe it wasn’t as unbalanced as I thought. But I wonder if the casual player who doesn’t go to the lengths I did will find the final boss truly impossible. Perhaps the game adjusts his levels based on the player.

Maybe there’s a little RPG fan in me after all, just waiting for a game like Puzzle Quest to draw it out. I doubt I’ll be digging into Final Fantasy any time soon, but I am now one of the countless hordes waiting for Puzzle Quest’s forthcoming Plague Lord expansion.

Writing Pause

2008-04-10T15:10:00.001-07:00

Once I start at Maxis, I'm going to cut back on my freelance writing. One reason is that the team will be moving toward crunch mode. But the main reason would have been true in any new job: I want to build my team's trust in me before I do writer things such as going to wine tastings in the middle of the day or visiting wineries or whatever. My current employer has given me a lot of freedom in this regard, but it didn't at first.

My clients have been pretty understanding. One asked me to do a piece because it wouldn't require any fieldwork (plus, they said, my copy is pretty clean, and they were a bit desperate for it). (I turned them down.) I bowed out of a Chronicle lede because I won't have time to give it the research it deserves, but then my editor asked if I'd be willing to write a 400-word review/commentary of a new book.

Overall, my life as a published writer is taking a breath. But I'm not letting it sit idle. Instead, I have writing skills that I want to develop. I'll be working on essays — a genre that I adore above all other non-fiction but which I find difficult to write, despite the real argument you could make that each OWF post is an essay exercise. I'm also planning to work on my narrative nonfiction abilities; I want to be the type of writer who casts features in the form of conflicts, crises, and resolutions. I want my pieces to have a story flow that brings the reader into the prose. And I want to continue to grow my newfound library research knowledge. If I may use a metaphor from my newest development project, those skills are currently in the Tidepool phase and I want them to be at the Civilization phase. My existing research skills have earned compliments from Ed Behr, which counts for a lot, but I want to go further.

But aside from my writing skills, I intend to devote some time to my programming projects. My recent job quest and my parallel investigation of the latest, greatest technologies have reminded me how much I like programming. I go through little phases where I believe I no longer do, but it doesn't usually take much to remind me of the sheer joy of coding. I'm working on a "wine notebook/cellar management" system, which I may or may not make more available. The application allows me to have a project where I have to explore technologies in depth but not spend months and months and months working with them. I can do a Spring/Hibernate/YUI version, as I am now, or a Mac OS X/iPhone version, or a Ruby on Rails version, all using the same database. Doing any one of those gives me enough of a grounding in the technologies to work on more complicated projects (at least in theory — I already know a fair amount about Spring, something about Hibernate, but nothing at all about YUI).

Dashcode/Spore Countdown Widget

2008-04-07T09:59:00.000-07:00

For obvious reasons, I’ve been reading all the press about Spore. At one point, I decided to do the cheesy, geeky thing and find a Dashboard widget that counts down to Spore’s release. Only I couldn’t find one that I liked. So I did the even geekier thing and wrote my own.

Melissa made noises about Kool-Aid and cults, but if you’ve ever played with DashCode, you’ll realize that my widget is a barely-modified version of the default countdown template. I fiddled with some colors, set some attributes, and added a graphic from the fansite kit. Other than downloading the fansite kit, which took eons on our slow connection, the whole widget probably took 30 minutes of my time.

I’ve played with DashCode before, but always with ambitious projects in mind that demanded more than the Dashboard infrastructure would allow. I was impressed with the application at the time, but ultimately my widget ideas didn’t go anywhere. Having a project go from start to finish in a short time made me appreciate just how easy DashCode is. I didn’t need to do any coding for the widget; any moderately technical person could have set it up.

AppleScript: Open A Bunch Of Links In Safari

2008-03-21T10:28:00.000-07:00

I recently judged the American Wine Blog Awards. My list of nominees came as a spreadsheet with the URLs of the blogs and their names, grouped by category. Because there were 30 or more nominations in some categories, I didn’t want to open each link individually in Safari. I turned to my old friend and nemesis: AppleScript.

It took a bit of online sleuthing to figure out the quirks in Safari’s dictionary, but I eventually got a script that opens a set of URLs, each in its own tab, in a new window. I’ve included it below for anyone who might find it useful. It works well enough, given that I only needed to run it 8 times. It’s pretty brain dead as it is: There is no error checking and you have to have the URLs in a return-delimited list, which is what you get when you copy URLs out of a column in Excel. If you run it, you’ll also notice that it opens an extra tab at the “beginning” of the new window. If it bothered me, I would figure out how to remove the tab, but for 8 runs I could just click the close box. I only tested it on Mac OS X 10.5 and Safari 3.0.



set clipText to the clipboard

set AppleScript's text item delimiters to {"
"}
set urlList to text items of clipText
set AppleScript's text item delimiters to {""}
tell application "Safari"
 set newDoc to make new document
 set currWindow to front window
 repeat with currentURL in urlList
  make new tab at the end of tabs in currWindow with properties {URL:currentURL}
 end repeat
end tell

Writing I Like: Nicholson Baker Writes About Wikipedia

2008-03-03T13:02:00.000-08:00

I think I’ve liked every piece of Nicholson Baker nonfiction that I’ve ever read, but I thought I would call out his piece about Wikipedia for my new-and-exciting Writing I Like category. Ostensibly, the essay is a review of Wikipedia: The Missing Manual, but in true NYT Review of Books fashion, that’s a lightweight skeleton supporting the piece’s muscle.

Baker has whole battlefronts of conflict at his disposal to spice up his piece: He paints the modern-day Wikipedia as an mostly-unseen war between the keepers of the encyclopedic truth and its would-be spammers and trolls. Even within the legitimate ranks, he finds tension: There are aggressive purgers debating against article inclusionists. (And, really, is it any surprise that the author of Double Fold sides with the “let’s include everything” camp?) There is even his conflict between his life as a newly enthusiastic Wikipedia editor and his life as a father and husband with household obligations.

But this piece really shines with its use of specifics. Baker has a finely tuned eye for detail backed by an obsessive knowledge-seeking mind. Consider his accounts of Wikipedia vandalism:

Some articles are vandalized a lot. On January 11, 2008, the entire fascinating entry on the aardvark was replaced with "one ugly animal"; in February the aardvark was briefly described as a "medium-sized inflatable banana."

He doesn’t bother cracking jokes. Who needs to with source material such as this?

As with the Wired piece about the Netflix Prize, Baker’s piece shines because of his presence in the piece (a more overt presence than that of the writer of the Netflix piece). He talks about how he got drawn in to Wikipedia editing, the battles he won (and lost) to keep articles in the system, the addictive pull of debates with other editors and conflicts with vandals.

I finished the piece feeling a little lighter, a little happier, and a little more inclined to edit Wikipedia articles.

Writing I LIke: Wired's Psychologist And The Netflix Prize

2008-03-02T20:55:00.000-08:00

I spend a lot of time here griping about laughable published writing that shouldn’t have slipped past an editor’s red pen. Let’s look a piece that makes me smile in a good way: Wired’s piece about an English psychologist/operations engineer who has rocketed up the leaderboard for the Netflix prize, a $1 million reward for anyone who can improve the company’s recommendation system by 10 percent.

First of all, hats off to the author, Jordan Ellenberg, for distilling complex math into a usable form for the smart, but not expert, reader. Ellenberg is a mathematician in his own right, and he summarizes the high-math concepts used by the competitors into common English, using analogies to illustrate his points. As someone who increasingly finds himself writing technical, “wine geek” wine pieces for a mainstream, layperson audience, I am impressed by his skill.

But the biggest draw of this piece is the compelling narrative: A classic “little guy beats the big guy” scenario. Good fiction, which is the model for narrative non-fiction, revolves around conflict, and the author has rightfully used the inherent battle — a single psychologist and his high-school daughter, the math consultant, trouncing teams of math and computer science professionals working with sophisticated programs — as the axis of his piece. From conflict comes crisis, the boiling point, and Ellenberg provides it with the current status: all the contestants close to the final prize from a numerical point of view, but very far from a realistic point of view. Ideally, one wants a resolution as well, but that remains in the future, an acceptable ending for a newsy narrative nonfiction piece.

The author doesn’t lock himself into this story, though. “Digress often, but never for long,” reads one of the few axioms laid out in the classic The Art and Craft of Feature Writing. Ellenberg spins a quick history of Netflix, a brief description of the prize, a look into the minds of the Netflix statisticians, the surprising collaboration of the competing groups, and more, all through short digressions that linger just long enough: As soon as you start to think, “Get back to the psychologist!” he does.

My final point — though there are other things to like in this piece — is that Ellenberg allows himself to be in the piece. Feature wells don’t often permit first-person narrative for obvious reasons: Too much first-person, and the reader begins to wonder why s/he should care about the writer so much. Among my clients, only The Art of Eating finds it natural, though others allow it when it makes a difference. But Ellenberg sometimes steps away from his story to give his own view: “He refers to the psychological model underlying their mathematical approach as ‘crude.’ His tone suggests that if I weren’t taping, he might use a stronger word.” Ellenberg exposes himself to the reader, but in doing so draws a more detailed picture of the person you actually care about. And he doesn’t forget to show and not tell, though he has a harder time in his straightforward reporting sections: Small details like the notebook and the elderly Dell allow the reader to paint a more vivid picture.

I might try to incorporate a bit more color and rhythm into the prose, were I writing this, but the story is good enough that only someone looking for nits will drill down on that.

Google Chart

2008-02-20T21:50:00.000-08:00

I recently discovered Google’s Chart service, which allows you to query Google with a bunch of parameters and get back a dynamically generated chart. The idea intrigued me, but I couldn’t think of an immediate use for it until I remembered that I had yet to publish the results of a survey I ran on OWF in September. Now was the time, I thought.

After a day of playing with the system, I have to say I’m pretty impressed. You get a wide range of styles, you can set the data in a variety of ways, and you can fiddle with labels and legends. It has some quirks: I couldn’t get the horizontal axis to quite line up with my data points on my bottom graph, and you have to give the data for the bar chart as a percentage instead of a straight value. But once I got the hang of it, the graphs came together quickly and looked nice.

Now I just need an excuse to put a Venn diagram on OWF.

Amusing Grammar Slip

2008-02-02T06:39:00.000-08:00

Mediabistro, a paywalled website for writers, recently posted the transcript of a seminar on grammar. I read through it — it’s all stuff I know, unfortunately — and noticed this sentence in the section on pronoun agreement: “The next example is probably my favorite because I didn't make it up. I get silly text messages from Virgin Mobile for their promotional crap, and it had a lot of errors in it. ”

I know it’s just a transcript. It may even be what the teacher said. But that particular sentence in that particular section tickled me.

Link Blogs

2008-01-31T14:15:00.000-08:00

One of the many blog genres is the link blog, a site that just provides links to other sites. It sounds boring, but some of the blogosphere’s most popular sites are link blogs: Jason Kottke adds little content at kottke.org; nor do the boing boingers.

Without realizing what I was doing, I started a link blog in July of 2007. I put up a “snacks” feature on OWF, a light feature that would just point people to interesting items on the Web. I made it easy to maintain, and I started populating it.

I’ve come to realize just how addictive such a feature is. Its main purpose is still to provide links that my readers may or may not find interesting, links that may fuel a coffee break at work. But I often find myself “snacking” a link that I want to be able to retrieve at some later point. Blogger makes this even more compelling: I host the main snack blog on blogspot, and the default Blogger header has a search box for the entire site. And once Jack prompted me to add descriptions to my links (which show up in the RSS feed but not yet on OWF), I realized I could add keywords that would speed up the retrieval.

So have I read The Mathematics Of Change Ringing [pdf] yet? Well, not completely. But I can revisit it at some later point without trying to remember where I saw the link. To find the link just now, I went to the main snack page, typed the word “bell” into the navbar at the top of the screen, and went right to the entry.

It’s been liberating in an odd way. I can tuck links into this little corner of the Internet and recover them at will. It’s become a Pensieve of sorts. I can snack a link, close the window, and revisit it when I have time.

The Atlantic's Variety Cryptic

2008-01-26T22:36:00.000-08:00

The blogosphere has been abuzz with The Atlantic’s recent decision to tear down its long-standing paywall.

While most have mentioned trawling the archives for this or that article, few have made the obvious observation: The variety cryptic, by Cox and Rathvon, is part of the package. Perhaps that’s because the boing boingers and Jason Kottke don’t actually do variety cryptics, but I feel like it hasn’t even shown up on my puzzle lists. Maybe it’s always been online, and I’ve never looked?

Cryptic crosswords are crossword puzzles in which the clues have the word definition, but they also have a hefty amount of wordplay to get you to the answer. Thus each clue in a cryptic is really two clues. It’s up to you to determine where one ends and the other begins, and deciphering the clue can be tricky: I often figure out what word should go in the space provided and then try to reverse-engineer the wordplay.

A variety cryptic adds yet another dimension. In a variety cryptic, the answer to the cryptic clues don’t necessarily fit into the grid the way you’d expect. You might have to add a letter, drop a letter, change one fragment to another, or make words “warp” from one part of the grid to another. I remember one variety cryptic, published in The Enigma, of course, where words that contained the names of European currencies had to be transformed so that those letters became EURO. Remarkable became reeuroable, for instance.

Technology Rules

2008-01-16T10:11:00.000-08:00

Some high school students discovered an asteroid. Neat.

But the best part is the summary of connections mentioned on Slashdot. The Wisconsin students, “used a telescope in New Mexico, belonging to a college in Michigan, that they controlled over the Net.” Gotta love the Internets.

Quote Of The Day: Starting Fires With Fish

2008-01-15T13:46:00.000-08:00

From Joystiq, describing the latest news of Uwe Boll, the German director most famous for his big-screen videogame adaptations:

…the infamous German director and pouting pugilist will see his future endeavors financially constrained after his latest $70 million video game adaptation, "In the Name of the King: A Dungeon Siege Tale," set the box-office alight with all the effectiveness of two moist fish furiously rubbed together.

Quote Of The Day: This Is Why People Don't Like Goblins

2008-01-14T10:59:00.000-08:00

Jerry, from Penny Arcade, writes of his re-entrance into World of Warcraft.

After seven blissful levels of wondering why I ever quit WoW, I ended up on some featureless beach in Feathermoon waiting for some stupid, almost nonsensical drop. After that, I talked to a Goblin whose most pressing concern (in a world whose very crust was cracked by perpetual War) was his tremendous thirst. Would I go and get something for him to drink? I've got water, but he doesn't want it. He only wants to drink the glands of some twelve-foot tall plant man. It's like, listen. This is why people don't like Goblins.

Randomness

2008-01-11T00:04:00.000-08:00

When Pim and I looked at the prizes for Menu For Hope, we were surprised to see that a number of people had won more than one prize. I know, theoretically, that an even distribution among prize winners is one possible outcome while lists with duplicates are much more likely. But it's one thing to know it, and another thing to accept it. An iPod's shuffle feature is truly random, but it often seems as if the shuffle is favoring one artist or album. (That's why iTunes now offers a "smart" shuffle that diminishes randomness in favor of distributing artists/albums.) And in the TV show Numb3rs, an evenly distributed set of points is a clue that someone has tried to make something look random. True random numbers aren't spread out equally.

Just to convince myself, I wrote some code to demonstrate this unintuitive aspect of randomness.

First, draw 1 number from a set of 10. The probability for each number is 1/10th. Do this 10,000 times to get a bunch of results to chart, and you end up with an almost even distribution across those 10 numbers, demonstrating the fairness of the random draw:
1 1024
2 1042
3 992
4 952
5 974
6 1034
7 995
8 993
9 1001
10 993

In that run, each "turn" was a single draw. Now make each turn 10 draws. In Menu For Hope, many contributors entered multiple raffles, and drawing more than once from the same pool simulates this. We expect each number to come up once, because that's what probability tells us. But it doesn't work that way, at least not necessarily. On one run, I got this set of results:
2
5
6
7
2
10
2
6
8
3

Even though the probability of each number is 1/10, you end up with single numbers repeated. There are three 2s and 2 6s.

When I ran the 10-draw turn 10,000 times, I got these results:

1 unique numbers 0
2 unique numbers 0
3 unique numbers 6
4 unique numbers 187
5 unique numbers 1274
6 unique numbers 3457
7 unique numbers 3528
8 unique numbers 1373
9 unique numbers 169
10 unique numbers 6

Here I counted the unique numbers in each set. 10 unique numbers means a perfect distribution. 1 unique number means that every random draw hit the same number. I didn't implement the logic to separate duplicates (the 6 in the previous example) from the triples (the 2 in the prior example). Or quadruples, for that matter. This was a simple simulation.

The odds of randomly getting an even distribution are pretty low. You are roughly 600 times more likely to have 6 or 7 unique numbers, meaning that some of those numbers are duplicated (or tripled or whatever). You’re 200 times more likely to have a scenario where half the numbers are missing. You’re 30 times more likely to have just 4 unique numbers: a whole host of repeats. The numbers looked similar on multiple runs.

This makes sense if you think about it. In the second draw of the 10-draw turn, you have a 10 percent chance of drawing a duplicate. If you don’t, you have two unique numbers, and on the third draw you have a 20 percent chance of duplicating one of the existing values. By the time you get to the tenth draw, there’s a 90 percent chance that you’ll draw a number that’s already been drawn, assuming all the others were unique.

Menu For Hope, of course, is much more complicated. Raffle ticket purchasers can stack the odds in their favor by buying more tickets for a given raffle. Prizes with fewer bidders have different odds than those with more bidders. And it was about 100 draws across 9,000 raffle tickets. But even the simple version in this post, where each of just 10 numbers has an equal probability, shows that duplicates and triplicates should be commonplace. So it's not surprising that some people won multiple prizes: It would have been more surprising if none had.

Which Paragraph Looks Better?

2008-01-10T19:48:00.000-08:00

In my article about Franconian beer, I have these paragraphs:

But even the best Franconian brewers aim for the everyday drinker, not the thrill-seeking beer snob. Waltman contrasts Belgium's prestigious brews with Franconia's farmhouse ales.

Belgian beers kind of hit you over the head," he says. "They have unusual flavors; they're big. In Belgium, if you ask for recommendations for 10 breweries, they would all be cafes: People sitting around sipping these strong beers. In Germany, and especially in Franconia, beer is what you drink all day long. The beers are designed to be drunk." Simple flavors, well-balanced hops and low alcohol create a drink that can go with your weeknight dinner or your afternoon break.

This is the text as I submitted it and as the Chronicle ran it. But when I had a chance to review the piece before publication, I looked at it again and posed a question to my editor. Should it read, "Waltman contrasts Franconia's farmhouse ales with Belgium's prestigious brews"?

One of my writing mantras — one of the ones in my screen saver — is “Keep related concepts together.” Following that rule, the paragraph should read the way I asked. The first “Belgium” segues into a digression about that country’s beers. But then your internal reader sees “Franconia Franconia Belgium Belgium.“ Because those are proper nouns, it feels, for lack of a better word, loud.

What do you think?

Paging An Editor: An Article About Champagne Vinegar?

2007-12-26T09:35:00.000-08:00

I think the point of this New York Times article is that Champagne vinegar is good. But that point only shows up in the last two paragraphs: The rest is a stretched analogy about the flip side of Champagne's connotations of luxury. So what is the point of this article, anyway?

No. Really. No Ads.

2007-12-17T21:28:00.000-08:00

I got an email today asking something about placing content on OWF. I gave my standard response — I don't take ads on OWF. The author wrote back. Clearly I misread her intent: I thought she wanted to take out ads.

The text links I referred to in my previous email will not consume any advertisement space (header/footer/side bar) on your web pages and are to be placed within the existing content of your web pages.

Each of these paragraphs will be custom-written to go nicely with the rest of your webpage and will add unique content to your webpage. These paragraphs will contain natural text links only and will not affect your page ranking with Google.

I will write keeping the content, quality and audience of your website in view. After reviewing your website thoroughly, I'll write the paragraphs' content accordingly.

Would an ad by any other name still smell … ? If I say no ads, I mean it. It doesn't matter how they look.

Heroes Fans Fan Service?

2007-12-17T18:39:00.000-08:00

On my way home, I listened to the most recent episode of The 10th Wonder, a podcast about Heroes. The hosts announced that they'd be taking a break for the next few weeks, in contrast to last season's hiatus, where they kept the podcast going as "a service to the fans."

My inner schoolboy ("inner?" you're thinking) emerged, and I giggled. Three guys giving fan service? I'll pass, no matter what my friends might think of me. Call me when French Maid TV does it.

Oh, right. I guess FMTV is almost all fan service.

Gmail Ads

2007-12-17T10:37:00.000-08:00

I was amused by the current juxtaposition of context-sensitive ads appearing alongside my gmail pane:

Retaining Walls
Genesis Stoneworks Freeee! in home estimates Call 800-287-5400

Random Number Generation
Stanford finance risk mgmt courses. Learn new ideas & proven techniques

Official Gaiam Yoga Store
Official source for Gaiam yoga mats yoga kits, props & accessories.

SOHAKSC

2007-12-06T12:54:00.000-08:00

Today's Slylock Fox, shown near the bottom of this Comics Curmudgeon post, asks readers — children, in theory — to rearrange the letters in six words to spell the names of items in the picture.

Go ahead, try to figure out what SOHAKSC spells. You can probably guess the word based on how English structures these things, but do you know what it means? Can you find it in the picture? How many kids do you know who could nail this one?