How NOT to Fill Out a BJCP Scoresheet

Homebrew competitions invite organizers and judges to strike a delicate balancing act: How to honor, on the one hand, the inherently fun and casual nature of the homebrewing community, while at the same time being serious and diligent enough to give entrants their money's worth in terms of feedback and attention.

Those who have been through a quality judging course – as part of BJCP training or otherwise – are familiar with some of the best practices: fill out the entire sheet; comment on all aroma/flavor/etc. characteristics as prompted by the sheet; write legibly; tell the brewer where deducted points went and how to reclaim them.

With this in mind, take a look at the two scoresheets presented here (click the images for a larger view), corresponding to a couple entries of mine from a recent competition. Both were filled out by the same judge (same flight), an "experienced" judge who, based on information included on the scoresheet, seems to have recently taken the BJCP exam and is awaiting his score and rank.

Neither scoresheet is exactly a case study in how to evaluate beer. On the first scoresheet, there is plenty of unused white space, the handwriting is poor, and within each scoring section there are characteristics the judge does not comment on. Still, at the end the judge does provide an evaluative statement and offers a recommendation for improvement.

Consider now the second scoresheet. The handwriting is practically illegible and as your eye moves down the page, it encounters less and less writing, to the point where the final scoring section (the one where, incidentally, the most space is given for comments) is left entirely blank. The brewer is left simply to guess as to how the judge arrived at the assigned score, for there is little besides careless pencil scratches to offer any clues.

It's natural to wonder whether the judge had simply "evaluated" too much beer by this point and was worse off for it. Indeed, and in some measure of fairness, Exhibit A was judged fairly early in the flight; Exhibit B fairly late. Nevertheless, the second judge (usually entries are evaluated by a pair of judges) managed to write perfectly legible and thorough comments on both sheets, and at any rate a brewer should not have to fret over whether his beers will be evaluated by adequately sober judges.

In terms of providing useful feedback on how to improve the beer in question, much less providing a careful analysis of the entry, these scoresheets (the second one especially so) are unfortunate failures. I can only say it's good for my sake that I trust my own evaluative abilities enough that I do not enter competitions, generally, looking for feedback on how to improve my beers. (On this particular go-around I had been experimenting with blending beers and entering off-style; the judges tended not to be terribly impressed and my scores reflected that, as you can see; I had half expected as much.)

As a judge, I know that fatigue can set in near the end of a flight or after a long day of evaluating beers. Nevertheless, I do believe that each entry is entitled to the same thorough critique and feedback as is every other one. To see such woefully inadequate scoresheets is discouraging, but even more so coming from a person just now entering the ranks of the BJCP. Know that I do not write these words out of sour grapes – I am not troubled by the scores nor personally distressed by the sparse comments so much as I am dismayed by what appear to be bad habits in the making and the prospect that the next victim will be a brewer who truly relies on judging feedback to improve his beer.

Competition organizers and the BJCP had better take heed: I don't think it's too much to say that the very credibility of homebrew competitions, the BJCP and my fellow BJCP judges hinges in no small measure on the quality of judging entrants receive in exchange for their time, effort and entry fees. We can, and should, do better.

The History of American Brown Ale & American Pale Ale (Kind Of)

Today, it's easy for us beer people – the committed and the casual alike – to take styles for granted. After all, we have organizations like the BJCP and competitions from the GABF to the World Beer Cup to myriad homebrew comps that help keep beers more or less tidily segmented and compartmentalized for us. Then there are beer-rating sites like BeerAdvocate and RateBeer that allow beer drinkers to see how their favorite commercial brews stack up against their category-mates.

Yet it wasn't so long ago that many of the styles we are so familiar with now were either a mere figment of a brewer's imagination, or even if sitting right under our noses, had yet to be given the proper recognition and codification that today seems like such a no-brainer.

In the former category falls American Brown Ale. As far back as the early 1980s, when all that the world knew of Brown Ales was Newcastle and the like, homebrewers out in California were whipping up their own imaginative take on the style. Scott Birdwell, owner of Defalco's Home Wine and Beer Supplies in Houston, and a seasoned pillar of the homebrewing scene in his own right, tells of a trip to San Rafael, Calif., sometime during the Reagan years:

I was visiting a couple of friends who owned a homebrew shop there (Jay Conner & Byron Burch - Great Fermentations). They had a flyer for a recipe for Purple Passion Dark Ale with John Bull Dark Malt Extract, crystal & chocolate malts, and a ton of hops. This was a popular recipe with their customers and did well in local and regional homebrew competitions, but got slammed in the AHA & HWBTA Nationals for not meeting the style guidelines for "Brown Ales" (assumed to be British brown ales). It was true these beers didn't conform to traditional brown ale standards: they were too dark and too bitter. But, man, they were popular on the West Coast, becoming increasingly popular on the Gulf Coast, and were damn good beers!

Now here's where those devoted, zany and innovative Houston-based homebrewers known as The Foam Rangers (my former club) enter the picture. Their annual Dixie Cup competition, which these days ranks among the world's largest annually, would play a vital role in bringing this unique California concoction into greater prominence. Scott "Da Birdman" again:

We were in the infancy of the Dixie Cup at that time and I decided to include a category for these brews. We already had a category called "No Commercial Comparisons," in which the entries didn't meet commercial standards (at least the commercial beers available at that time). We decided to call the new category "California Dark" in deference to our friends on the West Coast. The category was an immediate success, even if we weren't overwhelmed with entries. ... The AHA immediately picked up on the California Dark category, but curiously decided to name the style "Texas Brown Ale" in deference to us (nice, but we weren't the brewing innovators, just the competition innovators). Obviously this struck a note with homebrewers all over the country as this proved to be a popular style, and eventually the name evolved into "American Brown Ale." This is probably an appropriate name given its widespread popularity. These days I consider "Texas Brown Ales" to be "extreme" American Brown Ales: O.G. at least 1.060 and 40 IBU's, but that may just be me.

For all the strictness and even arbitrariness modern style parameters seem to reflect, this tale reminds us that styles do in fact arise organically – such that even a well-known style like American Brown Ale can be traced to a San Francisco suburb by way of a humble competition in Texas.

But that's not all. Consider this final anecdote from Scott:

As far as I know, Dixie Cup VII in October 1990 was the first time any homebrew contest featured a category entitled "American Pale Ale." In those days, when the number of beer style categories were considerably more limited, we (meaning the Foamers) did not limit ourselves to AHA and/or HWBTA style categories (this was before the BJCP established it's own guidelines). We felt free to establish our own categories with style descriptions for the Dixie Cup. This is how (American Pale Ale) came into being. ... That first year for APA (1990), it was the largest single category we had for that year's Dixie Cup (I seem to recall we received over 30 entries). The AHA jumped on this and incorporated APA the very next spring for the NHC.

Wow, talk about trail blazers. It's no huge leap to say this club and this competition were responsible for the formal recognition of two major American beer styles. And in the case of APA, consider that the very paradigm of the style, Sierra Nevada Pale Ale, had been in production for fully 10 years before the national beer-agenda setters, spurred by the Foamies, picked up on the style. Within two years, the GABF got on board, and APA has never looked back since. Ttoday there are over 2,000 commercial APAs listed on BeerAdvocate, more than any other single style.

Behold the power of homebrewers.

The Session #33 – Framing Beer

A mighty interesting, open-ended topic for this month's Session, hosted by Andrew at I'll Have a Beer and announced here. Our task is to consider how context and framing influence the way we consider and evaluate beers.

There are any number of ways to go with this thought-provoking topic, but in the interest of space and time I'll try to keep my focus tight. And given that many of my thoughts – my beer-related thoughts especially – fixate on homebrew, that seems like a good place to take this Session effort.

Homebrew competitions, by and large, are fine exercises for neutralizing many of the effects of context when it comes to evaluating beer. There are no names, so a brewer's reputation can't influence the judges. Extract vs. all-grain is not specified, so those biases are off the table. Recipes are not divulged, so expectations based on ingredients used cannot be considered. All that the judges have to work with is a declared style and a set of style guidelines against which to measure the entry.

And yet here is where we see how even in such a context-neutral environment as a homebrew competition, framing and context do indeed play key roles. More on that in a second.

As a card-carrying beer judge, I enjoy working competitions and trying my best to provide entrants with constructive, impartial and informed evaluation of their beers. But I know full well that, try as we might, judges are often vulnerable to context and framing influences, just as we are in "real-world" beer-drinking situations.

Let's start with the matter of categorization. So that beers can be properly evaluated against one another, entries are sorted into categories, with sub-categories further specified by the entrant. In so doing, the beers are pitted not only against each other but also a standardized set of guidelines specific to each style, with detailed notes on appearance, aroma, taste, mouthfeel, et cetera all assisting the judge in diagnosing the beer's quality. Thus the beer is framed before it even passes the drinker's lips – it is generally assumed that beers entered into a given category do indeed fall within its parameters; thus judges will tend to evaluate the beer as if it at least roughly fits the guidelines. Variation, where not plainly egregious, is often considered to amount to a mild departure from this or that prescribed quality.

This framing tendency is easily testable. I once entered a dark beer fermented with Witbier yeast and seasoned with coriander and citrus peel in the Brown Porter category. It scored fairly high marks, with none of the evaluators picking up on ingredients that, beyond a doubt, would be officially forbidden in the category. (And had I entered the beer in Specialty and declared the additions, I am certain they would have been commented upon.) Another amusing trick is to cross-enter the same beer in different, though similar categories in the same competition. (For example, Robust Porter and one of the Stout categories.) If the beer receives high scores in both cases, one has to wonder whether power of suggestion had prevailed or there was simply not enough daylight between the categories to reveal one entry as fraudulent. Perhaps a little of both.

Evaluating a beer against its category is only half the judge's task; the other is to suss out flaws in the production of the beer. A whole roster of off-flavors is usually available for consultation (though the experienced judge should already be familiar with these) and it is up to the evaluators, where appropriate, to call a brewer out for them and penalize the beer accordingly. (And, of course, to make suggestions for how to overcome the flaw next time.)

Judges are by no means assured of finding the same flaws in every beer they taste together. Some individuals simply are less sensitive to certain flavors than others – for a long time I did not believe I could easily pick up on diacetyl or oxidation – others may even have a reputation for being extra (or excessively) sensitive to some. I have been on judging panels where, once the score cards are compared, one would think we had sampled entirely different beers. (This makes it all the more gratifying when, in what itself is no rare instance, judges independently pick up on the same things.)

On larger panels, groupthink can become an issue, where a particular idea gains traction and suddenly the entire table becomes convinced that a certain flaw or characteristic is present. Sometimes consensus arrives by way of a particularly strong personality, or others' lack of confidence, or power of suggestion.

I don't mean to impugn homebrew competitions unfairly; on the whole I'd say judges tend to get the calls right, and at the end of the day the best beers are rewarded while the less-than-stellar ones are not. But beware the veneer of objectivity and the assumption it can be achieved to an absolute degree. It just may be that framing and context in beer evaluation – as in just about all other aspects of life – could be unavoidable after all.