How NOT to Fill Out a BJCP Scoresheet

Homebrew competitions invite organizers and judges to strike a delicate balancing act: How to honor, on the one hand, the inherently fun and casual nature of the homebrewing community, while at the same time being serious and diligent enough to give entrants their money's worth in terms of feedback and attention.

Those who have been through a quality judging course – as part of BJCP training or otherwise – are familiar with some of the best practices: fill out the entire sheet; comment on all aroma/flavor/etc. characteristics as prompted by the sheet; write legibly; tell the brewer where deducted points went and how to reclaim them.

With this in mind, take a look at the two scoresheets presented here (click the images for a larger view), corresponding to a couple entries of mine from a recent competition. Both were filled out by the same judge (same flight), an "experienced" judge who, based on information included on the scoresheet, seems to have recently taken the BJCP exam and is awaiting his score and rank.

Neither scoresheet is exactly a case study in how to evaluate beer. On the first scoresheet, there is plenty of unused white space, the handwriting is poor, and within each scoring section there are characteristics the judge does not comment on. Still, at the end the judge does provide an evaluative statement and offers a recommendation for improvement.

Consider now the second scoresheet. The handwriting is practically illegible and as your eye moves down the page, it encounters less and less writing, to the point where the final scoring section (the one where, incidentally, the most space is given for comments) is left entirely blank. The brewer is left simply to guess as to how the judge arrived at the assigned score, for there is little besides careless pencil scratches to offer any clues.

It's natural to wonder whether the judge had simply "evaluated" too much beer by this point and was worse off for it. Indeed, and in some measure of fairness, Exhibit A was judged fairly early in the flight; Exhibit B fairly late. Nevertheless, the second judge (usually entries are evaluated by a pair of judges) managed to write perfectly legible and thorough comments on both sheets, and at any rate a brewer should not have to fret over whether his beers will be evaluated by adequately sober judges.

In terms of providing useful feedback on how to improve the beer in question, much less providing a careful analysis of the entry, these scoresheets (the second one especially so) are unfortunate failures. I can only say it's good for my sake that I trust my own evaluative abilities enough that I do not enter competitions, generally, looking for feedback on how to improve my beers. (On this particular go-around I had been experimenting with blending beers and entering off-style; the judges tended not to be terribly impressed and my scores reflected that, as you can see; I had half expected as much.)

As a judge, I know that fatigue can set in near the end of a flight or after a long day of evaluating beers. Nevertheless, I do believe that each entry is entitled to the same thorough critique and feedback as is every other one. To see such woefully inadequate scoresheets is discouraging, but even more so coming from a person just now entering the ranks of the BJCP. Know that I do not write these words out of sour grapes – I am not troubled by the scores nor personally distressed by the sparse comments so much as I am dismayed by what appear to be bad habits in the making and the prospect that the next victim will be a brewer who truly relies on judging feedback to improve his beer.

Competition organizers and the BJCP had better take heed: I don't think it's too much to say that the very credibility of homebrew competitions, the BJCP and my fellow BJCP judges hinges in no small measure on the quality of judging entrants receive in exchange for their time, effort and entry fees. We can, and should, do better.

The Session #33 – Framing Beer

A mighty interesting, open-ended topic for this month's Session, hosted by Andrew at I'll Have a Beer and announced here. Our task is to consider how context and framing influence the way we consider and evaluate beers.

There are any number of ways to go with this thought-provoking topic, but in the interest of space and time I'll try to keep my focus tight. And given that many of my thoughts – my beer-related thoughts especially – fixate on homebrew, that seems like a good place to take this Session effort.

Homebrew competitions, by and large, are fine exercises for neutralizing many of the effects of context when it comes to evaluating beer. There are no names, so a brewer's reputation can't influence the judges. Extract vs. all-grain is not specified, so those biases are off the table. Recipes are not divulged, so expectations based on ingredients used cannot be considered. All that the judges have to work with is a declared style and a set of style guidelines against which to measure the entry.

And yet here is where we see how even in such a context-neutral environment as a homebrew competition, framing and context do indeed play key roles. More on that in a second.

As a card-carrying beer judge, I enjoy working competitions and trying my best to provide entrants with constructive, impartial and informed evaluation of their beers. But I know full well that, try as we might, judges are often vulnerable to context and framing influences, just as we are in "real-world" beer-drinking situations.

Let's start with the matter of categorization. So that beers can be properly evaluated against one another, entries are sorted into categories, with sub-categories further specified by the entrant. In so doing, the beers are pitted not only against each other but also a standardized set of guidelines specific to each style, with detailed notes on appearance, aroma, taste, mouthfeel, et cetera all assisting the judge in diagnosing the beer's quality. Thus the beer is framed before it even passes the drinker's lips – it is generally assumed that beers entered into a given category do indeed fall within its parameters; thus judges will tend to evaluate the beer as if it at least roughly fits the guidelines. Variation, where not plainly egregious, is often considered to amount to a mild departure from this or that prescribed quality.

This framing tendency is easily testable. I once entered a dark beer fermented with Witbier yeast and seasoned with coriander and citrus peel in the Brown Porter category. It scored fairly high marks, with none of the evaluators picking up on ingredients that, beyond a doubt, would be officially forbidden in the category. (And had I entered the beer in Specialty and declared the additions, I am certain they would have been commented upon.) Another amusing trick is to cross-enter the same beer in different, though similar categories in the same competition. (For example, Robust Porter and one of the Stout categories.) If the beer receives high scores in both cases, one has to wonder whether power of suggestion had prevailed or there was simply not enough daylight between the categories to reveal one entry as fraudulent. Perhaps a little of both.

Evaluating a beer against its category is only half the judge's task; the other is to suss out flaws in the production of the beer. A whole roster of off-flavors is usually available for consultation (though the experienced judge should already be familiar with these) and it is up to the evaluators, where appropriate, to call a brewer out for them and penalize the beer accordingly. (And, of course, to make suggestions for how to overcome the flaw next time.)

Judges are by no means assured of finding the same flaws in every beer they taste together. Some individuals simply are less sensitive to certain flavors than others – for a long time I did not believe I could easily pick up on diacetyl or oxidation – others may even have a reputation for being extra (or excessively) sensitive to some. I have been on judging panels where, once the score cards are compared, one would think we had sampled entirely different beers. (This makes it all the more gratifying when, in what itself is no rare instance, judges independently pick up on the same things.)

On larger panels, groupthink can become an issue, where a particular idea gains traction and suddenly the entire table becomes convinced that a certain flaw or characteristic is present. Sometimes consensus arrives by way of a particularly strong personality, or others' lack of confidence, or power of suggestion.

I don't mean to impugn homebrew competitions unfairly; on the whole I'd say judges tend to get the calls right, and at the end of the day the best beers are rewarded while the less-than-stellar ones are not. But beware the veneer of objectivity and the assumption it can be achieved to an absolute degree. It just may be that framing and context in beer evaluation – as in just about all other aspects of life – could be unavoidable after all.