The button-down collar creates a crisper, more professional look, so that if the shirt fits well and is accompanied by a good tie, its inherent casualness or dweebiness is completely offset.
The entire procedure operated with a casualness to it that seemed to make fun of the serious nature of what was happening, while the rules seemed more important than what they were designed to accomplish.