I've helped judge a few contests here and there, and even hosted/organized EASTER one year. If there's one thing I've learned, it's that judging film is a very tricky art.

If, for example, one film has small and simple movements smoothed to perfection, and another film has larger, far more complex movements, yet contains a few imperfections, which do you rank higher? How do you compare the hand waving of a calm conversation with the mad dash of a panicked crowd at the approaching monster? Can you assign a completely objective value to that? And if one film has perfect animation, yet the concept is stale and lifeless, should that rank higher than an original and creative movie with blander sets?

Those are many of the questions that plague the minds of brickfilm judges. Hence, discussion.

Now to address the quote from SillyPenta, so far as I understand it.
Here's why most contest judging can't be that specific. Take two of our own events as examples: THAC is very unlike the Animation Challenge, and thus, judging cannot as straightforward. The Challenge looks to one thing: The animation itself. Creativity plays a part as well, but the Challenge can be judged by very specific criteria. THAC, on the other hand, cannot be so easily taken care off. Often, films carry various imperfections, the theme is more nebular and open to interpretation, and often, in my experience, these films become more (or sometimes less) than the sum of their parts. One judge may find something to be more creative than another, or one may see the animation of one as being better than another, and different tastes may begin to color individual rankings. Plus, some flawed films may simply be far more enjoyable than those whose presentation may be of a higher quality. That's why it's tricky.

The description guidelines for judging don't seem, at least here, to contradict that. No hint is given as to which factors overrule others, nor that the clear victor in one category will necessarily be the winner overall. If one person put all of their might into crafting the smoothest animation this world has ever seen, but his story wasn't original, the judges will need to discuss where the film should rank.

Some contests have gotten more specific in the past, however, that has it's own barrel of monkeys to deal with. As it is, the judges do try to collaborate and discuss things. This helps balance the many different factors. Could they do better? Probably. We're all human. I have never agreed 100% with the results in any contest, (even the EASTER that I ran) but that's life. Sometimes, life isn't fair. The judges can try their hardest, but nothing can ever be quite what is expected. Again, too many variables.

There's a difference between strict guidelines, and general guidelines. While THAC and other contests try, they certainly lean more toward the general guideline side of things. Why? Because it's impossible to judge this variety and number of films on incredibly specific criteria, as doing so cannot account for the number of variables involved.

As to collaborations, I'm for them. I think this year's winner is the first that I remember to really showcase that potential in a THAC, but it doesn't seem to be a problem for me. Many of the best animators in the world work in teams, some friends, some family, some across continents, some across the room. Anybody is free to collaborate, or not collaborate, as they please. A collaboration has great potential for good, but also bad. The jerk between animation styles can be a boon, or a curse. This is one year, and in the history of THAC, how many times has a team won? Some, but not all. Have I ever felt a hint of jealousy at others having a brother or friend help them on their film? Oh yeah. But that doesn't mean they should not be allowed to do it.

Now, all that being said, your concerns are certainly justified. Judges are flawed humans, and it does good for them, and us, to think over our process and look for areas of improvement. Sometimes, honestly, judges do a better job than other times. Sometimes, the films they are handed are more easily and obviously judged than at other times. I can't speak for this contest, having only seen a handful of the films, and having no part of the judging, but these are still great talking points that I hope are picked up by those with more involvement and familiarity with this specific contest. With the coming SHAC having four judges instead of THAC's three, maybe that'll help even things out a bit more.

Also, great job stating all this so civilly and carefully! That means a lot, and will help it be considered by those to whom it is addressed.

Woof, someone's got their pants on backwards.

With any contest, the theme generally serves two purposes:

1. To prevent cheaters by adding yet another stipulation to the contest to make sure entries are made within the time limit/previously completed films aren't submitted as a "new work."

2. Spur creativity by giving all entrants one common ambiguous theme to base their films off of.

Themes, IMO, are more-so meant for those purposes, rather than as strict judging guidelines. And, no matter if you're having the judges freely rank the entries, or if they go off of a weighted number ranking system, it will always be inevitable that you’ll end up with two films competing for the same spot - one a better fit to the overall theme - the other a better overall brickfilm.

This has happened with nearly every brickfilming contest since I've been a member of the community and have paid attention to such things. And honestly, I usually tend to disagree with the final judging as well. But, at the end of the day, it's not whether we made a theme-fitting film, but rather, that we made the best film we could in the time limit.

I remember immediately disagreeing with the final rankings for THAC X back in the day – Especially the 1st place entry. But, after I thought about it more (and a bit of time passed) I eventually came to appreciate those brickfilms more than I initially did – the 1st place entry even becoming one of my favorite all time brickfilms!

Edit after posting:
Just thought I'd add that I wasn't the biggest fan of "The Exquisite Corpse," either. But, again, that could change in time.

I decided to host my own contest (potentially 2… more on that later) partially because of my want to give back to the community, and partially due to my own disagreements with past contest judgments. I’ve set my goals and hope to stick with them, even though I know some may not agree with the final rankings. I’m not saying you’re complaining, but, at the same time, you’ve laid down your opinion, but have no way to act upon it.

If you think there could be more improvements made to brickfilming contests, why not contact Nathan, or even host your own contest! At the end of the day, I don’t care who wins, so long as everybody had fun and all winners receive their prizes.

Last edited by Dyland (February 8, 2018 (04:16pm))

As the designer of the current BiM judging system I feel I should respond to this. I posted a similar thread to this one back in 2014. I proposed a new system which is basically the system we use today.  http://www.bricksinmotion.com/forums/po … 3/#p330073 I do realize that it has been a while since this kind of thing has been explained so people are not familiar with how the judging system works. So I will explain it here. The current BiM judging system works like this. Judges are selected from a pool of respected member of BiM or the general film community. For bigger contests like the Summer Contest they like pulling in people from outside the brickfilming world for a fresh perspective. Once the judges are selected and the films submitted the judges will independently watch all the films and rank their top 15 films. The top 15 leaves a decent sized buffer for a top 10. I collect everyone's lists and place it into a spreadsheet. I have a program I wrote that then does instant run off voting from the rankings. IRV leads to rankings that mathematically the judges are most happy with. Films not in a judges rank list are given a ranking of 16. The sum of the rankings are added up for all the films and the one with the highest total is removed and given a rank. So if there were 20 films the film would be given a rank of 20. The the judges rankings get adjusted for the removal of that film. So if one of the judges had that film ranked 11 then their 12th ranked film will now be ranked 11th. Their 13th film will be ranked 12th and so one. This process is done until there is only one film left. This film is ranked first. This is how the IRV rankings are generated. This is not the final rankings however because this ranking list is then sent back to the judges and the talk to each other and agree to move the rankings around however they see fit. The IRV leads to a good starting point and for the most part the judges only move around a few films after this point. Once the judges are happy with the final rankings they are announced to the forum.

This is fair I would totally prefer to have had more judges. THAC has traditionally been judged by a panel of three to five and most commonly three people. I personally prefer five but we couldn't make that happen so we made three work.

Art is subjective and because of that breaking a film into categories and giving scores to those categories is not a good solution. This is how THAC used to be judges and how judges scored all the categories was wildly different. I ran the number from past THACs that used this system and the judges has a standard deviation of 20 points on a 100 point scale. That is crazy. Art isn't as simple as creating a rubric and checking boxes. We have moved to a system where we are looking at the film as a whole and ranked it compared to another film. It's a subjective process and you cannot get around that.

Why should we judge a film by a team differently than a film made by one person? Are we going to start keeping track of how many people work on a film and dividing their score by that number? Does asking other people to voice act in your THAC film count as a team? This seems fundamentally unfair and unrealistic. People with more money for better equipment are also going to have an easier time. A faster computer for editing and a faster internet connection is a huge advantage. We aren't trying to judge films based on the resources they had to make the film we are judging the film.

 
What's a professional way to judge a film? As I pointed out before splitting up judging a film into categories is not good. You have now multiplied the subjectivity. When it comes down to it the rankings are the films that the judges felt where the best overall films. We look at and discuss the originality, production values (animation, set design, cinematography, sound, etc.), story, and creative interpretation of the theme of a film. These things are not ranked evenly however.

 
This is not true. The IRV makes it that outlier opinions are not given more weight than the value of the group as a whole.

 
I'm sorry you feel this way but if you are only making films to get the attention of a handful of judges you might to reevaluate why you are making films in the first place.

I'm always open to ways to make the judging better and improve the system. One way is I think I need to document the process in a formal way that can get linked in future contests.

Odysee ~ YouTube ~ Website ~ Twitter