<?xml version="1.0" encoding="utf-8"?>
<rss version="2.0">
	<channel>
		<title><![CDATA[Forums - Bricks in Motion - BiM Contest Judging]]></title>
		<link>https://bricksinmotion.com/forums/topic/19549/bim-contest-judging/</link>
		<description><![CDATA[The most recent posts in BiM Contest Judging.]]></description>
		<lastBuildDate>Wed, 06 Aug 2014 15:48:41 +0000</lastBuildDate>
		<generator>PunBB</generator>
		<item>
			<title><![CDATA[Re: BiM Contest Judging]]></title>
			<link>https://bricksinmotion.com/forums/post/330362/#p330362</link>
			<description><![CDATA[This is a very interesting thread. I think it's important to discuss this topic because it's healthy to periodically question and revisit established procedures to make sure they are still 'up to snuff' and adequately meet the needs of an evolving community. I'm also glad to see that the discussion has, for the most part, stayed on point and has not turned into a forum to dissect specific judges and contest results.

There are only three things in life you can count on; death, taxes and someone questioning the judging results of a BiM contest. Often, the questioning seems like sour grapes, but sometimes valid questions are raised. However, due to the inherent nature of judging, you will never be able to eliminate some questioning of results. Because of that, there needs to be a level of transparency into the judging process. These things help with transparency; methodology must be clearly defined and knowledgeable and respected judges should be used.

So many excellent points have been raised here. The problem is that every solution has a downside. For example, Repelling Spider makes an excellent point about the individual components vs. the sum of the parts. However, if you take that too far and minimize, or even eliminate, category judging, you open yourself up to questions of how fair your judging was, and not giving entrants any guidance of criteria.

The key is to find the right balance. I think a hybrid approach should be used. I like the idea of not ranking all entries but having the panel select the top X entries to judge. It's like the academy awards, hundreds of films are produced each year and 'submitted for consideration' for judging, but there are only a handful of nominations. The judges produce their lists of the top X films and then they get together and determine, through discussion, the group of top finalists that will be judged. (This could be a separate announcement in the contest, The Finalists) The finalists are then judged in a two step process. I think you need to establish traditional categories (theme interpretation, cinematography, animation, etc.) to make sure all the judges are on the same page. Each judge comes up with their ranking of the finalists based on these established categories. Then, the judges share their rankings with each other and discuss them to determine the final results. The category rankings are simply the starting point for the final judging discussion. You need both; standardized judging criteria (think: required elements of a figure skating routine) and discussion of overall presentation and impression.

Some notes: 

 - Judging methodology must be clearly communicated in the announcement of the contest. 
 - Ranges for the number of Finalists should also be established prior to the contest. As the number of entries will not be known, ranges should be used: 1 - 20 Entries = 3 - 6 finalists; 21 - 40 Entries = 8 - 15 finalists; etc. (admittedly, this one is tricky but I think it is a necessary step to avoid accusations of fudging results)
 - A head judge should be named who would settle any stalemates in the final judging

I suspect this is unofficially how many of the contests are already being judged, but I don't think that is being disclosed. Full disclosure up front is key. Ideally, independent judges would be best to avoid any accusations of playing favorites but given the size of the community this would be very difficult to achieve.]]></description>
			<author><![CDATA[dummy@example.com (AlNickelsFilms)]]></author>
			<pubDate>Wed, 06 Aug 2014 15:48:41 +0000</pubDate>
			<guid>https://bricksinmotion.com/forums/post/330362/#p330362</guid>
		</item>
		<item>
			<title><![CDATA[Re: BiM Contest Judging]]></title>
			<link>https://bricksinmotion.com/forums/post/330319/#p330319</link>
			<description><![CDATA[Remember, this thread is about how to improve contest judging for the [b]future[/b], and not about the previous BRAWLs. Let's stay on topic.]]></description>
			<author><![CDATA[dummy@example.com (Nathan Wells)]]></author>
			<pubDate>Tue, 05 Aug 2014 15:03:21 +0000</pubDate>
			<guid>https://bricksinmotion.com/forums/post/330319/#p330319</guid>
		</item>
		<item>
			<title><![CDATA[Re: BiM Contest Judging]]></title>
			<link>https://bricksinmotion.com/forums/post/330316/#p330316</link>
			<description><![CDATA[[quote=Squid]Imagine now We the Pumpkins Three, except instead it's called 3 Pumpkins.  The Pumpkins are just blank orange heads on unfitting red shirts, whilst just one of them has the correct head but it's turned the wrong way.  The elf looks kinda weird and it's just a normal yellow head turned round.  The set is just a few things of old grey.  The script doesn't rhyme, it's just me yelling into a mic making an intro and the elf's like, "yo can I have a face?" and the pumpkins would be like "yeah sure girl, we just need you to get us some stuff to develop it."  "K." and it says in comic sans "LATER" and then she comes back, she gets her face, she cries, and the pumpkins explain the moral less than eloquently without properly rhyming and then it ends.
All of it animated and shot just like BRAWLING DURING BRAWL.

Now this would all likely look disgusting[/quote]
Are you kidding? This would be a thrilling finale to the successful trilogy. I want this film in front of my eyes immediately.]]></description>
			<author><![CDATA[dummy@example.com (Mickey)]]></author>
			<pubDate>Tue, 05 Aug 2014 14:51:11 +0000</pubDate>
			<guid>https://bricksinmotion.com/forums/post/330316/#p330316</guid>
		</item>
		<item>
			<title><![CDATA[Re: BiM Contest Judging]]></title>
			<link>https://bricksinmotion.com/forums/post/330251/#p330251</link>
			<description><![CDATA[[quote=brickelodeon][quote]The theme interpretation isn't rated on how obvious/apparent the theme was in a film, rather its interpretation, how clever/unique it was. BRAWLING DURING BRAWL was not a serious film, it was a joke, intended to get last place, so don't take the rating on that seriously.[/quote]

This makes a bit more sense. Thanks for the clarification. Although I still hold that the score of theme interpretation was directly proportional to presentation.[/quote]

I feel like that if I had actually put effort into BRAWLING DURING BRAWL it would likely have gotten a much higher score in nearly every respect even if I left the story and theme basically untouched.  I could have lengthened it out and had it play itself more seriously with well-built sets, dramatic lighting, better minifigures, proper voice acting, nice music, and perhaps a few more jokes.  Maybe even a little bit more background showing how the doctor got his ambition, showing his wife and parents dying of disease.
Now, that would all likely be a much better film, however, the theme and story would be basically as they were, however I would guess get a higher score on all accounts, even theme and story.  They wouldn't be much different, yet they would be presented much better and seem much nicer (or maybe just not horrible).  And the film might've placed decently.

But the same would likely go the other way.  Imagine now We the Pumpkins Three, except instead it's called 3 Pumpkins.  The Pumpkins are just blank orange heads on unfitting red shirts, whilst just one of them has the correct head but it's turned the wrong way.  The elf looks kinda weird and it's just a normal yellow head turned round.  The set is just a few things of old grey.  The script doesn't rhyme, it's just me yelling into a mic making an intro and the elf's like, "yo can I have a face?" and the pumpkins would be like "yeah sure girl, we just need you to get us some stuff to develop it."  "K." and it says in comic sans "LATER" and then she comes back, she gets her face, she cries, and the pumpkins explain the moral less than eloquently without properly rhyming and then it ends.
All of it animated and shot just like BRAWLING DURING BRAWL.

Now this would all likely look disgusting, however, it would have exactly the theme interpretation and story as the actual film, though I have little doubt it would easily place last or very nearly there.

As Aquamorph pointed out, the criteria for the judging is a little meaningless.
I sometimes feel a little like every good brickfilm is just a bad brickfilm in disguise, that or every bad brickfilm is just a very good one in disguise.  A bad story can be made to look good, a good story can be made to look bad.  The whitest cleanest LEGO brick turns black in the dark, a black LEGO brick will shine brightly if you position it to reflect the light the right way.]]></description>
			<author><![CDATA[dummy@example.com (Squid)]]></author>
			<pubDate>Mon, 04 Aug 2014 19:11:54 +0000</pubDate>
			<guid>https://bricksinmotion.com/forums/post/330251/#p330251</guid>
		</item>
		<item>
			<title><![CDATA[Re: BiM Contest Judging]]></title>
			<link>https://bricksinmotion.com/forums/post/330249/#p330249</link>
			<description><![CDATA[[quote=AquaMorph]Each of the judges ranks the films from first to last. These rankings are tallied together.[/quote]
This is how we did the past two THACs. First place got 78 points, last place got 1, whoever got the most when all the scores were added up won.]]></description>
			<author><![CDATA[harry@bossert.com (Hazzat)]]></author>
			<pubDate>Mon, 04 Aug 2014 18:21:00 +0000</pubDate>
			<guid>https://bricksinmotion.com/forums/post/330249/#p330249</guid>
		</item>
		<item>
			<title><![CDATA[Re: BiM Contest Judging]]></title>
			<link>https://bricksinmotion.com/forums/post/330244/#p330244</link>
			<description><![CDATA[[quote=AquaMorph]The first in political science terms is called instant runoff voting. Each of the judges ranks the films from first to last. These rankings are tallied together. I think the easiest way to do this is to say a first place vote is one point a second place vote is two and so on. You tally up the point and eliminate the film with the most points. That film gets the last place ranking. From there the ballots are adjust removing the film that was just ranked. If the on the ballot the film was not the lowest ranked film the films that are lower ranked move up. This process is repeated until all the film are ranked and you have a winner. This entire process is easily automated and only takes each judge filling out one ballot. Statistically it leads to the judges being happier with the results.[/quote]

This seems to be the most familiar and straight-forward solution. I think it would be good to explore it further for future contests.

[quote=AquaMorph]The second is currently only a theory but it is based on the Elo ranking algorithm. This is used in a lot of things for example chess and most video games with ranking systems. How it works is all films would start out at the same level. For example, 1500. The judges would then compare one film against another and select the one they like better. From there the algorithm gives points to the winner and subtracts them from the looser. The amount of point is determined by the expected outcome of the face off. So for example, if a 1800 and a 1200 film face each other and the 1800 film wins it will not that gain that many points because it was heavily favored. However, if the 1200 film wins it will gain much more points. After a few rounds a clear ranking is developed.[/quote]

This is also an intriguing solution. It might be harder to convince the general public about it, since it seems a little more complex and uncommon (even though, as you point out, it is used in many ranking systems). I think if this method is used, it needs to be explained clearly so people aren't turned off by it's unfamiliarity.]]></description>
			<author><![CDATA[dummy@example.com (Nathan Wells)]]></author>
			<pubDate>Mon, 04 Aug 2014 16:49:48 +0000</pubDate>
			<guid>https://bricksinmotion.com/forums/post/330244/#p330244</guid>
		</item>
		<item>
			<title><![CDATA[Re: BiM Contest Judging]]></title>
			<link>https://bricksinmotion.com/forums/post/330222/#p330222</link>
			<description><![CDATA[[quote]The theme interpretation isn't rated on how obvious/apparent the theme was in a film, rather its interpretation, how clever/unique it was. BRAWLING DURING BRAWL was not a serious film, it was a joke, intended to get last place, so don't take the rating on that seriously.[/quote]

This makes a bit more sense. Thanks for the clarification. Although I still hold that the score of theme interpretation was directly proportional to presentation.]]></description>
			<author><![CDATA[dummy@example.com (Brickelodeon)]]></author>
			<pubDate>Mon, 04 Aug 2014 03:17:32 +0000</pubDate>
			<guid>https://bricksinmotion.com/forums/post/330222/#p330222</guid>
		</item>
		<item>
			<title><![CDATA[Re: BiM Contest Judging]]></title>
			<link>https://bricksinmotion.com/forums/post/330221/#p330221</link>
			<description><![CDATA[A lot of good ideas are being tossed around in this thread! It's nice to see a discussion on the topic of judging. There will certainly be changes to the judging system next year. I haven't made a decision on what system to use yet (I don't need to for quite some time), so please feel free to keep suggesting judging systems/techniques you know of or thought of.

[quote=brickelodeon]Just throwing this out there:
I have nothing against Nathan Wells, he is an incredible brickfilmer, but I don't see why Blinders won BRAWL this year. When I looked at the spread sheet, I was shocked to see that Blinders had the highest score in theme interpretation. I've watched it over and over, but I can only find tiny references to development. Galan505's film, The Website, was obviously about development, it couldn't have been more blatant, and yet it got a 27 and Blinders got a 35? I feel like the judging was incredibly biased in theme interpretation this year. The score of theme interpretation was directly proportional to presentation. BRAWLING DURING BRAWL, while it was a joke film, got an 11.7 even though it was obviously about developing a cure for all diseases. THIS IS MESSED UP!!

There. Now look what you made me do. You made me rant.

Take it or leave it, I feel like my post is relevant to this thread.[/quote]

The theme interpretation isn't rated on how obvious/apparent the theme was in a film, rather its interpretation, how clever/unique it was. BRAWLING DURING BRAWL was not a serious film, it was a joke, intended to get last place, so don't take the rating on that seriously.]]></description>
			<author><![CDATA[funtasticfilms@gmail.com (Fun Sucker)]]></author>
			<pubDate>Mon, 04 Aug 2014 02:52:43 +0000</pubDate>
			<guid>https://bricksinmotion.com/forums/post/330221/#p330221</guid>
		</item>
		<item>
			<title><![CDATA[Re: BiM Contest Judging]]></title>
			<link>https://bricksinmotion.com/forums/post/330216/#p330216</link>
			<description><![CDATA[I think we need to be careful not to dis individual judges who are volunteering their time to make the community a better community.  We should do as the OP suggests, keep conversation to the general topic of competition judging.]]></description>
			<author><![CDATA[dummy@example.com (togfox)]]></author>
			<pubDate>Mon, 04 Aug 2014 00:56:25 +0000</pubDate>
			<guid>https://bricksinmotion.com/forums/post/330216/#p330216</guid>
		</item>
		<item>
			<title><![CDATA[Re: BiM Contest Judging]]></title>
			<link>https://bricksinmotion.com/forums/post/330212/#p330212</link>
			<description><![CDATA[Just throwing this out there:
I have nothing against Nathan Wells, he is an incredible brickfilmer, but I don't see why Blinders won BRAWL this year. When I looked at the spread sheet, I was shocked to see that Blinders had the highest score in theme interpretation. I've watched it over and over, but I can only find tiny references to development. Galan505's film, The Website, was obviously about development, it couldn't have been more blatant, and yet it got a 27 and Blinders got a 35? I feel like the judging was incredibly biased in theme interpretation this year. The score of theme interpretation was directly proportional to presentation. BRAWLING DURING BRAWL, while it was a joke film, got an 11.7 even though it was obviously about developing a cure for all diseases. THIS IS MESSED UP!!

There. Now look what you made me do. You made me rant.

Take it or leave it, I feel like my post is relevant to this thread.]]></description>
			<author><![CDATA[dummy@example.com (Brickelodeon)]]></author>
			<pubDate>Mon, 04 Aug 2014 00:37:24 +0000</pubDate>
			<guid>https://bricksinmotion.com/forums/post/330212/#p330212</guid>
		</item>
		<item>
			<title><![CDATA[Re: BiM Contest Judging]]></title>
			<link>https://bricksinmotion.com/forums/post/330175/#p330175</link>
			<description><![CDATA[I have some ideas.

[b]1)[/b] Get 6 judges for popular contests. For each category for each film, the highest and lowest scoring judge gets their score removed (like they do in scoring games in the olympics), and an average is calculated.
[b]2)[/b] Have clear categories. Theme, story, presentation, and sound design.
          [b]a:[/b] To me, story makes or breaks any film. This should be 40% of the total score.
          [b]b:[/b] For timed contests, theme is important, otherwise someone could write their story and perfect it long before the contest even starts. In this case, the theme interpretation should probably have either a large bearing, or contestants get penalised to an extent on story if it doesn't fit the theme. I think that 30% for theme is important for timed contests. For every other contest, then it's not as important so could probably be 20%
          [b]c:[/b] Presentation and sound design are both important, but technical parts of the animation process. Some are good at one, but not the other. Some both, and some not. This should be a point where the best can get extra points, but the worst don't loose so much on. Technically a better story should beat a better presentation any day. Fo timed competitions, I think these should each be worth 15%, and for not timed competition, 20%.]]></description>
			<author><![CDATA[dummy@example.com (Kieren Barnett)]]></author>
			<pubDate>Sun, 03 Aug 2014 14:31:47 +0000</pubDate>
			<guid>https://bricksinmotion.com/forums/post/330175/#p330175</guid>
		</item>
		<item>
			<title><![CDATA[Re: BiM Contest Judging]]></title>
			<link>https://bricksinmotion.com/forums/post/330073/#p330073</link>
			<description><![CDATA[I'm not complaining about BRAWL. I have been annoyed at the way judging has been treated for a while. This year's BRAWL was excellently run from what I saw. Questions were answered in a timely manner and the judging was prompt. This was general designed to get stuff going for future contest while people still cared about contest stuff. I have a few ideas for improving judging.

First, I think giving a score and ranking to every film is a huge waste of time. My suggestion is do the judging in stages. With a few judges select the pool of films that will be judge. I would not set a hard coded value or percentage because the amount of good entries will vary on the contest. After that go through all the select entries and rank them. Then announce a top 10 or top 5 or whatever is appropriate for the contest. That way being ranked would be more of an horror and it also eliminate a lot of the complain of films being under ranked.

Now as far as ranking systems I have two ideas. 

The first in political science terms is called instant runoff voting. Each of the judges ranks the films from first to last. These rankings are tallied together. I think the easiest way to do this is to say a first place vote is one point a second place vote is two and so on. You tally up the point and eliminate the film with the most points. That film gets the last place ranking. From there the ballots are adjust removing the film that was just ranked. If the on the ballot the film was not the lowest ranked film the films that are lower ranked move up. This process is repeated until all the film are ranked and you have a winner. This entire process is easily automated and only takes each judge filling out one ballot. Statistically it leads to the judges being happier with the results.

The second is currently only a theory but it is based on the Elo ranking algorithm. This is used in a lot of things for example chess and most video games with ranking systems. How it works is all films would start out at the same level. For example, 1500. The judges would then compare one film against another and select the one they like better. From there the algorithm gives points to the winner and subtracts them from the looser. The amount of point is determined by the expected outcome of the face off. So for example, if a 1800 and a 1200 film face each other and the 1800 film wins it will not that gain that many points because it was heavily favored. However, if the 1200 film wins it will gain much more points. After a few rounds a clear ranking is developed.]]></description>
			<author><![CDATA[dummy@example.com (AquaMorph)]]></author>
			<pubDate>Fri, 01 Aug 2014 23:12:14 +0000</pubDate>
			<guid>https://bricksinmotion.com/forums/post/330073/#p330073</guid>
		</item>
		<item>
			<title><![CDATA[Re: BiM Contest Judging]]></title>
			<link>https://bricksinmotion.com/forums/post/330068/#p330068</link>
			<description><![CDATA[Disclaimer: BRAWL is not a BiM contest in the official sense. So maybe the judging on BRAWL is not good, I don't really follow it closely enough to say, and I will make no attempt to defend it. The fact that I can't supervise contests closely in every instance is part of why BiM only has a few official contests that I put my stamp of approval on; in the past few years it has only been THAC.

I think THAC judging is generally pretty good, though I would say that after 10th place or so it becomes very difficult to judge degrees of goodness or badness because everything is on a mediocre-to-bad spectrum at that point.

That said, for the non-THAC contests I have overseen, such as the summer contests like TOY and Avant-Garde, a more thorough judging process was employed, and I think that it achieved the best results. The problem with the systematic approach used in THAC and (as far as I know) BRAWL is what Repelling Spider has touched on.

With the summer contests, I created a private forum where judges discussed the rankings of the winning entries until they had settled on an overall ranking they felt was accurate. We then (yes) fudged with the points to make the rankings fit the way we wanted. This was more about making the point averages something we could all agree on and reflect the results we wanted than, say, tampering to reflect our biases or something like that. This method isn't practical for THAC because THAC gets too many entries and the overall quality is not quite as high because of the 24-hour nature of the contest. Inevitably there are some compromises to make judging a contest like this efficient.]]></description>
			<author><![CDATA[dummy@example.com (Sméagol)]]></author>
			<pubDate>Fri, 01 Aug 2014 21:29:32 +0000</pubDate>
			<guid>https://bricksinmotion.com/forums/post/330068/#p330068</guid>
		</item>
		<item>
			<title><![CDATA[Re: BiM Contest Judging]]></title>
			<link>https://bricksinmotion.com/forums/post/330066/#p330066</link>
			<description><![CDATA[I also support this idea.
It may sound a little self-serving since I could have placed a little better in this BRAWL, but even when I won BRAWL last year with Odoriferous, I felt a little weird about it because there were two entries I topped which I thought might beat me.  And, even if they I were to top them, I was pretty sure that I would do so only slightly and they would follow directly after mine, and yet they were a few entries behind, which all felt weird.

If it can be managed I think more judges would be an excellent idea.  Also, as Flying Minifig mentioned, if possible getting an outsider to be added to the judging panel may be interesting as it could give some rather unbiased opinions.  I feel as it is judging deviates somewhat from the community's general opinion of the films.  But more judges may help to make the general opinion of the judges somewhat closer.
Of course, I doubt we can ever make everyone happy, but we might be able to make the system to some degree slightly better.

[quote=FlyingMinifig]Maybe the judging criteria should be restructured as well.  40/100 for theme interpretations seems pretty excessive.  I'm guessing it might be to get people to adhere to the theme and deter cheating, but I found the allocation of points in this contest pretty random.  I think 10-20/100 should be enough.  Story should be given a greater proportion of the criteria, and I feel there should be separate categories for animation, cinematography and sound design--I found it odd that they were all lumped together in one category for BRAWL 2014.  Also, how is "creativity" defined as?  It doesn't seem like a very objective criteria.[/quote]

I think one of the points Christian was trying to make in his spreadsheet thingy with his fancy math is that the criteria didn't actually matter that much, as regardless of having several different points of criteria, they all tended to be rated relatively similar percentages.
In THAC X the judges simply picked their favourite films, and I think that may in general do better, but especially so with more judges, and possibly even more so with Aq's other idea of cutting off the high and low ratings, of course, I'm not sure how that would turn out, but it sounds like an interesting idea.  A good film is good because it's really good, and choosing it as good from a few categories is a bit weird.

Though, theme is an especially weird category to have lead, as I've always thought of the theme as more of a way to prove that it was made within the time constraints, as well as a sort of almost running gag throughout our films so we could obviously theme-drop.  I've never watched a BRAWL or THAC film and thought I liked it because of the way the theme was interpreted was especially good, but I've always liked it more because it was just a good film.

I've judged several brickfilming contests before and been given some random criteria, sometimes it does seem like a sort of roll of the dice since you're not quite sure exactly how much it should get in some certain category, so I tended to just lean toward more points or less points depending upon how much I liked the film.
I also feel that judging the film as a whole takes somewhat less time than thinking about specific categories.]]></description>
			<author><![CDATA[dummy@example.com (Squid)]]></author>
			<pubDate>Fri, 01 Aug 2014 20:39:59 +0000</pubDate>
			<guid>https://bricksinmotion.com/forums/post/330066/#p330066</guid>
		</item>
		<item>
			<title><![CDATA[Re: BiM Contest Judging]]></title>
			<link>https://bricksinmotion.com/forums/post/330064/#p330064</link>
			<description><![CDATA[I agree that there is a problem with the judging and I suggest bringing back the Enjoyment category from the early BRAWLs. This was basically a category for what Spider is getting at; at the end of the day, what matters most is just how good of a watch it was. I believe that with this category, the results for this BRAWL would look vastly different and one of the entries in second or third would have won.]]></description>
			<author><![CDATA[sillypenta@gmail.com (Penta)]]></author>
			<pubDate>Fri, 01 Aug 2014 20:33:31 +0000</pubDate>
			<guid>https://bricksinmotion.com/forums/post/330064/#p330064</guid>
		</item>
	</channel>
</rss>
