ROAR Skew for Scenarios out of Balance?

[Aaron Cleavin]

Somehow, this thread devolved in a "ROAR is more valuable" versus "ROAR is less valuable" debate. I don't think that is where Aaron is coming from. I think Aaron is merely identifying a case that is skewing the numbers in a particular direction. I generally agree with his hypothesis. ROAR has its value and flaws. This observation transcends those ROAR attributes. Aaron, please correct me if I don't understand your position...
Steve

Yes not arguing for ROAR being less or more valuable. Just offering an observation, the major basis being about 500 games between Chad Cummins and I (SP 1-252, J1-157, FrF 1-64, Crucible of Steel (- Flying Torretes). We alternate attacker defender and after each scenario do an assessment on perceived balance, the effect of scenarios seeming more out of balance than ROAR indicates seems to be quite consistent (Note we only look at ROAR after playing the scenario). I know the data set is not huge but enough I feel to offer an observation.

 

[lluis61]

I think all this is pointless. ROAR is what it is and no more, specially when one considers that the same scenario Will be different from the personal choices of setup. ASL is not chess. So, ROAR is merely orientative.

 

[Will Fleming]

I don't mean to knock ROAR and use it as a check quite often when deciding on scenarios to play. JR does us a service by providing this info in an easy to consume format.

 

[von Marwitz]

I don't mean to knock ROAR and use it as a check quite often when deciding on scenarios to play. JR does us a service by providing this info in an easy to consume format.

This.

Nevertheless, I am always interested in deliberations on chances & odds. So I find Aaron's thoughts interesting.

von Marwitz

 

[Ray Woloszyn]

IMO the most important stats on ROAR are the rankings. It's nice that a scenario "seems" balanced, but knowing that a lot of other players had fun while playing it is ultimately a better reason to play it yourself.

Well put!

 

[Mister T]

On the contrary the value of ROAR is to provide an objective ballpark assessment on a scenario's balance. While i believe Aaron's assumption, it affects only already-largely unbalanced scenarios and its estimated magnitude remains small. ROAR remains a useful tool for having a first hint at a scenario's balance.

I never look at rankings, as fun assessments are subjective, personal and can easily be derived from your own observation of the playing card.

It takes seconds to determine whether a scenario is fun or not, while it would take minutes for an equivalent assessment on balance, that's where the ROAR comes handy (provided there are 'enough' playings).

 

[boylermaker]

Would agree there probably is the effect mentioned given that sides are determined randomly. I would suggest that the better player can *probably* select the better side as well, which may be a counter example if sides are selected and not random.

This is something you could test for! This effect will be stronger in tournaments than in causal play, because in tournaments the better player is more motivated to use this advantage to his own benefit. Collect a bunch of tournament playings; if scenarios look less balanced in a tournament context than on average, that's good evidence this phenomenon exists.

If ROAR saves the order in which playings were added, you could also look for signs of Aaron's original idea.

 

[Steven Pleva]

Balance is way to complicated to boil it down to one number. As someone up thread observed, balance is not usually consistent across all skill levels. Some scenarios are balanced at a high level, but not at a low level -- and vice versa. Peaches and I noticed this often when playtesting for Albany...
Steve

 

[boylermaker]

I would argue that's super easy to boil down to one number: calculate the balance between players of equivalent skill for Level 0 players, Level 1 players, Level 2 players, all the way up to whatever the highest level is. Then average all those balances, weighting each by the proportion of games played by Level 0, Level 1, Level 2 players, etc. That's the balance of the scenario.

That's also what ROAR reports.

(Yes, ROAR also reports games between players of different skill levels, but based on this conversation I'm inclined to think that those games aren't likely to change the balance in one direction or another).

Steve is right that the most granular possible metric of balance would be a 3-D surface with Axis Skill on the x-axis, Allied Skill on the y-axis, and proportion-of-Axis-wins on the z-axis. But I this approach has some sticking points too--how do you know how good you and your opponent are? Can you figure this out with enough precision compared to the size of the fluctuations in the Balance Landscape that there is any improvement upon using a single balance number? I don't know the answer to that.

 

[Gunner Scott]

What ever happened to Trezza? I kinda miss his unique colloquial's on life and ASL. lol

Balance is way to complicated to boil it down to one number. As someone up thread observed, balance is not usually consistent across all skill levels. Some scenarios are balanced at a high level, but not at a low level -- and vice versa. Peaches and I noticed this often when playtesting for Albany...
Steve

 

[Steven Pleva]

Trying to "fix" ROAR is a fool's errand. Understanding some of its limitations is about the best we can hope for. I think that Aaron's OP tries to shine a little more light on the subject. Overstating or understating the value of ROAR is probably an attempt by those who like certainty trying to "justify" their view point. Some things in life defy simplicity...
Steve

 

[Steven Pleva]

What ever happened to Trezza? I kinda miss his unique colloquial's on life and ASL. lol

I miss the big lug, too. I talk to him every so often and he is a happy camper...
Steve

 

[von Marwitz]

What ever happened to Trezza? I kinda miss his unique colloquial's on life and ASL. lol

Good question. He posted some of the best AARs around here from time to time (those with the 'Uh-oh'...).

von Marwitz

 

[Sparafucil3]

Balance is way to complicated to boil it down to one number. As someone up thread observed, balance is not usually consistent across all skill levels. Some scenarios are balanced at a high level, but not at a low level -- and vice versa. Peaches and I noticed this often when playtesting for Albany...
Steve

One need look no further than the Singling mini-CG from operations. Top players think the American's have too much. The rest of us schmucks think they need some help. -- jim

 

[Jacometti]

We generally design scenarios in such a way that poor players will publicly denounce them as unbalanced.