ROAR Skew for Scenarios out of Balance?

[Aaron Cleavin]

If 35-40% of games reported to ROAR are from players of significantly different skills levels then ..

If a scenarios is 50/50 or close then I think the stronger player getting either side will change the probability of that side winning but about the same amount.

Where one side is favored by say 80/20 though if the get the favored side it might of the win chance to 90-95% while if the get the un-favored side then the favored side might move to 50-55%.

I think this skew might move out of balance scenarios back to the center by about 5% Depending on what assumptions one makes.

Thoughts?

 

[Robin Reeve]

It depends how many plays have been recorded.

 

[Aaron Cleavin]

It depends how many plays have been recorded.

Clearly I am thinking that scenarios which show up as 70-30 or more after 50 playings might be considered to be 5% more skewed than the record indicates
due to the effect as noted.
Getting experimental evidence would require a power of work but it is an effect worth considering the possible effect of.

 

[JOS]

... scenarios which show up as 70-30 or more ... might be ... 5% more skewed ... an effect worth considering ....

Whether ROAR labeled a scenario 75-25 instead of 70-30 would have little impact on my decision to play it, or on which balance to employ. ROAR is a useful tool, but as with most statistics better predicts population rather than individual patterns. It boots little to parse its results to this level of detail, in my opinion.
Enjoy, Joshua

 

[Eagle4ty]

Clearly I am thinking that scenarios which show up as 70-30 or more after 50 playings might be considered to be 5% more skewed than the record indicates
due to the effect as noted.
Getting experimental evidence would require a power of work but it is an effect worth considering the possible effect of.

In as much as several tournaments use ROAR to some extent to help pick scenarios for their agenda, in my limited experience with tournament play I would guess less than 10% of scenarios showing a rather favorable edge to a side (say ~60%+) actually play out to be wildly biased beyond that percentage. I certainly do not have all the pertinent data to back up such a claim, but my personal experience and gut feeling is that a relatively balanced scenario showing on ROAR is usually borne out in a tournament setting with only a few exceptions and those are usually scenarios with a low record of playings..

 

['Ol Fezziwig]

If 35-40% of games reported to ROAR are from players of significantly different skills levels then ..

If a scenarios is 50/50 or close then I think the stronger player getting either side will change the probability of that side winning but about the same amount.

Where one side is favored by say 80/20 though if the get the favored side it might of the win chance to 90-95% while if the get the un-favored side then the favored side might move to 50-55%.

I think this skew might move out of balance scenarios back to the center by about 5% Depending on what assumptions one makes. Play for the fun of it?

Thoughts?

 

[bprobst]

Thoughts?

So what you are saying is that good players who are good are more likely to win than other players who are not as good?

I don't think you need to look at ROAR numbers to arrive at that conclusion.

ROAR statistics do not describe a scenario, they are merely a single point of information, and only of very limited scope because of the very loose method that generates the statistics in the first place. Trying to squeeze "truth" out of them is a pointless exercise. Practical experience of the players is far more valuable. There are many scenarios on ROAR that appear to be "near perfect" if the ROAR statistics are the only thing that you consider, but the actual reality of playing them will generate a different opinion; and of course a significant disparity in player skills will skew results on the gaming table.

IMO the most important stats on ROAR are the rankings. It's nice that a scenario "seems" balanced, but knowing that a lot of other players had fun while playing it is ultimately a better reason to play it yourself.

 

[Aaron Cleavin]

So what you are saying is that good players who are good are more likely to win than other players who are not as good?

Not really what I am saying Bruce. I am noting an ASYMMETRY of the chances of an unbalanced scenario when played between players of differing skill levels depending on side taken.

 

[fanatic+1]

I have noted that some scenarios are balanced when played by experienced players, but are not balanced when played by noobs.

 

[bprobst]

Not really what I am saying Bruce. I am noting an ASYMMETRY of the chances of an unbalanced scenario when played between players of differing skill levels depending on side taken.

In other astonishing news, the sun rose in the east this morning.

I have no clue what you're trying to suggest here. Don't take ROAR statistics at face value? We've known that for as long as ROAR has been around.

When judging the potential balance of any scenario, the experiences and skills of the players will count for 90% or more. What ROAR says about the scenario will count for maybe 5% (and more typically, a lot less). Anyone who tries to make ROAR more meaningful than that is just asking for disappointment. Applying random variables based on hearsay is unlikely to improve the situation.

 

[Aaron Cleavin]

In other astonishing news, the sun rose in the east this morning.

I have no clue what you're trying to suggest here. Don't take ROAR statistics at face value? We've known that for as long as ROAR has been around.

When judging the potential balance of any scenario, the experiences and skills of the players will count for 90% or more. What ROAR says about the scenario will count for maybe 5% (and more typically, a lot less). Anyone who tries to make ROAR more meaningful than that is just asking for disappointment. Applying random variables based on hearsay is unlikely to improve the situation.

Suggesting that a scenario that ROAR suggest is somewhat out of balance may be more out of balance than indicated by the numbers.

 

[Steven Pleva]

Where one side is favored by say 80/20 though if the get the favored side it might of the win chance to 90-95%

I generally agree with your premise regarding the difference of ability on the ROAR results. However, I think in the example above where two players have a significant difference in ability playing an 80/20 scenario the better player playing the favored side will win 97-99% of the time. 80/20 is a big hill to climb...
Steve

 

[von Marwitz]

I have noted that some scenarios are balanced when played by experienced players, but are not balanced when played by noobs.

Indeed.

Maybe you can even go so far to say that some scenarios are unbalanced when played by experienced players but balanced when played by noobs. This will be the case less frequently than the other way around as scenario designers are usually rather experienced. But I am thinking of some old scenarios with 10 turns. While way back, more Prep Firing and less Movement seemed to be a widespread approach, it took longer to get things done. Over the years, tactics have evolved and spread in breadth, so today the same old scenario might only be considered balanced with 9 or 8 turns.

I think there are too many factors that influence the reported ROAR results so that Aaron's original (and quite interesting) thought is of no real practical consequence.

• Differing skill levels of the opponents.
• Different skill levels per se - if you don't know some things, some scenarios will simply not work.
• Scenario balance.
• Scenario balance in relation to the differing skill levels of the opponents and sides picked by them. I.e. if the better player plays the unfavorable side, the scenario might appear more balanced, which might skew reported results.
• Solo plays reported on ROAR.
• Scenarios played and reported multiple times by the same player. Some might not 'grok' it in their first playing but get a feeling for the timing and for what to do on a subsequent playing.
• Scenarios played multiple times by the worse player vs. a first playing by the better player.
• An insufficient number of reported plays as to make the ROAR rating empirically reliable.
• People conceding prematurely. I have seen this quite a lot of times. Some people don't like hanging on for the end-game being bashed for an extra turn or two either because they have no fun in it, don't have confidence in their chances or do not see that things might suddenly take a turn for the worse for the opponent (it is ASL after all...)

The list could go on.

So ROAR is best to be taken as a rough indicator about the given scenario - not more nor less.
The most interesting bits of information are:

• Number of playings
• Fun rating
• Balance rating

Or rather the combination of these three factors. Looking at the combination of these factors will usually allow you to determine if the balance rating has some significance, if the scenario is fun to play, and if it is popular. But even if a scenario has many playings, say 70 - 100, I think due to the many factors laid out at the top, it will be impossible to determine of whether the ratings are skewed in the range of some 5% by one factor or the other.

von Marwitz

 

[boylermaker]

My guess would be that the give-the-better-player-the-disfavored-side effect exists, and it skews ROAR records towards 50-50.

On the other hand, should we take it into account when trying to guess whether scenarios are unbalanced? In the grand scheme of things, Bruce is of course right that there are much better ways that a good ASL player can use to determine scenario balance. But that is a true fact that isn't helpful to me, or to other inexperienced players, who don't have the skill to look at a scenario card and figure out the balance. I don't really have any better source than ROAR, and ROAR is a huge improvement on my own intuition, so it's worth it for players like me to try to divine what the various ROAR biases are.

But, how about this bias: some players consult ROAR to determine which scenario to play. If a scenario has a lopsided record, it looks like a dog and is less likely to be played. Some scenarios are balanced, but have lopsided records due to chance alone. Once a scenario has a lopsided record, people play it less, which means that it won't get the number of playings needed to "return to the mean". This effect biases ROAR records towards 0-100.

Are the two effects comparable in magnitude? I dunno.

 

[bprobst]

Suggesting that a scenario that ROAR suggest is somewhat out of balance may be more out of balance than indicated by the numbers.

And your suggestion is based on which hard facts, exactly?

What? You don't have "hard facts"? So you're thinking you should modify what is already a vague estimate based on other vague estimates?

You honestly don't see a problem with that? Well, I suppose it's not so much a "problem" as a complete waste of time. Your time to waste, I guess, but if you're really concerned about the actual balance of a scenario my gut instinct would be to try the time-honoured approach of "looking at the scenario card and seeing how it jibes with my personal ASL experience and knowledge". Maybe even -- gasp! -- discuss it with your opponent. And if all else fails -- play the damn thing. Or not. Agonising over what ROAR seems to suggest about the scenario would be way down on my list of priorities.

ROAR is just a data point, one of many, and not a hugely important one at that.

I don't really have any better source than ROAR, and ROAR is a huge improvement on my own intuition, so it's worth it for players like me to try to divine what the various ROAR biases are.

Yes you do, no it isn't, and no it isn't.

I'm not saying ROAR is worthless. Quite the opposite! It has a lot of value, but it so happens that "providing an estimate of the approximate balance of a scenario" is pretty much the least valuable service it provides. Too many players think that it's the only service that it provides, and even worse, they assume that it's really good at doing it.

 

[Steven Pleva]

Somehow, this thread devolved in a "ROAR is more valuable" versus "ROAR is less valuable" debate. I don't think that is where Aaron is coming from. I think Aaron is merely identifying a case that is skewing the numbers in a particular direction. I generally agree with his hypothesis. ROAR has its value and flaws. This observation transcends those ROAR attributes. Aaron, please correct me if I don't understand your position...
Steve

 

[Ric of The LBC]

Isn't this game played for fun?

 

[dlazov]

Isn't this game played for fun?

Except when your losing then it’s unbalanced.

 

[TopT]

Except when your losing then it’s unbalanced.

That's why I lose all the time!

 

[Will Fleming]

Would agree there probably is the effect mentioned given that sides are determined randomly. I would suggest that the better player can *probably* select the better side as well, which may be a counter example if sides are selected and not random.

[one way people pick scenarios]
Say I play a guy named "Player-A" and he is a much better and more experienced player. I send him 3 scenarios to choose and he sees an advantage for a side in a particular scenario and goes with it. Instead of the scenario being unbalanced 80/20 given equal skill levels, it is now 95% or so in this example. So, allowing the stronger player to pick sides may affect things in a different direction (my claim, not Aaron's).

[a second way]
If we roll a dr for sides instead, his skill advantage will tend to balance ROAR no matter the real balance as per Aaron's original post. Whatever side he gets will win more often than not because I lack the skill set to give him a fair contest.

[and a third for some who have lots of time]
Say "Player-A" plays me in "The unbalanced dog" and beats me as the Russians. We then flip sides and he beats me as the Germans. Both games reported to ROAR which shows it perfectly balanced based upon our two playings, but all that really happened was the better player won both games.

==> While I have no idea on 5% being a fair number to define the shift, I think Aaron is correct in his basic claim.