That's quite a few questions, of which for some the answer would be a manual - wait a minute, Mr Pitcavage already did that.
I can only speak for Friendly Fire of course. How other people go about this business, I do not know.
I was wondering what the process or philosophy was for playtesting and then publishing new scenario's. I know publishers playtest, yet once published some come out as dogs while others are 50/50 after 100 or 1000 plays, based on what I see in Roar for example. I see far different numbers of plays as well for scenario's published from the same pack and it got me to wondering how important is "balance" during playtesting?
Balance is very important, but it is not only hard to archive but also hard to know when you've actually archive it or not - even long after publication and many, many ROAR reports. You might even find it hard to define what balance is.
And balance is (obviously?) not only important in a competitive environment - it's also more fun when the game is close when you play "for fun".
Is the goal 50/50 or 60/40 or is there really no goal?
Our goal is 50/50 given ideal play from both sides, but I would say 60/40 is also OK.
Are playtesters generally so experienced that their skill level can overcome things that the average or new player may not be able to, and a scenario that they like becomes completely one sided once sold to the public?
Yes, I think that happens, and that is one of the reasons for it being so hard to understand if your scenario is balanced based on ROAR records. Not only need you have very many playings before you can say anything with some certainty, but also, just as you say, will ROAR reflect average play. But then, it might be the case you prefer to know the "average play" balance and not the balance under ideal conditions (without saying those ever appear).
With "simple" (meaning, less likely for people to screw up in one way or the other) scenarios, ROAR will say more.
What causes some scenario's to become far more popular then others, can publishers guess that before release?
Yes, pretty much. You know what most people appreciate and even more so what they don't want. For example, I knew already before we printed Friendly Fire Pack 3 that FrF21 Cavalry Brigade Model wasn't going to be the most popular scenario in the pack, even though I thought it was probably my best design. And there's a simple reason for it: it has Cavalry. People generally avoid having to read new rules (quite understandably - I do too).
But it does happen you're wrong, of course. I didn't expect so many to play FrF20 Adolf's Amateurs. What I was right about was many people having strong opinions about it.
Especially playing the attacker (Germans) is both frustrating and hard, and it easy to screw up and lose the game already at set up. I included it in the pack because I like it myself, and because a few players have really loved it.
I'm sure you could analyze and find a cook book for scenario design. But that would probably be counter-productive. Because I believe the single most important factor determining if a scenario is going to be good (innovative, balanced, high quality etc) or not is how much effort you are willing to put into it. And since we're not paid to do ASL R&D, often the effort follows how fun it is. Following a receipt is no fun.
Is the goal 50/50 or 60/40 or is there really no goal?
Our goal is 50/50 given ideal play from both sides.
Are playtesters generally so experienced that their skill level can overcome things that the average or new player may not be able to, and a scenario that they like becomes completely one sided once sold to the public?
I think you're best off having a playtesting team that have a little of everything. But of course, the experienced, and more importantly, skilled players are those contributing the most. And yes, I think it happens a lot the ROAR record is misleading (both ways of course - the scenario might be both more or less balanced than it seems). Generally what contributes to this kind of discrepancy is that the scenario is very different - those scenarios only the skilled and creative players can handle the first playing or that it utilize rely on some generally misunderstood or unknown rule that make a large impact on balance.
I've read how some scenario's need only minor changes from playtesting while others have needed major renovations. How do those do once released?
For us, playtesting and designing is an iterative process. I can't see a pattern between number of iterations and how the scenarios do once they are out in the wild. What you should avoid when using this process it to have too many iterations - you will simply grow bored.