Beta testing of CMN: What is the process, and what have they learned?

[hirr]Leto

Varmint Croonie
Joined
Jan 29, 2008
Messages
1,124
Reaction score
13
Location
Saskatoon
Country
llCanada
We know that there were huge gaps in testing that lead to the crummy release of CMSF. I have several questions to pose:

1) What is the process involved with Beta testing? Do individual beta testers have different jobs, strengths (etc, bug spotting as opposed to scenario testing).
2) What are the management processes involved with beta testing, especially when it comes to managing volunteers?
3) Because of the selection process of beta testers posted by BFC, is there a chance that the ranks are once again posted with yes men, and that major issues with the game may be overlooked due to the nature of the community (ie. testers too close to the product, and possibly the CMSF engine to successfully bridge game theatres)?
4) What is the general process to beta testing a game: are there general steps that are taken (basic logic model stuff) and what are the potential roadblocks, challenges, areas where the most issues come up?
5) Obviously, BFC is not an office shop, but is a run from home business (Charles working from his home etc), so what is the main interface for command and control of a business like this: web based, skype based, phone based, message board based? In other words, is there a main beta testing mb that can organize testing (and concommitantly, be used to slag off other website posters, ahem).

Cheers!

Leto
 

Geordie

CM Moderator
Joined
Jan 27, 2005
Messages
2,111
Reaction score
13
Location
Scotland
Country
llUnited Kingdom
Havent a clue. I would hope there was a structure to it. Guys are allocated specific Beta tasks and stick to them.

Some Beta Tester would have to answer this one though.

It clearly didnt work for the initial release and the Marines release but I saw no problems with the particularly good Brit release.
 

Michael Dorosh

der Spieß des Forums
Joined
Feb 6, 2004
Messages
15,733
Reaction score
2,765
Location
Calgary, AB
First name
Michael
Country
llCanada
[hirr]Leto;1290829 said:
We know that there were huge gaps in testing that lead to the crummy release of CMSF. I have several questions to pose:

1) What is the process involved with Beta testing?
You won't get straight answers to any of your questions due to NDA. I can give you some general answers.

Do individual beta testers have different jobs, strengths (etc, bug spotting as opposed to scenario testing).
In an ideal world, you would have a beta team leader who would organize things either one way or the either - testing both bugs and scenarios simultaneously, or separately. I don't know that there is an "ideal" method. Certainly there is overlap regardless, and you could have teams testing campaign routines exclusively, for example, and if they happened to find a game-crashing bug, you would have a reporting process for it.

2) What are the management processes involved with beta testing, especially when it comes to managing volunteers?
What kind of information would you expect to get from this question?

3) Because of the selection process of beta testers posted by BFC, is there a chance that the ranks are once again posted with yes men, and that major issues with the game may be overlooked due to the nature of the community (ie. testers too close to the product, and possibly the CMSF engine to successfully bridge game theatres)?
It sounds like you're answering your own question.
4) What is the general process to beta testing a game: are there general steps that are taken (basic logic model stuff) and what are the potential roadblocks, challenges, areas where the most issues come up?
You should probably volunteer for a beta test somewhere as the best way to find out some of this stuff.
5) Obviously, BFC is not an office shop, but is a run from home business (Charles working from his home etc), so what is the main interface for command and control of a business like this: web based, skype based, phone based, message board based? In other words, is there a main beta testing mb that can organize testing (and concommitantly, be used to slag off other website posters, ahem).
How do you know Charles doesn't commute to a rented, borrowed or shared office space to do his work? You are assuming a lot of things about someone you've never even seen a picture of.
 

Redwolf

Member # 3665
Joined
Sep 2, 2002
Messages
5,113
Reaction score
43
Location
MA, USA
Country
llUnited States
You have very little control over the beta testers. You can't really assign them tasks, unless you build it up so that they feel special from having that task assigned to them.

In general what is needed is look for very specific bugs, in particular with continuation bugs, that means bugs that sneaked in that prevent the player from progressing in the game through a certainly stage. In wargames that isn't that much of an issue but e.g. for campaigns or linked operations it is. This stuff can break and re-break at any time.

Beta testers are supposed to shut up about design issues. If foliage provides insufficient cover due to game mechanics that is nothing you want Betas to comment on. At that phase there will be no changes there.

Beta selection will in a good company try to get idiots in. You want people who never read the manual and are mouse-clumsy in the extreme. You need some of them because there are bugs, namely UI bugs, that the more controlled powergamers will never see.
 

Michael Dorosh

der Spieß des Forums
Joined
Feb 6, 2004
Messages
15,733
Reaction score
2,765
Location
Calgary, AB
First name
Michael
Country
llCanada
Beta testers are supposed to shut up...
Not specific to BFC but to testing in general, I note from my experience on some of the ASL playtesting groups that riding herd over testers via email is a difficult thing to do, and getting testers to "shut up" about design issues is only half as annoying as stopping testers from shutting up altogether. It's when they suddenly go quiet that you start to wonder - like the six year olds upstairs who were raising merry hell just ten minutes ago - why things are all of a sudden TOO quiet. The answer is usually that your volunteers have either stopped playing, or have just stopped looking for bugs. And so you have to prod them again to keep working, or find something else for them to look for.

The tendency to want to talk about design issues can be strong - again, I speak from an ASL perspective, I don't recall that this was particularly "problematic" or not in CM:SF - I was actually quite happy to follow the party line while I was testing. But everyone fancies themself a game designer; it's up to the actual developer to see how much they want to indulge the testers on questions of design and development. As RW points out, the testing phase is not really the avenue or forum for those kinds of discussions - you've done your prep work and its time to ensure that what you've created is working. Discuss what doesn't work and find solutions, but if you're still doing drawing board discussions, you're probably in trouble.
 

[hirr]Leto

Varmint Croonie
Joined
Jan 29, 2008
Messages
1,124
Reaction score
13
Location
Saskatoon
Country
llCanada
You won't get straight answers to any of your questions due to NDA. I can give you some general answers.



In an ideal world, you would have a beta team leader who would organize things either one way or the either - testing both bugs and scenarios simultaneously, or separately. I don't know that there is an "ideal" method. Certainly there is overlap regardless, and you could have teams testing campaign routines exclusively, for example, and if they happened to find a game-crashing bug, you would have a reporting process for it.


What kind of information would you expect to get from this question?

The question was more academically inclined, and I guess I did not expect to really get an answer here


It sounds like you're answering your own question.

Not really, but it was set up in a way to make it seem leading. My main concern is that the testing of the game is important to it not being released in the shape that CMSF was... but from further exploration of yours and Redwolf's comments, it may be more design issues than successful or unsuccessful testing of those designs to uncover any errors. Perhaps beta testing is NOT that important in relation.

You should probably volunteer for a beta test somewhere as the best way to find out some of this stuff.

I would like to and may just do that in 2011 when I will have more free time.

How do you know Charles doesn't commute to a rented, borrowed or shared office space to do his work? You are assuming a lot of things about someone you've never even seen a picture of.
Not assuming anything at all. There is a BIG difference between inductive reasoning and assumption. I based my statement on evidence produced on the BFC boards that suggested that they mainly work from their homes. I will find the link if you really need to know what I am basing this on, but that might take some time... other than that, you'll have to trust me that I did see this evidence, valid or not.

I guess many people are saying that CMN has been beta tested by the consumers for 3 years running now and that it will be in much better initial release state because of it. My fear is that the transition back toward WW2 may not be a smooth one, and that major design and testing issues will be overlooked, or simply some of the soul of the game will be hollow due to the close cohesive nature of the people testing it.



Cheers!

Leto
 

Michael Dorosh

der Spieß des Forums
Joined
Feb 6, 2004
Messages
15,733
Reaction score
2,765
Location
Calgary, AB
First name
Michael
Country
llCanada
[hirr]Leto;1290853 said:
I will find the link if you really need to know what I am basing this on, but that might take some time... other than that, you'll have to trust me that I did see this evidence, valid or not.[/COLOR]

I'd be interested if you do remember where it is; it has been discussed from time to time and might even go towards the "do they have day jobs or not" debate.

I guess many people are saying that CMN has been beta tested by the consumers for 3 years running now and that it will be in much better initial release state because of it. My fear is that the transition back toward WW2 may not be a smooth one, and that major design and testing issues will be overlooked, or simply some of the soul of the game will be hollow due to the close cohesive nature of the people testing it.
It all comes down to organization and having someone with a firm grasp of what needs to be tested, who is testing what, what kind of reporting procedures you have in place, how you are tracking data, etc. If you have a model in which all you have are a bunch of guys playing games over and over and posting "stuff" that happens when they encounter it randomly, it's probably less useful than defined reportage using specially selected or developed software that is purposely tailored to a testing environment. You wouldn't have that with a public beta, either formal or informal.

I think the scope of the CM project would be challenging for even a large "corporate" body with access to larger test groups in-house and even on payroll. As a garage project, perhaps CM:BO and add-ons tested the limits comfortably of what they could accomplish. Maybe we'll see if they prove that statement wrong.
 

[hirr]Leto

Varmint Croonie
Joined
Jan 29, 2008
Messages
1,124
Reaction score
13
Location
Saskatoon
Country
llCanada
And that is kind of my interest... it boils down to "how do they effectively manage people, objectives and outcomes, monitor progress, evaluate and report on it, and integrate controls. I teach this kind of stuff to the 3rd year biz students, and you'd be amazed at how greek this actually may be. From my experience, engineers are very good at this kind of thing, software guys included, with the rest of the social science, marketing and finance guys pretty clueless about it. Even a business of BFC's size would have some kind of controls with well known and followed policies for these kind of things. Otherwise, it would be, like you say, a hodgepodge of data that would be very hard to actualize into anything concrete, outside of the "found a bug --> go fix it" process.

I know that I read it the link about Steve calling Charles at home to discuss some new features such as "buildings at angles" in some of the old links that you recently posted.

Cheers!

Leto
 

Michael Dorosh

der Spieß des Forums
Joined
Feb 6, 2004
Messages
15,733
Reaction score
2,765
Location
Calgary, AB
First name
Michael
Country
llCanada
[hirr]Leto;1290875 said:
I know that I read it the link about Steve calling Charles at home to discuss some new features such as "buildings at angles" in some of the old links that you recently posted.
Steve posts comments about calling Charles "at home" all the time. I only infer from that that Charles has a home and occassionally is found there. You infer from those comments that he works from his home, but when I hear someone calling their boss "at home", I generally get the sense that the words "at home" are being used to make the distinction clear that he wasn't "at work". I've talked to supervisors of mine after hours in their residences on many occasions. It didn't mean they were employed where they lived.

Charles may very well work out of his home, I have no idea. I just haven't seen any of Steve's comments as being evidence of it.
 

Michael Dorosh

der Spieß des Forums
Joined
Feb 6, 2004
Messages
15,733
Reaction score
2,765
Location
Calgary, AB
First name
Michael
Country
llCanada
The key, IMO, is not the beta testers, it's their manager and their testing plan.

-dale
Yes, exactly. But I don't think you would get anyone to comment on it due to the NDA the testers sign. BFC might entertain direct questions on that point on their own forum, assuming they have a testing plan. I don't know how interesting nuts-and-bolts type stuff like that is to people; I personally find behind the scenes things like that quite entertaining.
 

dalem

Member
Joined
Jan 4, 2009
Messages
2,298
Reaction score
62
Location
Columbia Heights, MN
Country
llUnited States
Yes, exactly. But I don't think you would get anyone to comment on it due to the NDA the testers sign. BFC might entertain direct questions on that point on their own forum, assuming they have a testing plan. I don't know how interesting nuts-and-bolts type stuff like that is to people; I personally find behind the scenes things like that quite entertaining.
I think it's possible to draw the conclusion that, whatever their management and plan was for CMSF, it was insufficient and overwhelmed. I don't need too go behind the scenes to figure that out. :)

-dale
 

KG_Jag

KG Vice Kommandir
Joined
Aug 5, 2002
Messages
1,782
Reaction score
180
Location
New Braunfels, TX/Reno, NV
Country
llUnited States
I get the impression that in 2007 BF was doing far more crisis management than product quality management.

The deadline pressure from the contract that they had entered into with Paradox was the major factor, coupled with the state of the product at the contractually determined (including an extension) release date--and all the fixes and missing features that remained to be addressed post release.

This was not only their initial product with their new engine, which had been in development since before CMBB was released, but also the game engine upon which their future--indeed continuing existence--depended/still depends.
 

thewood

Member
Joined
Feb 19, 2004
Messages
2,594
Reaction score
12
Location
Boston
Country
llUnited States
And that is exactly why I don't understand their tie in with Paradox. For all of Steve's ranting about big distribution companies, and he goes and bets the company on one. And worse, its Paradox. A company that has self admittedly shoveled titles out the door in a poor state, just to let the developer fix it. I still can't fathom Steve's thinking on that.

No army of beta testers can solve that problem.
 

dalem

Member
Joined
Jan 4, 2009
Messages
2,298
Reaction score
62
Location
Columbia Heights, MN
Country
llUnited States
Total worst-case USELESS speculation:

- development of CM2 engine starts along with CMBB
- main idea is Steve's "Build a slot for everything under the Sun and turn off what you don't need per game"
- at some point 2004-ish, initial development is trashed, engine junked. New RT platoon shooter idea born
- this compresses timeline, forces consideration of Paradox (some have said they got money from P)
- history then unfolds as we now saw it.

-dale
 

Jeph

Member
Joined
Nov 22, 2009
Messages
188
Reaction score
4
Location
Oz
Country
llAustralia
[hirr]Leto;1290829 said:
We know that there were huge gaps in testing that lead to the crummy release of CMSF.
We don't know that at all. Such has been assumed or accused but the ones who know aren't telling.

I'm sure the guys at BFC aren't so retarded that they missed all the numerous and massive bugs, but the game had to go out, finished or not. Which brings us to:

And that is exactly why I don't understand their tie in with Paradox. For all of Steve's ranting about big distribution companies, and he goes and bets the company on one. And worse, its Paradox. A company that has self admittedly shoveled titles out the door in a poor state, just to let the developer fix it. I still can't fathom Steve's thinking on that.
You make a big assumption that it was Steve's idea.
 

KG_Jag

KG Vice Kommandir
Joined
Aug 5, 2002
Messages
1,782
Reaction score
180
Location
New Braunfels, TX/Reno, NV
Country
llUnited States
You make a big assumption that it was Steve's idea.
It does not matter whose idea it was, only that BF (as a partnership, sub-S corp. or in whatever form) made it.

That said, I would be very surprised if Steve did not at least concur in the decision and its details.
 

thewood

Member
Joined
Feb 19, 2004
Messages
2,594
Reaction score
12
Location
Boston
Country
llUnited States
Exactly...its not a big assumption. There were less than four people who could have come up with the idea, but its not a stretch at all to think Steve had to OK it.
 

[hirr]Leto

Varmint Croonie
Joined
Jan 29, 2008
Messages
1,124
Reaction score
13
Location
Saskatoon
Country
llCanada
We don't know that at all. Such has been assumed or accused but the ones who know aren't telling.
Not assumptive at all, but very much deductive: evidence of the many bugs could be realized by anyone who bought the paradox release version of the game and followed some of the early bug reporting. Simply put, testers either did not have the time, direction or ability to detect some of the bugs... I do agree that there were multiple factors that exacerbated their inability to iron out some of the really bad ones, and also agree that even the best and most thorough testing can miss potential problems in the software.

Cheers!

Leto
 
Top