Letter to Scott Regarding Beta Testing

[HercMighty]

Scott,

There has been a lot of interest over at the Warefare HQ forums of volunteering to do a structured Beta of POA2 in an effort to squash remaining bugs and gameplay issues. We do not know how the game is being tested now other than AI vs AI and us reporting stuff to you. Unfortunatly we have found that our testing is haphazard with no real direction, giving you snipets and random reports depending on what the particular person is interested in. For example:

Also can people comment on whether on the targeting XO staff officer screen that Last Turn Fire Results and Next Turn Expected Fire entries are working? Even with FOW off I am not getting what I think I should.

Is something I am interested in. Most have agreed that in order for this work we need to have your support, and help to at least get us started into a structure that helps you. We have at least a couple of people who are able to construct test scenarios and I believe we will be able to get enough testers. Hopefully the increased message traffic on the forums will once again lure people back, get ones on the fence interested and get the word out that POA2 is a viable simulation\game that is in the process of being completed.

Regards
Dustin and the members of the Warefare HQ Forums.

[HercMighty]

This is the message I received from Scott:



I that setting up something on WHQ would be great. To date what Jeff and I have been doing, aside from responding to reports, is to follow a "2-track" kind of testing. On the one track we look at general games, with a full range of weapons and support available, and then let them run AI vs. AI to see if we hit any crashes (c++ errors, mostly), and also to see that the game ends up in a reasonable fashion. The other thing we do is to make small scenario specifically designed to test one aspect of the system - ATGM's, or AI rotation, or AI targeting arty, or things like that.

Uncoordinated testing works well for general debugging, since it allows us to find things that are, for lack of a better term, "player specific". There have been dozens of things like that so far, where Jeff and I never hit the problem because we tend to fall into our own patterns, and just never set up a scenario or used the program as need to bring the issue to light. It also works well to test the system over a wide range of situations, since everyone creates and plays differently.

But I agree that a focused approach works better for specific issues. There have been so many changes to POA-2 in the last year that we are working through testing it top to bottom. In particular, we'd like to look again at night/illumination and weather effects, AA/SAM engagements, mines and engineering, bridge demolition, and the chem stuff (going in/out of MOPP, losses, etc.). So if the guys on the forum want to start looking into these that would be a great help.

I'd also like to continue to receive reports along the lines of your example, though, where players don't think something is correct. All I ask is that when folks report things they give us a saved game file and enough detail to figure out what they think should have happened and what they are actually seeing. It may be a problem, it may be correct, or it may be FOW. But if nobody tells me about it, I can't verify it or find another less-confusing way to present the info.

Oh, and in case Jeff or I didn't reply about the US arty firing chem - for most gun systems there are two versions defined. One has purely conventional ammo, and one is NBC-capable. We did this for two reasons. The first is that in most situations, arty units are going to carry little or no chemical rounds. So it makes sense that when players set up a scenario, this is what they will get. The second is that the database has a limit of 8 ammo types per gun/launcher, and for ease of use and flexibility we didn't want to take away a common conventional munition by default and replace it with much more rare chemical one. So we created a new type for the NBC firing weapons that players could select for those special situations where chem might be used, which has a mixture of both ammo types. Theses weapons systems are designated with "(NBC)" after their nomenclature, e.g., "M109A4 How (NBC)". The same thing applies for the other countries, too. I'd appreciate it if you could post that one the site. I think a number of people have wondered about it. BTW - it's simple to change a weapons system when setting up a new scenario; just bring up the Unit Info window (right click on the unit or force tree), and then left click the weapon field.

[HercMighty]

But I agree that a focused approach works better for specific issues. There have been so many changes to POA-2 in the last year that we are working through testing it top to bottom. In particular, we'd like to look again at night/illumination and weather effects, AA/SAM engagements, mines and engineering, bridge demolition, and the chem stuff (going in/out of MOPP, losses, etc.). So if the guys on the forum want to start looking into these that would be a great help.

So what do people think of this paragraph? Do we need more clarification or can we work with this?

[WNourse]

One way to handle it would be for each person to take one or more designated areas to test (e.g. engineering ops, chemical ops, artillery) and establish a set of test cases to evaluate against. We can set up forum threads for each major set of functionality to be tested and post ideas and results there. I know Scott doesn't generally follow the forums, but this might be profitable for him to follow, particularly if he spots assumptions that are being made that are incorrect, which would help keep testing focused and moving forward in the right direction.

When running tests, it helps to keep the number of variables to a minimum and to increase complexity over time (starting simple). The test cases would be run each time a build is released, to determine if anything that was working has been broken, what changes to functionality have been introduced, etc.

For testing purposes, I'd suggest that a series of very simple maps be created, along with a set of stock forces which would be used. Obviously, those forces would need to changed depending upon what functionality is being tested (can't test airdrop capability with an armored company). Would any of the resident map-makers be willing to take on the task? I would see us needing three or four maps, all fairly small, with a very limited variety of terrain and elevation (one should include ocean terrain in order to handle seaborne and amphibious units).

My 2 cents,

Will

[HercMighty]

I do have the map making software to make maps. I haven't really had time lately to put it through its paces but simple maps ought not to be to hard. People would just need to post a general discription of what they need.

[MikeSinn]

So what do people think of this paragraph? Do we need more clarification or can we work with this?

This seems like enough direction to at least get things rolling...

I like Wnourse's suggestions on how to segment and cooridinate testing the various components. I can take a stab at testing the NBC options within the game (as that fits into the work I am already doing with artillery). I'll try to get some initial test plans posted to a thread later today or tomorrow.

Thanks,
Mike

[HercMighty]

I think that one thing that may help is screenshots. By using CTRL-PRNT SCRN and then pasting into MS Paint or similar program and then crop the image to what is needed I find helps to understand.

Also having the program export out the combat reports file and posting this information would probably help. I have been using detail level 5 most times and detail level 6 when I am testing something. Even posting this information when you do not understand something would be great, it can read like a AAR.

[thewood]

I'll take a shot at Night/Illumintion. I'll set up a couple of test scenarios that include tanks and infantry to see how the sighting routines work in non-daylight operations. It may take a week to complete due to work.

[HercMighty]

Scott Requests:

1. Night/Illumination & Smoke\Dust (thewood)
2. Weather Effects (HercMighty)
3. AA/SAM engagements
4. Mines
5. Engineering
6. Bridge Demolition
7. Chem Stuff (going in/out of MOPP, losses, etc.) (MikeSinn)
8. NBC options (MikeSinn)

User Requests:
1. ATGM's (MikeSinn)

I will update as people sign up and as they are completed.

[MikeSinn]

1. night/illumination (thewood)
2. weather effects (HercMighty)
3. AA/SAM engagements
4. mines
5. engineering
6. bridge demolition
7. chem stuff (going in/out of MOPP, losses, etc.)
8. NBC options (MikeSinn)

I will update as people sign up and as they are completed.

Herc,

I plan on including the "Chem stuff" in my NBC testing, so you might just combine those two entries.

While not on Scott's original list, I believe ATGMs should be added to this plan as well. My testing this weekend leads me to believe that these aren't functioning correctly yet. A structured test would probably go a long way to either clarifying the issues.

Thanks,
Mike

[HercMighty]

I think that once we get an option tested like ATGMs we will then want to do a DAY/Night test and then a Weather test also. It also looks like to me that 4,5,6 could also go together. So I think all of these will at one point be tested all together.


For now though we need to keep the variables down so this will work. Some of the things Mike is doing does just makes sense to group them together. For weather I am planning on doing a few different scenarios. Weather affects terrain and affects different units in different ways. In a way it is to bad that Scott has a built in randomness factor as this can make testing difficult. I will have to email him to see if there is a way to turn it off.

Regards

[thewood]

My initial tests on illumination and night will be in clear dry weather to minimize variables. After we feel comfortable with that, I can add in a few weather effects. I will include smoke and dust in the initial tests. I plan on running tests in hotseat and all different FOW options. I'll then run tests comparing how the AI reacts in daylight vs. night vs. illumination. Does that seem like a valid plan?

[HercMighty]

My initial tests on illumination and night will be in clear dry weather to minimize variables. After we feel comfortable with that, I can add in a few weather effects. I will include smoke and dust in the initial tests. I plan on running tests in hotseat and all different FOW options. I'll then run tests comparing how the AI reacts in daylight vs. night vs. illumination. Does that seem like a valid plan?

Sounds great. By the time you get to weather I will have some data on my tests to compare to.

[MikeSinn]

Below is a rough outline on how I plan to test the games implementation of NBC weapons.

Testing environment:
1. Open, clear terrain.
2. Daylight hours
3. Initialy test with on FOW to record and validate results. Increase FOW as results are recorded.

Chemical Warfare Testing:

Effectiveness of GB weapons
Test drift frequency, patterns and duration of GB cloud
Test lethality of GB against various weapon systems
Infantry in the open
Non NBC sealed vehicles (i.e. trucks, jeeps...)
NBC sealed vehicles (tanks APCS...)

Evaluate AI reaction to GB gas. MOPP level changes
Evaluate impact on movement routines
Nuclear Warfare Testing:


Effectiveness of Nuclear weapons
Test lethality of nuclear against various weapon systems.
Infantry in the open
Non NBC sealed vehicles (i.e. trucks, jeeps...)
NBC sealed vehicles (tanks APCS...)

Evaluate AI reaction to nuclear weapons. MOPP level changes
Evaluate impact on movement routines
I plan on posting my findings and general thoughts on each of these test as well as any relevent screenshots, game files and combat logs.


Hopefully, by the time I finish this first iteration of testing, we will have a better understanding of weather effects. I will then incorporate that knowledge into my next rounding of testing.

Thanks,
Mike

[MikeSinn]

Scott Requests:

1. Night/Illumination & Smoke\Dust (thewood)
2. Weather Effects (HercMighty)
3. AA/SAM engagements
4. Mines
5. Engineering
6. Bridge Demolition
7. Chem Stuff (going in/out of MOPP, losses, etc.) (MikeSinn)
8. NBC options (MikeSinn)

User Requests:
1. ATGM's (MikeSinn)

I will update as people sign up and as they are completed.


Herc,

It may make sense to start a new thread with this information and ask Don to "sticky" it. Each of the testers can then post our results there.

Thanks,
Mike

[HercMighty]

Herc,

It may make sense to start a new thread with this information and ask Don to "sticky" it. Each of the testers can then post our results there.

Thanks,
Mike

I created the thread and am trying to get someone to sticky it.