The long term francises work that way. Quake, Doom, Unreal. They all allow people to make any kind of scenrio and add anything including 3D models. Heck they even allow you to plug in your own AI. Starting in 1996.Actually, I don't think that that is how most games in the world operate. AFAIK, most games provide a base game and then sell expansion packs or whatever you call them with new content. BFC could never charge enough for a game to make money if they include every unit from the East Front for the whole war, as in CMBB.
However, I'm not sure what you mean when you refer to an upgraded engine and scenario compatibilty--do you mean they should have updated CMBB with the CMx1 engine? That is not a course that they were interested in pursuing.
You can make a best effort reading CMx1 scenario data into a CMx2 game. It would come out half decent if you let people tinker with the translation programs. Sigh.