If you're attacking at 2:1 in CC then, yeah, he'll think you're crazy. Your opponent gets to roll with HtH numbers, too.
The rules allow HtH in certain situations, however, and optional use - use in situations that normally don't allow it - is frowned upon [by some]to say the least. The CCPh is abstracted and elimination in CC doesn't mean that 10 men are dead - it means they're hors de combat. Decreasing a sides OB/pieces on the board with optional use of rules affects balance. Free HtH combat would tend to affect tactics: you'd tend to see an increase in deploying to hs's and advancing into CC at low odds since a '5' or '6' would eliminate a lot of enemy pieces. If you and your opponent want to experiment, go for it; It's experimentation, though, don't forget.