Dwarves? As in the Fantasy Race?
Because "Midget" is more of an informal/prejudiced term for a person with any variety of Dwarfism, ya know...
If you do indeed mean the Fantasy Race type of Dwarf, the a Dwarf, but they'd beat most humans anyway if they could catch us. As they are generally thought as having or depicted to be stout, sturdy and muscular, having powerful but short limbs and very strong wills and pretty tough skin, so yeah, Humans period would lose in hand to hand combat with Dwarves.