To map keyboard keys to buttons in DOSBox, start your selected program then perform the key combo to get back to the Retroarch menu. In the Quick Menu, go to 'Controls' and set the User 1 Device Type to 'Mapper', then in the 'Options' tab, half-way down the page there should be a set of "RetroPad" buttons you can map specific keys to. After that, you can save your config and resume emulation, and the buttons should now work.
Hope this helps!