The "right" resolution for docking is 1080p and for handheld 720p. Because the resolutions are different, custom aspect ratios and overlays designed for one resolution won't work on the other. Normally you would have to make separate versions of each and switch between them. There's other benefits to rendering for the native output resolution so this is absolutely correct for default behavior, an option to disable might be nice but none are built in AFAIK.
Using ReverseNX you *might* be able to force retroarch to always act as if it is handheld, which will render 720p while docked. You would do this by turning reverseNX on for an installed title, then override that title to launch hbmenu and then retroarch. Or at least I think, haven't tried myself, but that is what I would look into.