Try a different headphone/IEM. Earbuds aren't that great, especially the Apple ones.
Apple ear buds may suck, but ear buds in general not so much, i have some with sort of a silicone gasket that blocks out outside noise and puts out about 19 ohms...only costs about $20
I find it to be really low using Apple earbuds, even with max volume. Anyone else agree?