I think you need to change the way you remember the layout itself. For example not something like X is above B and Y is left of A like in normal N controllers.
But something like A is the big middle button where B of the other colour is on its bottom left, and then just remember Y is above and X is right for those two similar buttons.
But something like A is the big middle button where B of the other colour is on its bottom left, and then just remember Y is above and X is right for those two similar buttons.