So, it's just another bad port because it was outsourced to a company (or rather team) who has no idea what they're doing.
A lot of the games Virtuos has worked on perform well (Spyro Reignited, Dark Souls Remastered for Switch, Fable Anniversary) but there are a couple I've played that aren't very good, including this and Battlefield V. The fact that we can improve the graphics without causing worse performance tells me the issue is somewhere else.
Issue lays in code itself, not in graphics settings.
As stated in Saber Interactive presentation about Witcher 3 Port ARM processors can handle some tasks within similar time, but they require different approach. With RED Engine fact is that it's solely C/C++ based engine, when UE4 has blueprints and C/C++ modes.
My guess is that original game was made with blueprints mainly, while Virtuos is more coding staff. Because of relying on blueprints documentation is lacking in many ways, which makes Virtuos job harder as they need to understand on their own what is happening and why is happening on low level (so it needs big amount of time and money).
The biggest issue here is streaming assets and AI.
Streaming assets is taking too much time, probably asset management is badly optimized and it takes forever to decide what is needed, and what not (or game has too much assets, they would need to change approach on how assets are stored). AI code is too complicated, so by fighting with Idk 3 or more enemies CPU starts to struggle.
For me it's probably the specific of development with UE4 which makes it harder. And add to this fact that this was made originally by masters of bad optimization - Obsidian.
This port just shouldn't happen. I think Saber Interactive or Panic Button would struggle to fix CPU issues in the same time Virtuos had. Maybe only graphics would be better.
And about BF V - Frostbite was always criticized for lack of documentation - that only DICE understands their own engine and other studios frequently needs help from their team about understanding issues. So this may be connected somehow.