OK fixed my own issue but I thought I'd post my findings. Since reading through problems in this thread I think it may be related.
So what I was doing was adding a PS5 and naming it "PS5 Local" or "wifi" or whatever which is fine the first time you add and pair it however you cannot access it the second use. Also when this adding the first host it was finding a "PS5-119" and adding it to my list of hosts but you could not then access this and pair etc.
After I few resets and deleting everything, starting over I worked out it will only work with one host in the .conf file and (as the instructions say) you must name the host as it appears in the PS5 system settings, in my case "PS5-119"
I haven't tested my next theory but I presume this may be why some people can play locally then have issues connecting remotely (and seem to be sure all other settings are correct) since they are adding an extra host and running against the same problem, this may worth a shot anyway.