Not really, but it depends on what you do. Cheaper phones will have cheaper components in general, so if you don't play phone games but want phone pictures/video that don't look awful, you'd need a decent phone.
With that said, there are many mid-range phones that are fairly powerful and have not-great-but-still-decent cameras.
Regarding a high-end phone doing things faster than a low end phone, it isn't as noticeable today as it was a few years ago, unless you're talking about an absolute bottom of the barrel phone.
Speaking from experience, I have an LG Stylo 2 Plus, it's about as middle of the road as it gets, crappy camera, so-so gpu and cpu, only 2GB of ram, but I couldn't justify the cost of a higher end phone for my personal use. It does have trouble with Sketchfab, but other than that in day to day use it doesn't feel slow.
One thing to keep in mind is that the absolute cheapest phones will usually have a very short life, so I'd still recommend buying at least something from a reputable brand, cheap electronics are expensive in the long run.