What are your goals in this? Teach you some maths, a project for some course, CV fodder, something you want to use? We already have the likes of matlab, mathml, all sorts of stats languages
https://www.datacamp.com/community/tutorials/statistical-language-wars-the-infograph#gs.XeX3zBM and this could go on for a while. To that end if you want to do something useful from scratch you face an uphill battle, on the other hand any number of open source projects out there could use some help.
Ignoring that then after basic calculator most people make a scientific calculator (logarithms, trigonometry, maybe basic stats), maybe then a graphics calculator, somewhere in the mix a programming calculator (mainly just output and calculate in binary, hex, float and maybe octal just because you can) and then after all that is done people look at equation solving. So you probably start with either solving basic y=mx+c things, then quadratics with real results, maybe figure out maxima, minima, or saddle points and then you are either inventing your own programming language or just implementing an existing one. Facing that they get bored with all that and maybe do geometry (surface areas, volumes....) or dip into physics/engineering (mostly electrical equations but occasionally basic mechanics), or they go down the rabbit hole of making sure their random numbers are random.
So yeah build your calculator if you want but to do something useful you would be far better off finding an existing project and seeing how you could expand that a bit.
like a 250 dollar kit back when DDR4 was Intel only
