How do kernels stop programs from executing high privelige/potentially malicious instructions

[StackMasher]

^^^

 

[Mrrraou]

they dont

 

[nero99]

silly OP, kernels are for popping in the microwave, not stopping code

 

[x65943]

One common method is to make user programs only run in userspace (a specific section of ram) that cannot access/execute kernel commands.

 

[FAST6191]

If you want to look specifically at the x86 family then have a look at the concept called rings. You might have met it before when people speak of a ring 0 rootkit.

Otherwise yeah would be here all day discussing security.

Edit. Another thing that might be worth looking at.
I recall linking a talk from possibly Defcon or Blackhat (one of the big US security/hacker conferences anyway) detailing return oriented programming, however part of the run up to that was going through the previous measures that ultimately led to ROP becoming a thing. DEP and how it failed, ASLR and how it failed and so on. I can't recall specifically which talk and ROP is a fairly hot topic among hackers which is unfortunate if you go searching but it might be worth looking for.

 

[Futurdreamz]

What kind of Kernel are we talking about? There are very many different types, with their own practices. Some outsource everything to subcomponents with limited access, some require all executable code to be cryptographically signed before they will be executed, and some use use an obscure and unpopular OS so nobody will want to hack it.

 

[gudenau]

The way that it is handled is multiple CPU modes of operation, normally "supervisor" and "user".

The CPU processes instructions differently when in the different modes, the lower the privilege level the less instructions can be executed. When the CPU attempts to execute an instruction that the current privilege mode does not have access to it throws an exception that an upper layer catches. This upper layer is normally the kernel, but can be other things depending on virtualization extensions and other low level details. From there whatever handles the exception will either terminate the process that executed it, crash or make the process think that it worked.

It varies from architecture to architecture, but that is basically how it works.