Indeed. The point is you KNOW whether or not you have done anything, the police DON'T. Conversely, the police KNOW what has happened locally / what they've noticed / been told which gives them the grounds to stop / speak to / search you. You DON'T.
So the fact you know you are innocence is entirely irrelevant to whether or not they are acting lawfully. If they have grounds to suspect we have given them to power to find out more.
Unfortunately the combination of some people giving it the "I've done fuck all, why you harassing me" from the outset and some cops who take an arrogant / rude approach from the outset leads to many stops degenerating into a fracas that need not happen.