Now I know you've never had one of these jobs.
The thing about being a cleaner, a sweeper etc, is that you have a shit job, you have no money, and yes, there is still 'office politics', but it often involves people who are stupid or insane, or both. Sometimes, your boss is one or both of those things. At least in an office, an insane boss has to rein it in a little, but if you're out working as a sweeper and your boss, the foreman, comes by every hour in a truck, he can do and say whatever he wants, because there's no one else around.
Often, the foreman or boss at these low level jobs, is someone who knows just a bit more than the sweeper, and therefore thinks that he was destined for greatness, but has been undermined by the world. As a result, he's bitter, and he takes it out on his subordinates.
Your attitude sounds a lot like the 'joy of labour' espoused by the monied classes, as applied to those beneath them.