I always give up my seat to someone who needs it more, however you have to be sensitive about it. For example, lots of older men who are clearly having trouble standing up feel humiliated by being offered a seat by a woman (I've had the experience of men refusing the seat even though they obviously needed it).
On such occasions the decent thing to do IMO is to get up without offering them your seat as though you enjoy standing or need to move down to bus to look at some fascinating thing outside, allowing them to take the seat without obviously being offered it. Obviously if there's a bus / train full of desperate commuters this may not work as some other person will grab the seat first.
Then there's the issue of the many people who have no obvious disability but who need a seat more then anyone else.
I'm really feeling very well now, but when I was really ill with fibromyalgia and hypermobillity syndrome, or even with asthma, it would have helped a lot to have a badge of some sort as to everyone else I looked young and healthy. There were times when I was on a bus and in a lot of pain and if the bus stopped suddenly while I was standing I dislocated the joints in my knees, hips and shoulders which was really painful.
I ended seeing my GP loads of times about injuries I'd received when travelling on busses and my GP told me that they see dozens of elderly people very week with bus injuries caused when they get thrown around and injure themselves when the busses stop or even when people push them. Elderly people can break bones and get serisouly injured from bus injuries so I'm always willing to offer my seat to an elderly or an ill person.
