There's a bit of a problem here in not knowing what you do or enough about your "business" to know what work is involved and/or what you could reasonably expect to pay your accountant. For example, I have 2 clients in the exact same business. One is charged £350, the other £850pa - neither is VAT registered, neither is a Ltd company and neither has more than 5 "customers" ... the massive difference in billing is due to the amount of work involved in getting the information to be able to do the returns, the state the information arrives in, and "risk" of a revenue investigation. (It's an all-in fee, he won't be charged more if revenue investigate and I have to spend hours on it).
Personally I save clients far more than I charge - as long as they take the advice given. It's not as simple as turnover vs no. of customers vs statutory returns = bill, it's about how much work is involved to obtain all the information needed, sort everything provided, and a judgment call on anything that doesn't have documentary back-up.
It's possible you're not helping yourself in the way you "present" information to your accountant. It's also possible (s)he's priced it highly because it's not really the sort of work they want to be doing.
The problem isn't whether you should be self-employed or a Ltd Company, but that your accountancy fees "feel" too high - ask him/her what YOU can do to reduce the fees/time required on your accounts. If (s)he's straight, they'll be only too happy to tell you (believe me, I've been telling the £850 one for years!) and if they can't tell you how you yourself can reduce these fees simply by changing the way you record/present things to them, then certainly look around.
E2A I'm no longer in the UK so I'm not looking to take on any extra clients, just talking about the variance in behaviour of clients who do the exact same thing who can save themselves money by changing how they do things.