On August 18, 1955, two days before the first game of the season against Newcastle United, Mercer was appointed to replace Sheffield United manager Reg Freeman who had died during the close season. As a manager he began inauspiciously. He lacked experienced and his first season ended in relegation.
The rest of his time as manager was spent in the Second Division and in December 1958, wanting to move to a bigger club, he resigned and moved to Aston Villa who were bottom of the First Division. Although he led them to the FA Cup semi-finals he was relegated to Division Two for a second time.
He moulded a talented young side at Villa and his team became known as the 'Mercer Minors'. He led Villa to victory in the inaugural League Cup in 1961 but was then sacked in 1964 on grounds of ill health.