The other point is that when it was built, it was all countryside around there, but since then the city has engulfed it, and AFAIC we need to keep all the green spaces we've got! If you go and stand on that ridge, the views are stunning. You can see kestrels and sparrowhawks hovering overhead and catching things in the bushes, pick blackberries, or sit on a sphinx with a can of beer.
slowjoe is right as well. It's all very well saying "Rebuild the Crystal Palace", but what FOR exactly, what's it going to do? If commerically viable uses are just an afterthought it's really not going to work very well, and if it's commercially driven then it has no place on parkland. All round bad idea!
And another thing... those who bang on about the history of the area only seem to be able to see back as far as 1851. What about before that when the area was farmland, or before that when it was part of the Great North Wood?