My answer to this question, is that there is very little we can trust. Books, accounts, are certainly easy to fake. Antiquarians, historians are not trustworthy. And apart from the fact that accounts may be false (and lots left out) there is the fact that environments change. The new generation simply doesn't have the context that older generations assume - eg why would a child born today know about phone boxes, taxis, going to the shops, asking a girl out on a date, etc once common place ideas fall out of consideration. The same will be true for us - eg no one considers the different infrastructure that must have been in place to run a world on horses, as it has all gone.
When you say the wall, I think you are closer. Physical structures are far harder to fake or change. And they can have their own story, that one might be able to get a clue about, via personal investigation. Additionally, you can look at the old plans and configurations of villages, towns, etc - you can confirm that the clothes shop used to be a butcher or bakers or whatever.
