How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation