I've been totally enjoying Greg Milner's Perfecting Sound Forever: An Aural History of Recorded Music which offers a history of audio recording from Edison up to the software tricks of modern pop singers. Excellent material of interest to any self-respecting audiophile, including visits to Mikey Fremer's house.
While on the subject of what any audiophile should read, I am a great admirer of Daniel Levitin's This Is Your Brain on Music: The Science of a Human Obsession which I reviewed for Affordable Audio a couple of years back. It's one thing to study gear but to really understand how it all works, you need to understand that machine between your ears. Levitin explains it better than anyone I've read.