Math for Computer Science
This book should be read by everyone who's serious about computers. It will give you the necessary background to work on optimization software, cryptographic algorithms, analysis of algorithms and several other subjects that are far beyond just "writing programs".
It is not an easy book, however. You have to read carefully and slowly, and you will need time to work on the exercises. But it is well worth it.