Zoom H2 is inexpensive but has decent sound. For the price it's great value.
This video posted on the forums recently will give you an idea what it can do.
The zoom does a good job of recording audio but you'll also need to listen critically to your recordings. That means an audio interface, studio monitors (or a decent hi-fi) and headphones. Onboard audio isn't up to the task.
A Mackie Blackjack would be good value. Driver problems on windows but apparently they're rock solid on macs.
An Apogee Duet would be more expensive but has excellent audio quality. It's more or less the standard by which the others are judged. Get one if you can.
Good acoustic guitar mics which won't break the bank are the ADK A6 and Oktava Mk012. Get two if you can to record in stereo.
These are all items which I think stand out as being particularly good value for the audio quality they provide. A lot of gear pitched at home studio use can be disappointing.