Olga, check out Shuzen-ji Onsen on the Izu peninsula. It's a very small town but it has some lovely temples, a small bamboo forest, and onsen, of course! You can get there by train and bus or taxi. It's often overlooked COMPLETELY by foreign visitors and is definitely a more "off the beaten track" destination. But if you want to escape the crowds, this is a good place.
Chichibu, in Saitama Prefecture, is nice. It is located within a national park, has a number of old temples and shrines, and there are onsen too. The ropeway up the mountain offers some great views as well. Chichibu is less than 2 hours from Tokyo. The JNTO (Japan National Tourism Organization) and Seibu Railway web pages (both in English) are helpful.
On the outskirts of Tokyo, in the locality of Hachioji, there's a Yakuo-In Yuki-ji, a buddhist temple on Mt Takao. It dates back as early as the 8th century, and is a close enough ride from Shinjuku, around 50minutes by train. The view there is also pretty good, plus during summer there's a beer garden buffet.
I really like the Edo open-air museum, you can get there by train from Seibu-shinjuku station, then a short walk. They have some well preserved relocated/ reconstructed buildings from early Meiji and Edo periods. It's nice as it's outdoor, and you can walk in and out of the buildings. It only cost 400 yen for access to the whole site. Really good value when museums usually change 1000-2000 yen.