I went to Heirinji Zen Temple during the last week of November and it turned out to be an excellent choice. The autumn colors were at their peak, and as the temple grounds are very large, there was a lot to explore. It was a weekday, so there were no crowds, just a few other middle-aged people like me with their cameras, but have a look yourself!
Getting there
To get there take the Tobu-Tojo line express train from Ikebukuro, Tokyo, to Asakadai station (15-20 minutes). There are regular bus services from there, which take another 15 minutes or so. Alternatively you can walk directly to the temple from JR Niiza station (30 minutes).