The Koishikawa Botanical Garden is located right in the middle of central Tokyo. It is run by the University of Tokyo and started out as a medicinal herb garden in 1684 during the Tokugawa Shogunate. Thus, it is the oldest in Japan.
I have been living in Japan for about 12 years now and had never heard of this place! It is not on the beaten path at all. I basically just discovered it by looking at an old Tokyo street atlas then on Google Maps looking for someplace interesting to take photos.
The garden is basically divided into a lower and upper half. The lower half is dominated by ornamental ponds and bushes with open spaces to take in the beauty. When I was there, there were many people drawing and painting the scenery. The upper half is on a hill and is mostly forested with many types of trees and lots of shade - very refreshing during the summer.
There are many paths throughout the garden. It is fun to explore and just let a split in the path take you to someplace totally different. It might be a good idea to wear long sleeves and pants since some of the paths are quite small and overrun by scratchy vegetation and the ponds in the lower half attract no small number of mosquitoes in the summer.
The garden is relatively large and can take you a couple hours to just leisurely wander around the ponds and woodland. I personally enjoyed the forested upper half. There is still open space up there, but shaded by clumps of trees. It is kind of like an orchard in places - perfect place for a little picnic. Another great thing about this garden is that you can pretty much wander anywhere. There are no roped off areas like in many similar parks in Tokyo - so feel free to spread yourself out on the grass under a tree generous with its shade.
You do have to pay to get in since it is a research area (you can tell with all the labels on the trees), but it is only ¥400. It is open from 10:00 to 15:00 every day except Thursdays and national holidays. You can find a map (in Japanese) on their website.