Adjacent to the Kenroku-en Garden are the grounds of the Kanazawa Castle which was built in 1583. In fact the Garden was built for the Lord that lived in the castle.
Unfortunately the original castle burned down about 350 years ago and its several replacements have all either burned down or been destroyed by earthquakes. So today there is no castle, but there are several support buildings and gates that have survived.
The castle grounds sit atop a hill in central Kanazawa, so we actually thought the best part was the view you got of the surrounding area.