Surrounding the Castle Pittormie possesses beautiful gardens, manicured lawns and a huge variety of magnificent trees. Cedars, Oaks Redwoods, Monkey Puzzles and many more are scattered across the estate. The garden and grounds have been laid out and developed by successive generations and owners of Pittormie. They have been lovingly brought back to their original grandeur in recent years. The mature trees are a particular feature of the property and many fine specimens provide a lovely setting for the house. The gardens have been developed through the Victorian era to provide interest and colour at all seasons. The informality of the parkland and arboretum contrast with the lovely herbaceous borders and topiary gardens. The rhododendrons and azaleas are a blaze of colour in the early summer, whilst the formal west garden with its historic fountain is an ideal setting for an afternoon tea party. Beyond The Steadings is an area given over to sports with a practice area, paddocks, tennis court and croquet lawn. The Castle itself is set amongst a back drop of mature trees which provide interest and colour throughout the year. Immediately in front of the house is a generous lawned area planted with bulbs and hybrid rhododendrons flanked by a small copse. To the east of the Castle is a parkland arboretum leading down to the main entrance. The garden winds round the house and to the north a wrought iron gate gives on to the herbaceous garden with fine perennial shrubs sheltered by an ancient high wall clad in clematis and roses. The plantings have been carefully designed to allow colour at all times of year.

The magnolias and pieris are particularly colourful in the late spring. This area leads on to a formal paved garden centred on a leaded statue with topiary box hedging and rose bushes beyond. There is a carp pond with lawns edged with primula and stone terracing, progressing on to a gravelled drum with stone balustrade and rock garden which now acts as the terrace for dining. Beyond the wall a further path edged with azaleas and bordered by lime leads through a rhododendron garden to the principal lawn with a terracotta fountain at its centre with summerhouses. A walkway through a small woodland with a flight of steps brings you to the south area of the policies with tennis court, croquet lawn edged with laurel and with two paddocks with shelters. There is a golf practice area with a newly planted orchard and kitchen garden edged with beech hedge and meadow land beyond. In front of the house is an interesting arboretum with a fine collection of specimen trees, including horse chestnut and sweet chestnut, Atlantic cedar, walnut, beech and copper beech, sequoia and sycamore as well as a mature Chilean pine nursing a young crop.