Joseph Samuel Swift was born on 25 May 1965 in Newcastle, England. His father was actor Clive Swift, best known for playing Richard Bucket in Keeping Up Appearances, and his mother is novelist Dame Margaret Drabble. His parents divorced in 1975; his mother later married writer and biographer Sir Michael Holroyd in 1982. His siblings include brother Adam Swift, a political philosopher, and the late sister Rebecca Swift, a poet and founder of The Literary Consultancy, who died in 2017.

After leaving school, Joe went to art college but dropped out and became a guitarist in a band called “The Tumbling Dice.” He left the band to go travelling and first got into gardening whilst in Israel working on a Kibbutz. He then worked as a landscaper in North London and later lived and worked in Australia, continuing as a landscaper and drawing influence from different environments.

Joe returned to study a degree in Garden Design at The English Gardening School at the Chelsea Physic Garden. One of his first jobs after qualifying was to design his mother a garden. His break into television came in the late 1990s when his engaging personality and deep horticultural knowledge caught the attention of BBC producers. His first major TV role was as a presenter on BBC Two’s Gardeners’ World in 1998.

In 2003, Swift co-founded Modular Garden, an award-winning design-and-build brand for which he serves as Design Director. In 2018, Joe designed Horatio’s Garden at Stoke Mandeville, calling it “the most important garden I’ve ever designed.” Joe and his wife share two children and reside in Hackney, London. In 2026, he joined Garden Rescue Series 11 as a new designer.