In 1903 Sir Henry Pellatt purchased 25 lots from developers Kertland and Rolf. Sir Henry commissioned Canadian architect E.J. Lennox to design Casa Loma with construction beginning in 1911, starting with the massive stables, potting shed and Hunting Lodge (a.k.a. coach-house) a few hundred feet north of the main building. The Hunting Lodge is a two storey 4,380 square foot house with servant's quarters. As soon as the stable complex was completed, Sir Henry sold his summer house in Scarborough to his son and moved to the Hunting Lodge. The stables were used as a construction site for the castle (also served as the quarters for the men servants), with some of the machinery still remaining in the rooms under the stables. The house cost approximately $3.5 million and took a team of 300 workers three years to build from start to finish. Unfortunately, due to the start of World War I, construction on the house was halted. At 98 rooms, it was the largest private residence in Canada. Notable amenities included an elevator, an oven large enough to cook an ox, two vertical passages for pipe organs, central vacuum, two secret passages in Sir Henry's ground-floor office and three bowling alleys (never completed).

Most of the third floor was left unfinished, and today serves as the Regimental Museum for The Queen's Own Rifles of Canada. Pellatt joined the Regiment as a Rifleman and rose through the ranks, eventually becoming the Commanding Officer. He was knighted for his dedication to the Regiment. Later, Pellatt served as the Honorary Colonel and was promoted to Major-General upon retirement.

During the depression that followed World War One, the City of Toronto increased Casa Loma's annual property taxes from $600 per year to $1,000 a month, and Pellatt, already experiencing financial difficulties, was forced to auction off $1.5 million in art and furnishings for only $250,000. Sir Henry was able to enjoy life in the house for less than ten years, leaving in 1923. It was later operated for a short time as a luxury hotel. During the late 1920s Casa Loma was also a popular nightspot. The Orange Blossoms, later known as Glen Gray and the Casa Loma Orchestra, played there for eight months in 1927–1928. Shortly thereafter, they went on tour of North America and became a major swing era dance band.

The city seized Casa Loma in 1933 for $27,303 in back taxes.

The castle was extremely run down and the city was motioning for the castle to be demolished. In 1937, however, it was leased by the Kiwanis Club of Toronto (currently known as the Kiwanis Club of Casa Loma). Today, Casa Loma is undergoing a 15 year exterior restoration.

During World War II, Casa Loma was used to conceal research on sonar, and for construction of sonar devices (known as ASDIC) for U-boat detection.

Contrary to popular belief, Casa Loma has never been an official residence of either the city or the Province of Ontario. In 1937 it was opened to the public for the first time as a tourist attraction operated by the Kiwanis Club of Toronto. Coincidentally, this is the same year that Chorley Park, the Government House of Ontario, was closed by the provincial government. The house is still operated by the Kiwanis Club. Today it is one of Toronto's most popular tourist attractions.

Casa Loma has five acres of gardens. An underground tunnel connects Casa Loma to the Hunting Lodge and to The Stables (Garage, Potting Shed, Stalls, Carriage Room and Tack Rooms).

Sir Henry imported artisans from Europe to design much of the furniture and other features of the castle, a few of which can be seen in the images below.

Casa Loma is on Austin Terrace, at the north end of Spadina Road on an escarpment (Davenport Hill) above Davenport Road. Davenport runs along the bottom of the escarpment which was the shoreline of Lake Iroquois, the predecessor of Lake Ontario (coordinates 43°40′41″N 79°24′33″W / 43.678°N 79.4093°W / 43.678; -79.4093). Casa Loma affords views down the escarpment and Spadina Avenue into the heart of Toronto.