IMO in spite of the long history of auto racing in the area its never really appealed to a broad mass of folks; I don't think there is any question that those who are into the sport, and especially those who remember the well known drivers from back in the day including Roy Smith, Roy Haslam, Billy Foster etc., are equally as passionate about their interest in motor racing locally and otherwise as fans of any other sport, including those sports which have a much, much broader mass appeal.
But in my experience growing up in Victoria it seemed far more niche-y: my teenage cohort went to Western Speedway perhaps a dozen times altogether; they were in fact the only times in my life I ever went, the last of those being in about 1974. The experience remains memorable to me mainly because sitting in those stands watching a Saturday Night 75-Lap Main Event, sometime in about 1973-4, was the first time I ever smoked a joint. Beyond that - and lets honest here - the main reason *most* casual fans went to the old track was to see the pile-ups.
Those experiences aside none of my friends were motor head racing fanatics, even the ones who liked to tinker under the hood of their (usually) crappy first cars; nor did their friends. Nor did my parents or any of their friends identify particularly with auto-racing, either on TV or locally at WS.
Aside from the promise of a spectacular wipe-out no one in my junior HS ever talked about it; nor when I was in HS at Mt Doug, nor later in my UVic years. It was usually viewed - fairly or not, probably not - as a "Langford" thing that primarily attracted people from what we now call the West Shore. We called it something else less polite back then . Remember though 40-50 years ago that entire region looked nothing like today, being far more rural and even farming oriented with little to no built up developed 'core', therefore at the time it was also an ideal location for a motor speedway because it was a long, long way from any really populated neighborhood.
It'll be interesting given the broad growth 'everywhere' in the CRD where they ultimately find a piece of real estate large enough to meet their needs without also putting them within earshot of, and possible conflict with, 'suburbia'.