Welcome to VV, David.
Some parts of Esquimalt were somewhat shady up until about 15 years ago but certainly no worse than some parts of the big municipality of Saanich or the Westshore (Langford, Colwood, etc.).
But that being said, most folks probably don't know that James Bay was once a very shady area until several decades ago when ports were closed and industrial zones were transformed. Today James Bay is held up as one of the most desirable areas of the region (how times change, hey?).
Ideas and misconceptions die long, hard deaths. The truth about Esquimalt is its always been a quasi extension of the City of Victoria in the sense that it's close to downtown. It has amazing waterfront properties and in the real-estate world has been a quietly held secret. Check out Rockheights, Saxe Point, Selkirk and the Parklands areas. West Bay already has two newer condos overlooking the harbour built by the same guys behind Westbay Quay and some of the units are worth millions. Would folks buying multi-million dollar condos do so in a "no way" part of town?
West Bay Marina has a huge amount of upwards value potential thanks to its long-held anti-development attitude that's slowly being chipped away. Have a look for yourself and chat with the locals. You'll learn more than listening to folks who have so clearly intimated that they have a absolutely no idea what they're talking about if indeed they are telling you "no way."
Pretty much this: I grew up in Gordon Head as a kid and for half my life to age 27 lived a block from UVic on the Saanich-Oak Bay border. Later I lived for about 5 years in, ahem, 'upscale' Fairfield near the Cook Street Village: Esquimat was always perceived as this 'across the bridge', uber-working class, blue collar neighborhood where off duty navy guys got drunk on cheap beer in smoke-filled bars....
....and that is so 1970's-thinking and its also dead wrong; I've lived a block from the Selkirk neighborhood on the Gorge waterway now almost 6 years. Its a great place to live, raise your kids and its a hell of a lot quieter than Fairfield-Cook Street, which had emergency responder sirens going seemingly every night since Fairfield Road is a main west-east arterial route to RJH for ambulances and police etc.
By contrast we rarely ever see a cop car here, never mind hear it. Selkirk in the spring or fall is as beautiful a neighborhood as you'll find in the core; and Parklands is probably one of the most underrated locations anywhere in the CRD mainly because so few people are even aware it exists. So to answer your question David a resounding "NO" - Esquimalt really isn't that bad....