It's a long walk to downtown, especially the Empress and south of downtown, from West Bay, though a pretty one. It can be handled for sure, but you wouldn't want to do it on a regular basis. If you want to cycle, you have to go up a steep hill to Esquimalt Road and then have the bike path along Esquimalt towards the bridge. I'm not a fan of the area.
Great bus access to downtown, if that's how one chooses to get around. A quick walk to Esquimalt rd, hop on the #15 goes right to Fort St. At peak travel times it runs every 10 minutes.