I certainly don't think new buildings should have to "blend in" to the neighbourhood. That is a recipe for mediocrity.
It depends on what we mean when we're talking about fitting in. If a hypothetical district is defined by its ornate and distinctive buildings of varying forms and heights and styles and materials (and eras) then a new building can and should fit in by extending the effect, by being ornate and distinctive. In other words, DO NOT imitate the particulars, but DO adopt the attitude. Methinks there's way too much focus on the crudest material aspects and almost no regard at all for overall effect or atmosphere or the way elements come together and complement one another.
"Hey, the Empress Hotel is covered in a lot of red brick so therefore -- in order to fit in -- every new building on the inner harbour should also be covered in a lot of red brick (even though plenty of other old buildings were not covered in red brick)."
Baloney. It should be:
"Hey, the Empress Hotel and the Belmont Building and the Union Club and the steamship terminal (and the long lost post office) are all very different but very grand buildings, ranging widely in terms of size, height, style, materials, etc., so therefore -- in order to fit in -- any new building on the inner harbour should also be very grand but should also be unique and bear no resemblance whatsoever to the others."
In the case of Fairfield we're talking about a very different character, of course. But I hope you get my point.