Before urban development, Victoria Harbour consisted mostly of rocky shorelines with numerous pocket beaches, a few sand and gravel shorelines, intertidal mud flats, and salt marsh. Many areas were subsequently filled in, such as the marsh to the west of Point Hope, much of Rock Bay, most of the shoreline and coves between Odgen Point and Shoal Point, and the intertidal mud flats at the east end of James Bay, where the Empress Hotel now stands. Only a small percentage of the shoreline in Victoria Harbour remains in a relatively natural condition. The physical features of the harbour are now mostly characterized by altered shorelines such as wharves, seawalls, angular rock and a breakwater (at Ogden Point).
A few of the more ecologically valuable remaining shorelines include intertidal mud flats in West Bay, the rocky shoreline between Macaulay and McLoughlin points, and several bedrock islands.
Most of the subtidal substrate consists of sediment (mud/sand), with smaller areas of bedrock and gravel. Man-made debris also covers much of the harbour seafloor, particularly near historic industrial sites and public wharves.
There are relatively strong tidal currents in the harbour, particularly in the outer harbour near the entrance.