Hi, ozzis: Never having been a tourist on Vancouver Island, just born, raised and now live here again, I had no idea that such a tour was available. Thanks for pointing it out. I am assuming that the tour company allows you to "re-join" the following day as their website just talks about an Ultimate Day tour from Vancouver and back.
So, what to say....I think you can easily occupy yourselves in and around the Victoria and the inner harbour where Prince of Whales docks, for the remainder of your first day and possibly for the morning of your second. Looks like you have to board your shuttle to Butchart Gardens (14 miles north of Victoria) from the inner harbour so, unless you have a car, I don't think there is enough time to do much else.
You could visit the Empress Hotel for high tea and see Parliament Buildings in the Inner Harbour, the Royal BC Museum is nearby. You could tour Canada’s oldest Chinatown, visit the Maritime Museum of British Columbia, Helmcken House, Emily Carr House or Craigdarroch Castle. You could take a Tally-Ho horse-drawn carriage tour through Beacon Hill Park or double-decker bus tour around the city.
Note that Vancouver Island is the largest Pacific island east of New Zealand and is 460 kilometres (290 mi) in length and Victoria is at the bottom end and is the largest city on the island. Nanaimo, where H.& G. live is about 90 km/60 miles north of Victoria, is the second largest city. You could easily spend a week exploring the island - maybe on your next trip!
I am happy that you are able to come to Ye Olde Victoria - the "British" part of British Columbia. Maybe we can connect for a coffee...we live about 10 minutes from the harbour.