It depends on the vessel type and how long they're in port for.
Usually if it's only a short port call everyone will stay on watches, so those off watch will either sleep, go ashore or catch up on maintenance/other tasks - which is why you can't see anyone on deck.
If it's a longer port call then more deckhands will get called up for mooring, or alternatively they'll stay on watches until late that evening, when the nightwatchman will come on duty, and everyone else will switch to daywork.
When at anchor everyone will usually stay on watches, the deck officers certainly.
Ships don't have a lot of crew, either. My last ship had a total crew of 13 including myself, so you haven't got a huge amount of people to start with!