With ref to Darren's post, Railways and Ports are different beasts as far as photography are concerned. Railway Stations are classed as public places, and, as such, photography is permitted, provided the photographer is observing the by-laws in force at the location, e.g. not going beyond the "Do not pass" limits.
Following the ISPS code, ports are classed as private places, and access and photography are banned to the public, unless with the specific permission of the port operator.
Of course you can debate the rights and wrongs of this, but that is the case, however it is true that the rules are enforced more rigorously in some ports than others. Health and Safety rules also apply more now, which introduce their own restrictions.
However if you are taking photographs from a public place, even through a fence of a restricted area, only the police have a right to stop you, and then only if they have reasonable ground to suspect that you are involved in terrorist or criminal activity. They cannot do so just on a whim. If you are approached by a security guard in one of these area's, remind them of this. They must involve the police if they actually want to move or stop you, and the police get a bit annoyed if they are involved for no justifiable reason. Certainly in the UK airport security guards are trained in this, and might come and talk to you, but rarely involve the police. If they do it's usually because the person approached has got aggressive.
The same rules apply to airports.
It's all a bit of a nonsense of course. In some ports, like Goole, the port estate is crossed by several public roads from which photography is permitted and easy.