Here is an option if you're closer to the Inner Harbor. If you take Key Highway East after exiting I-95, you'll eventually see the Domino Sugar plant on your left. Right after you pass it there will be a left turn into a parking lot for the Under Armour buildings in North Locust Point, and if you are there during the weekend you should find plenty of parking. That is a good place to see bulk carriers discharging sugar.
Fort Smallwood is a decent vantage point, but there can be long waits for vessels. Also, you need to always be aware of your surroundings there.