We've been twice now and have stayed both in the Golden Horn area and at the Ritz Carlton. The port is very close to the Ritz Carlton/Intercontinental area ... about 5-10 minutes. It is a good 20 minutes from the Golden Horn due to traffic. You can see the ships and port from the Golden Horn. The traffic is dreadful in Istanbul and it can take ages to get from the Golden Horn to Taksim or the Port area. We had no problems, but at certain times you can have a dreadful time. We opted to get around from the Golden Horn by public transportation and it worked well (but the BO was dreadful).
Having stayed in both areas, even with the traffic and BO, we really preferred staying in the Golden Horn area due to the fact that just about everything you want to see is in that area and traffic is dreadful going to and from. You can take a ferry and go to the Asian side and walk around as we did if you want.
There are wonderful restaurants in the Golden Horn area too...there are really good restaurants all over Istanbul (I loved Turkish food)...LOL