In short: YES. Not only is it safe, but the water is excellent! So please don’t buy bottled water.
95% of all water in Iceland comes from springs in the ground, so it never comes in touch with any pollution. Unlike in many neighbouring countries, the Icelandic water is free of chlorine, calcium, and nitrate. The quality of tap water in Reykjavík is well monitored, so you are not to worry!