No matter what you may have been told, water stored in poly tanks or tanks with plastic lining should have no taste.
Of course, this depends on the maintenance of your roof, gutters and the tank itself. You should be aware however, that most rainwater tanks are not approved for drinking water by the local authorities in cities. This policy is not based on any "fears" about any of the materials used to manufacture tanks and relates back to the possibility of unclean drinking water from poorly maintained roofs and other contaminants that may enter the tank.
Steel tanks made from galvanised steel may leach excess concentrations of zinc into the water and this can alter its taste. It's important to note that this is not a health risk.
New concrete water tanks may leach lime thereby increasing the pH of water. This can also alter the taste. Concrete water tanks may need to be flushed before first use. Many find the taste of water from concrete tanks to be superior due to calcium and other minerals imparted by the concrete walls.