You can find Silver ore by fast traveling to the Ancient Spire in Albaneve Summits and heading South East (as shown in map screenshot).