Maris Skujiņš, who earlier this week won the support of Finance Minister Dana Reizniece-Ozola and was due to face a Saeima vote to confirm him in the high-profile position, said a smear campaign had taken place against him.
In a post on his Facebook account, Skujiņš said: "The lying campaign and their media agents have won. I withdraw my candidacy for the post of VID director."
"Recent days are a vivid testimony to the mire of political culture in the most unfortunate way. This episode is lost, but in the long run we will be the winners!" he wrote, adding that he had informed the Security Police about what he alleged were false rumors spread about him.
He also apologized to his family for the "last day of hell" they had endured.
The move creates another headache for the Latvian government as it strives to prove to the international community that it has matters well in hand, not only in the non-resident banking sector, but in the wider financial infrastructure of the country.
As previously reported by LSM, press reports had speculated on links between Skujiņš and a controversial businessman, Maris Martinsons.
The government press office confirmed in the evening that Skujiņš had pulled out of the running, a move that will force a competitive contest for the job to be run all over again, a process likely to take weeks.
Reizniece-Ozola said that future contests would need to "pay much more attention" to make sure the situation in which a candidate was named and publicly endorsed did not pull out at the last minute again.