The fraudsters, who claimed to be from banking group Santander, phoned Mr Williams to say they needed remote access to his bank account.
After gaining access they transferred three payments totalling £53,000 into a new account.
The scam left Mr Williams “distraught”, as he had planned to use the money to buy an apartment by the seaside to spend time with his wife, Carol, who had terminal cancer, and has since died.
Instead the couple were forced to take out a mortgage to purchase the property in Weston-super-Mare.
“At the time I had been told my wife was terminally ill, so that was more important,” said Mr Williams, of New Bryngwyn Road, Newbridge.
“I was distraught but I knew I had phoned Santander (during the scam) and believed I could get most of it back.
“I told them and they did not prevent it happening.”
The retired phone engineer said it was only due to his persistence and an investigation by an ombudsman that he was able to recoup the money though.
Mr Williams said his initial request for a refund, along with an appeal and complaint, was rejected by the bank.
After pointing out he had attempted to contact the bank during the scam – to show he had not acted with “negligence” – Mr Williams said he was offered a £33,000 payment from Santander as a “goodwill gesture”, though the bank made clear it did not accept liability.
A further £12,500 was recovered by Santander, while the remaining £9,145 is due to be refunded in the coming weeks after an ombudsman investigation.
Santander has also offered to pay 8% interest on the cash, dating back to the time of the scam, as well as money in recognition of the “trouble and upset” caused, according to an email from an ombudsman investigator seen by the Local Democracy Reporting Service.
“Justice has been done,” Mr Williams said.
“I did not think I was going to get all the money back.
“It shows what a fantastic service the ombudsman is for the little guy to go against the might of the institutions and banks.”
Santander said it “will continue to work hard with all relevant parties to stop the people who carry out these crimes.”
A spokeswoman said: “Following an internal review and guidance we have now refunded all the money taken from Mr and Mrs Williams’ accounts.
“We appreciate that this fraud happened at a particularly difficult time for our customers and we would like to extend our condolences to Mr Williams on the death of his wife earlier in the year.”