The Welsh Government has been accused of failing to take its own expert infrastructure body seriously amid warnings crucial advice is being ignored and delayed.