Speaking to BBC Good Morning Ulster, South Down MP Mr Hazzard said the approach of the Conservative party to Brexit shows the importance of planning for future political outcomes.
"I don't know of a single person who is advocating for an Irish unity referendum today, tomorrow or next week," he said.
But he said Mr Varadkar and others in his party, Fine Gael, need to "end the Jekyll and Hyde approach to constitutional change".
A Citizen's Assembly on the issue of Irish unity should "lead the way" for Irish unity along with the Irish government, he added.
It's after South Antrim MLA Declan Kearney said Mr Varadkar's comments were "lacking in ambition and politically weak".
On the same programme, DUP MP Sammy Wilson agreed a poll would be divisive.
"Sinn Fein want to increase division in Northern Ireland and that's why they're pushing a referendum," he said.
The Belfast Agreement makes it clear the conditions for a referendum haven't been met and people in Northern Ireland "don't want" this constitutional change, the East Antrim MP added.
Provisions for a possible border poll on Irish reunification are included in the peace agreement, which states the Northern Ireland secretary must call a border poll if it at any time it appears "likely" to that a majority of people in Northern Ireland would vote for a united Ireland.
The DUP's aim is to have stable government in Northern Ireland while fighting against the "divisive impact" of the Northern Ireland protocol, Mr Wilson said.
On Sunday, Mr Varadkar described the aspiration to a united Ireland as a "legitimate one" but said a border poll would "not be appropriate or right" at this time.
He told BBC NI's Sunday Politics show that the focus should be on getting the Northern Ireland Assembly "up and running" and said the "tests around when a border poll can and should happen" needed to be clarified.