The secretive mobster has seen his one-time iron-clad grip on the UDA crime gang loosen in recent months, leaving him Brigadier in name only.
The organisation is now in the grip of a committee of senior paramilitary veterans with Fisher forced to toe the line.
Also his appearance at a Remembrance Sunday service two weeks ago has left those close to him in shock.
His gaunt appearance had people raising questions about his health.
His cheeks appeared hollow, he has shaved his head and those who have seen him describe him as 'haunted'.
Sources have told us the crime boss is a man under intense pressure.
The Sunday World understands he is desperate for an exit strategy that will take him out of the firing line.
His organisation has come under intense scrutiny from the Paramilitary Crime Task Force and he is known to be anxious to get out of the crime business before he is scooped.
Sources in South East Antrim have told us Fisher has been trying to engineer an exit strategy with enough money to sustain a life away from the UDA.
As head of a cash-rich organised crime gang, Fisher has amassed a considerable personal wealth, but even as brigadier his hands are tied when it comes to walking away.
"It is notoriously difficult to leave," a senior loyalist source told us.
"All you have to do is look at history, there are very few commanders or brigadiers who were simply allowed to leave.
"Putting it bluntly they were either stood down, expelled or shot - that's the only way to get out."
He cited former UFF chief Johnny Adair as one of the few to have got out with the skin on his back.
"He was lucky to be let out alive, he knows it, but he's the exception rather than the rule."
Fisher is no longer the main man in South East Antrim, he has seen his close circle of trusted sidekicks dissolve.
Close pal Clifford 'Trigger' Irons is currently out on bail awaiting trial on drugs charges. And one-time trusted lieutenant Colin Simms is in disgrace after being linked with the murder in Carrick a year ago of Glenn Quinn.
Convicted killer driver and drug offender Simms was once Fisher's muscle entrusted with keeping a small army of drug dealers in Carrick in check.
Fisher associates subjected Simms to a punishment shooting after he was found with his hands in the till.
Simms' demise was the catalyst for the disintegration of Fisher's power base.
He moved from the SEA power base in the Rathcoole estate to the sleepy backwater of Greenisland as his grip on SEA weakened.
Carrick's drug trade is now a mish-mash of conflicting independent criminal gangs.
Convicted gunrunner Thomas Morgan, a one-time member of the UDA, has muscled in on the town's drug scene.
Dissident groups and LVF-linked mobsters have also moved in to claim a slice of the action.
It has brought the town under close scrutiny from the Paramilitary Crime Task Force.
Law enforcement agencies see the loosening of Fisher's control as an opportunity.
"When Gary was in complete control, there would have been no suggestion of any leaks, it was a tight, tight operation.
"Fisher is vulnerable and he knows it.
"Fisher can't make a decision, he's not allowed to make a decision and they're just keeping him there, partly because no one wants the job and they're waiting until the heat from the Task Force eases.
"They'll let Fisher take the fall before replacing him, they don't see any point in putting someone else in now only for him to be targeted."
The Sunday World also understands Fisher has become increasingly paranoid, believing there is an informant who is passing information about his movements to the PSNI.
There is a belief in the organisation that the Task Force is building a case against Fisher and senior figures in SEA are content to leave him in place.
"He'll take the hit, he's like a deer in the headlights, there's nowhere for him to go."