Jack Harrison’s first-half opener was cancelled out by Burnley substitute Maxwel Cornet’s superb free-kick early in the second period.
But Stuart Dallas’ brilliant 77th-minute strike raised the roof at Elland Road and Bielsa celebrated substitute Dan James’ stoppage-time header by hugging assistant coach Pablo Quiroga.
Leeds’ first league win since November halted a run of three straight defeats and when asked if he felt relieved, Bielsa said: “Yes, of course.
“I don’t ignore the situation that we are in. I also don’t ignore what my responsibilities are. That can’t not influence my state of mood.”
Leeds’ fourth league win of the season lifted them eight points clear of Burnley, who remain third from bottom.
Bielsa’s side had dominated the first half and typically wasted a hatful of chances, but Burnley improved with the arrival of Cornet after the break and made a real game of it before Leeds finished strongly.
“We managed a fair victory,” Bielsa added. “The difference I thought was also fair. It’s very difficult to play against Burnley. They’re very clear on how they want to play and always keep the differences close.”
Leeds welcomed back Junior Firpo and Diego Llorente and raised their game following their home defeat to Arsenal two weeks ago, despite still being without the likes of Kalvin Philips, Rodrigo and skipper Liam Cooper.
Bielsa added: “Even if the performance today was a step forward, we need to maintain that level of performance against any opponent and in any pitch.”
Burnley’s solitary league win this season was at the end of October and boss Sean Dyche admitted his players were lacking in confidence.
He said: “I spoke to the players about that weird feeling that creeps in, usually when results are not going your way and everyone is looking to someone else to do the next thing.
“When we talk about confidence and winning and results, you find when you’re in a good moment, the game becomes fluid, both defensively and in attack.
“At the minute it’s like we’re waiting for someone to do their job and if it goes wrong we’re not active, we’re not anticipating.”
Dyche, whose side are two points from safety but with a game in hand on Watford, was complimentary in his assessment of Leeds.
“I don’t want to discredit Leeds,” he added. “They played forward, they worked hard. Leeds try and play, but a lot of it was played straight, getting forward and running hard, playing that physical game with an edge to it.
“Their first goal came from that. It came from a clearance and their lad (Harrison) runs it down like his life depended on it.
“That’s sometimes the edge that it takes and we weren’t there for that, that kind of feel, for the whole game.”