Aamer Anwar also told the inquiry that providing free legal advice to the family of Mr Bayoh, who died in police custody, jeopardised his business.
Mr Bayoh, 31, a father of two, died after he was restrained on the ground by police officers in Kirkcaldy, Fife, on May 3 2015.
The inquiry has been investigating the circumstances of his death and whether race was a factor.
His family were provided with pro-bono legal assistance by human rights lawyer Aamer Anwar for five years, before an inquiry was announced, and Mr Anwar paid tribute to Mr Bayoh, who would have celebrated his 41st birthday on Monday, praising the family’s “tenacity and perseverance”.
Mr Anwar, 56, originally from Liverpool, branded the Scottish justice system “institutionally racist” and said “at least 50%” of his workload was taken up by the case.
He said he was warned by Imran Khan KC, who represented the family of murdered teenager Stephen Lawrence, “not to respond” to racism.
He also said he was warned by Michael Mansfield KC: “It’s fine for me, I’m white, I’m middle-class. I’m seen as a knight in shining armour who rides to the rescue of poor, vulnerable black people. But when you do it, Aamer, or Imran does it, you’re playing the race card, you must be prepared to be attacked.”
Giving evidence on Tuesday, Mr Anwar said he had received death threats and had to move house because of his work, experienced paranoia and had been put under police surveillance, which he branded “unlawful” but said he had not made a formal complaint due to the risk of diverting attention from the case.
Mr Anwar said he told the family “we cannot publicly mention the word racism”, which made them so angry they considered not instructing him, the inquiry heard, and that he advised them to trust in the Crown Office and Police Investigations and Review Commissioner (Pirc).
Mr Anwar said: “Fundamentally, all of them failed to deliver justice. The criminal justice system in this country is institutionally racist.”
In a statement, Mr Anwar said Kenny MacAskill, a former Scottish justice secretary, claimed he chose controversial cases as it was “good for business” – which Mr Anwar rejected, saying that he feared his business “would not survive” due to the pressures of pro-bono work on the Bayoh case.
Mr Anwar said: “People say you’re paranoid – I know exactly how they look at me and talk about me, and what they say about me behind my back. Imran Khan said ‘you will be attacked, they will come for you, do not respond because you will be falling into the trap of what the state does’.
“It wasn’t good for business, it was bad for business.”
He said the impact of the case had been “traumatic” and that he had been subjected to “attacks” professionally and had a nervous breakdown.
The inquiry heard Mr Anwar warned the Bayoh family not to carry out door-to-door inquiries, expecting Pirc would – however, Mr Anwar claimed it never happened.
He also said that a post-mortem examination which the family requested be postponed so family members could attend was carried out regardless.
Mr Anwar said: “We now know police officers were present who shouldn’t have been present but the family weren’t entitled to be present.
“It was not an investigation into death in custody, it was amateur hour for Pirc.”
Mr Anwar said he was advised of the possibility of police surveillance by Mr Khan KC and that he found out during the inquiry it had happened.
He said: “I was subject to targeted surveillance. If this was a white lawyer doing the job I was doing would that be acceptable, or would there be uproar?
“Because it’s Aamer Anwar and the death of a black man, there isn’t any outrage.”
A text message from Mr Anwar to then-lord advocate Lord Mulholland on May 5 2015, just before 10am, expressed concern the situation could go “tits up”, followed by a verbal discussion about the request to wait to carry out the post-mortem examination until Mrs Bayoh arrived, which was agreed, the inquiry heard.
However, in an afternoon meeting between Mr Anwar and the family, a phone call with Pirc revealed the post-mortem examination had already happened, causing relatives to “become visibly upset and angry”, the inquiry heard.
It heard that a family liaison officer from Pirc later claimed Mr Anwar was told on May 4 that the post-mortem examination had happened – which the lawyer branded “utter nonsense” and said Lord Mulholland was “astonished” it had gone ahead.
In a statement, Mr Anwar said no “satisfactory explanation” was given, adding: “This was one of many issues with Pirc that shattered trust.”
Mr Anwar told the inquiry: “What was the haste? That morning as well they didn’t tell me. After the LA intervenes it’s like ‘oh it’s gone ahead’.”
A meeting took place with Pirc on May 6 where the cause of death was given as “unascertained”, the inquiry heard.
Inquiry chairman Lord Bracadale said: “I want to recognise the enormous amount of work you have put into the case over the years.”
A spokesperson for the Crown Office and Procurator Fiscal Service said: “The Crown examined the death of Sheku Bayoh in great detail and instructed appropriate lines of investigation.
“Following careful consideration, and then a thorough review, of all the available evidence by the most senior prosecutors in Scotland, it was concluded that there should not be a prosecution in this case.
“Although the available evidence available would not justify criminal proceedings, the Crown reserved the right to prosecute should evidence in support of that become available.
“Day in and day out, Scotland’s prosecutors fulfil their responsibilities with professionalism and skill. They take hard decisions and do so independently, rigorously and in accordance with the evidence. They secure the public interest in fair and effective administration of justice.
“The Crown Office and Procurator Fiscal Service is fully supportive of the inquiry and does not consider it appropriate to comment on evidence.”
The inquiry continues in Edinburgh.
Police Scotland said it would not be appropriate to comment on evidence led at the inquiry, however said that Chief Constable Jo Farrell outlined her ongoing commitment to building an anti-racist, anti-discriminatory Service which delivers for all communities during a meeting of the Scottish Police Authority in May.
A spokesperson for Pirc said: “It would be inappropriate to comment while the public inquiry is ongoing.”