Asimov got many of those from his Black Widower mystery stories from real life.
For example, in "The Family Man," a Black Widowers story which appeared in a 1976 issue of Ellery Queen Mystery Magazine, the Black Widower's guest is an IRS investigator. His presence prompts Thomas Trumbull to say, "Good God. And you offer that as justification for your existence? Horse-whipping with barbed wire is what it justifies."
Later on in the story, Trumbull says, ""What the hell has 'unwilling' got to do with it? You enforce your own interpretation of the rules, act as prosecuting attorney and judge, hound us relentlessly, treat us as guilty till we prove ourselves innocent, and are perfectly ready to jail us if you can. What do you care if we're unwilling?"
and you know Asimov was thinking about the IRS when he had Geoffrey Avalon say this:
"I don't conscientiously pad, but I must admit that the IRS and I might not agree on just what constitutes a deductible expense in the first place."
And he has his IRS character say, "Then you deduct it till we tell you otherwise. That's the tax man's version of keeping you innocent until proven guilty."
Asimov never got in trouble with the IRS, but he was questioned about a deduction he took when he donated his papers to Boston University in 1968. (8 years before the events of "The Family Man."
From Yours, Isaac Asimov (a collection of Asimov's letters and postcards to people):
In late 1968, Isaac got a notice from the IRS that it had questions about his 1966 tax return.
1 November 1968
My accountant kept calming me and calming me and telling me he knew I was honest and the tax people knew I was honest and he would tell them flatly that it was a waste of time to question a man of my integrity, etc., etc. Then he looked through my tax return and said, "Aha-a-a-a." And I said, "What, what, what, what, what?" Then, to make it perfectly clear, , I said "What?" (That's a Wodehousian touch.
In 1966, I was giving my manuscripts and papers to BU, and for the first time, I received a note from them which included a notarized statement from some expert in such matters which gave an evaluation of the worth of my contribution. He placed it at $3,500. BU informed me I could deduct this from my income for tax purposes and so I did.
My accountant said, "Any sunstantial contribution in something other than money sets off an alarm in the computer and everyone comes running."
I said, "But I had nothing to do with it. I didn't ask for it. BU sent it of their own accord, and this guy is a recognized expert employed by them and this is a notarized letter."
And he said, "It doesn't matter. They want to argue it. They're going to question whether the material is worth that much."
I said, "For my part, it isn't worth anything. I used to burn it all before I started giving it to BU. But the university says its worth a great deal to them and to future generations of scholars."
He said, "I'll argue it with them and we'll see what happens."
Anyway, I'm not mad anymore because I think this is a legitimate investigation. In my case, I think the deduction is reasonable enough, but I can see where this sort of thing could lend itself to great abuse and the government should investigate it. And I'd rather be investigated myself in a careful guard against abuse than to escape it out of the general corruption and inefficiency of the government. Because, to be selfish about it, a rotten government would be far more expensive to me in the long run than the enforced (even unjustly enforced) payment of a few hundred dollars.