Intelligence for the business carried about by our intelligencies agencies, is, like the title "Defense" for the business carried out by our military, a frame shaping deception. Yes, the CIA and the like engage in information gathering, sometimes by means which would be illegal if carried out by you and I, but that isn't the reason that people fear the CIA.
The reason to fear the CIA and related agencies is that they are tasked with engaging in "covert operations" with very little political branch supervision. This covers things like kidnappings, assassinations and disinformation campaigns.
Indeed, these covert operations are also one of the main reasons that the agency must have so much secrecy. While information gathering can be compromised to some extent if one's methods for doing so are known, there is no reason for the products of the information gathering to be as closely held as they are, which means that we spend tens of billions of dollars a year for information that doesn't reach everyone who needs to know it. The exposure of a covert operations, in contrast, is politically explosive, and worrisome, because it involves the carrying out of policies that haven't been subject to the public debate usually a part of the policy process in a democracy.
The linkage between human intelligence and covert operations (both houses in the same division of the CIA to a great extent) has also resulting in funding for human intelligence, which is quite useful, at times, being undermined by liberal critics of covert operations.
Maybe we'd all be better off if the two functions were more effectively seperated. The CIA could become a little more open (if major media organizations can manage to protect its sources will widely disseminating their information, so can the CIA to a much greater extent). And, if covert operations were not buried in a much larger bureacratic infrastructure, they would be easier for public officials to supervise (while giving the true intelligence officials more latitude to act) and it would be easier for them to keep secrets because they fewer people who are in the loop, the easier it is to keep a secret.
Of course, there is bound to be mischief. What category does kidnapping someone and questioning them fall under (and should we be in that business at all)? Perhaps it isn't possible. But, it would seem to me that there is, at least, some room for a rethinking of the issue.