An explanation of why a panopoly of legal theories to collaterally attack a state court domestic relations decision is set forth here. Many of the theories discussed apply more generally. This is really a short treatise on why you can't sue the government in certain situations and also can't sue people involved in the court process.
This type of lawsuit, brought without a lawyer, isn't that unusual. People unhappy with the system, who no longer trust lawyers, bring them with some frequency, usually to similar results.