I've come across a number of argument-structuring tools in the past. I think doing this right is much harder than people give it credit for.
The core problem is that most of the action in a quality discussion doesn't actually consist of claims and counterclaims. For example, I went to Truthsift and the first claim I found on their randomized front page was "vaccines are safe" (http://truthsift.com/search_view?topic=Are-Vaccines-Safe-?&id=406&nid=4083). The response I wanted to make is that vaccine is too broad a category; a well-managed discussion would first clarify that the discussion was limited to the sort of FDA-approved vaccines one would normally be prescribed, not to random research chemicals, then break into sections about specific vaccines and vaccine additives, the trustworthiness of the FDA approval process, and so on. But Truthsift wouldn't let me say that, first because I'd have to somehow mash that into a pro-and-con structure, and second because I'd have to merge that into an incompatible structure that someone else set up. This is entirely representative of my experiences with argument-structuring sites: I show up, find an elaborate structure that partitions the question in a way I think is wrong, and bounce.
Arbital is currently better because it doesn't try to structure everything; it leaves space for all the hard-to-tag irregular stuff to happen in comment threads. I think there's a large valley in between comment threads and a structure that can incorporate all the irregular things that happen in discussions, and that TruthSift's mistake was that it tried to squeeze out the irregular stuff.
Comments
Andrea Gallagher
Great analysis of problems with TruthSift. Perhaps we should start a list of irregular and natural behavior that Arbital needs to handle gracefully. Hmm, I wonder what kind of page that should go on?
Can you point to other argument-structuring tools?
I see the point about this being a hard problem - it increases the likely investment needed to get the return, so my "high ROI for improving discussion quality" is a claim at risk. But if we can keep our eyes out for a low-effort way to solve the problem, the return still feels high.
Alexei Andreev
It's very likely that because we haven't seen any solution to this problem, the ones that do exist are not low-effort. This is part of Eliezer's argument (as you'll probably get from the doc).
Andrea Gallagher
After reading the Doc(tm), I think there is still design space to explore. For most readers, and many authors, keeping track of the points in an argument is actually pretty hard mental work. Just a little help would make them be better at it.
The trick would be to create "just enough" structure that is easy to fold into the process of authoring a long post or writing a quick comment. I don't think we need to reproduce all the elements of formal analysis here (and if we tried, it would be unusable).
In the end, I might argue myself back into the model where there are just "claims", not "claims" and "evidence".