Discussions on Less Wrong have delved into complex systems of voting and moderation to improve the quality of discourse in a community. I think creating structures to a discussion beyond threaded comments will be more productive and have a higher ROI. For claims, I care less about comments than about whether someone can provide evidence in support or against it, and evidence can be strong or weak and have it's own evidence-children. Summing up across all of that evidence would be interesting.
Comments
Andrea Gallagher
So, how do we measure value in elevating accuracy or truth?
Ben Pace
My main intuition against: I think that the value that an arbitrary researcher gives to a field is highly heavy tailed, whereby a few key people move the whole field forward, and so we want clear indicators of who those people are, and to listen to them.
My main intuition for, however, is that the whole premise of arbital is that you can get value from increasing the communication quality in the whole network, which is what increasing the quality of evidence available is (this isn't a product aimed at the few, it's like wikipedia, which gets its value from raising the whole level of discourse).
At a first pass I'm weakly in agreement, but not sure that one is significantly more valuable than the other (or that they necessarily require trading-off).
Andrea Gallagher
Good catch about reputation being critical. Perhaps I should remove that, since I'd like to see reputation systems, but have them depend on broad data (including claims, evidence, voting, etc).
Eric Bruylant
I foresee good reputation systems being extremely valuable (essentially necessary to scale while maintaining quality), with high credence on that being more important than argument structuring features.
Andrea Gallagher
I see reputation systems as being necessary, but not sufficient. Without argument structuring features, how is Arbital different than Reddit or Stack Exchange?
Eric Bruylant
Reddit's reputation system gives new arrivals equal weight to long-standing highly trusted members of the community, and does not include priors about content quality based on poster's history. It's the simplest thing which could barely work, and does not allow for high quality discussion to scale without relying heavily on moderators or other social things not present in all communities and not able to resist certain forms of attack. It also lacks adequate indexing and browsing by topic, making discussions temporary rather than able to produce lasting artifacts and be continued easily.
SE's reputation system is a little better (you need to prove to the system you can productively engage with the topic before your votes have any weight), but it's very focused on QA, which is not a great format for extended truth-seeking discussion.
Cool argument structuring seems like an optional bonus (still great to have, but not necessary for the thing to work), but features that give users reason to expect their high-quality content gets more eyeballs (particularly the eyeballs which most need that specific content) seem core and essential.
Andrea Gallagher
This is great. I think my next core argument needs to be for why argument structuring is more than an optional bonus, and the user goal you frame up helps a ton to focus my case.
Alexei Andreev
My main point of disagreement is that it's sometimes hard to say if some evidence is "for" or "against." It might also be subjective. Or it might be ___, for which I don't have a word, but an example would be General Relativity as it relates to Newtonian Physics.
Andrea Gallagher
I imagine you could propose evidence as "for" or "against", and then the discussion steps down a level to the quality and direction of the evidence.
This would all need to be hierarchical, and what I call "evidence" may really be a "sub-claim".
Alexei Andreev
Even if the platform had pros and cons, you'd still need to decide for any given claim whether some piece of evidence is a pro or a con and by how much. I'm not even sure what a good solution to that might look like that isn't basically a reputation system.
Andrea Gallagher
Again I think I erred in including "reputation system" in the claim. I was trying to draw a distinction between voting up or down on a claim/comment/post and between providing evidence for or against.
While I'd like to think it could be evidence all the way down, in reality you would need voting at least at the leaf nodes. And the voting is probably useful at each level.
I wasn't trying to make it exclusive, rather that conversational structures to support things like evidence, cruxes, and tests is an unexplored area in the field, while voting systems are both well known and hard to perfect. So the ROI feels higher for the first.
Paul Crowley
I think that Scott's essays The Control Group Is Out Of Control (particularly the idea of the Ouroboros of Scientific Evidence) and Why I Am Not Rene Descartes are relevant here. In difficult cases, assessing evidence properly is a matter for intricate discussion.
Andrea Gallagher
Thanks for those links. My view is partly inspired by the first post, and the second is new to me. (The bit about Descartes' dirty hands is pure Scott).
I think the value of structured discussion tools is driven by that need for an intricate discussion to asses evidence. Today, discussion platforms don't have methods of weaving arguments together, so it becomes too hard to follow a debate with detailed and nested evidence.
Instead, the crowd throws up it's collective hands in defeat and "trusts the scientists", assuming that any given study is the best and only way to get an answer. But within scientific literature, it's still very hard to trace the crux of a theory and the quality of evidence for it.
Scott is a master at the science literature tear-down, but he's not sufficiently scalable. I'm hoping that by baking some of the best logical structure and process into the tools, we can make the crowd just a bit smarter.
Andrea Gallagher
Thanks for picking apart my claim, folks! Rather than modify this claim, I think I'll work on a Post approach, probably with a few different linked claims in it.
In retrospect, I don't like the value comparison structure of the claim.
I'm used to formulating pseudo-hypotheses in a way that feel testable to me, and relative comparison can be easier than picking and measuring some absolute value. And I do think that in any project each effort is traded off against other efforts. But the claim focused as much on the value of reputation systems, rather than the value of structured discussions, and that muddied the discussion.
Since I'm using this to help get a feel for how good claims, posts, etc are crafted, this was very helpful.