Survey redux: bad science make Brynn angry!

[brynndragon]

I referred to a survey the other day that appeared in various polyamory-related communities on Livejournal (example, another example, a third example). It's actually even worse that is seems: the survey hasn't been reviewed by either of the groups mentioned in the survey (which was said repeatedly in the comments on the links above).

From The Kinsey Institute's communications director: "Just to clarify, this is not a project that is sanctioned or affiliated with The Kinsey Institute. Our researchers were not consulted regarding this study.
The Kinsey Institute library accepts donations of materials from the public that will help to shed light on sexual behavior and the human experience. We do have a collection of materials on polyamory, and welcome contributions to that collection. We do not, however, collaborate or support research that is conducted without the protection of a human subjects committee or institutional review board."

From the coordinator of The Community-Academic Consortium for Research on Alternative Sexualities Research Support Program: "CARAS has no involvement in this project. It was not submitted for review by the CARAS Research Advisory Committee, a process that is described on the CARAS website: https://www.caras.ws/index.php/researchsupport/index/"

(The link in that last quote? Includes a process by which someone with no academic affiliations can find mentors and reviewers for doing exactly this work in an ethical and scientifically valid manner. Which makes the whole thing doubly upsetting - it's like the surveyor found the organization through which they could do this properly but for whatever reason didn't actually do so.)

I advise any and all people who consider taking the survey to not do so, and any who did take the survey to request that any information on you (including your answers and IP address) that was recorded by Surveymonkey be permanently deleted immediately. This is bad science masquerading as useful research that's pretending it's merely community outreach. I have no idea what joreth thinks they're doing, but it's certainly not to the benefit of the poly community.

As for the moral of the story: if you want to do amateur science, go with the hard sciences. That way you'll probably just blow yourself up.

Comments

[zandperl.livejournal.com]

So what's wrong with someone who's just curious about the demographics of polyamory doing an informal survey? Or are you thinking that it's some evangelical Christian group who's going to twist the results to their end?

[benndragon.livejournal.com]

I'm not so concerned it'll fall into the hands of some group that hates poly folk. What I'm concerned about is someone who actually cares about poly folk taking the results from this thing seriously when it neither protects its subjects nor offers real information. After all, it's intended to be used in exactly that fashion (from the survey: "The results will be going to the Kinsey Institute and made available to CARAS and any other academic group or organization for the purposes of directing the focus of their research.").

If that line isn't enough to convince you this isn't simple curiosity, here are some comments made by the author: "I accepted feedback from researchers and academics during the review stage before the survey was posted" (source); "Yes I consulted other people, including members of the Kinsey Institute and CARAS" (source). That crosses the line from casual to serious, certainly enough to warrant, at the very least, some basic protections for those subjects such as immediate destruction of data upon request and name(s) and professional contact information of anyone gathering data (i.e. those who will take responsibility for what happens with the data).

FWIW, I would be totally fine with someone tossing together a poll/survey and asking people to answer purely out of curiosity or personal edification. I see that sort of thing all the time and participate in them pretty frequently just for fun. I hope I've demonstrated that this particular survey is nothing like those casual questions.

[zandperl.livejournal.com]

Do you have that little faith in research scientists to think that Kinsey and CARAS would actually use a data set this illegitimate? A corrupted data set could pique their interest, but they would then have to go retake the data themselves.

[benndragon.livejournal.com]

I am concerned about "any other academic group or organization" who might get the results without being aware of the poor methodology used in arriving at them due to not seeing the survey itself or due to being misled by the implication that KI and CARAS are somehow involved. As for KI and CARAS themselves, my concerns are different: it is possible that other research scientists will see this survey and think that KI/CARAS are associated with it, which would reflect very poorly on those institutions due to its sloppiness and lack of basic human subject standards. That's part of why I brought it to their attention in the first place.

My other concerns include the human subjects whose information may be used against them (e.g. outing via IP logging) with no recourse. If you would not consider it sufficient that I care about the well-being of people who participate in such research, I also find it in my interests that lay people be able to discern good quality human research from poor quality human research *before* it bites them in the ass, causing them and members of their social network to decide to not participate in research in the future. Which leads into another concern: human research already has a bad enough name, it doesn't need to be dragged further through the muck by well-meaning yet irresponsible pseudo-researchers. Finally, badly done science is a pet peeve of mine, right up there with people who use "schizophrenic" when they mean Multiple Personality Disorder.

[metahacker.livejournal.com]

Ngrh.

I think it's time for my post on "Surveys CANNOT be used to gather data; they can only be used to provide evidence for or against a hypothesis".

[benndragon.livejournal.com]

I'd totally read that.

[earthdragon.livejournal.com]

I've seen the same person posting in yet another online venue, and playing the word games of not claiming to be affiliated, but not willing to admit that mentioning names creates that implication. Also still making the claim of being surprised that people are taking things the wrong way and getting defensive there too.

Anyone who is surprised when they get critiqued multiple times for the same issues needs a visit with a clue-by-four.

[benndragon.livejournal.com]

Hmm. I'm remembering this comment about not being liked by some poly folks, as well as multiple references to not liking the hostility of LJ communities. I'm thinking this is not their first encounter with the clue-by-four, but they have decided it is a sign of personal dislike rather than a reaction to bad behavior (like a weird sort of fundamental attribution error).

[earthdragon.livejournal.com]

Well, I did see a claim that was 'will apologize if you convince me I made an actual mistake, but not if I have offended you' which strikes me as a sign of someone who does not really care about how their actions are perceived.