Friday, August 11, 2017

MTurk tips

2 easy hacks to make MTurk/Web/Qualtrics data collection/management easier:

1. isn't actually a hack, just a recommendation to send users to an experiment that is either hosted on your own server or goes through your own server as a landing page (before redirecting to qualtrics or whatever). This serves a couple of purposes. You have complete control and complete records of absolutely every interaction anyone has with the website. You can see the IP and entry point and time of everyone who loads the site, you can record exactly when and where they click, you can manage random assignment to treatment groups yourself in a way that keeps samples balanced or meets any other constraint you have, you can see how many people open the survey and decide not to do it, etc. I also send users back to my server (with an automatic redirect from qualtrics) so I can mark them off as having completed the study.

2. is a hack assuming you do #1: make sure your server keeps all access logs from a long enough time span that at the end of the study you can keep a full record of all interaction. I cannot tell you how many times this data has been useful to have. With the timestamps and IPs I can tell when someone in the lab changed computers because he entered the wrong url and got an error on his first computer, I can tell you who used their phone, who restarted, who finished but just didn't click "submit" at the end, etc etc etc. There are always a few mystery people in the data who didn't do things the right way or had technical issues and I've always been able to track down what happened this way.

3. is a hack assuming you do #1 and #2 in combination with using qualtrics (or any similar third-party platform). Instead of hosting photos or other imported media on the third-party platform, host them on your own server and load them via url on the other platform. Every access of this kind will show up in the access logs, giving you better timing information and IP tracking than those platforms will usually tell you. Even qualtrics, which is phenomenal and and will give you timing information for every page of every survey, only records the time of the first and last click, but if the page is loaded and immediately exited you won't be able to tell, and if something is clicked that opens a sub-question and then closed to avoid the sub-question you won't be able to tell, and if someone fails a verification check and has to re-do a page you can't tell, and so on and so forth. You can make hacks of this kind arbitrarily fancy with custom code.

No comments: