This document contains information for reviewers, requesters, and other users of Turkopticon. It addresses:
This document is a rewrite of the 2014 FAQ, which it replaces.
1. Turkopticon, people, and background
1.1. What is Turkopticon?
1.2. Why should Turkers have a voice?
1.3. Who organizes Turkopticon?
1.4. Who has been involved with Turkopticon in the past?
1.5. Is Turkopticon associated with Amazon?
1.6. Why was Turkopticon built?
1.7. Can I give a donation?
2. Reviews, flags, and comments
2.1. How can I get the Turkopticon Script?
2.2. What are the requester "attributes" ("comm", "pay", "fair", and "fast")?
2.3. How do I post a review?
2.4. I am not getting the verification email.
2.5. How can I write a good review?
2.6. Can I delete my review?
2.7. What is a comment and how can I post one?
2.8. I asked for commenting over a week ago, but I still can't comment and I haven't heard back.
2.9. Can I delete my comment?
2.10. What is a flag and how can I post one?
2.11. Can I delete my flag?
2.12. What are appropriate uses of flags and comments?
2.13. I am a requester. Can you delete or censor a review of me?
3. Miscellaneous technical issues
3.1. My display name is based on my email address (e.g., s...@g...). How do I change it?
3.2. Can I delete my account?
3.3. Does Turkopticon collect personalized data about me?
3.4. I am a requester and I changed my name on Mechanical Turk. Can you change my name on Turkopticon?
4. The Turkopticon application programming interface (API)
4.1. What is the Turkopticon API?
4.2. Why does it exist?
4.3. How do I use it?
4.4. Do I need permission to use it?
4.5. Do I need permission to use it?
4.6. Can I see the code?
5. Known issues and future plans
6. Privacy
6.1. I am a requester. A reviewer posted an email I sent them in a review. Can you remove it?
7. Information for researchers
7.1. I am a researcher. Can I use average requester data from Turkopticon?
7.2. I am a researcher. Can you give me anonymized data from the Turkopticon database?
7.3. I am a researcher. Can I run a field experiment on Turkopticon?
Turkopticon used to be a volunteer run software project and now it is a worker-run organizing and advocacy project, and software is one of the things we do.
Have you ever had an issue with an Mturk requester? If you have, then you know that some experiences are not ideal and leaving a review is not enough. We know one voice alone doesn’t usually make a difference. When we unite those voices, the squeaky wheel gets the oil. This is where Turkopticon comes in and why we need to build the voices for Turkers. We need to stand up and acknowledge the issues and work together to make Mturk better for us all.
There is lots Turkers don’t agree on. But there’s one thing we do agree on. MTurk could be better. That’s why we all need to be in the room. We need your voice to show that Turkers do care and your unique experience needs to be heard by other Turkers. With unique voices, we can find ways of making Turk better that truly help us all.
The committee is made up of Alex, Ashley, Brook, Krystal, Clearblue, Phil, Krista, with other dedicated Turk workers.
Turkopticon is maintained by the committee and volunteers from Tech Workers Coalition.
Turkopticon was founded by Lilly Irani and Six Silberman when they were kids. Lilly is now a professor at the University of California, San Diego. Six is an engineer based in London.
Turkopticon was founded by students at a university, Lilly Irani and Six Silberman. Lilly and Six had worked as programmers and wanted to support Turkers in finding better work. Along the way they got help from Turkers, from people who worked with crowdsourcing, students, and from moderators who were Turkers and the daily lifeblood of the community.
If you want more details, please refer to the 2014 FAQ 7.2.
No. It is not and it has never been.
Turkopticon started in 2008 as a class project to raise awareness about the unaccountability of Mechanical Turk requesters. We saw requesters and researchers celebrating how cheaply they could get work done on Mechanical Turk and we felt this was wrong. We didn't understand why Amazon didn't build in a requester reputation system, like Amazon Marketplace and eBay have. Building Turkopticon was our way of making this point. We never expected it to be used by so many people, or to become a fixture in the Turker community.
Yes! This will ensure that we can continue to provide a worker run review site where we focus on the human part of reviews. Turkers and requesters understand how to make things better.
You can donate to the project at this link.
If you cannot make a monetary donation we are always looking for volunteers to make Turk better for us all. If you are interested in building a voice for Turkers, please contact us at volunteers@turkopticon.net.
In order to run the script, you will need a userscript manager such as Tampermonkey or Greasemonkey
Once you have a userscript manager, you can grab the latest version of the turkopticon userscript here <>
Note: The generosity attribute is being replaced by pay/hr.
The "attributes" used to describe requesters answer the following questions:
We have added a pay/hr attribute in place of the 1-5 scale rating for pay/generosity. The 1-5 rating system doesn't take into account workers being spread across the globe and other various influences that affect judgement of pay. We hope this will help workers better determine hits they want to do, and remove the frustration some workers were experiencing with the pay ratings in the past.
The easiest way to post a review is to click the link "Report your experience with this requester" in the add-on drop-down.
However, you can also review any requester by going to turkopticon.ucsd.edu/report (Turkopticon login required). You'll need to know their Mechanical Turk requester ID, though (e.g., A1BC2DE34FGHJ5). If you don't include a Mechanical Turk requester ID, your ratings won't be included in the average ratings shown by the add-on.
This is probably because of a problem with our mail sending system. Fixing this is on our list of things to do. In the meantime, email us at moderation@turkopticon.net from the email address associated with your Turkopticon account and we'll verify it manually.
Give the ratings you feel best describe your experience.
You may wish to put the name(s) of the HIT(s) you are reviewing in the "HIT Name(s)" field.
If one of the fields is not relevant, leave it blank.
For example, if you didn't need to communicate with the requester, leave that field blank.
Don't include or link to profanity, racial slurs, or ethnic slurs.
Don't personally insult or attack requesters or other workers. Calling a requester "cheap" is okay, but anything stronger is subject to being hidden.
Don't include personal information about the requester unless it is part of the HIT.
Don't write your review in ALL CAPS.
Don't include screening information, information about attention checks, or completion codes.
If you're a requester, don't review yourself. Comment instead.
Reviews that are profane or insulting or appear to be requester self-reviews may be hidden.
The most up-to-date version of these guidelines will always be posted on the review page (Turkopticon login required).
For now, no.
If you managed to post a review of the same requester twice (i.e., there was a technical error), email us and we will delete the duplicate, as long as nobody else has flagged or commented on it yet.
The main reason we don't allow deletion of reviews is accountability. The world around Mechanical Turk, including Turkopticon, is generally one of mistrust and suspicion. If we allowed untraceable deletion of reviews, people might worry that we were deleting reviews at the request of requesters. We don't do that, and we don't want to jeopardize the usefulness of Turkopticon by giving anyone any reason to imagine that we might. So we don't delete any reviews, even at the request of the author. (Duplicates are the only exception.)
A second reason is that if we were to delete a review that had comments or flags on it, those comments or flags would disappear. This could be confusing, and confusion causes frustration and suspicion, which we try to avoid.
Sometimes people post a review they later regret posting. This often happens when other people flag the review or leave mean comments. The review author often wants to delete the review in order to delete the comments or flags. In our view this is unfair to the comment authors or flaggers, even if they said mean things. We do not have a policy of censoring something if it is merely mean—only if it is profane. In the long run, usually the person who said the mean thing is more embarrassed than the person they said it to. If you are in this situation, you can always edit your review!
All that said, in the future, we may allow review authors to delete their own reviews. If we do this, we will find a way to keep a record that the review existed, and keep comments and flags accessible. For now, if you have the urge to delete a review you posted, the best thing to do is edit it instead, or leave a comment under it.
A comment is a piece of text that responds to a review. (See figure at right.) You post a comment "on" a review.
New Turkopticon accounts cannot post comments. Once you have posted some reviews, you can request commenting (Turkopticon login required) by clicking on the previous link or on the link that says "request commenting ability" that appears between the "settings" and "logout" links at the top of every Turkopticon page while you are logged in.
Commenting requests are reviewed manually, usually within a few days, by our administrators. If you have only posted a few reviews, please expect to be told that you will need to wait to be able to comment.
If you are a requester and you want to post comments in reply to reviews, please email us.
If your comments and/or reviews have been the subject of heated community discussion in the past (for example, you have been involved in flame wars with other people on the site, or have repeatedly posted comments that were hidden for profanity or racism), your commenting request may have been ignored.
For now, no. You can edit it, though.
A flag is a sign for moderators to consider hiding a review that may violate Turkopticon's civility guidelines (see above, Section 1.5, "How can I write a good review?"). If a moderator agrees that the review violates civility guidelines or is otherwise inappropriate, the moderator will also flag the review. If a review has at least two flags, and at least one is from a moderator, the review is hidden. Ratings from hidden reviews are not included in the average ratings shown by the add-on.
Only people who can comment can flag reviews. (See above, Section 2.7, "What is a comment and how can I post one?" for information on commenting and getting access to commenting.)
Also, if a comment on a review violates civility guidelines, please don't flag the review. Post to turkopticon-discuss instead.
For now, no. You can edit it or convert it to a comment, though.
Flag a review if it:
In general, flags are for reviews that violate our civility guidelines or threaten the integrity of the review database. If you simply disagree with a reviewer, but believe they are reviewing with good intent, even if you think they are reviewing "wrongly", comment instead. Flags, generally, are for reviews which appear to have some sort of malicious intent.
Moderators and administrators may ignore flags at their discretion.
If you're not sure if you should flag or comment, comment.
As in reviewing (see above, Section 2.5, How can I write a good review?), when commenting, please:
Comments violating these guidelines may be edited by administrators or moderators.
Delete? No. (See above, Section 2.6, "Can I delete my review?")
Censor? Probably not.
With the exception of technical errors (e.g., a review posted twice), we do not delete reviews, even at the request of the author.
In general, requesters are encouraged to comment on reviews rather than flag. But if a review clearly falls within the flagging guidelines (see above, Section 2.12, "What are appropriate uses of flags and comments?"), feel free to flag it. If a moderator agrees that the review violates Turkopticon's civility guidelines (see above, Section 2.5, "How can I write a good review?"), they will also flag it, and the review will be hidden. The ratings included in hidden reviews are not included in the averages displayed by the add-on. (For more on flagging, see above, Sections 2.10–2.12.)
Personal information about requesters:If personal information about a requester is available in the HIT (for example in a consent form workers must read before doing a HIT as part of an academic study), we allow that information to be posted in a review.
We reserve the right to remove or censor other personal information, but we do not guarantee that we will do so, even at the requester's request.
In general, once your display name is set, it is supposed to stay as it is. But if there was a mistake and somehow your display name was set to the default display name (which is based on the email address you used to sign up for Turkopticon), email us and we will "unset" it, after which you can pick a different display name.
If you already have a "real" display name, we will not unset it. The point of this is to create a little bit of accountability.
For now, the best we can do is "close" your account.
We realize this can be frustrating. "I own all of my reviews!", you might say, "I have a right to delete them!" In principle, we agree. Unfortunately there are technical issues. The main one is that if we actually deleted your account—deleted the entry in the Turkopticon database that represents your account—we would have a bunch of reviews left over associated with a nonexistent account. When the web site tried to display those reviews, it would choke. "So delete them!", you might say. If nobody else had commented on or flagged any of your reviews, we could do that without any technical issues—although we would still have the accountability issues discussed in Section 2.6, "Can I delete my review?". But this is not usually the case. And if we delete a review that someone else has commented on, we are effectively disappearing their comment, which is not fair to them (see above, Section 2.6, "Can I delete my review?", for more on this topic).
So, for now, the best we can do is "close" your account. A closed account cannot be logged into. In the future it is possible we will hide reviews from closed accounts in the same way we hide reviews that violate civility guidelines, although for the moment we see no real need to do this.
As of 5 August 2013, we have not closed many accounts (in fact we have closed exactly one), so our policy on this topic is not very well developed. We don't understand fully why people are motivated to delete their accounts, so we don't know exactly what else we should do when an account is closed. We are open to discussing ways to improve how we handle it.
To close your account, email us. In the future, we may set up a way to do it automatically.
No.
A version of Turkopticon posted in 2010–2011 did collect anonymized data automatically. This code was written by an economist who wanted to make actual wage data available to Turkers, but it was removed because of privacy concerns.
Yes. Email us.
The Turkopticon API is a PHP file that accepts Mechanical Turk requester IDs and returns the information about those requesters available in the Turkopticon database. Specifically, for each requester ID, it returns the requester's name, average ratings for the four requester attributes ("comm", "pay" (this may be zero for new requesters with reviews in lieu of the pay/hr), "fair", and "fast"; see Section 2.2, "What are the requester 'attributes'?"), pay/hr ("pay_bucket"), the number of reviews for the requester in the Turkopticon database (excluding hidden reviews), and the number of those reviews that reported that the requester violated the Mechanical Turk Terms of Service.
This information is returned as a JSON object [wikipedia entry; example JSON object; example from Turkopticon API].
It is used by the Turkopticon user script and browser add-ons to quickly retrieve data about requesters from the Turkopticon database.
Call
https://turkopticon.net/api/multi-attrs.php?ids=foo,bar,baz,batwhere foo, bar, baz, and bat are the Mechanical Turk IDs of the requesters you want information for. For example, calling
https://turkopticon.net/api/multi-attrs.php?ids=A35VLPB12A0TTY,A3VTXG9MRA65HP,A3OO8KUVWDSZPVreturns information about the requesters IMP AMT and rohzit0d, and CrowdSource.
You may pass as many IDs as you like to multi-attrs.php. It has not been tested with more than a hundred or so, however. Calls with hundreds of IDs or more may be slow, or fail entirely.
No. However, if you plan to use it on an ongoing basis, please also tell us, so we can warn you if we plan to make any changes that may break your application. Also, if you use it a lot over a short period of time, the server may crash. This will make your application fail and cause problems for other users. Please ask us before doing this so we can keep an eye on server load.
Yes. If you call the API more than once a second, you will receive an error message.
Yes. It is on GitHub here. It is licensed under the GNU Affero General Public License.
The software team has been recently reinvigorated with fresh volunteers, and the team is working on improving Turkopticon, both with features and under the hood.
Almost certainly no.
While you may feel that emails you send to workers are implicitly private, such emails are in our view not 'personal communication' but 'business communication'. While requesters are not employers in the legal sense, requesters are in a position of power over workers by virtue of making payment decisions that may have significant practical consequences. Therefore if a worker feels threatened, intimidated, condescended to, or even just frustrated by an email sent by a requester, we feel it is reasonable for them to be able to post it publicly in a place where other workers or potential workers can see it. We may not always agree that it is a good idea for a worker to do so, but in general we will not censor reviews or comments that include emails sent to workers by requesters.
There are three main exceptions:
Yes. You may use the publicly-accessible Turkopticon API (see above, Section 3).
If you publish or present anything using this data, or if you discover something interesting about it that we might not know, please tell us!
If you plan to use the API on an ongoing basis, please also tell us, so we can warn you if we plan to make any changes that may break your application.
We have not done this so far, although we might be willing to consider it. We would consult publicly with workers first, so if you want to make a request like this, please ask well in advance of any deadlines you may have.
We recommend strongly that you do not do this, even if you think you can do it undetected. A previous field experiment that we were unaware of was discovered by workers while in progress and resulted in formal IRB complaints to the researchers' home institution and severe stress and large amounts of lost time, money, sleep, and effort for the researchers.
While Turkopticon was built by, and continues to be operated partly by, researchers, we do not use Turkopticon data to produce research. We have published papers about Turkopticon, but we do not publish papers using Turkopticon data. Use of Turkopticon is not implicit consent to being a research subject. Turkopticon's primary research contribution is to help imagine more responsible approaches to the design and operation of computing systems. Thus Turkopticon's first responsibility is to the community of workers that relies on it.