- Joined
- Jan 10, 2016
- Messages
- 4,862
- Reaction score
- 13,275
- Points
- 863
- Gender
- Female
Working on Amazon Mechanical Turk can be both underwhelming and overwhelming at first. Underwhelming in terms of your earning potential, and overwhelming logistically. If you persevere, however, it can become a pretty reliable source of income.
Notice that I said become. If you come to mturk expecting high hourly earnings right away, you will almost certainly be disappointed. Think of your first 1000 HITs as a training period. Have realistic goals. It takes time to learn how the site works, what work is available to you, how to find work, and how to optimize your browser to maximize your time and work as efficiently as possible. You are already doing something very right by signing up on this site and exploring the educational resources available here.
This thread is broken down into (hopefully) easily digestible sections about the mturk site itself, turking tools, and the knowledge you will need to succeed in your first 1000 HITs and beyond. Welcome!
What is Amazon Mechanical Turk (mturk)?
Turking Scripts
Before You Start: The Dashboard
Before You Start: Qualifications
The Probation Period (Your First 10 Days)
On Your Way
500!
Surveys vs. Batches
1000!
Keeping Good Records
More About: Qualifications -- Closed Quals and Why You Should Always Be Qualing
After Your First 1000 HITs: A Word on Hourly Rate
Welcome to the grind! I hope this information helps you understand what turking is about, and whether it is a good fit for your abilities and aspirations. When you get to 1000 approved, please post in that day's daily work thread so that we can celebrate with you! And good luck!
Copyright jklmnop 2015-2016. All rights reserved.
Notice that I said become. If you come to mturk expecting high hourly earnings right away, you will almost certainly be disappointed. Think of your first 1000 HITs as a training period. Have realistic goals. It takes time to learn how the site works, what work is available to you, how to find work, and how to optimize your browser to maximize your time and work as efficiently as possible. You are already doing something very right by signing up on this site and exploring the educational resources available here.
This thread is broken down into (hopefully) easily digestible sections about the mturk site itself, turking tools, and the knowledge you will need to succeed in your first 1000 HITs and beyond. Welcome!
What is Amazon Mechanical Turk (mturk)?
Amazon Mechanical Turk (mturk) is a marketplace for work hosted by Amazon. It is also known as a crowdsourcing platform. Other crowdsource work platforms exist, but mturk is arguably the largest and most well-known.
Now, if you've spent any time on the mturk website, you've probably noticed that the site does not make any serious attempt to guide workers through the process or help them work productively. Amazon takes a very hands-off approach to the site, so at first glance "turking" can seem very daunting and maybe even a waste of time. That's presumably why you're here, and it's also why we're here and why this thread exists.
Now, if you've spent any time on the mturk website, you've probably noticed that the site does not make any serious attempt to guide workers through the process or help them work productively. Amazon takes a very hands-off approach to the site, so at first glance "turking" can seem very daunting and maybe even a waste of time. That's presumably why you're here, and it's also why we're here and why this thread exists.
Turking Scripts
Much of the worker side of mturk has not been updated in several years. You can fix this to some extent and increase ease-of-use and productivity by installing some basic turking tools and scripts. Depending on your computer proficiency, motivation, and goals, you may want to install just the essentials or go all out.
As turking tools have matured, the mturk work marketplace has become more competitive. When I posted the first version of this article in April 2014, several tools now in daily use barely existed then. I asked newer workers for one or two things they know now that they wish they'd known when starting out. Every single one of them said some variation on "I wish I'd installed/learned how to use scripts and tools sooner."
At a minimum you should install the following browser extensions and scripts before you start turking (or ASAP if you've already started):
As turking tools have matured, the mturk work marketplace has become more competitive. When I posted the first version of this article in April 2014, several tools now in daily use barely existed then. I asked newer workers for one or two things they know now that they wish they'd known when starting out. Every single one of them said some variation on "I wish I'd installed/learned how to use scripts and tools sooner."
At a minimum you should install the following browser extensions and scripts before you start turking (or ASAP if you've already started):
- Tampermonkey extension (Chrome) or Greasemonkey extension (Firefox). These tools allow you to install small scripts that improve your turking experience.
- Turkopticon script (works with either of the above). Turkopticon, aka TO, is a feedback/review system for workers. The quality of work and pay on mturk varies wildly. Get into the habit of checking TO early on in your turking career (and add your own reviews, too!) TO helps workers determine whether the work is worth our time. (NOTE: There is also a TO browser extension available, but the script appears to operate faster, is updated first when changes are made, and is more easy to enable/disable at will.)
- A page refreshing solution. Page refreshing is predominantly used to grab HITs that are in high demand. The use of rapid page refreshing for this purpose is known as PandA, short for "previewandaccept."
Once you are comfortable with the above, then consider installing Turkmaster, Hitscraper, Hit Monitor, and AutoHotKey. All four are extremely useful tools.
Before You Start: The Dashboard
There are a few stats worth pointing out on the mturk dashboard. The most relevant numbers to this guide are your HITs Submitted, Approved, Rejected, and Pending.
Each time you submit a HIT, you should get a message on the next page stating, "Your results have been submitted to 'requester name' and will be approved or rejected shortly." Then that HIT will be added to your HITs Submitted count. As requesters go through and approve your HITs, or as they are auto-approved (also known as "AA"), your accepted HIT count will increase and your pending HITs will decrease.
If one or more of your HITs are rejected, the HIT(s) will be counted in the HITs Rejected row and you will not be paid for that work. Just about everyone has had HITs rejected at some point. Your first rejection will suck, it will hurt and you might even get upset, but we've all been there. Depending on the situation, you may have some recourse, but that's outside of the scope of this guide. (If you have a question about a rejected HIT, feel free to bring it up in the latest Awesome HITs thread.)
The reason I bring up rejections now is because rejections tie into your account's approval percentage. Most requesters require a high approval percentage (98% or even 99% of HITs approved) to work on their tasks. Right now you want to pay close attention to any mention of requesters who reject and, frankly, avoid them. Rejections will hurt an account that has <100 approved much more than someone with 10K+ approved. Early rejections will restrict your options to push forward and get to the better paying work. You should be very hesitant about doing HITs from requesters who are known to reject until you have at least 1000 HITs approved. As you build your numbers you can start to manage risk a bit differently.
Each time you submit a HIT, you should get a message on the next page stating, "Your results have been submitted to 'requester name' and will be approved or rejected shortly." Then that HIT will be added to your HITs Submitted count. As requesters go through and approve your HITs, or as they are auto-approved (also known as "AA"), your accepted HIT count will increase and your pending HITs will decrease.
If one or more of your HITs are rejected, the HIT(s) will be counted in the HITs Rejected row and you will not be paid for that work. Just about everyone has had HITs rejected at some point. Your first rejection will suck, it will hurt and you might even get upset, but we've all been there. Depending on the situation, you may have some recourse, but that's outside of the scope of this guide. (If you have a question about a rejected HIT, feel free to bring it up in the latest Awesome HITs thread.)
The reason I bring up rejections now is because rejections tie into your account's approval percentage. Most requesters require a high approval percentage (98% or even 99% of HITs approved) to work on their tasks. Right now you want to pay close attention to any mention of requesters who reject and, frankly, avoid them. Rejections will hurt an account that has <100 approved much more than someone with 10K+ approved. Early rejections will restrict your options to push forward and get to the better paying work. You should be very hesitant about doing HITs from requesters who are known to reject until you have at least 1000 HITs approved. As you build your numbers you can start to manage risk a bit differently.
Before You Start: Qualifications
At the most basic level, qualifications are settings on HITs that allow some workers to access and do the work while excluding others. If you log on to mturk right now and go to the most basic HITs listing page, you may notice that there are a lot of HITs you cannot do. That is because some HITs have special qualifications, or "quals" set on the batch so that only specific workers may access and do the work. Other HITs may have no qualification parameters whatever. You can see the Qualifications Required
for any given HIT by clicking on the HIT name (top left for each listing).
Most work is qualified by the number of HITs you have or your approval percentage. The default qualification setting on the requester/employer side is 1000 HITs approved. That is why this thread is designed to help you work toward that most common milestone rather than 100 or 500 or 5000 approved . You can always see what HITs you are qualified for here.
Mturk qualifications may be obtained in a few different ways. Some quals open to everyone may be accessed directly from the Qualifications tab on mturk. For example, if you are okay with adult content, then you can and should obtain the site-wide Adult Content Qualification right now by going here and requesting the qualification from the third listing from the top (the one with over 200,000 qualified users). The standard qual value is 1.
Other qualifications require you to take a test for the qual, and then the test is either graded on the spot by software or sent to the requester for review. If you'd like to grab another qual right away, go back to your Qualifications tab and search for "venue quality" -- only one result should pop up. Take the qual test (qual should be auto-generate with a value of 80).
A third way requesters screen workers is to have us perform qual HITs (whether a single HIT or a batch of HITs). Qual HITs can be the actual work you will be doing, or a demographic screener, or some combination thereof. Depending on the requester, after you do a qual HIT(s), you might find out about your score right away, or you might get an email, or you may just have to periodically check your Qualifications tab. (Some, but certainly not all, requesters will notify workers of new quals by the email associated with your mturk account.)
for any given HIT by clicking on the HIT name (top left for each listing).
Most work is qualified by the number of HITs you have or your approval percentage. The default qualification setting on the requester/employer side is 1000 HITs approved. That is why this thread is designed to help you work toward that most common milestone rather than 100 or 500 or 5000 approved . You can always see what HITs you are qualified for here.
Mturk qualifications may be obtained in a few different ways. Some quals open to everyone may be accessed directly from the Qualifications tab on mturk. For example, if you are okay with adult content, then you can and should obtain the site-wide Adult Content Qualification right now by going here and requesting the qualification from the third listing from the top (the one with over 200,000 qualified users). The standard qual value is 1.
Other qualifications require you to take a test for the qual, and then the test is either graded on the spot by software or sent to the requester for review. If you'd like to grab another qual right away, go back to your Qualifications tab and search for "venue quality" -- only one result should pop up. Take the qual test (qual should be auto-generate with a value of 80).
A third way requesters screen workers is to have us perform qual HITs (whether a single HIT or a batch of HITs). Qual HITs can be the actual work you will be doing, or a demographic screener, or some combination thereof. Depending on the requester, after you do a qual HIT(s), you might find out about your score right away, or you might get an email, or you may just have to periodically check your Qualifications tab. (Some, but certainly not all, requesters will notify workers of new quals by the email associated with your mturk account.)
The Probation Period (Your First 10 Days)
You can accept 100 HITs per day the first 10 days you work on mturk following signup. After the 10 day introduction period, the floodgates open and you can work on as many HITs as you like, up to a maximum of 3800 HITs per day (including returns). The work day on mturk begins at midnight Pacific Standard Time.
For your first 1000 HITs you may want to concentrate on approval milestones rather than $$$. It's not that the money is not important, but most of the better-paying requesters require 1000/5000/10000+ approved HITs to do their surveys or work on their batches.
A common term you'll see mentioned, especially as regards newer workers and accumulating approved HITs, is "numbers" HITs. Numbers HITs are typically penny or otherwise low-paying HITs that offer terrible pay and usually mind-numbing tasks BUT also tend to approve quickly and are never or very rarely rejected. You can find safe, reliable numbers HITs in the Numbers HITs thread, and you will also find some mentioned in the daily work threads from time to time. Be sure to read any helpful text on these regarding approvals/rejections!
If you've already decided that you're going to take turking seriously, then focus on maxing out your probationary period to get to 1000 approved ASAP. Otherwise, work according to your preferred pace. While you're amassing the approved HITs, keep checking for numbers HITs and check the daily work thread for HITs that you are able to do.
Daily work threads serve as both an information source and a virtual water cooler for Mturk Crowd members. Whether new turker or old hand, everyone is welcome!
Get into the habit of occasionally checking your HITs Assigned To You tab to make sure you haven't forgotten about any HITs (remember, returned and expired HITs count towards your 100 max in the first 10 days). Returned and abandoned (expired) HITs are still recorded by the mturk system, but they have little impact on your overall numbers; very few requesters have ever used these metrics for qualifications.
Pay attention to what people say about Turkopticon ratings, i.e., "good TO" or "bad TO," so as to avoid rejections this early on. Ask questions. Everyone who knows what they're doing, or sounds like they know what they're doing ? They were a newbie once, and we all realize how confusing it can be to get started. Ask questions!
For your first 1000 HITs you may want to concentrate on approval milestones rather than $$$. It's not that the money is not important, but most of the better-paying requesters require 1000/5000/10000+ approved HITs to do their surveys or work on their batches.
A common term you'll see mentioned, especially as regards newer workers and accumulating approved HITs, is "numbers" HITs. Numbers HITs are typically penny or otherwise low-paying HITs that offer terrible pay and usually mind-numbing tasks BUT also tend to approve quickly and are never or very rarely rejected. You can find safe, reliable numbers HITs in the Numbers HITs thread, and you will also find some mentioned in the daily work threads from time to time. Be sure to read any helpful text on these regarding approvals/rejections!
If you've already decided that you're going to take turking seriously, then focus on maxing out your probationary period to get to 1000 approved ASAP. Otherwise, work according to your preferred pace. While you're amassing the approved HITs, keep checking for numbers HITs and check the daily work thread for HITs that you are able to do.
Daily work threads serve as both an information source and a virtual water cooler for Mturk Crowd members. Whether new turker or old hand, everyone is welcome!
Get into the habit of occasionally checking your HITs Assigned To You tab to make sure you haven't forgotten about any HITs (remember, returned and expired HITs count towards your 100 max in the first 10 days). Returned and abandoned (expired) HITs are still recorded by the mturk system, but they have little impact on your overall numbers; very few requesters have ever used these metrics for qualifications.
Pay attention to what people say about Turkopticon ratings, i.e., "good TO" or "bad TO," so as to avoid rejections this early on. Ask questions. Everyone who knows what they're doing, or sounds like they know what they're doing ? They were a newbie once, and we all realize how confusing it can be to get started. Ask questions!
On Your Way
Start out with reasonable goals. Check TO on any work you do. Do safe numbers HITs from the numbers thread, and look for surveys from known-good requesters. Do Google HITs if they come up. HITs from most Google requesters are the epitome of safe as long as you follow the directions set out in each HIT and don't rush. If you are within your first 10 days, you should be trying your best to max out the HITs available to you each day.
500!
500 approved opens up...more surveys for the most part. (Disappointed? Well, yeah, that's mturk for you. No one said it would be easy!) At this point you should continue to do safe numbers HITs, browse the daily work thread regularly for new HITs that you are eligible to do, and become more familiar with the feedback people post about HITs as well. Spend some more time installing and learning how to use the tools available to you. If you feel comfortable with it, learn how to PandA (but keep a very close eye on your queue! Filling it up with HITs can be overwhelming at first and may lead to more mistakes, stress, and/or expired HITs.) Continue to work, read, and learn.
You might have a received a rejection or five. When you get your first rejection(s), try not to take them personally. Rejections hurt, but sometimes you simply cannot prevent them from happening. Sometimes you just mess up. Sometimes requesters mess up. Your best bet is to keep using TO and keep up with feedback from other workers to avoid bad requesters and HITs. You might try contacting the requester to politely (!) ask what you did wrong in the HIT and see if it can be reversed; a few requesters sometimes reverse rejections when asked.
You might have a received a rejection or five. When you get your first rejection(s), try not to take them personally. Rejections hurt, but sometimes you simply cannot prevent them from happening. Sometimes you just mess up. Sometimes requesters mess up. Your best bet is to keep using TO and keep up with feedback from other workers to avoid bad requesters and HITs. You might try contacting the requester to politely (!) ask what you did wrong in the HIT and see if it can be reversed; a few requesters sometimes reverse rejections when asked.
Surveys vs. Batches
By now you may have tried both surveys and batches (multiple HITs under one name involving the same type of repetitive work). You may even know which work you'd prefer to do day in and day out. Surveys may offer more monetary rewards to you on the way to 1000, while batches of HITs might help you get to 1000 faster.
It is unlikely at this point that you will have access to really great-paying batch work, so you might want to concentrate on surveys on your way to 1000 if maximizing your income is the goal. Batch work can occasionally be more profitable than surveys <1K, but it cannot be overstated that as a newer worker, you want to protect your approval percentage. The easiest way to do that is to stick with HITs (batch, survey, or otherwise) from requesters who basically never reject anyone. Doing 100 HITs for one requester on one batch of work is a far greater risk to your account than doing 100 different surveys from 100 requesters (all of whom you've vetted by checking their TO reviews, right? )
There is a Worst Case Scenario script that will add, well, worst case scenarios to your dashboard in terms of how many rejects would leave you at 98% approved or 95% approved. Consider installing that. And then just keep plugging away to get to 1000 ASAP.
It is unlikely at this point that you will have access to really great-paying batch work, so you might want to concentrate on surveys on your way to 1000 if maximizing your income is the goal. Batch work can occasionally be more profitable than surveys <1K, but it cannot be overstated that as a newer worker, you want to protect your approval percentage. The easiest way to do that is to stick with HITs (batch, survey, or otherwise) from requesters who basically never reject anyone. Doing 100 HITs for one requester on one batch of work is a far greater risk to your account than doing 100 different surveys from 100 requesters (all of whom you've vetted by checking their TO reviews, right? )
There is a Worst Case Scenario script that will add, well, worst case scenarios to your dashboard in terms of how many rejects would leave you at 98% approved or 95% approved. Consider installing that. And then just keep plugging away to get to 1000 ASAP.
1000!
Whether it takes you a couple of weeks or a couple of years, by the time you hit 1000 approved you will have a better idea of how mturk works in general, how the scripts and extensions help you find things to do and work more efficiently, how much time you want to spend turking, and what sort of monetary or time goals you want to set on the way to 5000 approved. You may also find that mturk work is not really for you, and that's fine, too! It is not easy to make steady $$$ from turking. It takes time and perseverance to learn how the platform works, how to find HITs, and how to optimize your workflow with scripts and browser extensions (and hand stretches!)
We all have the same overarching goal for turking ($$$) but different folks have different short- and long-term goals, schedules, and prefer some tasks over others. The advantage of mturk is that it gives you the possibility of earning money in a manner and on a schedule of your choosing. To maximize your earnings, you may occasionally need to work when you'd rather be doing something else, because the HITs are up now but will be gone in an hour. The great part about that, though, is that the choice is yours. Once you've learned the ropes, figure out what works for you in terms of schedule, type of work, and monetary goals. Keep learning and asking questions, too.
You might even find that you know the answer to another new person's question! Feel free to share your experience, but also respect any comments/replies from more experienced turkers. Use any suggestions or feedback as a learning opportunity, because ultimately we are all here to help each other (and share pizza gifs).
We all have the same overarching goal for turking ($$$) but different folks have different short- and long-term goals, schedules, and prefer some tasks over others. The advantage of mturk is that it gives you the possibility of earning money in a manner and on a schedule of your choosing. To maximize your earnings, you may occasionally need to work when you'd rather be doing something else, because the HITs are up now but will be gone in an hour. The great part about that, though, is that the choice is yours. Once you've learned the ropes, figure out what works for you in terms of schedule, type of work, and monetary goals. Keep learning and asking questions, too.
You might even find that you know the answer to another new person's question! Feel free to share your experience, but also respect any comments/replies from more experienced turkers. Use any suggestions or feedback as a learning opportunity, because ultimately we are all here to help each other (and share pizza gifs).
Keeping Good Records
Install some sort of recordkeeping tool, such as HitDB or Export Mturk History, within 45 days from doing your first HIT (or from the date you started turking more seriously if you have an older account). Amazon only keeps your last 45 days of HITs on mturk.com, so you need an external system of some sort to keep a longer history of your work. You may consider using additional spreadsheets or recordkeeping tools to record bonuses or goals.
Money earned from work completed on mturk is considered income and you may be liable for taxes on it. You are responsible for your own federal and state tax filings. No one on this website is a tax professional.
Money earned from work completed on mturk is considered income and you may be liable for taxes on it. You are responsible for your own federal and state tax filings. No one on this website is a tax professional.
More About: Qualifications -- Closed Quals and Why You Should Always Be Qualing
It's worth repeating: as browser tools have matured, the mturk marketplace has become more competitive. "Closed" qual work, i.e., work that is restricted to hundreds or even dozens of workers, is increasingly (though not always!) where you will find the most lucrative work on mturk. Sometimes gaining access to closed qual work comes down to dumb luck, as closed quals are often awarded from working an open batch of qualification HITs that are up on mturk for a limited period of time. You must complete those qual HIT(s) while they are online, and then hopefully get the qual as a result of your quality effort. It's probably wise to note here that nobody can be online 24/7 so there will always be a few quals you miss, others where you luck out, and still others where you do the qual HITs and don't get the qual. It happens to everyone.
Your access to closed qual work should increase over time as you do more sets of qual HITs. Thus my overarching mantra on mturk if you plan to do this for any length of time is always be qualing. I would put that phrase in old-school blinky text if I could. If you see HITs that have a qual test, or if someone in the daily thread mentions a qual test of any sort, take it even if you don't think you would want to do the work. Qualifications on mturk = OPTIONS. The more options for work you have, the steadier your income from mturk can be, and the more flexible your schedule can be. You can make money without any major quals, plenty of turkers are proof of that. But having a variety of quals makes it easier to find work at the times when you want to/can work. Always be qualing.
Your access to closed qual work should increase over time as you do more sets of qual HITs. Thus my overarching mantra on mturk if you plan to do this for any length of time is always be qualing. I would put that phrase in old-school blinky text if I could. If you see HITs that have a qual test, or if someone in the daily thread mentions a qual test of any sort, take it even if you don't think you would want to do the work. Qualifications on mturk = OPTIONS. The more options for work you have, the steadier your income from mturk can be, and the more flexible your schedule can be. You can make money without any major quals, plenty of turkers are proof of that. But having a variety of quals makes it easier to find work at the times when you want to/can work. Always be qualing.
After Your First 1000 HITs: A Word on Hourly Rate
Finally, a word on hourly rate. This is purely my opinion here, some may disagree, but once you reach 1000 HITs approved, while I would still encourage you to continue to do numbers HITs up to at least 5000 approved, I would also suggest that you start to be more selective about the work you do in general. The standard acceptable metric for time-$ on mturk is currently 10 cents per minute. That results in $6/hr (before taxes!), which I guess is okay if you just want to turk while watching TV or something, but let's face it -- that's not even minimum wage. Nobody in the United States can live on $6/hr. And working for such low wages simply will not benefit you in the long run.
Unless you are in extremely dire financial straits (e.g., risk of prosecution/eviction), I strongly suggest that if the only work you can find on mturk pays less than minimum wage, then you are better off spending that time doing something more productive. Your time is valuable, and you can never get it back!
So if all you can find on mturk are crappy HITs with crappy slave wages, then take this opportunity to give yourself a free pass and do something better with your time! Several crowdsourcing platforms exist now and some are quite active, so look for other online work opportunities. Go to EdX and enroll in a free online class. Go to Code Academy and learn how to program. Go to Duolingo and start learning a new language. Go to Khan Academy and pick a subject you're interested in.
Check sites like Free Stuff Finder or the myriad other free stuff sites and look for deals on things you normally buy anyway, and save money that way.
Go to YouTube and learn something new, or learn something new about something you already do. Go make something, or learn how to make something. Even cleaning/organizing your living space or de-stressing with a video game is more beneficial to you in the long run than working for less than minimum wage for a requester that clearly does not care about you, nor the larger world he/she lives in, enough to pay people a reasonable wage. Don't let those bottom-feeders take advantage of your situation. You would gain so much more by using that time to better yourself.
Unless you are in extremely dire financial straits (e.g., risk of prosecution/eviction), I strongly suggest that if the only work you can find on mturk pays less than minimum wage, then you are better off spending that time doing something more productive. Your time is valuable, and you can never get it back!
So if all you can find on mturk are crappy HITs with crappy slave wages, then take this opportunity to give yourself a free pass and do something better with your time! Several crowdsourcing platforms exist now and some are quite active, so look for other online work opportunities. Go to EdX and enroll in a free online class. Go to Code Academy and learn how to program. Go to Duolingo and start learning a new language. Go to Khan Academy and pick a subject you're interested in.
Check sites like Free Stuff Finder or the myriad other free stuff sites and look for deals on things you normally buy anyway, and save money that way.
Go to YouTube and learn something new, or learn something new about something you already do. Go make something, or learn how to make something. Even cleaning/organizing your living space or de-stressing with a video game is more beneficial to you in the long run than working for less than minimum wage for a requester that clearly does not care about you, nor the larger world he/she lives in, enough to pay people a reasonable wage. Don't let those bottom-feeders take advantage of your situation. You would gain so much more by using that time to better yourself.
Welcome to the grind! I hope this information helps you understand what turking is about, and whether it is a good fit for your abilities and aspirations. When you get to 1000 approved, please post in that day's daily work thread so that we can celebrate with you! And good luck!
Copyright jklmnop 2015-2016. All rights reserved.
Last edited: