The use of crowdsourcing has been growing in popularity for several years, and now has a wide variety of applications for both consumers and businesses. Some crowdsourcing applications are effective, especially for consumer services, and they have inspired innovation, new ideas, and collaboration. However, in some forms crowdsourcing can pose serious risks, especially for businesses and organizations that require quality and reliability from their service providers.
When it comes to services, crowdsourcing means a job that is traditionally performed by a professional team member is outsourced to a large group of anonymous volunteers, often for very low pay. It is the anonymity of this outsourced workforce that often makes crowdsourced services unreliable, inconsistent and a potential privacy risk.
We discussed before how newer captioning companies have cut corners to lower costs, and crowdsourcing is one of the processes used to do this. We’ve seen prices advertised by these companies for as low as $1.00 per minute of audio, even with promises of a 24 hour turnaround. For some captioning needs, a crowdsourced service may be good enough. When quality and privacy are a concern however, it probably isn’t the best choice.
With a short and clear audio file, it’s possible a crowdsourced captioning service will work out just fine for your captioning needs, but know that with anything more complicated, problems are bound to happen. This all makes sense when you think about who exactly is transcribing your video.
At first glance you’ll see most popular crowdsourced captioning services promise a 99% accuracy rate, but dig deeper and you’ll find customer reviews indicate the results are completely inconsistent. This is because your captioning project could go to anyone, trained or unqualified, native-speaker or ESL-speaker, motivated or just trying to pocket extra cash. Often transcription is not part of this individual’s skill set or career experience and so mistakes are more likely to happen. Also lack of compensation and accountability means less motivation to get the job done well. This is not to say that there are no crowdsourced workers willing to put in the effort, but there is always a possibility of seeing three positive results in a row, then a terrible quality transcript that you will have to have redone or spend the time to correct yourself.
If your video is long, it is more likely that a crowdsourced company will not meet the promised turnaround time. This is because the crowdsourced transcribers are able to pick and choose which projects they want to tackle, and they’ll often give up on more difficult audio when there are easier projects they can get done faster. Not only do these transcribers have a low pay rate to begin with, but they are also paid by the length of the video, rather than the time it takes them to transcribe it. It makes sense that they would choose easier projects they can get done faster, and therefore get paid more. In reviews of crowdsourced captioning services you’ll sometimes see that the customers received notices that their projects were delayed due to high volumes. Others say that only a portion of the transcript was completed in the allotted time, and even some that say they later were up-charged due to length or poor audio quality.
Also due to the anonymous nature of crowdsourcing, there are concerns about privacy and security of sensitive information. Although some are concerned with the security of the platforms used for crowdsourcing, the real risk is with the individuals who have open access to your video and audio files. Some companies that utilize crowdsourced captioning require all transcribers to sign a nondisclosure agreement before receiving captioning jobs. However, if the company is using crowdsourced transcribers who are inexperienced or based in other countries, your jobs could be given to someone who does not care, or does not understand the importance of following these legally binding agreements.
In 2014 researchers ran an ethics test with over 1000 participants on Amazon’s Mechanical Turk, a crowdsourcing platform. They paid individuals 5 cents to transcribe photos of fake credit card numbers. The good news is that only 30% were willing to complete the illegal task, as opposed to 75% who were willing to do a similar task with a series of random numbers. The bad news is that still 30% were willing to do it, and since the numbers were fake, there is no way to know if any of them ended up trying to use the credit card numbers.
It is always possible that your experience with a crowdsourced caption service will be fine, but if you’re relying on the service to get it right the first time, have a guaranteed quick turnaround or perfect spelling and grammar, you have a strong chance of being disappointed. In some cases this could be a costly mistake, causing you to miss a deadline or end up editing transcripts for hours on your own.
Tell us about your experience with a crowd-sourced service in the comments.