An onlineupdating algorithm on probabilistic matrix factorization with active learning for task recommendation in crowdsourcing systems
 ManChing Yuen^{1}Email author,
 Irwin King^{1} and
 KwongSak Leung^{1}
Received: 24 November 2015
Accepted: 14 September 2016
Published: 1 October 2016
Abstract
Background
To ensure the output quality, current crowdsourcing systems highly rely on redundancy of answers provided by multiple workers with varying expertise, however massive redundancy is very expensive and timeconsuming. Task recommendation can help requesters to receive good quality output quicker as well as help workers to find their right tasks faster. To reduce the cost, a number of previous works adopted active learning in crowdsourcing systems for quality assurance. Active learning is a learning approach to achieve certain accuracy with a very low cost. However, previous works do not consider the varying expertise of workers for various task categories in real crowdsourcing scenarios; and they do not consider new workers who are not willing to work on a large amount of tasks before having a list of preferred tasks recommended. In this paper, we propose ActivePMFv2, Probabilistic Matrix Factorization with Active Learning (version 2), on a task recommendation framework called TaskRec to recommend tasks to workers in crowdsourcing systems for quality assurance. By assigning the most uncertain task for new workers to work on, this paper identifies a flaw in our previous ActivePMFv1, Probabilistic Matrix Factorization with Active Learning (version 1). Therefore, ActivePMFv2 can give new workers a list of preferred tasks recommended faster than that of ActivePMFv1. Our factor analysis model considers not only worker task selection preference, but also worker performance history. It actively selects the most uncertain task for the most reliable workers to work on to retrain the classification model. Moreover, we propose a generic onlineupdating method for learning the model, ActivePMFv2. The larger the profile of a worker (or task) is, the less important is retraining its profile on each new work done. In case of the worker (or task) having large profile, our onlineupdating algorithm retrains the whole feature vector of the worker (or task) and keeps all other entries in the matrix fixed. Our onlineupdating algorithm runs batch update to reduce the running time of model update.
Results
Complexity analysis shows that our model is efficient and is scalable to large datasets. Based on experiments on realworld datasets, the result shows that the MAE results and RMSE results of our proposed ActivePMFv2 are improved up to 29 % and 35 % respectively comparing with ActivePMFv1, where ActivePMFv1 outperforms the PMF with other active learning approaches significantly as shown in previous work. Experiment results show that our onlineupdating algorithm is accurate in approximating to a full retrain of the learning model while the average runtime of model update for each work done is reduced by more than 80 % (decreases from a few minutes to several seconds).
Conclusions
To the best of our knowledge, we are the first one to use PMF, active learning and dynamic model update to recommend tasks for quality assurance in crowdsourcing systems for real scenarios.
Keywords
Crowdsourcing Task recommendation Matrix factorization Probabilistic matrix factorizationBackground
Crowdsourcing is an idea of outsourcing a task to a large group of networked people in the form of an open call to reduce the production cost [1, 2]. In recent years, crowdsourcing systems attract much attentions at present [3, 4]. Some examples of crowdsourcing systems are Amazon Mechanical Turk (or MTurk) [5], CrowdFlower [6], Taskcn [7] and TopCoder [8]. In a crowdsourcing system, the output quality of a completed task in a crowdsourcing system is “the extent to which the provided outcome fulfills the requirements of the requester” [9]. For quality assurance, a requester has to verify the quality of every answer submitted by workers, and it is very timeconsuming. Alternatively, requesters highly rely on redundancy of answers provided by multiple workers with varying expertise, but massive redundancy is very expensive and timeconsuming. “If we ask 10 workers to complete the same task, then the cost of crowdsourcing solutions tends to be comparable to the cost of inhouse solutions” [10]. Therefore, it is important to investigate on how to support task requesters to verify correct answers on crowdsourcing platforms easily and effectively. On the other hand, it is not efficient that the amount of time for a worker spent on selecting a task is comparable with that spent on working on a task, but the monetary reward of a task is just a small amount. To address this problem, task recommendation can help to provide a list of preferred tasks to workers in crowdsourcing systems. However, new workers do not want to work on a large number of tasks or wait for a long time before having a list of preferred tasks recommended. Therefore, it is important to help workers to find their right tasks as quick as possible and minimize the number of task assignments to achieve a target output quality [11]. The worker performance history makes it possible to mine workers’ preference on tasks and to provide an indication of worker quality on tasks. Based on worker performance history, an active learning approach on task recommendation can be used to help requesters to receive good quality output quicker with lower cost, thus achieve quality assurance in crowdsourcing systems. Moreover, by assigning the most uncertain task for new workers to work on, it not only helps new workers having a list of preferred tasks recommended faster, but also improves the output quality of crowdsourcing systems.
Task recommendation can help requesters to receive good quality output quicker as well as help workers to find their right tasks faster. Probabilistic Matrix Factorization (PMF) [12] is the stateoftheart approach for recommendation systems. A factorization model has to be trained and learned before the model can be applied for prediction. In realworld applications, the performance of a factorization model is highly affected by how the model is updated, and thus dynamic updating a model is very important [13]. When updating a worker’s profile, the profile will not change much if the worker having large profile; while the profile will have great change if the worker having small profile.
Active learning is a learning approach to improve the prediction accuracy with a low cost. By performing active learning in a task recommendation model, it can guarantee the accuracy of recommendations with a very low cost, but it still needs to consider the minimizaton of the user waiting time. The tasks recommended to the new workers have to carefully selected, because new workers are not willing to work on a lot of tasks before having their preferred tasks recommended. Therefore, systems that provide recommendations in large user waiting times are not suitable for realworld applications [14]. Moreover, it does not make sense to retrain the model from scratch whenever a worker of large profile completes a task, because the performance improvement by retraining the model in the case is tiny but the cost of retraing model is high. Furthermore, when a large number of workers are working in the crowdsourcing system at the same period of time, the computational complexity is very high if the model is retrained after each worker completes a task. Batch update provides a method of reducing both user waiting time and computational complexity.

First, we propose a way for quality assurance by performing ActivePMFv2, Probabilistic Matrix Factorization with Active Learning (version 2), for task recommendation in crowdsourcing systems, where ActivePMF is an active learning approach on factor analysis based on probabilistic matrix factorization, such that the worker latent feature space, task latent feature space and task category latent feature space are learned. ActivePMF considers the varying expertise of workers for different tasks in real crowdsourcing scenarios. The most informative task and the most skillful worker are selected to learn the factor analysis model.

Second, we first assign all new tasks to the most reliable workers based on the task categories in our proposed ActivePMFv2, so new workers can receive a list of preferred tasks recommended faster than that of ActivePMFv1 [15]. Our proposed ActivePMFv2 also has better output prediction quality than that of ActivePMFv1.

Third, we propose a generic onlineupdating method for learning a factor analysis model, ActivePMFv2. In our proposed onlineupdating approach, our onlineupdating algorithm applies on a learned PMF model without having retrain the whole model. The proposed update methods are generic and appliable for all PMF models.

Fourth, we demonstrate the performance of our proposed ActivePMFv2 by using the real word dataset. The experimental results show that ActivePMFv2 outperforms ActivePMFv1 by 29 % in the MAE results and 35 % in RMSE results, where ActivePMFv1 outperforms PMF with various active learning approaches significantly (PMF is the stateoftheart approach for recommendation systems).

Fifth, we demonstrate the performance of our onlineupdating algorithm by using the real world dataset. The experiment results show that the prediction of onlineupdating ActivePMF on TaskRec model approximates to that of a full retrain of ActivePMF on TaskRec model while the running time of onlineupdating algorithm is significantly lower than that of a full retrain of the model. By using onlineupdating algorithm, the average runtime of model update for each work done is reduced by more than 80 % (decreases from a few minutes to several seconds).

Finally, complexity analysis shows that our model is efficient and is scalable to large datasets.
Related work
Crowdsourcing systems

Motivate workers of diverse background to work on crowdsourcing tasks in long run. Currently, on crowdsourcing sites, most workers only provide moderate contributions [16] and there is a significant population of young and welleducated Indian workers [17]. It can attract more workers to contribute their efforts in long run if a worker find a suitable task on a crowdsourcing site easily.

Improve the quality of work. Workers perform better if they are familiar with the tasks. Chilton et al. [18] showed that task workers only browsed the first few pages on crowdsourcing sites when searching for tasks. The task list for a worker of Amazon MTurk site is usually displayed on hundreds of pages. A worker selects a task from the list of available tasks sorted by a specified feature of tasks such as task creation date and reward amount. When the tasks posted on the first few pages are not suitable for a worker, the worker might choose a task that he does not familiar with and try to complete it to earn the rewards; otherwise, the worker does not select any task. Working with a unfamiliar task might decrease the quality of work.
Recommendation systems
Broadly speaking, recommendation systems are based on either content filtering approach or collaborative filtering approach. The content filtering approach creates a profile for each user or product, for example, a movie, to characterize its nature. The profiles of users and products allow programs to associate users with matching products. The advantage is that it can address the system’s new products and users. However, the profile information might not be available or easy to collect. On the other hand, the collaborative filtering approach relies only on past user behavior. This approach analyzes relationships between users and interdependencies among products to identify new useritem associations. It is generally more accurate than content filtering approach. However, collaborative filtering cannot address the system’s new products and users [19], which is the coldstart problem.
To address the coldstart problem, latent factor models are an alternative approach that can approximate the ratings by characterizing both users and items on a number of factors inferred from the ratings patterns. “Some of the most successful realizations of latent factor models are based on matrix factorization.” [19] Matrix factorization has a lot of applications [20, 21]. Although matrix factorization can solve the coldstart problem, it is not scalable. Probabilistic matrix factorization (PMF) model [22] can scale linearly with the number of observations, and performs very well on large, sparse, and imbalanced datasets.
Recently, several probabilistic matrix factorization methods [23] have been proposed for collaborative filtering approach in recommendation systems. These methods focus on using lowrank approximations to model the useritem rating matrix for making further predictions. The premise behind a lowdimensional factor model is that there is only a small number of factors influencing preferences, and that a user’s preference vector is determined by how each factor applies to that user. The above approaches are used for user recommendation in social tagging systems.
Task recommendation in crowdsourcing systems
A recommendation system can improve the performance of crowdsourcing systems by providing task requesters some output quality controls based on a number of parameters, such as task requirements, task properties, worker interests, worker incentives, and costs [24]. Based on collaborative filtering, Organisciak et al. [25] proposed two approaches for capturing personal preferences in personalized item recommendation in crowdsourcing systems; and they are tastematching and tastegrokking. Tastematching uses workers’ taste to infer the requester’s taste where workers and the requester have similar tastes, while tastegrokking uses workers’ explicit prediction on the requester’s taste. Both tastematching and tastegrokking have better performance than the use of generic workers. Later, Organisciak et al. [26] demonstrated the performance of personalized crowdsourcing in a complexer environment by carrying out case studies on personalized text highlighting in film reviews. The results show that both approaches have better performance than a nonpersonalized baseline. Besides, the tastegrokking approach performs well in simpler tasks and the tastematching approach performs well with larger crowds and tasks with latent decisionmaking variables. Ambati et al. [27] proposed classification based task recommendation approach to recommend tasks to users based on implicit modeling of skills and interests. However, these approaches can not solve coldstart problem. Besides, task recommendation is much difficult than product recommendation, and workers do not have to give ratings to tasks to indicate the extent of their favor of each task. A crowdsourcing system needs some signals indicating types of available tasks, and the number of tasks workers select and complete [28].
Active learning
Active learning is a learning approach to achieve certain accuracy with a very low cost. Broadly speaking, active learning systems are based on either streambased approach [29] or poolbased approach [30]. The streambased approach considers one unlabeled instance each time, and decides whether to query its label or ignore it. This approach is useful when unlabeled instance is continuously available but cannot be stored easily, such as sensor data. However, as the streambased approach relies on a real underlying input distribution, it is difficult to decide whether to query the label of an instance or ignore it at its arrival time. On the other hand, the poolbased approach ranks all unlabeled instances in order of informativeness, and queries the label for the most informative unlabeled instance in the pool. The advantage is that large amount of unlabeled data are available in many domains at present, this approach is very important. However, it is difficult to find a good way to choose good queries from the pool. In both streambased approach and poolbased approach, a query selection strategy is required to achieve high accuracy with as low labeling cost as possible.
Several query selection strategies are commonly used in active learning systems. For example, uncertainty sampling selects instances, which the current model is the most uncertain about, to query. There are many ways to measure the uncertainty, such as smallest margin [31], least confidence [32] and maximum entropy [33]. Another example of query selection strategies is query by committee [34]. This approach maintains a committee of independently trained classification models, and queries the instance for which the committee models disagree the most. Among all query selection strategies, uncertainty sampling is one of the simplest and widely used strategies in active learning systems [35].
Active learning for task recommendation in crowdsourcing systems to achieve quality assurance
Our motivation is the observation of the increase of difficulty for requesters to obtain good output with low cost. Currently, active learning approach has been applied on a number of quality assurance methods in crowdsourcing systems [35–37] because active learning can achieve certain accuracy by using fewer annotations even in noisy annotation scenarios. Laws et al. [36] demonstrated that actively selecting instances for label query can achieve performance gains in natural language processing tasks. However, they did not consider actively selecting annotators. To further improve the output quality, Yan et al. [35] proposed to use uncertainty sampling to select the most uncertain instance to query, and also uses an optimization formulation to choose multiple confident workers to query from. However, it is costly to pay for multiple labelers. To characterize the strength of each worker and improve the compentency of weak workers, Fang et al. [37] proposed a SelfTaught Active Learning paradigm, where a weak worker can learn complementary knowledge from a strong worker. Each labeler has a knowledge set (a set of confidence scores), which is used for worker selection. By using the knowledge set of the most reliable labeler to replace that of the most unreliable labeler, the most reliable labeler can teach the unreliable labeler. However, the learning curves of different labelers vary in real scenarios. Later, Fang and Zhu [38] proposed to use diversity density to characterize the oracle’s uncertain knowledge. However, an oracle does not exist in realworld applications.
In recent years, a number of research works [39–42] proposed recommendation systems based on a Probabilistic Matrix Factorization (PMF) model to improve the output quality in crowdsourcing systems, where Probabilistic Matrix Factorization (PMF) is the stateoftheart approach for recommendation systems. Jung and Lease [39] proposed to use a PMF model to infer unobserved labels to reduce the bias of the existing crowdsourced labels, thus improve the quality of labels. Later, Jung [40] proposed to use a PMF model to improve the quality of crowdsourcing tasks. Experimental results proved that the strength of PMF over Singular Value Decomposition (SVD) and baseline methods. However, it is not suitable for a huge number of tasks on crowdsourcing systems in reality. Yuen et al. [41, 42] considered various task categories in real scenarios in crowdsourcing systems and proposed a PMF model for task recommendation in crowdsourcing systems. They proved that considering task categories in PMF can improve the performance. However, it does not consider to reduce the labeling cost by applying active learning approach. Later, Yuen et al. [15] proposed Probabilistic Matrix Factorization with Active Learning (version 1) for task recommendation systems. The model outperforms the PMF model with other active learning approaches, but new workers have to wait for a long time before having a list of perferred tasks recommended due to lack of worker performance history for all new workers.
Task recommendation for new workers in crowdsourcing systems
In crowdsourcing systems, workers prefer to have a list of recommended tasks, but they are not willing to work on a large number of tasks before having a list of preferred tasks recommended. Since new workers have not worked on a lot of tasks yet, it is difficult for a recommendation system to make a better recommendation for new workers due to the small working profiles of new workers. When performing active learning in recommendation systems, besides accuracy of recommendations and minimization of cost, it is also important for new workers to have a list of preferred tasks recommended as soon as they start working in a crowdsourcing system.
Dynamicupdating crowdsourcing systems for realworld scenarios
Some previous works proposed various ways to improve the performance of recommendation systems for realworld scenarios [13, 14]. Rendle et al. [13] proposed an onlineupdating algorithm for three kernal matrix factorization models, they are linear, logistic and linear nonnegative matrix factorization model. They demonstrated that the output quality of their proposed onlineupdating algorithm approximates to that of fully retraining the models. However, the three kernal matrix factorization models are designed for useritem matrices. Besides, the user preference on item category, which is necessary for realworld applications, is not considered in the models. Karimi et al. [14] proposed an active learning method for aspect model in recommendation systems. They observed that “users are not willing to provide information for a large amount of items, thus the quality of recommendations is affected specially for new users” [14], and a full retrain of an aspect model needs a long time especially for a system of large number of existing users. In their onlineupdating method, the aspect model is updated to learn user latent factors for new users only and not the other users. Experimental results show that the user waiting time of their method is significantly less than that of bayesian method, but the prediction accuracy of their method is not always better than that of bayesian method.
Our motivation
Our motivation is the observation of the increase of difficulty for workers to find their preferred tasks [18, 43, 44], the increase of demand on task recommendation in crowdsourcing systems [45–47], and no previous works on task recommendation model for crowdsourcing systems that considers dynamicupdating for reducing the user waiting time on model update. To achieve certain output quality with a very low cost, we propose ActivePMFv2, Probabilistic Matrix Factorization with Active Learning (version 2), for task recommendation in crowdsourcing systems to actively select the most uncertain tasks and the most reliable workers for retraining the classification model. To reduce the user waiting time on the learning model update, we propose a generic onlineupdating method for learning the model, ActivePMFv2.
Task recommendation framework
Our task recommendation framework (TaskRec) is based on matrix factorization method, to perform factor analysis to learn the worker latent feature, the task latent feature and the task category latent feature. To reduce the labeling cost and guarantee the output quality, Probabilistic Matrix Factorization with Active Learning (version 2) selects the most informative task to be learned and selects the best worker to query from.
The problem we study in this paper is how to effectively predict the missing values in the workertask performance matrix so as to select the most informative task for the best worker to query from to achieve high accuracy with as few instances as possible. We define the problem of quality assurance in crowdsourcing systems as follows:
Definition 1
Quality assurance problem: Given a set of workers WS = \(\{ w_{i} \}^{m}_{i=1}\), a set of tasks VS= \(\{ v_{j} \}^{n}_{j=1}\), a set of ratings R = {r _{ ij }} associated between worker w _{ i } and task v _{ j } where \(r_{ij} \in \mathbb {R}^{M\times N}\) and only certain elements of R are initially known. The binary matrix I = {I _{ ij }} of the same shape as R represents the known points, so that I _{ ij } is 1 if r _{ ij } is observed and 0 otherwise. The set of (i, j) indexes where I _{ ij } = 0 is denoted by PS. Predict the set of the unknown elements of R = {r _{ ij }} where (i,j)∈P S. The aim of ActivePMF is to query the most informative tasks selected from the set of the unknown elements of R = {r _{ ij }} where (i,j)∈P S, and to query from the most reliable workers.
Basic notations throughout this paper
Notation  Description 

WS= \(\{ w_{i} \}^{m}_{i=1}\)  WS is the set of workers, w _{ i } is the ith worker, m is the total number of workers 
VS= \(\{ v_{j} \}^{n}_{j=1}\)  VS is the set of tasks, v _{ j } is the jth task, n is the total number of tasks 
CS= \(\{ c_{k} \}^{o}_{k=1}\)  CS is the set of task categories, c _{ k } is the kth task category, o is the total number 
of task categories  
\(l \in \mathbb {R}\)  l is the number of dimensions of latent feature space 
\(W \in \mathbb {R}^{l \times m}\)  W is the worker latent feature matrix 
\(V \in \mathbb {R}^{l \times n}\)  V is the task latent feature matrix 
\(C \in \mathbb {R}^{l \times o}\)  C is the task category latent feature matrix 
R = {r _{ ij }},  R is the workertask preferring matrix, r _{ ij } is the extent of the favor of task v _{ j } 
\(R \in \mathbb {R}^{m \times n}\)  for worker w _{ i } 
U = {u _{ ik }},  U is the workercategory preferring matrix, u _{ ik } is the extent of worker w _{ i }’s 
\(U \in \mathbb {R}^{m \times o}\)  preference for category c _{ k } 
D = {d _{ jk }},  D is the taskcategory grouping matrix, d _{ jk } indicates the task category c _{ k } that 
\(D \in \mathbb {R}^{n \times o}\)  task v _{ j } belongs to 
(i,j)∈P S  PS is the set of indexes where the rating r _{ ij } is unknown 
N(xμ,σ ^{2})  Probability density function of the Gaussian distribution with mean μ and variance σ ^{2} 
Probabilistic Matrix Factorization on Task Recommendation Framework
Worker Behavior  Value  

Worker’s work done is accepted by requester.  →  5 
Worker’s work done is rejected by requester.  →  4 
Worker completes a task and submits the work done.  →  3 
Worker selects a task to work on but not complete it.  →  2 
Worker browses the detailed information of a task.  →  1 
Worker does not browse the detailed information of a task.  →  0 
In some cases, the ratings based on value transformation of worker behavior would be inaccurate on reflecting workers’ task preference. For example, a worker’s work done is being accepted, but he might not like the task very much.
Workertask preferring matrix factorization
We have m workers, n tasks. The workertask preferring matrix is denoted as R, the element r _{ ij } in R means the extent of the favor of task v _{ j } for worker w _{ i }, where values of r _{ ij } are within the range [0, 1]. Without loss of generality, we first map the ratings that inferred from worker behavior 1,..., 5 to the interval [0, 1] using the function f(x)=x/5. Hence, we are given a partially observed workertask preferring matrix, R, with m workers and n tasks.
To learn the workers’ preference on the tasks, we employ matrix factorization, more specifically, Probabilistic Matrix Factorization (PMF) [12], to recover the workertask preferring matrix. Given the partial observed matrix R, we aim at decomposing the matrix R into two ldimensional lowrank feature matrices, W and V, where \(W \in \mathbb {R}^{l \times m}\) is the latent feature matrix for workers with column vector W _{ i }, and \(V \in \mathbb {R}^{l \times n}\) is the latent feature matrix for tasks with column vector V _{ j }.
Workercategory preferring matrix factorization
where g(.) is the logistic function, and f(w _{ i },c _{ k }) represents the number of times worker w _{ i } completes a task of the category c _{ k } where the task is accepted.
where N(xμ,σ ^{2}) is the probability density function of the Gaussian distribution with mean μ and variance σ ^{2}, and \(I^{U}_{ik}\) is the indicator function that is equal to 1 if worker w _{ i } has at least one completed task of the category c _{ k } being accepted and equal to 0 otherwise.
Taskcategory grouping matrix factorization
where f(v _{ j },c _{ k }) is an indicator variable with the value of 1 if the task v _{ j } belongs to the category c _{ k }, and 0 otherwise.
where N(xμ,σ ^{2}) is the probability density function of the Gaussian distribution with mean μ and variance σ ^{2}, and \(I^{D}_{jk}\) is the indicator function that is equal to 1 if the entry d _{ jk } is observed and equal to 0 otherwise.
A unified matrix factorization for TaskRec
where g ^{′}(.) is the firstorder derivative of the logistic function. To reduce the model complexity, we set θ _{ W }=θ _{ V }=θ _{ C } in our experiments. The training time for our model scales linearly with the number of observations.
ActivePMFv2  active learning for concurrent selection of task and worker
We first query all new tasks, which have not been tried by anyone, from the most reliable worker. Then, we query the most uncertain task from all new workers. Next, we query the most uncertain task, and select the most reliable worker for the task to query from. Our proposed ActivePMFv2 is presented in Algorithm 1.
Random new task selection for reliable worker
In Algorithm 1, Step 6 represents the process of new task selection for most reliable worker in the category.
Uncertainty sampling for task selection for randomly selected new worker
In Algorithm 1, Step 8 represents the process of most uncertain task selection for new worker.
Uncertainty sampling for task selection for reliable worker
In Algorithm 1, Step 9 represents the most uncertain task selection for the most reliable worker.
After annotation, the selected task is removed from the unlabeled data set. Next, the selected task and its rate are added to the set of labeled dataset. The model is then retrained on the labeled tasks and the informativeness of the remaining tasks in the unlabeled data set is reevaluated.
Onlineupdating on ActivePMFv2  onlineupdate on active learning for concurrent selection of task and worker
In this section, we present our onlineupdating approach for learning the matrix factorization model, ActivePMFv2 model. The onlineupdating approach of ActivePMFv2 is presented in Algorithm 2, while the full retrain approach of ActivePMFv2 is presented in Algorithm 1. The onlineupdating approach uses the same sampling heuristics and the same active learning approach in query method as that of the full retrain approach. The main difference between the onlineupdating approach and the full retrain approach is the model update methods they used. In the full retrain approach, it retrains the whole learning model after each work done. In the onlineupdating approach, it has two main parts: (1) It retrains the learning model in batch mode where model update occurs after a number of work done; (2) For each work done related to a worker (or task) having profile larger than the threshold, it updates the whole feature vector of the worker (or task) and keeps all other entries in the feature matrix fixed. On the other hand, for work done related to a worker (or task) having profile smaller than the threshold, it updates the whole feature matrix.
Partial update
The impact on retraining the whole learning model decreases as the profile size of the worker (or task) increases. Especially when work done by new workers or work done on task having small profile, updating the feature matrix is crucial. For a new worker, each work done by him will result in much change in his task perference in his worker profile; while for a worker that has already completed a lot of tasks, each work done by him will not change much in his worker profile. Updating feature vectors for a worker (or task) having smaller profile results in a much better model. As a result, for a worker (or task) having large profile, we observe that the model learned from retraining the feature vector of the worker (or task) only is approximate to that learned from a full retrain.
In Algorithm 2, Step 12, 14 and 16 represents the process of partial update. For a worker (or task) having profile size larger than the threshold, the algorithm retrains the feature vector of the worker (or task) and keep all other entries in the matrix unchanged; otherwise, the algorithm retrains the feature vectors of all workers and all tasks.
Batch update
The time for retraining a learning model is proportional to the computational complexity of the model and the amount of information stored in the model; while the amount of information depends on the number of workers, the number of tasks and the number of work done. Retraining a large learning model takes a long time. For a large realworld crowdsourcing system, it is inefficient if the whole model is retrained from scratch once a worker completes a task.
In Algorithm 2, Step 24 and 25 represents the process of batch update. When the number of work done is smaller than the batch size, the learning model is not retrained. On the other hand, when the number of work done is larger than the batch size, the algorithm retrains the learning model.
General update problem
Our proposed onlineupdating approach can also be applied in a general update problem. In a general recommendation system, a new rating r _{ u,i } might affect the features of both user u and item i. The partial update method works based on the following conditions: (1) If the profiles of both user and item are large, it could update the feature vectors of both the selected user and the selected item; (2) If the user profile is large (but the item profile is small), it could update the user feature vector only; (3) If the item profile is large (but the user profile is small), it could update the item feature vector only; (4) If the profiles of both user and item are small, it could update the feature vectors of all users and all items. Besides, based on the number of new incoming ratings, the batch update method retrains the whole learning model. A general update algorithm is shown in Algorithm 3.
Complexity analysis
To compute the complexity of our ActivePMFv2, we consider both the computation of the gradient descent methods and the computation of selecting the most uncertain task for the most reliable worker. The main computation of the gradient descent methods is evaluating objective function E and corresponding gradients on variables. Because of the sparsity of matrices R, U, and D, the complexity of evaluating the objective function in Eq. (13) is \(\mathcal {O} \left (n_{R}l + n_{U}l + n_{D}l \right)\), where n _{ R }, n _{ U } and n _{ D } are the number of nonzero entries in matrices R, U, and D respectively, and l is the number of dimensions of latent feature space. By using the similar approach, we can derive the complexities of Eq. (14), Eq. (15) and Eq. (16). For the computation of selecting the most uncertain task for the most reliable worker, the complexity of selecting the most uncertain task in Eq. (21) is \(\mathcal {O} \left (n_{R}l \right)\), the complexity of selecting the most reliable worker in Eq. (22) is \(\mathcal {O} \left (m \right)\), and thus the total complexity of assigning a task to a worker is \(\mathcal {O} \left (m + n_{R}l \right)\). As a result, the total complexity for one iteration is \(\mathcal {O} \left (m + n_{R}l + n_{U}l + n_{D}l \right)\). It means that the complexity is linear with respect to the number of workers and the number of observations in the three sparse matrices. The complexity analysis shows that ActivePMFv2 can scale to very large datasets.
To apply onlineupdating approach when learning ActivePMFv2 model, we consider both the partial update method and the batch update method. Since only some nonzero entries in matrices is updated in the partial update method, the complexity of evaluating the objective function in Eq. (13) is \(\mathcal {O} \left (z_{R}l + z_{U}l + z_{D}l \right)\), where z _{ R }, z _{ U } and z _{ D } are the number of nonzero entries to be updated in matrices R, U, and D respectively, and l is the number of dimensions of latent feature space. As a result, the total complexity for one iteration is \(\mathcal {O} \left (m + z_{R}l + z_{U}l + z_{D}l \right)\), where z _{ R }<<n _{ R }, z _{ U }<<n _{ U }, z _{ D }<<n _{ D }. Based on our observation, the complexity of learning ActivePMFv2 model can be highly reduced by using the partial update method. Besides, the batch method can greatly reduce the number of iterations, thus further reduce the computational complexity.
Experimental analysis
 1.
How is ActivePMFv1 approach compared with PMF with various active learning approaches?
 2.
How is ActivePMFv1 approach compared with PMF with various active sampling heuristics?
 3.
How is our ActivePMFv2 approach compared with ActivePMFv1?
 4.
How is the partial update part of onlineupdating approach compared with the full retrain when learning ActivePMFv2 model?
 5.
How is the batch update part of onlineupdating approach compared with the full retrain when learning ActivePMFv2 model?
Description of dataset
Statistics of our dataset
Number of workers  1,592 
Number of different tasks  6,639 
Number of categories  43 
Total HITs from all tasks  19,815 
Number of ratings  19,815 
Max number of HITs of a worker  2,691 
Min number of HITs of a worker  1 
Average number of HITs of a worker  12.4 
1st quartile (25th percentile) of number of HITs of a worker  1 
2nd quartile (50th percentile) of number of HITs of a worker  2 
3rd quartile (75th percentile) of number of HITs of a worker  5 
Task categorization by both language and keywords given by MTurk in our dataset
1  EnglishAfrikaans translations  23  EnglishRomanian translations 
2  EnglishAzeri translations  24  EnglishRussian translations 
3  EnglishBulgarian translations  25  EnglishSlovak translations 
4  EnglishBangla translations  26  EnglishSomali translations 
5  EnglishBosnian translations  27  EnglishAlbanian translations 
6  EnglishWelsh translations  28  EnglishSerbian translations 
7  EnglishSpanish translations  29  EnglishTamil translations 
8  EnglishBasque translations  30  EnglishThai translations 
9  EnglishFarsi translations  31  EnglishTurkmen translations 
10  EnglishIrish translations  32  EnglishTagalog translations 
11  EnglishHindi translations  33  EnglishTurkish translations 
12  EnglishIndonesian translations  34  EnglishTatar translations 
13  EnglishKorean translations  35  EnglishUkrainian translations 
14  EnglishKurdish translations  36  EnglishUrdu translations 
15  EnglishLatin translations  37  EnglishUzbek translations 
16  EnglishLatvian translations  38  English annotations 
17  EnglishMongolian translations  39  Spanish annotations 
18  EnglishMaltese translations  40  Arabic annotations 
19  EnglishNepali translations  41  English relevance judgment 
20  EnglishPunjabi translations  42  English creative writing 
21  EnglishKapampangan translations  43  English transcription 
22  EnglishPolish translations 
Evaluation metrics

T[Rand]W[Rand]: It assigns a randomly selected task to a randomly selected worker.

T[MaxDiff]W[Rand]: It assigns the task of maximum difference between its observed values and its predicted values to a randomly selected worker.

T[Rand]W[Reli]: It assigns a randomly selected task to the most reliable worker in the task category.

T[MaxDiff]W[Reli]: It assigns the task of maximum difference between its observed values and its predicted values to the most reliable worker in the task category.

T[N]W[Reli]+T[MaxPredictErr]W[Reli]: It has two parts. First, it assigns all new tasks to the most reliable worker in the task categories. Second, it assigns the task of maximum average prediction error of all its predicted values to the most reliable worker in the task category.

T[N]W[Reli]+T[MaxEntropy]W[Reli]: It has two parts. First, it assigns all new tasks to the most reliable worker in the task categories. Second, it assigns the task of maximum Entropy on the posterior variance to the most reliable worker in the task category.

T[N]W[Reli]+T[MaxDiff]W[Reli] (ActivePMFv1): It has two parts. First, it assigns all new tasks to the most reliable worker in the task categories. Second, it assigns the task of maximum difference between its observed values and its predicted values to the most reliable worker in the task category.

T[N]W[Reli]+T[MaxDiff]W[N]+T[MaxDiff]W[Reli] (ActivePMFv2): It has three parts. First, it assigns all new tasks to the most reliable worker in the task categories. Second, for all new workers, it assigns the task of maximum difference between its observed values and its predicted values to a randomly selected new worker. Third, it assigns the task of maximum difference between its observed values and its predicted values to the most reliable worker in the task category.
where r _{ i,j } denotes the rating that indicates the extent of the favor of task j for worker i, \(\hat {r}_{i, j}\) denotes the predicted rating, and N is the total number of testing ratings.
Performance comparison
To show the prediction performance improvements of ActivePMFv2, we first compare ActivePMFv1 [15] with PMF with various active learning approaches and different active sampling heuristics, where Probabilistic Matrix Factorization (PMF) [12] is the stateoftheart approach for recommendation systems. Next, we compare our proposed ActivePMFv2 with ActivePMFv1.
MAE comparison among various active sampling heuristics in PMF (A smaller MAE means a better performance)
PMF with active sampling heuristics  

No. of selected samples  T[N]W[Reli]+ T[MaxDiff]W[Reli] (ActivePMFv1)  T[N]W[Reli]+ T[MaxPredictErr]W[Reli]  T[N]W[Reli]+ T[MaxEntropy]W[Reli] 
1000  0.3658  0.3658  0.3658 
2000  0.2778  0.2778  0.2778 
3000  0.1955  0.1955  0.1955 
4000  0.1473  0.1443  0.1482 
5000  0.1151  0.1184  0.1262 
6000  0.0977  0.0977  0.1100 
7000  0.0945  0.0839  0.0963 
8000  0.0899  0.0664  0.0809 
9000  0.0652  0.0479  0.0536 
10000  0.0334  0.0340  0.0366 
11000  0.0176  0.0221  0.0237 
RMSE comparison among various active sampling heuristics in PMF (A smaller MAE means a better performance)
PMF with active sampling heuristics  

No. of selected samples  T[N]W[Reli]+ T[MaxDiff]W[Reli] (ActivePMFv1)  T[N]W[Reli]+ T[MaxPredictErr]W[Reli]  T[N]W[Reli]+ T[MaxEntropy]W[Reli] 
1000  0.7101  0.7101  0.7101 
2000  0.6097  0.6097  0.6097 
3000  0.5110  0.5110  0.5110 
4000  0.4164  0.4364  0.4583 
5000  0.3180  0.4025  0.4409 
6000  0.2621  0.3626  0.4300 
7000  0.2566  0.3411  0.4195 
8000  0.2563  0.2862  0.3856 
9000  0.2054  0.2342  0.3023 
10000  0.1259  0.1975  0.2390 
11000  0.0880  0.1532  0.1786 
MAE comparison among various active learning approaches in PMF (A smaller MAE means a better performance)
PMF with active learning approaches  

No. of selected samples  T[N]W[Reli]+ T[MaxDiff]W[Reli] (ActivePMFv1)  T[MaxDiff]W[Reli]  T[MaxDiff]W[Rand]  T[Rand]W[Reli]  T[Rand]W[Rand] 
1000  0.3658  0.4731  0.4728  0.4179  0.3607 
2000  0.2778  0.4579  0.4592  0.3580  0.2739 
3000  0.1955  0.4567  0.4567  0.3083  0.2340 
4000  0.1473  0.4325  0.4371  0.2642  0.2090 
5000  0.1151  0.4089  0.4122  0.2296  0.1866 
6000  0.0977  0.2427  0.3887  0.1941  0.1677 
7000  0.0945  0.1872  0.2646  0.1716  0.1435 
8000  0.0899  0.1334  0.1908  0.1447  0.1122 
9000  0.0652  0.0898  0.1214  0.0994  0.0728 
10000  0.0334  0.0357  0.0583  0.0394  0.0406 
11000  0.0176  0.0247  0.0312  0.0287  0.0388 
RMSE comparison among various active learning approaches in PMF (A smaller MAE means a better performance)
PMF with active learning approaches  

No. of selected samples  T[N]W[Reli]+ T[MaxDiff]W[Reli] (ActivePMFv1)  T[MaxDiff]W[Reli]  T[MaxDiff]W[Rand]  T[Rand]W[Reli]  T[Rand]W[Rand] 
1000  0.7101  0.8407  0.8396  0.7745  0.6715 
2000  0.6097  0.8417  0.8417  0.7112  0.5355 
3000  0.5110  0.8423  0.8420  0.6578  0.4833 
4000  0.4164  0.8017  0.8089  0.6052  0.4533 
5000  0.3180  0.7597  0.7655  0.5643  0.4247 
6000  0.2621  0.6918  0.7245  0.5191  0.3978 
7000  0.2566  0.6067  0.6150  0.4501  0.3605 
8000  0.2563  0.5186  0.5381  0.3205  0.2991 
9000  0.2054  0.3899  0.4844  0.2952  0.2210 
10000  0.1259  0.1683  0.2139  0.1843  0.1524 
11000  0.0880  0.1245  0.1357  0.1342  0.1396 
MAE comparison between ActivePMFv2 and ActivePMFv1 (A smaller MAE means a better performance)
No. of selected samples  T[N]W[Reli]+T[MaxDiff]W[N]+ T[MaxDiff]W[Reli] (ActivePMFv2)  T[N]W[Reli] + T[MaxDiff]W[Reli] (ActivePMFv1) 

1000  0.3658  0.3658 
2000  0.2778  0.2778 
3000  0.1955  0.1955 
4000  0.1052  0.1473 
5000  0.1001  0.1151 
6000  0.0948  0.0977 
7000  0.0897  0.0945 
8000  0.0879  0.0899 
9000  0.0652  0.0652 
10000  0.0283  0.0334 
11000  0.0156  0.0176 
RMSE comparison between ActivePMFv2 and ActivePMFv1 (A smaller RMSE means a better performance)
No. of selected samples  T[N]W[Reli]+T[MaxDiff]W[N] + T[MaxDiff]W[Reli] (ActivePMFv2)  T[N]W[Reli] + T[MaxDiff]W[Reli] (ActivePMFv1) 

1000  0.7101  0.7101 
2000  0.6097  0.6097 
3000  0.5110  0.5110 
4000  0.2710  0.4164 
5000  0.2594  0.3180 
6000  0.2513  0.2621 
7000  0.2479  0.2566 
8000  0.2494  0.2563 
9000  0.2030  0.2054 
10000  0.1116  0.1259 
11000  0.0845  0.0880 
In Tables 4 and 5, we compare among PMF with three sampling heuristics on selecting tasks, which are Maximum Difference (ActivePMFv1), Maximum Prediction Error and Maximum Entropy. The performance results of the three approaches are similar when the number of selected samples is small. However, as the number of selected samples increases, Maximum Difference approach (ActivePMFv1) outperforms the other two approaches.
In Tables 6 and 7, we compare among PMF with different active learning approaches on task selection and worker selection. When the number of selected samples is very small, assigning tasks to workers randomly gives the best performance in both the MAE results and the RMSE results. However, as the number of selected samples increases, assigning new tasks in the first stage can greatly improve the performance in both the MAE results and the RMSE results. Compared with random selection on task and worker to the PMF learning model (i.e. T[Rand]W[Rand]), ActivePMFv1 can greatly improve both MAE and RMSE performance.
In Tables 8 and 9, we compare our proposed ActivePMFv2 with ActivePMFv1. The performance results of the three approaches are similar when the number of selected samples is small. However, as the number of selected samples increases, ActivePMFv2 outperforms ActivePMFv1 in both MAE and RMSE results. Compared with ActivePMFv1, the MAE results and the RMSE results of ActivePMFv2 are improved up to 29 % and 35 % respectively.
Comparison on a FullRetrain with Partial Update on OnlineUpdating Approach on ActivePMFv2 model learning (Feature k = 20; No of Work Done = 11,000; Batch = 1)
ActivePMFv2 model  MAE  RMSE  Avg Runtime per Work Done (min) 

Full Retrain  0.0156  0.0845  3.839 
OnlineUpdating (Partial = 0.001)  0.0156  0.0845  3.374 
Comparison on a FullRetrain with Batch Update on OnlineUpdating Approach on ActivePMFv2 model learning (Feature k = 20; No of Work Done = 11,000)
ActivePMFv2 model  MAE  RMSE  Avg Runtime per Work Done (min) 

Full Retrain  0.0156  0.0845  3.839 
OnlineUpdating (Partial = 0.001; Batch = 1)  0.0156  0.0845  3.374 
OnlineUpdating (Partial = 0.001; Batch = 10)  0.0191  0.0914  0.675 
OnlineUpdating (Partial = 0.001; Batch = 50)  0.0313  0.1353  0.142 
OnlineUpdating (Partial = 0.001; Batch = 100)  0.0515  0.2103  0.137 
OnlineUpdating (Partial = 0.001; Batch = 150)  0.0768  0.2977  0.049 
OnlineUpdating (Partial = 0.001; Batch = 200)  0.0513  0.2033  0.038 
OnlineUpdating (Partial = 0.001; Batch = 500)  0.1022  0.3445  0.017 
Conclusion
In this paper, we have proposed Probabilistic Matrix Factorization with Active Learning (version 2), ActivePMFv2, on Task Recommendation framework, TaskRec, for quality assurance in crowdsourcing systems. It first randomly assigns new tasks to the most reliable worker in the task categories. Second, it actively selects the most uncertain task, and then request new workers to complete the task. Third, it actively selects the most uncertain task, and then request the most reliable workers to complete the task for retraining the classification model. Experimental results show that our ActivePMFv2 outperforms ActivePMFv1, where the MAE results and the RMSE results of ActivePMFv2 are improved up to 29 % and 35 % respectively. Previous work shows that ActivePMFv1 outperforms the PMF with other active learning approaches significantly.
Moreover, we have proposed onlineupdating on the task recommendation model to reduce the runtime of retraining the model. In a largescale crowdsourcing system, it does not make sense to retrain the model from scratch whenever a worker of large profile completes a task, because the performance improvement by retraining the model in the case is tiny but the cost of retraing model is high. Moreover, when a large number of workers are working in the crowdsourcing system at the same period of time, the computational complexity is very high if the model is retrained after each worker completes a task. The larger the profile of a worker (or task) is, the less important is retraining its profile on each new work done. In case of the worker (or task) having large profile, our onlineupdating algorithm retrains the whole feature vector of the worker (or task) and keeps all other entries in the matrix fixed. Our onlineupdating algorithm runs batch update to reduce the running time of model update. Experiment results show that our onlineupdating algorithm is accurate in approximating to a full retrain of the learning model while the average runtime of model update for each work done is reduced by more than 80 % (decreases from a few minutes to several seconds).
Declarations
Acknowledgements
This research was in part supported by grants from the National Grand Fundamental Research 973 Program of China (No. 2014CB340405), the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. CUHK 413212), and Microsoft Research Asia Regional Seed Fund in Big Data Research (Grant No. FY13RESSPONSOR036).
Authors’ contributions
All authors made substantial contributions to conception and design of the work. MC carried out experiments, data analysis and data interpretation. Besides, MC participated in drafting and revising the manuscript. All authors read and approved the final manuscript.
Competing interests
The authors declare that they have no competing interests.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Authors’ Affiliations
References
 Howe J. The rise of crowdsourcing. Wired. 2006;14(6).Google Scholar
 Howe J. Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business. New York: Crown Business; 2008.Google Scholar
 Yuen MC, King I, Leung KS. A survey of crowdsourcing systems. In: SocialCom ’11: Proceedings of The Third IEEE International Conference on Social Computing. Boston: IEEE Computer Society: 2011. p. 766–73.Google Scholar
 Yuen MC, Chen LJ, King I. A survey of human computation systems. In: CSE ’09: Proceedings of IEEE International Conference on Computational Science and Engineering. Vancouver: IEEE Computer Society: 2009. p. 723–8, doi:10.1109/CSE.2009.395.Google Scholar
 Amazon Mechanical Turk. https://www.mturk.com/.
 CrowdFlower. http://crowdflower.com/.
 Taskcn. http://www.taskcn.com/.
 TopCoder. http://www.topcoder.com/.
 Allahbakhsh M, Benatallah B, Ignjatovic A, MotahariNezhad HR, Bertino E, Dustdar S. Quality control in crowdsourcing systems: Issues and directions. IEEE Internet Computing. 2013; 17(2):76–81. doi:10.1109/MIC.2013.20.View ArticleGoogle Scholar
 Ipeirotis PG, Provost F, Wang J. Quality management on amazon mechanical turk. In: HCOMP ’10: Proceedings of the ACM SIGKDD Workshop on Human Computation. New York: ACM: 2010. p. 64–7.Google Scholar
 Karger DR, Oh S, Shah D. Iterative learning for reliable crowdsourcing systems. In: Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a Meeting Held 12–14 December 2011, Granada: 2011. p. 1953–1961. http://papers.nips.cc/paper/4396iterativelearningforreliablecrowdsourcingsystems.
 Salakhutdinov R, Mnih A. Probabilistic matrix factorization. In: NIPS ’07: Proceedings of the TwentyFirst Annual Conference on Neural Information Processing Systems. Taipei: Curran Associates, Inc.: 2007.Google Scholar
 Rendle S, SchmidtThieme L. Onlineupdating regularized kernel matrix factorization models for largescale recommender systems. In: Proceedings of the 2008 ACM Conference on Recommender Systems. RecSys ’08. New York: ACM: 2008. p. 251–8, doi:10.1145/1454008.1454047. http://doi.acm.org/10.1145/1454008.1454047.Google Scholar
 Karimi R, Freudenthaler C, Nanopoulos A, SchmidtThieme L. Active learning for aspect model in recommender systems. In: Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining, CIDM 2011, Part of the IEEE Symposium Series on Computational Intelligence 2011, April 11–15, 2011, Paris: 2011. p. 162–7, doi:10.1109/CIDM.2011.5949431. http://dx.doi.org/10.1109/CIDM.2011.5949431.
 Yuen MC, King I, Leung KS. Probabilistic matrix factorization with active learning for quality assurance in crowdsourcing systems. In: ICWI 2015: Proceedings of The IADIS International Conference WWW/Internet 2015. Greater Dublin: 2015.Google Scholar
 Stewart O, Lubensky D, Huerta JM. Crowdsourcing participation inequality: a scout model for the enterprise domain. In: Proceedings of the ACM SIGKDD Workshop on Human Computation. HCOMP ’10. New York: ACM: 2010. p. 30–3, doi:10.1145/1837885.1837895. http://doi.acm.org/10.1145/1837885.1837895.Google Scholar
 Ross J, Irani L, Silberman MS, Zaldivar A, Tomlinson B. Who are the crowdworkers?: shifting demographics in mechanical turk. In: CHI EA ’10: Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems. New York: ACM: 2010. p. 2863–872, doi:10.1145/1753846.1753873. http://doi.acm.org/10.1145/1753846.1753873.
 Chilton LB, Horton JJ, Miller RC, Azenkot S. Task search in a human computation market. In: HCOMP ’10: Proceedings of the ACM SIGKDD Workshop on Human Computation. New York: ACM: 2010. p. 1–9, doi:10.1145/1837885.1837889. http://doi.acm.org/10.1145/1837885.1837889.Google Scholar
 Koren Y, Bell R, Volinsky C. Matrix factorization techniques for recommender systems. Computer. 2009; 42(8):30–7. doi:10.1109/MC.2009.263.View ArticleGoogle Scholar
 Yang S, Ye M. Global Minima Analysis of Lee and Seung’s NMF Algorithms. Neural Process Lett. 2013; 38(1):29–51. doi:10.1007/s110630129261x.View ArticleGoogle Scholar
 Yang S, Yi Z. Convergence Analysis of NonNegative Matrix Factorization for BSS Algorithm. Neural Process Lett. 2010; 31(1):45–64. doi:10.1007/s1106300991260.View ArticleGoogle Scholar
 Salakhutdinov R, Mnih A. Probabilistic matrix factorization. In: Advances in Neural Information Processing Systems: 2008.Google Scholar
 Zhou TC, Ma H, King I, Lyu MR. Tagrec: Leveraging tagging wisdom for recommendation. In: Proceedings of the 2009 International Conference on Computational Science and Engineering  Volume 04. Washington: IEEE Computer Society: 2009. p. 194–9, doi:10.1109/CSE.2009.75. http://dl.acm.org/citation.cfm?id=1632710.1633781.
 Allahbakhsh M, Benatallah B, Ignjatovic A, MotahariNezhad HR, Bertino E, Dustdar S. Quality control in crowdsourcing systems: Issues and directions. IEEE Internet Computing. 2013; 17(2):76–81. doi:10.1109/MIC.2013.20.View ArticleGoogle Scholar
 Organisciak P, Teevan J, Dumais ST, Miller RC, Kalai AT. A crowd of your own: Crowdsourcing for ondemand personalization. In: Proceedings of the Seconf AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2014, November 2–4, 2014, Pittsburgh: 2014. http://www.aaai.org/ocs/index.php/HCOMP/HCOMP14/paper/view/8972.
 Organisciak P, Teevan J, Dumais S, Miller RC, Kalai AT. Matching and grokking: Approaches to personalized crowdsourcing. In: Proceedings of the 24th International Conference on Artificial Intelligence. IJCAI’15. Pittsburgh: AAAI Press: 2015. p. 4296–302. http://dl.acm.org/citation.cfm?id=2832747.2832856.Google Scholar
 Ambati V, Vogel S, Carbonell J. Towards task recommendation in microtask markets. In: AAAI ’11: Proceedings of The 25th AAAI Workshop in Human Computation. Pittsburgh: AAAI Publications: 2011.Google Scholar
 Lin CH, Kamar E, Horvitz E. Signals in the silence: Models of implicit feedback in a recommendation system for crowdsourcing. In: Proceedings of the TwentyEighth AAAI Conference on Artificial Intelligence, July 27 31, 2014, Québec City: 2014. p. 908–15. http://www.aaai.org/ocs/index.php/AAAI/AAAI14/paper/view/8425.
 Atlas L, Cohn D, Ladner R, ElSharkawi MA, Marks II RJ. Training connectionist networks with queries and selective sampling. Adv Neural Inf Process Syst. 1990; 2:566–573.Google Scholar
 Lewis DD, Gale WA. A sequential algorithm for training text classifiers. In: SIGIR ’94: Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Dublin: Springer: 1994. p. 3–12.Google Scholar
 Scheffer T, Decomain C, Wrobel S. Active hidden markov models for information extraction. In: IDA ’01: Proceedings of the 4th International Conference on Advances in Intelligent Data Analysis. Cascais: Springer: 2001. p. 309–18.Google Scholar
 Culotta A, McCallum A. Reducing labeling effort for structured prediction tasks. In: AAAI ’05: Proceedings of the 20th National Conference on Artificial Intelligence  Volume 2. Pittsburgh: AAAI Press: 2005. p. 746–51.Google Scholar
 Dagan I, Engelson SP. Committeebased sampling for training probabilistic classifiers. In: ICML ’95: Proceedings of the Twelfth International Conference on Machine Learning. Tahoe City, California: Morgan Kaufmann: 1995. p. 150–7.Google Scholar
 Seung HS, Opper M, Sompolinsky H. Query by committee. In: COLT ’92: Proceedings of the Fifth Annual Workshop on Computational Learning Theory. Pittsburgh: ACM: 1992. p. 287–94.Google Scholar
 Yan Y, Rosales R, Fung G, Dy JG. Active learning from crowds. In: ICML’11: Proceedings of the 28th International Conference on Machine Learning. Bellevue: Omnipress: 2011. p. 1161–1168.Google Scholar
 Laws F, Scheible C, Schütze H. Active learning with amazon mechanical turk. In: EMNLP ’11: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Edinburgh: Association for Computational Linguistics: 2011. p. 1546–1556.Google Scholar
 Fang M, Zhu X, Li B, Ding W, Wu X. Selftaught active learning from crowds. In: ICDM: 2012. p. 858–63.Google Scholar
 Fang M, Zhu X. Active learning with uncertain labeling knowledge. Pattern Recogn Lett. 2014; 43:98–108. doi:10.1016/j.patrec.2013.10.011.View ArticleGoogle Scholar
 Jung HJ, Lease M. Improving quality of crowdsourced labels via probabilistic matrix factorization. In: Human Computation Workshop at the TwentySixth AAAI Conference on Artificial Intelligence: 2012.Google Scholar
 Jung HJ. Quality assurance in crowdsourcing via matrix factorization based task routing. In: International Conference on World Wide Web 2014: 2014.Google Scholar
 Yuen MC, King I, Leung KS. Taskrec: Probabilistic matrix factorization in task recommendation in crowdsourcing systems. In: ICONIP (2): 2012. p. 516–25.Google Scholar
 Yuen MC, King I, Leung KS. Taskrec: A task recommendation framework in crowdsourcing systems. Neural Process Lett. 2015; 41(2):223–38. doi:10.1007/s110630149343z.View ArticleGoogle Scholar
 Yuen MC, King I, Leung KS. Task matching in crowdsourcing. In: CPSCom ’11: Proceedings of The 4th IEEE International Conference on Cyber, Physical and Social Computing. Boston: IEEE Computer Society: 2011. p. 409–12.Google Scholar
 Yuen MC, King I, Leung KS. Task recommendation in crowdsourcing systems. In: KDD ’12: Proceedings of ACM KDD 2012 Workshop on Data Mining and Knowledge Discovery with Crowdsourcing (CrowdKDD). New York: ACM: 2012.Google Scholar
 Schnitzer S, Rensing C, Schmidt S, Borchert K, Hirth M, TranGia P. Demands on Task Recommendation in Crowdsourcing Platforms  The Worker’s Perspective. In: CrowdRec Workshop. Vienna: 2015.Google Scholar
 Geiger D, Schader M. Personalized task recommendation in crowdsourcing information systems – current state of the art. Decision Support Systems. 2014; 65:3–16. doi:10.1016/j.dss.2014.05.007. Crowdsourcing and Social Networks Analysis.View ArticleGoogle Scholar
 Aldhahri E, Shandilya V, Shiva SG. Towards an effective crowdsourcing recommendation system: A survey of the stateoftheart. In: SOSE. Boston: IEEE: 2015. p. 372–7. http://dblp.unitrier.de/db/conf/sose/sose2015.html\#AldhahriSS15.Google Scholar
 Dueck D, Frey BJ, Dueck D, Frey BJ. Probabilistic sparse matrix factorization. Technical report, University of Toronto. 2004.Google Scholar
 NAACL 2010 Workshop. http://sites.google.com/site/amtworkshop2010/data1.