MODERN METHODS FOR CALCULATING THE LEVELS OF THE DIFFICULTY OF JOBS AND LEVELS OF PREPARATION OF TESTED

In the conditions of rapid development of modern technologies, the role of the human factor and the risks associated with it increases in enterprises, therefore, the requirements for methods of checking the level of professional training of employees increase. The level of professional training of specialists can be estimated by the results of their training at the University, training courses, etc. Incorrect evaluation of the results of the test in the future can lead to various problems in the workplace, such as material losses of the enterprise or even the risk of human casualties.

The relevance of this work is to reduce the error of the evaluation systems of learning outcomes, thereby increasing the level of training of the test and reducing the level of risk at enterprises. The object and subject of the study are the theory of latent variables based on Rush models and the theory of refined calculation of estimating the difficulty of diagnostic tools and the level of training of test subjects. For tasks that have more options for the outcome of the event (for example: true, partially true, incorrect), consider the polyatomic model of Rush.

The use of the theory of latent variables, based on Rush models, allows to make independent estimation of the calculated values of the latent parameter «level of preparation of tested» (LT) from the values of the level of «difficulty of tasks» (DT). This helps to obtain more objective assessments of the level of training of students.

The term «Latent variable (parameter)» is usually understood as a theoretical construct that characterizes some hidden property or quality (for example, the level of training of the student, the difficulty of the test task), which can not be directly measured [1, p. 7].

In the theory of latent variables, each test participant is assigned one value of the latent parameter that determines the observed test results. The result of each test task depends on the difference between the values of the latent parameters of the test and the task. In contrast to the classical theory of tests, where the individual score of the test is considered as a constant number, in the theory of latent variables, this parameter is considered as a variable, the value of which is on the empirical data and then sequentially refined.

Since the level of training of the student is a latent parameter and can not be directly measured, it can be obtained by testing knowledge with a certain probability. It is impossible to say exactly whether the student will be able to correctly solve the proposed task, but it is possible to calculate the probability of this outcome of the event.

The system of recurrent ratios can be used to calculate the latent parameters of the UT of the tested  and UT tasks (indicators) :

                                    (1)

                                 (2)

where  – level of preparation of the tested;

 – the level of difficulty of the task;

 and  – individual test scores;

 – test subject number;

 – job number;

t – iteration number;

– number of test takers;

 – the number of items (indicators);

– probability distribution density of the latent variable.

This probabilistic model, represented by formulas (1) and (2), is a model obtained by the Danish mathematician G. Rush to assess the level of educational achievements of students, which was called the dichotomous or basic model of Rush.

Since the theory of latent variables is probabilistic in nature, it assumes that there is a one-dimensional continuum of the latent variable on which the probability distribution of the latent variable with density occurs  [3]. In the dichotomous Rush model, this distribution is described by the logistic function:

                                                          (3)

By transforming the initial test scores into a scale of natural logarithms, G. Rush introduced a general logarithmic measure for the parameters of the model   and , which he called logit.

Logit is a unit of measurement of the level of readiness of the test participant and the difficulties of test tasks within the framework of logistic models of testing [4].

Figure 1 shows the characteristic curve of the j-th test task, showing the relationship between the values of the independent variable θ and the value that was calculated using the formula (3).

Picture 1. Characteristic curve of the j-th task

From formula (3) and figure 1 you can see that the probability of successful completion of the test depends on the interaction of two variables: LT test  and DT test . It is logical to assume that if the LT of the i-th test subject is higher than the j-th task, then the test subject is likely to correctly answer this task. The opposite is also true – if the abilities of the i-th test subject are lower than the difficulty of the j-th task, then the subject will not be able to perform this task correctly.

Formulas (1) and (2) were obtained using the maximum likelihood method of R. Fisher and Newton's method for solving nonlinear equations:

                                                       (4)

                                                     (5)

where  – error of calculation of LT tested;

 – calculation error DT job.

In the new approach, as a quantitative measure of the LT of the tested, it is proposed to use the ratio of the sum of the difficulties of all correctly performed tasks to the sum of the difficulties of all the tasks of the diagnostic tool used – the test. And as a quantitative measure of DT tasks is – the ratio of the sum of the levels of training of students who incorrectly completed the task to the sum of the levels of training of all students in the sample.

A new unified iterative process of computing ratings of LT under test and DT assignments was experimentally proved. The new approach takes into account the contribution of students of different levels of training and tasks of different difficulties in the evaluation. On the example of processing the results of centralized testing of schoolchildren, it was shown that the estimates of the level of difficulty of tasks calculated using the proposed iterative process can be 30 ÷ 66 percent different from the DT tasks calculated by individual points. As a result, new recurrence relations were derived [2, p. 105]:

                                                   (6)

                                           (7)

where  – vector LT test subjects;

 – the vector DT jobs;

N – number of test takers;

M – number of jobs;

 – the result of the run j-th task i-th students;

k – iteration number.

On the basis of relations (6) and (7) a mixed sequence of approximations to the desired vectors  and is constructed [2, 105]:

                                                   (8)

As initial conditions, you can select either a vector  or a vector , all values of which are equal to one. The process will converge faster if you select vectors (9) and (10) as the initial vectors:

                                                           (9)

                                                          (10)

                                                             (11)

                                                        (12)

where  – share of correct answers of the i-th student;

 – the share of incorrect answers of students to the j-th task;

 and  – individual test scores.

Latent parameters  and  in logit are calculated by the formulas:

                                                                 (13)

                                                               (14)

Calculation errors are calculated by formulas (4) and (5).

In contrast to the dichotomous model, which provides only two options for solving the problem, namely true and false, the polyatomic model allows more options, for example: true, partially true, incorrect.

Such tasks include test matching, multiple correct answers, etc. polyatomic model of Rasch is often used in processing the results of psychological and pedagogical tests.

The polyatomic task can be considered as a multi-step task, for the performance of which the student can get from 0 to m points. At the same time DT each step are different and independent from each other.

The probability that the n-th test taker will get k points for completing the i-th task is determined by the formula [5]:

                                             (15)

where – the probability that the n-th test taker will get k points for completing the i-th task;

 – the difficulty of performing the j-th step in task i.

Formula (6) defines a Rush model with arbitrary intermediate categories of task execution or a polyatomic Rush model. The conditional probability  of the test n to perform the k-th step in the task i, provided that the (k – 1)-th step is performed correctly, is determined by the formula [5]:

                                   (16)

The results of the polyatomic tasks can be represented as a three-dimensional matrix, the rows of which on the x-axis will represent the polyatomic profiles of the responses of each tested for all tasks. The columns on the y-axis are the polyatomic profiles of the responses tested for each job, and the rows within each column on the z-axis are the dichotomous profiles of the specific job steps.

The polyatomic rush model implies that a test participant can simultaneously perform x job steps that are in a certain area of the latent variable, and not perform the remaining  steps. The overall difficulty of the i-th task as the average value of difficulty of all the steps will be calculated according to the formula [5]:

                                                               (17)

In the polyatomic model, along with the already familiar latent variables  – LT of the students and  – DT of the task, the third variable  – the difficulty of each j-th threshold of the task is introduced. All these latent variables are computed by iterative methods, in particular the above-mentioned maximum likelihood method.

Список литературы:

1. Елисеев, И.Н. Математические модели и комплексы программ для автоматизированной оценки результатов обучения с использованием латентных переменных: диссертация / И.Н. Елисеев – Новочеркасск, 2013. – 371 с.
2. Шрайфель, И.С. Теоретическое обоснование единого итерационного процесса совместной количественной оценки трудностей заданий и уровней подготовки студентов / И.С. Шрайфель, И.Н. Елисеев // Сиб. журн. вычисл. математики / РАН. Сиб. отд-ние – Новосибирск, 2016. – Т. 19, № 1. – С. 103 – 119.
3. Точность моделирования латентных переменных с помощью модели Раша (Часть 1) [Электронный научно-практический журнал «Современные научные исследования и инновации»]//URL: http://web.snauka.ru/issues/2014/06/34399 (дата обращения 02.04.2019).
4. Инсай – вертикальное развитие: логит [Электронный ресурс]//URL: http://insai.ru/slovar/logit (дата обращения 03.04.2019).
5. Карданова, Е.Ю. Доказательство применимости политомической модели Г. Раша: Вестн. Новгородского. Гос. ун-та № 39 – Новгород: ИПЦ НовГУ, 2006. – 92 с.