DIAGNOSYS is a knowledge-based (or expert) system primarily for testing background knowledge of basic mathematics or other technical subjects. DIAGNOSYS has been obtained by over 60 universities and colleges in the UK and other countries, providing useful information to the student and to their tutors for individual attention, for identification of special groups, and for course design.
DIAGNOSYS v1.0 to 2.3 were the product of TLTP Phase 1 project number 17 based at the University of Newcastle upon Tyne with the collaboration of the other four North-East universities. The aim of the project was to produce a computer based diagnostic testing system for mathematics and other technical subjects. Development since then has been continued without TLTP funding, producing v2.4 and now v3.0.
With the huge increase in student intake, a wider variety of mathematical backgrounds has created a problem of identifying the weaker students who may benefit from extra tuition. In many departments, paper based diagnostic tests have been used to identify those students at risk. However, with the increasingly large numbers of students it becomes impractical to give feedback on large groups. The volumes of work involved with a paper based test also greatly increase the difficulties in obtaining statistics on how the students coped with their maths after the test (both those who did well and the those given remedial classes).
Much mathematical knowledge and skills has a hierarchical nature. That is, one can think of 'lower' skills and 'higher' skills, where it is not possible to learn the latter until the former has been mastered (except in a limited 'pattern matching' sense). This idea can be used to reduce the number of questions asked, in roughly the same way that a human interviewer would.
For example, expansion of (x+1)(x+2) implies an ability to expand x(x+2). Therefore, if a student were asked the first question, and could do it, we can infer that they could do the second.
Conversely, if they were asked the second, and couldn't do it, we can infer that they could not do the first. Thus we can approximately halve the number of questions asked.
DIAGNOSYS is therefore an 'expert system' designed around a network of skills. This network has a hierarchical structure with related skills linked. The network consists of 92 skills in the following mathematical areas: arithmetic, algebra, calculus, trigonometry, statistics and numeracy. The network has four levels of difficulty ranging from pre- GCSE (level 1) to approx. AS level (level 4).
On running the test the student is asked what is their highest maths qualification. Depending upon the student's answer the network is initialised. In the initialisation each question is marked as either 'possibly known' or 'unasked'. So, if the student has reached a GCSE standard in maths then all level 1 and 2 questions will be marked as 'possibly' and the level 3 and 4 as 'unasked'. The test is then started and the network of skills is traversed by using the following rules:
If a question on a given 'skill' is answered correctly (possibly after a second attempt, with a maximum number of 'lives'), then it is inferred that all linked prerequisite skills are probably known, and dependent skills (usually at a higher level) are 'possibly' known.
If a question is answered incorrectly, then it is inferred that all dependent skills are probably unknown.
At each point, the question chosen will be one that is 'possibly' known, with no dependent skill also possibly known.
The test terminates when there are no questions left to be asked or when a time limit is reached.
The complete skills network is split into six areas. To view an area of the network choose one of the following:
Please note that some of these files are quite large.
DIAGNOSYS is actually a 'shell', which means that just about any subject which can be described in a hierarchical structure can be tested. Already a diagnostic test on mechanics has been created using DIAGNOSYS.
Features of the basic mathematics test implemented on DIAGNOSYS are:
The maths input part has been written separately, in the form of a stand-alone module available to other developers. This offers maths typesetting (similar to that in MS Word) but offering output as a string suitable for parsing. Other features include run-time composition of maths expressions and of graphs, avoiding the storage of many large bitmaps.
The Mathinput tool is used for mathematical and numerical input throughout the test. The appearance and operation are very like the Equation Editor supplied with MS Word, but this editor provides an 'intelligent' output for mathematical programs.
MATHINPUT is now available as a DLL, with demonstration program to run it, and is FREE to developers. You can download it with DIAGNOSYS v3.0.
Feedback from the test comes in two forms: that intended for the student, and that intended for the tutor. On completing the test a text file is generated which contains the diagnosis of the student's performance.
On completion of the test various post processing programs allow the tutor access to information pertaining to the group of students tested. There are five post processing options available with the current version of DIAGNOSYS. These create a group profile of all the students tested, a ranked listing of the students in terms of score, tabulated answers actually given to all questions by all students (to highlight common misunderstandings), results of all questions and skills for subsequent spreadsheet analysis and a combined file of individual 'profiles' ready for printing.
These programs should help in targeting a course to fit the needs of its members and highlight which students may be in need of extra help.
Examples of output from DIAGNOSYS v3.0 follow:
An example student profile in bar chart form
Name: Appleby Date: 14 9 1998 Qu asked: 37 Qu correct: 23 Total mark (on ALL qu'ns) 74% ***************************** TOPICS numbers 100% **************************************** powers 80% ******************************** algbasic 100% **************************************** algmethods 50% ******************** equations 50% ******************** miscell 67% ************************** graphs 40% **************** areavol 56% ********************** GENERAL COMMENTS You should note any weaknesses in your basic mathematics. Your tutor will advise you on additional work needed, and which books or materials to use. Time used: 18 Lives: 5 Lives used: 5 - successfully: 3
Columns are %:
1: tested on topic (total) 4: qual{AS-level}
2: successful on topic (total) 5: qual{Other}
3: qual{A-level}
1 2 3 4 5
101 Multiplication of negative numbers 75 75 100 0 100
102 Multiply negative and positive 75 75 100 0 100
103 Negative Numbers 75 75 100 0 100
.........
203 Definition of negative powers 100 100 100 100 100
204 Rules for positive powers 100 100 100 100 100
205 Inverse ratios 75 75 100 0 100
206 Cancelling numerical fractions 75 75 100 0 100
.........
210 Expanding one bracket 75 75 100 0 100
211 Collecting terms 75 50 100 0 0
212 Solving linear equations 75 75 100 0 100
.........
312 Expanding two brackets 100 75 100 100 0
313 Difference of squares 100 50 50 100 0
314 Simultaneous Equations 100 0 0 0 0
.........
411 Solutions of a quadratic 75 50 50 100 0
441 Product rule 25 25 0 100 0
442 Integration of powers 25 25 0 100 0
.........
Total in each column: 4 4 2 1 1
Average mark: 68 62 84 65
Average time used: 7 10 3 5
Average lives used: 3.0 2.5 2.0 5.0
Av used successfully: 1.8 1.5 1.0 3.0
Av questions asked: 32 25 34 44
Av questions correct: 23 18 28 30
An example of an alphabetic listing
Appleby John 74 Davies Janet 84 Smith Frank 65 Williams Simon 49 Averages: mark: 68.0 lives used: 3.0 - successfully: 1.8 time: 7.0 questions asked: 31.8
Return to contents.
last updated 17/12/99