The recent focus on intelligent tutoring systems (ITS) raises the question of how the diagnosing and subsequent remediation of students' errors should be done in order to facilitate learning. In the ITS paradigm, it is traditionally believed that rather than simply reteaching students, it is more effective first, to point out their errors then reteach them. Empirical research, however, shows contradictory findings (Swan, 1983; Sleeman, Kelly, Martinak, Ward, & Moore, 1989; Bunderson & Olsen 1983). While both methods focus on procedural knowledge, other researchers suggest that the subjects' previous knowledge, whether procedural or conceptual, should have important roles to play in the tutoring of students (Gertner, 1982; Resnick, 1982; Cauzinille-Marmeche & Mathieu, 1988; Lee, 1993). This study is thus designed to attest the importance of conceptual knowledge in the remediation of procedural errors. A method called Conceptual Dissonance was developed and its effect on the remediation process was compared with that of previous methods. Results showed that students tutored with Conceptual Dissonance performed better in the retention test. The effects of tutoring seemed to last longer if conceptual knowledge was included in the tutoring process.

The use of computers for educational purposes has been in existence for approximately thirty years. In the early years, computers were used for the designing of adaptive systems to individualizing tutoring. This kind of system was referred to as the Computer Assisted Instructions (CAI). Builders of such systems must prespecify all available routes through the space of teaching possibilities. Every test, every decision, every branch to any remedial material and every exposition must be written in advance. (Goodyear, 1991). It was obvious that the combinatorial problem would make this kind of work too complex for practical use. Another use of computer in education comes from a completely different approach in which all available routes do not have to be prespecified. Instead, knowledge required for tutoring as well as the domain-specific knowledge would be represented explicitly in the system. Decisions on the use of expositions or remedial materials would be decided in real-time based on the knowledge in the system. Also materials would be generated when required. Thus, this kind of system, referred to as the Intelligent Tutoring System (ITS) does not require a large system to incorporate the large number of routes, is more flexible and may be applicable to more situations.

Reteaching and Model-based Remediation

One of the major differences between the designing principles of the traditional CAI and ITS is their different approach to remediation. In traditional CAI, the remediation is done by merely reteaching the materials in which the subject's errors are found. For example, when a student makes an error in solving algebraic equations, the rules of solving equations are retaught and the subject has another chance to repeat the problem. However, in ITS, a basic assumption is that errors are systematic and that the diagnosis is more difficult than the remediation. Once an accurate model of the student's error has been inferred, it is then relatively straight forward to use that model to direct a remedial dialogue (Sleeman, Kelly, Martinak, Ward & Moore, 1989). Sleeman et al. referred to this method of providing procedurally orientated remediation of specific errors found in a student's solution before reteaching a correct strategy as the model-based remediation (MBR). From the view of ITS designers (Brown & Burton, 1978; Resnick, 1984; Macnab & Cummine, 1986), this MBR works better than the traditional reteaching employed in CAI.

Empirical research, however, shows contradictory findings. Swan (1983) reports that a conflict approach (by pointing out errors made by students and demonstrating their consequences) is more effective than simple reteaching. On the other hand, no difference between the error-specific remediation and reteaching has been reported (Bunderson & Olsen, 1983; Martinak, Schneider, & Sleeman, 1987; Sleeman, Kelly, Martinak, Ward, & Moore, 1989). This research has had great impact on the design of ITSs. As reteaching was mostly done by reteaching in the "classical" computer-assisted instruction (CAI), while ITS relies mostly on the error-specific approach, the result would imply that CAI is as effective as ITS although the latter is more cognitive orientated.

One may attribute the similar effects resulting from the two remediation methods to the lack of attention on the part of the student. If student does not pay attention during the remediation processes, the use of either method would be immaterial since no learning would actually occur. However, empirical finding by Sleeman, Kelly, Martinak, Ward, & Moore (1989) shows that subjects do better in the posttest than in the pretest. Hence, it is clear that learning does occur and that lack of attention cannot be the reason for the similar effects resulting from the two methods.

Cognitive Engagement and Cognitive Dissonance

The error-specific approach was referred to as the model-based remediation (MBR) since the remediation is based on a student model inferred from what the student has used in solving the given problem. As the MBR method focuses on the student's specific errors, it should be more effective than Reteaching only since students' cognitive load would be much reduced by focusing only on the part where the error occurs. This might be what Sleeman et al. (1989) assumed initially, leading them to believe that the similar results obtained from MBR and Reteaching only come about because the students were not cognitively involved in the remediation processes. They then tried to add two additional components, namely inducing cognitive engagement and inducing cognitive dissonance, to the model-based remediation and then comparing the effects of these two new methods with that of Reteaching only. Cognitive Dissonance was created to students by demonstrating the unsound nature of the student's incorrect method (Macnab & Cummine, 1986) while Cognitive Engagement was done by having students verbally repeat the correct procedure back to the tutor (Sleeman, Kelly, Martinak, Ward, & Moore 1989). However, results of both the posttest and retention test again showed no significant difference between the two conditions. Finally, arising from the suspicion that the existence of unstable errors might dilute effects due to the two remediation methods, the researchers focused only on stable errors and attempted to find which of the methods would be more effective in reducing errors. Again, no significant difference was found.

Two reasons were given by Sleeman et al. (1989) to explain the little difference of effects resulting from the two remediation methods: viz. that MBR and Reteaching only are too similar and students in reteaching generated their own MBR. Further investigation is needed to establish whether these explanations are correct, however, a method better than the traditional Reteach only is required. Otherwise, there will be no theoretical basis for the tutoring methods currently employed in intelligent tutoring systems.

While both the methods MBR and Reteaching only focus on procedurally correcting students' errors, mathematics educators have already pointed out that meaningful learning based on the understanding of concepts related to the procedures is more beneficial to students than learning the procedures by rote (Davis, 1985). Even with the two additional components, Cognitive Engagement and Cognitive Dissonance, the tutoring processes are still procedural orientated: Students are only required to rehearse the rules in Cognitive Engagement condition. It is doubtful whether this would induce students to actually cognitively engaged in learning the rules, not to mention using these rules in later problem solving tasks.

On the other hand, in the Cognitive dissonance condition, dissonance is induced by having the students notice the incorrect results generated when their solutions are substituted back into the original equations. The idea may be that students would be more conscious of the errors made when dissonance is induced. However, though the students may be more aware of their errors, if there are no suitable means to help them to use the correct rules, this induced dissonance could do no more than point out their errors. It is commonly acknowledged that telling students they are wrong does not help them to correct their errors.

Connection to Conceptual Knowledge

Many researchers agree that students rote-learn mathematics algorithms without connecting them to the underlying semantic information. Systematic errors would then occur when the algorithms are misused (Resnick, 1982). Sternberg (1985) also pointed out that reasoning may consist of the manipulation of mental models that correspond to internal analogues of scenes of actors and errors made are due to that people fail to consider all the possible models of the premises. Both semantic information and mental models are, in a broad sense, conceptual knowledge related to the procedures where errors occur. Both researchers seem to suggest that increasing conceptual knowledge would be an effective way of reducing systematic errors.

Concerning mathematics, and especially on algebra, Cauzinille-Marmeche and Mathieu (1988) suggest that students encounter such difficulties in assimilating the rules for rewriting expressions because these rules seem to them an arbitrary collection, independent of each other and disconnected from their previous knowledge. These difficulties may be derived in part from students' and teachers' tendency to treat algebra as a purely formal system, without reference to the number relationship and situational constraints that give it referential meaning. Evidence from research by Hinsley, Hayes & Simon (1976) shows that students can have considerable difficulty in relating algebra equations to basic ideas such as equivalence and functional relationships.

Microworlds

In studying students' comprehension of literal or numerical expressions of the form a+b+c, with or without brackets around either the ab or the bc portion of the expression, Cauzinille-Marmeche and Mathieu (1988) argued that students' responses can be classified into three categories viz. the arithmetical, the algorithmic and the formal syntactic microworlds. In the first one, the expressions to be examined are analyzed as a chain of transformations applied to an initial quantity. In the second one, the expressions to be compared are examined by actually calculating the quantity, and if the expressions are literal, letters are replaced by numbers. Finally, in the formal syntactic microworld, the expressions are analyzed as chains of symbols. The researchers found that students, especially young ones, spontaneously refer to distinct microworlds and that these microworlds were not necessarily related to each other and did not form a coherent structure. They then suggested that inducing students to change their representation and to establishing links between different representations can help them understand and use the newly introduced algebraic rules. This instructional strategy was supported by experiments done on electricity and electronics (Gentner & Gentner, 1982) and elementary mathematics (Resnick, 1982).

Practice and Conceptual Links

While there is no conclusive agreement on whether MBR or Reteaching only would be more beneficial to students, it seems that the work done by Marmeche and Mathieu (1988) suggest a new direction for the remediation process. This strategy was also supported by Lee (1993). In building a neural network to explain the process involved in simplifying mathematics expressions, Lee suggested that during the process, several rules may be available to the student, from which he chooses only one of them. The choice depends on the relative strengths of the rules; stronger ones are more likely to be chosen. Hence, to correct errors, or more explicitly, to correct the procedure that produces errors, this procedure must be unlearned. According to Lee (1993), knowledge can only be unlearned through the learning of a new procedure with prerequisite conditions identical to the one to be unlearned. By increasing the strength of this new procedure, it has more chance to be chosen, at the same time reducing the chance for the old procedure to be selected. In this sense, the old procedure is unlearned. The learning of a new procedure should be done in two ways: practicing and building more links to subjects' precious knowledge. Building more links enables the correct procedures to be more easily accessed, while practice increases the strength of these new procedures. While both MBR and Reteaching allow subjects have more chances to practise, there is no direction on what content to practise. It is possible that the mal-rules as well as correct rules will be rehearsed which then reduces the effects of remediation. However, the most important consideration is that neither method can help students to build up more links to the previous knowledge. It is therefore understandable that neither method can produce better tutoring effects. The situation cannot be improved even if cognitive engagement and cognitive dissonance were added to MBR.

This argument leads to an assumption that linking students' errors to their previous knowledge may be a good way to helping students correct their errors. As errors are pieces of incorrect knowledge while previous knowledge is believed to be correct in most cases, this linkage would normally induce dissonance. Thus in the present study, a new tutoring method, called Conceptual Dissonance (CD), is designed based on this assumption. The effect of this method is compared with that of MBR and Reteaching only. Additionally, in order to substantiate the assumption pointed out by Lee (1993) that practising enhances learning of rules, the effect of practising was separately measured. It is hoped that in this study, a clear direction on strategies to be used in remediation process can be identified.

Materials

Materials used included a 20-item pretest, a 20-item posttest, a 20-item retention test, a Reteaching tutoring script, a MBR tutoring script, a CD tutoring script, and a list of tasks for practice. Items in both pretest and posttest were those used in Sleeman et al. (1989), while items in the retention test and in the list of practice tasks were made equivalent in form and difficulty to those in the pretest and posttest. Students could earn five marks for each item correctly done, making a total of one hundred marks for each of the three tests. The Reteaching and MBR tutoring script (see Appendix 3) were also those used in Sleeman et al. (1989) but were translated into Chinese. The CD script directed the tutor to point out the errors, induce dissonance on the students and then reteach the required procedure. There was also a set of manipulative rules (Appendix 1) and a set of remedial rules (Appendix 2) to be used in the tutoring scripts. Manipulative rules described the correct procedures for performing the tasks, while remedial rules, based on subjects' previous knowledge, explained why they were wrong and how to correct their errors..

Procedure

The experiment was performed in four stages, viz. pretest, tutoring, posttest and retention test. One hundred and twelve secondary two students from a Chinese Middle school in Hong Kong participated in the pretest. The academic standards of the students were assessed by their teachers to be below average. The pretest was done in June 1994. After the pretest, fifty-two students with scores less than seventy were identified as requiring tutoring. These students were randomly assigned into three tutoring groups: Reteaching, MBR and CD. Students in the Reteaching group were just retaught the parts in which they had made errors in the pretest, while those in the MBR group were given procedurally orientated remediation of specific errors found in their solutions before reteaching. Lastly, in the CD method, errors made were intentionally made dissonant to their previous conceptual knowledge. Half of the students from each group were randomly chosen to perform three more practice tasks while the other half did not. The students from the three tutoring groups were then combined as the Practice group, and the remaining students formed the Without Practice group.

The tutoring was done in three consecutive days approximately three weeks after the pretest. Each student was tutored individually on the tasks in which errors were found in the pretest. The tutors were postgraduate students and were trained to follow the tutoring scripts as shown in appendix 3. Tutoring of each task was complete when the student could redo the task correctly before the tutor. For the Without Practice group, tutoring of the next task then followed until all the tasks were dealt with. For those students in the Practice group, they were required to do three more identical tasks before they could continue with other tasks. The tutoring lasted for approximately 10 to 40 minutes, depending on which method was used, whether practice was required and how the student reacted. A summary of the average duration of the six categories is shown in Table 1.

Immediately after the tutoring, all students participated in a posttest. As the tutoring was taken in June, which was the end of the 1993- 1994 academic year, all the subjects had to participate in their final examination after tutoring and posttest,. The retention test was taken in September when the subjects returned to school after the summer vacation. It is believed that the students were not involved in any kind of formal learning between the two posttests. The final number of subjects was fifty, as two of the participants for various reasons could not take part in the retention test.

Result

Effects of tutoring

Table 2 shows the means and standard deviations of scores in the pretest, posttest and retention test by tutoring conditions.

Analysis of variance showed there was a significant overall mean difference among the three tests for all groups (p<.01). Post hoc analysis showed that mean scores in both posttest and retention test were significantly higher than those in the pretest (p<.01 and p<.05). The mean score in the posttest was significantly higher than that of the retention test (p<.01). The differences between the posttest and pretest as well as the difference between the retention test and pretest clearly showed that the remediation methods were beneficial to students. Although the effects of remediation deteriorated in the retention test held three months after the posttest, the score was still higher than that of the pretest. This latter result is contradictory to that of Sleeman et al. (1993) who claimed that the overall mean scores for the delayed posttest (two months after the first posttest) had reverted to the pretest levels. In order to investigate this discrepancy, the three tests for individual tutoring methods were compared. The results showed that for all three tutoring methods, significant differences were found among the three tests (p<.01). For the differences between pair of tests, the results of analysis are as shown in Table 3.

It can be seen that all the differences were significant except for the differences between the pretest and retention tests for the Reteaching and MBR groups. Hence, if according to Sleeman et al. only these two groups were considered, there should be no significant difference between the pretest and retention test. The significant difference between the pretest and retention test was due to the effect of the CD method, which in a way proves that CD is superior to the other methods in retaining the learning effect.

Scores in Posttest and Ceiling Effect

When the scores of Posttest were taken for the different tutoring methods including the groups with and without practice, no significant differences were found among the six groups. As all the mean scores in the posttest were quite high, it is possible that these scores had reached the ceiling that even better methods cannot show higher effects. Detailed discussion of this ceiling effect will be discussed in later sections.

Effects of Practice and Tutoring Methods on Retention test

Table 4 shows the mean scores in the retention test by practice and tutoring conditions. Table 5 shows the results by using two-way analysis of variance with pretest as covariate.

Significant difference was found between the practice and no-practice groups (p<.05) which proves that practice does enhance tutoring effects. For the three tutoring conditions, no significant differences could be found. However, the significant level was not far from being acceptable (p=.081). It had just been shown that practice improved the subjects' performance. However, if subjects' performance had reached their ceilings, then tutoring with better methods would make no difference. Hence the ceiling effect might again attenuate the effect brought about by tutoring methods, especially for the current rather low significant level. In order to clarify this, only the scores of those subjects who did not practice were analyzed. The result is presented in Table 6.

Significant differences among the three conditions were found (p<.05). This clearly showed that when the effects of practice were removed, effects due to the tutoring methods might surface. This confirms that different strategies have different effects on tutoring and that the effects are attenuated by the effects of practice.

When different tutoring methods were compared, it was found that CD condition was significantly better than MBR. When different tutoring conditions were compared with the overall effect, scores in the CD condition were found to be significantly higher (p<.05) while scores in the MBR condition were significantly lower (p<.01).

Conclusion

The result of the analysis showed that students tutored under all the tutoring methods scored much higher in the posttest. Although the effect was found to be deteriorated between the posttest and the retention test, yet students still fared better in the retention test than they did in the pretest. This proves that the teaching methods in general were effective in improving students' algebraic skills. Further analysis of the differences between pretest and retention test showed that significant difference could be found only in the CD group. This suggests that CD may be the best among the three strategies in helping subjects to retain the effects of learning for a longer time.

When different tutoring methods were compared, significant differences could only be found in the retention test but not in the posttest. This may be due to the ceiling effect. The ceiling effect on the posttest scores is obvious if we look at the mean scores: The overall mean score was found to be 85.39. For the six individual groups (three remediation methods either with or without practice), five of them obtained group means higher than 84. Consideration must be given to factors such as attention span, slips and tiredness because the subjects had to participate after school. These scores are too high for the effects of different methods to be differentiated.

On the other hand, when the retention test was taken after a long summer vacation, the average score decreased. At this time, the effects due to different tutoring methods and practice conditions could be observed. Although only different practice conditions showed significant differences at first, different tutoring methods also showed their effects with further analysis when subjects with practice chances were ignored. It seems that besides affecting the posttest, the ceiling effect also affected the retention test. In this case, comparison among the different tutoring methods showed CD seemed to be the most effective method while MBR fared the worst.

In contrast to previous findings which showed no difference between MBR and reteaching, this study found that the former was more effective. However, something must be done besides just pointing subjects' errors based on the subject models inferred and then reteaching. For the remedial tutoring to be effective, students should be induced to understand how the causes of errors would contradict their previous knowledge and why the correct rules should be used. Further, immediately after the students can correctly redo the problems, they should be asked to do some more practice on similar problems to consolidate the knowledge just learned.

As model-based remediation and reteaching are essential parts of ITS and CAI respectively, it seems that ITS should have better effects than its counterpart. It is suggested here that ITS would be a better tutoring tool than CAI if suitable conceptual knowledge can be incorporated.

Discussion

The effects of tutoring and practice conditions were found to be attenuated by the ceiling effect in the present study. Occurrence of the ceiling effect seems to be due to the fact that the tasks used were quite easy for the subjects. To further investigate the situation, it is suggested that more difficult problems be used in later studies. Also, there are two limitations on the generalization of the results obtained: As the test involved solving algebraic equations with one variable tutored by human tutors, it is doubtful whether the results obtained are applicable to other types of problems and computer tutors. This needs to be clarified before the strategies suggested could be incorporated into an ITS.

There are two findings in this study which do not quite agree with that from Sleeman et al. (1983). The first one is that the scores in retention test were found to be significantly better than those in the pretest while in the Sleeman et al. study, no significant difference was found. This discrepancy was further proved to have resulted from the effect of CD. Hence, the discrepancy may have resulted from the use of a method which the student had not used before.

The second discrepancy comes from the fact that in the study of Sleeman et al., no significant difference could be found between MBR and Reteaching only. Although the same result was found in the present study, it was further found that MBR had the least effect on remediation when compared with the overall results. This is strange since the best strategy found, viz. the CD method, is also a model-based remediation. The only difference between the two is that in CD condition, students were told why the rules used are incorrect and why the correct rules should be used. Hence, it would be possible that just pointing out subjects' errors without telling them the reasons would have the adverse effect of blocking learning rather than helping it.

Appendix 2: Remedial rules

1. 依你看?: (1+2)*3,跟 1+(2*3) 有什麼不同? 如二者不同的話, 1+2*3 應取那一個值呢?

請留意這式跟上述二者唯一不同的地方,是有沒有括號及括號的位置. (等候學生回答)

習慣上, 我們是先乘除後加減的. 即是說 1+2*3 = 1 + (2*3).

1. 因 為 我們的答案的形式是 x=____

如果不將所有沒x的項移往右邊,而左邊只保留有x的項, 我們怎能得到答案呢?

1. 將 +3 移往右邊, 要變成 -3 (-3 變成 +3, *3 變 /3, /3 變 *3), 如果不這樣做的話. 我們看看會怎樣:

x + 3 = 5 變成 x = 5 + 3

前式中的x 只要是2 便可以了, 但後式中的x 則變成 8 , 所以這是錯誤的.

1. 例如說: 1+2=3, 這是對的, 但如果我們只在左邊加 4, 即是說:

1+2+4= 3, 這對不對呢?

(可改為° 1+2-4 = 3, 1+2 /4 = 3, 1+2 *4 = 3)

1. 例如說: 2*(3+4) 應該是2*7 = 14, 而2*3+2*4=6+8=14, 所以2*(3+4)=2*3+2*4. 如果好像你剛才那樣做,....(模仿學生的做法,但以數字代入)...., 左邊便不等於右邊.

Appendix 3: Tutoring Scripts

Tutoring Scripts: Reteaching without rehearsal:

1. [FRESH PAPER]

2. Have student work the task aloud.

3. If wrong, say "這是錯的".

4. [FRESH PAPER]

5. say," 讓我做給你看...., 原因 是... (重覆 Manipulative rules 中適用者)".

Reteaching with rehearsal:

1. Repeat steps 1 to 5 in above

2. GIVE PRACTICE TASKS

MBR without rehearsal:

1. [FRESH PAPER]
1. Have student work the task aloud
1. After the student has completed the task, go back to EACH error, say:

" 看來你這兒做錯了, 原因 是... (重覆 Manipulative rules 中適用者)".

1. say," 讓我做給你?...., 原因 是... (重覆 Manipulative rules 中適用者)".

1. Repeat steps 1 to 4 in above

2. GIVE PRACTICE TASKS

MBR with conceptual dissonance:

1. [FRESH PAPER]
1. Have student work the task aloud
1. After the student has completed the task, go back to EACH error, say:

" 看來你這兒做錯了, 原因 是... (選用Remedial rules 中適用者)".

1. say," 讓我做給你?...., 原因 是 ...(重覆 Manipulative rules 中適用者)".

1. Repeat steps 1 to 4 in above
1. GIVE PRACTICE TASKS