These days, I am doing an extremely tedious job to look into a number of models that students in a course developed. (again using the i* model, which is not the important part. The point is to study how modeling notations and processes are applied) I am looking through their assignments that they developed by the i* notation, to find the mistakes they made (syntactical ones) and investigate why they made those mistakes. And the “why” part is actually the interesting part: Did they break the syntax of the notation, because they wanted to express something that the notation does not support? Was it visualization limitations on the screen or paper? Or they make the mistakes, because the way the models are developed with mistakes is more intuitive?
If one goes over enough number of models developed in real world practices, in academic papers, and in students' assignments, which we are doing it currently in our team as a large team work project, interesting patterns of mistakes emerge. If you realize the syntax and even the semantics of your notation is unconsciously or purposefully was broken, then you may need to change it in a way that is more acceptable and useable. On the other hand, there is a point that you need to stop, and decide after this decision line, the modelers and developers need more training and the notation was Ok. Finding where that line is located is hard. The harder job is to decide whether and how the notation and modeling process need to be changed based on the mistakes' pattern.
So, here we are, looking for common mistakes in one particular modeling notation (i*). I am wondering how we can generalize it for other modeling notations such as UML or even programming languages, to find out, how common mistakes can refine a notation.