Teachers get it handy!

[johnnycool]

My daughter has never got anything less than an A*/A in any exam got her A/S results this morning and was awarded AAB and is very unhappy. A complete Shitshow from CCEA, Weir has to go.

They evidently don't trust the teachers.

[johnnycool]

Ahem............

This is Engerland though"

[currychip]

My daughter has never got anything less than an A*/A in any exam got her A/S results this morning and was awarded AAB and is very unhappy. A complete Shitshow from CCEA, Weir has to go.

They evidently don't trust the teachers.

I don't think you can get A* in AS, only a max of A.  Maybe one of the teachers out there can confirm this.

[Milltown Row2]

My daughter has never got anything less than an A*/A in any exam got her A/S results this morning and was awarded AAB and is very unhappy. A complete Shitshow from CCEA, Weir has to go.

They evidently don't trust the teachers.

I don't think you can get A* in AS, only a max of A.  Maybe one of the teachers out there can confirm this.

That's true, was my mistake, A* in CGSE

Friend was speaking to admissions this morning at Queens, their son didn't get his course there (that being his first choice) was offered another college, but someone who was given unconditional offer on the same course managed to get in with a B-C-U when you needed a A and 2 B's! for the same course, now their son has to go to another college I think across the water to do same course after getting better grades..

But apparently that's all good say's Peter

[Jeepers Creepers]

Do these algorithms go back 25 years.

[Tony Baloney]

Do these algorithms go back 25 years.

it definitely screwed up my Physics result in 1994!

[CK_Redhand]

I don't comment here often, but at the risk of doxxing myself, here are my thoughts from a different forum.

I listened to Nolan this morning and felt the head of CCEA answered the questions very well and there were certain things he would like to have said, but couldn't as it would be a PR disaster.

1. Some of the anecdotes we heard were shocking but should be investigated a bit more before being taken as gospel. A couple of examples (trying not to disrespect these individuals, of course they must be angry this morning). A father saying his son never got a B grade in his life then later saying he thought he got one at AS level. A pupil wasn't sure how many were in his class (he thought it was 8 ). The St Cecilia's principal admitting the students had low class rankings, brushing over it, but not understanding why they got low grades.

2. Automatically upgrading every appeal, using predicted grades etc. like Scotland and Wales damages the credibility of the qualification. Teachers predict higher than deserved grades for many reasons, with extra incentive to do so this year, and using them would cause such a disproportionate number of higher grades. This would take away from the achievement of pupils who get that grade deservedly.

3. Nolan and the public in general don't understand how statistical models work. Explaining how is not easy, even to people with a good understanding of maths but I'll try... CCEA could only use 3 variables in their A level model: AS result, predicted result and class ranking. These are highly correlated, so much so that in a multivariate model some will have a reversal effect (negative coefficient when expecting a positive). They could remove the variable or manually fix coefficients up front, but all of these methods will result in anomalies, particularly when the results need to be standardised (eg. same/similar proportion of each grade as previous year). I'm sure CCEA tried their best but there is no perfect model or method to do this. They have tried to identify anomalies and make the schools aware, so that grades can be reviewed through the appeals process. Talking about particular cases on air wouldn't be appropriate for the head of CCEA to do.

4. At one stage Nolan was outraged that pupils grades were being balanced to NI nationwide previous results. Then 5 mintues later he was outraged that in a different model (GCSE) the results were being balanced to individual schools previous results. I don't know what Nolan wants to balance to if these are unacceptable to him.
...
Until the highly unlikely event of CCEA publishing more technical details of the model we have to make some assumptions. I didn't want to put too much in my initial post but here are some more considerations on the modeling side:

1. 25k pupils this year was the figure I heard for A level results. I'm open to correction on that. Let's say they used 10 years of data that's 250k observations on the training data. This is actually small for a predictive model eg. GLM and the associated error of predictions is larger when the sample size is small.

2. The predicted grade variable is likely to be of poor quality due to teacher optimism. Poor quality data affects the quality of predictions (garbage in, garbage out). Add to that the assumption that this year teachers were even more optimistic than in previous years, the predictions will have a higher margin of error. This affects the predictions of all pupils, bad and good.

3. Constraints such as the balancing/standardization one again makes for poorer quality predictions, especially for those at the boundary of grades.

In your example the difference may be within the margin of error given these constraints and assumptions. I don't think it's beyond the realms of possibility that the grades would change that much due to natural fluctuations from year to year.

CCEA can't do anything about these issues and I'm sure they tried their best to make the model "fair" but really they were put in an impossible position of having to provide grades in such circumstances.

[Milltown Row2]

Then why ask the teachers to rank them in the first place and give examples of why they have given the results if all they were going to do was use the model?

as for Nolan he'll look to have a 'story' regardless of whats being said

It was impossible for CCEA to keep everyone happy and just go with the predicted grades but it looks like they just dropped everyone a place.. so a low A* dropped to a A and a low B dropped to a C and so on

[Tony Baloney]

They haven't dropped everyone a place. My old school has their top achiever on Facebook already with 4 x A* and 1 A (AS). The missus said some school heads were on earlier and were delighted but equally Wallace looked like they were refusing to accept the results! It's a clusterfuck regardless.

[ONeill]

Then why ask the teachers to rank them in the first place and give examples of why they have given the results if all they were going to do was use the model?

as for Nolan he'll look to have a 'story' regardless of whats being said

It was impossible for CCEA to keep everyone happy and just go with the predicted grades but it looks like they just dropped everyone a place.. so a low A* dropped to a A and a low B dropped to a C and so on

Not as simple. Some pupils not dropped. Some by 1 grade. Some by 2 grades. Even by 3.

[Milltown Row2]

So bizarrely after speaking to the wife about her results, all her grades that she gave at AS and  A  level were kept?

I agree with Hardstation that yeah CCEA probably went "aye dead on" but it's the certain results that stand out are probably hard to swallow!

The colleges that gave out unconditional places have also fucked up as there are lots of students in there with results that wouldn't get them to tech! on a normal year

[ONeill]

Did boards not announce their system of awarding grades before predicted grades were asked for.

[David McKeown]

In the interests of full disclosure I make these comments as both a member of a board of governors and with plenty of relations who are either teachers or pupils this year. I have spoken to most of them as well as other governors.

The information provided about the statistical model is erratic. Some schools have been told that predicted grades were not factored in other than to check the accuracy of the model. I have been unable to confirm that.

The "anomalies" are plentiful. Examples I've seen included students with 11a* at GCSE, 4 at AS level and 4 in mocks now receiving 2A and 2B's with no explanation. Another pupil dropping three grades in one subject but going up 1 on another.  A departmental average dropping from 97% a*/a To 80%

The schools averages have fluctuated greatly too. I've spoken to teachers at schools whose results are down year on year despite the fact that comparisons between the GCSE and As results for this years Co-hort versus their equivalent predecessors would have predicted a double digit rise.

The statistical model doesn't seem to have allowed for teachers returning from illness, maternity leave etc.

Those for whom English is a second language seem to have been particularly hard hit. The fact they have a year or two more experience in the language will undoubtedly have had a major impact which doesn't seem to have been accounted for.

It doesn't seem to have Disproportionately impacted non grammar schools compared to grammar.

There's a real split between teachers over the efficacy of the system. Some though are concerned that if predicted grades were used instead then the teachers themselves may have been open to challenge.

The real problem I have with it is that In order to justify any individual's reductions you have to do so by reference to the model as a whole. For me that is unacceptable. When I got into uni it was on the strength of my grades not on the strength of the system that was in place. It's not fair now that a strong student is being denied that opportunity because of the outworking of afar from perfect statistical model. At the end of the day it's these individual kids who's lives are altered by this model.  If that system can't justify the discrepancies we are seeing (other than by arbitrarily suggesting that well as someone in a similar position to you had a bad exam so we assumed you might have this year) by reference to the individual then it is not fit for purpose.

I expect this to rumble on and I expect legal challenges. 

[CK_Redhand]

David, see my post earlier on this page for some of my thoughts from listening to the CCEA representative today.  I have a background in predictive model building and I am assuming they are using similar established techniques and best practices.

I will add that they would in fact probably have built a separate model for each subject, so that is why an individuals score could increase in one subject and decrease in another.

They deliberately would not have used the pupils school as an input as this would open up a whole other can of worms and accusations of bias and classism.  They couldn't have taken into account schools trends in previous years due to this.

They could not have used teacher maternity etc. data as they likely wouldn't have it, and even if they did they wouldn't have enough examples from previous years to get a statistically significant effect.

I too can forsee legal challenges and hopefully this will expose the inner workings of the models used.  Generally speaking these types of models are good at predicting things over a whole population rather than accuracy at an individual level.  I expect the models to be pretty weak with large margins of error.  They likely should not have been used but my guess is that a political decision was made hoping that this would cause the least disruption compared to England, Scotland and Wales.  They (the politicians) got that wrong and if any problems occur they can blame CCEA.

[David McKeown]

David, see my post earlier on this page for some of my thoughts from listening to the CCEA representative today.  I have a background in predictive model building and I am assuming they are using similar established techniques and best practices.

I will add that they would in fact probably have built a separate model for each subject, so that is why an individuals score could increase in one subject and decrease in another.

They deliberately would not have used the pupils school as an input as this would open up a whole other can of worms and accusations of bias and classism.  They couldn't have taken into account schools trends in previous years due to this.

They could not have used teacher maternity etc. data as they likely wouldn't have it, and even if they did they wouldn't have enough examples from previous years to get a statistically significant effect.

I too can forsee legal challenges and hopefully this will expose the inner workings of the models used.  Generally speaking these types of models are good at predicting things over a whole population rather than accuracy at an individual level.  I expect the models to be pretty weak with large margins of error.  They likely should not have been used but my guess is that a political decision was made hoping that this would cause the least disruption compared to England, Scotland and Wales.  They (the politicians) got that wrong and if any problems occur they can blame CCEA.

I had read that and thanks for it. My concern is using what is in effect a least worst option to determine something as important as the futures for these kids. To me If the model can't explain why on an individual basis someone went from all A*s in their entire school career to 2A's and 2B's a drop of 6 grades then it's not fit for purpose. The system didn't get rejected from medicine at her first two choice universities but she did. An appeal likely won't change that even if it corrects the anamoly. To me the inherent unfairness in that is shocking.