PSX Landing Metrics for Pilot Scoring

[Jeroen Hoppenbrouwers]

I propose this solution.

1. If there is a tie, Matt decides.
2. If Matt is in the tie, Matt wins.



Hoppie

[Hardy Heinlin]

I'm happy to accept further counsel on tuning the the graphs keeping in mind the above provides the reasoning behind all the scoring graphs shown.

I understand. To make it more difficult, I would demand more precision on the pitch, less on the landing distance (autoland aims for 2500 ft and that's a fine landing). E.g. with flaps 30 the pitch should be 4.0°. If you get 3.0° or 5.0°, that's already very bad. 2° or 6° is nearly catastrophic. In my opinion, you are way too tolerant with the pitch, and way to strict with the landing distance. I think it should be vice versa.


Cheers,

|-|ardy

[Roddez]

The biggest question that hasn't been raised here is which runway is this judged on?  The PSX runway, which you can't see in the sim or the P3D runway that is projected in front of your face?

Makes a huge difference...

[Steve - Browny]

What range are you expecting pilots to be able to land at?
Are you expecting the top score on a given leg to be about 95%, 90%, 80%? When formulating this, you need to have an expectation of the distribution curve of all the pilots. You have a a couple of pilots get above 90%, most around a defined average of a score you expect an average pilot to be at (e.g. 60%) and then terrible landings below that with a pretty bad landing being between 0-10% and catastrophic being 0-1% (random 0.x number to give the pilot hope) This was the reasoning behind the current system and was tweaked as we had more data come in after WF2019 as it was only myself and Matt flying the landings.

If you get someone who walks in off the street, do you want them to get 10% even if they land it alright for their level?

Based on the data in the current  formulas in the spreadsheet, pilots are averaging 40% with a standard deviation of 14? IS this what you expect to see and how you want the numbers to be? i.e. on average only 33% of pilots would be getting above 50% if it was normally distributed?

What does 50% mean? Does it mean that you landed pretty well?

[David Palmer]

In my opinion, you are way too tolerant with the pitch, and way to strict with the landing distance. I think it should be vice versa.

Thanks Hardy for your time examining the graphs and I accept your reasoning and implement the requested amendments.
Question... should I apply the same penalty for both short and long landings from the touchdown point or differently? In other words, is a short landing better than a long landing?

The biggest question that hasn't been raised here is which runway is this judged on?  The PSX runway, which you can't or the P3D runway that is projected?

Great question... as many PSX aircraft participating in WF are using P3D, a method to derive runway position and dimensions within P3D would be required and applied to the scoring system. I've read good reports on the accuracy of MSFS with respect to this accuracy so I'm not sure if there would be a development effort to do this when MSFS is 'just around the corner' with multi-view capability.

Browny... are you directing your post at me for a reply or Hardy?

[Steve - Browny]

Browny... are you directing your post at me for a reply or Hardy?

For a reply but also for consideration by others. Saying it's a score out of 100% and only able to achieve 60% isn't useful. It's like with Olympics and the technique-type sports like gymnastics or diving - there are occasions where people score the perfect score but that doesn't necessarily mean that there needs to be finer adjustment of the process. Even if two competitors got 9.8, people still can share the same medal.

[G-CIVA]

Don't forget that with regard to the metrics measuring landing performance measured against runway centreline offset & TDZ ... this is measured inside PSX.

But within the Sim you are landing visually using a combination of the PSX generated PFD & ND symbology (inside PSX) & visual cues created by the scenery generator.

The position of the runway centreline & TDZ limits inside PSX & the scenery generator DO NOT match in any way to sufficient accuracy to fulfil the level of accuracy you guys are looking for a cross the spectrum of ARPTs WorldFlight chooses on its yearly routes.

Yes, some ARPT RWYs fo match up but take it from a bloke that flies PSX with p3D everyday .... they are few & far between. Personally, I don't think your ever going to get a clean automated solution to this. I'd bet $50 that MSFS won't be better than LM & is actually worse at RWY coordinate positioning.

There are too many "Human Factors" involved in what your trying to achieve. There is no substitute for that extra set of eyes in the back with his clipboard.

[Hardy Heinlin]

If you get someone who walks in off the street, do you want them to get 10% even if they land it alright for their level?

Taking into account the personal fitness is a good point. I think, after the flight, the landing pilot should blow into an alcohol test tube. Just multiply the landing score by the measured value. So when you see two equally perfect landings -- one with alcohol and the other without -- the former is certainly the bigger masterpiece.


Cheerio,

|-|ardy

[Britjet]

That backs up my well-tested theory Hardy ;-)
Peter

[David Palmer]

What range are you expecting pilots to be able to land at?

No range... It will be a reduction in the penalty either side of the touchdown POINT. You land on the touchdown point, you get no penalty. Ahead or behind the touchdown point incurs a proportional penalty as shown on the graph. There'll be an increase in the pitch penalty the same way.

Are you expecting the top score on a given leg to be about 95%, 90%, 80%?

A pilot could have performed very well getting zero deductions for the metrics of bank, crab & VREF but very poorly with nose gear transition time, thus applying a penalty. The amount of deduction is based on the variance of the metric. No variance, no deduction... but a proportional deduction is applied when metric variance is present as shown in the graphs. No deductions = A Perfect Score of 100%.

If you get someone who walks in off the street, do you want them to get 10% even if they land it alright for their level?

If they exceed each scoring metric by an absolute amount, then the deductions should arrive at score of 0%. If they manage to get at least 1 metric without an absolute deduction, then they'll have a score above 0%.

Based on the data in the current  formulas in the spreadsheet, pilots are averaging 40% with a standard deviation of 14? IS this what you expect to see and how you want the numbers to be?

There will be some tweaking needed and this process is underway with Hardy providing his feedback on the Landing Distance and Pitch metrics.

What does 50% mean? Does it mean that you landed pretty well?

We could just retain the pilot landing score percentage as a hidden variable and use it to only display pilot rankings based on the score, such as 1st, 2nd, 3rd, Nth, Last. We also provide a printout of the landing metric values achieved by the pilot so they determine why they received that ranking ahead or behind other pilots.
This would take away the focus of the landing scores issue.

... Saying it's a score out of 100% and only able to achieve 60% isn't useful.

It's not impossible to score 100%.
The score determines a single winner from a group of participants using a process that is automated, transparent & objective. If we just showed rankings, as previously mentioned, there would be no issues with the score.

It's like with Olympics and the technique-type sports like gymnastics or diving - there are occasions where people score the perfect score but that doesn't necessarily mean that there needs to be finer adjustment of the process. Even if two competitors got 9.8, people still can share the same medal.

Many of these sports determine the winner from a subjective process that uses human judgment like the event examples you've given. Also there are also sports that determine the winner through comparison to a previous score based upon an objective measurement (time/distance/mass) such as swimming, long-jump, weightlifting etc.
As said before, I'm looking for an objective process to arrive at the scores where it's very difficult to achieve duplicate scores, but not impossible.

[Martin Baker]

Is PSXinfo available to download? I can't find a link.

Thanks,

Martin

[Steve - Browny]

Is PSXinfo available to download? I can't find a link.

Hi Martin,

PSXInfo Download
Then download the PSXInfo.rar file

I've made a few changes which removes a lot of the function that it is no longer used for plus added in an option for if someone has an accurate scenery generator or even uses PSX's view out the window, that you can have the offset and threshold distance to be assessed.

Steve

[Martin Baker]

Fantastic - thanks so much for this, Steve.

Martin