PSX Landing Metrics for Pilot Scoring

[David Palmer]

Some history...

Here is the link to a forum post, proposed by 'mikeindevon' 5 years ago, for a 'Pilot Assessment' add-on for PSX.

http://aerowinx.com/board/index.php?topic=3180.msg32090#msg32090

This proposed add-on was to pass judgement on a pilots performance flying PSX.

The thread also made reference to software called FSiPanel that passed judgement for an approach and landing for a select few 'study level' aircraft.
This 'Landing Report' was based on the approach and landing metrics of Landing Distance, Deviation from Centerline, Touchdown R.O.D., G Force, Bank angle at Touchdown and Pitch at Touchdown. It also provided a graph of Localiser deviation & Glideslope deviation.
Each metric in the report gave its value and accompanying traffic light colored text grades such as "Satisfactory", "Long Landing", "Unsafe Landing", "closed to hard Landing", "Very Good", "Positive Landing" & "Perfect" just to name a few.

Unfortunately the software never gave an objective, numerical score that considered the approach and landing metrics altogether.

Fast forward to today...

I'm leading the development of PSX landing metrics, with Steven Brown, to judge participating Sydney WorldFlight 2020 pilots given the WF2019 event gave us 2 winners who had scores of 100%. It appeared that the landing metrics were quite 'soft' and didn't have the required discrimination within the metrics to provide better separation of future pilot landing scores and avoid multiple, high/perfect score winners.

I submit here, to 744 Forum members, the following 9 landing metrics along with their minimum, target and maximum values and their conditional use plus each metric weight for your consideration and feedback. FYI... landing either side of the target metric will give an increasing penalty to the score up to the minimum or maximum values shown on each landing metric. Regardless how much a pilot exceeds a minimum or maximum metric landing value, they will just receive the greatest penalty for that landing metric.

Also, please keep in mind that any poor landing WX (gust effects) is considered 'roll of the dice' randomness in the scoring process as we don't want to get too bogged down into catering for different WX events. Pilots just have to do their best given their landing WX conditions but with the applied metrics shown for later objective score calculation.

I'm also interested in your feedback on other landing metrics that could be obtained or derived from PSX and/or P3D data.

Current development is only for providing an objective landing score but if successful, it could later be extended to other phases of flight.

Thanks in advance for your feedback.

David & Steve

[Hardy Heinlin]

Some suggestions:

Main gear sinkrate:
On wet or short runways a hard landing is better than a soft one. Allow 300 or even 400 fpm.

Distance from threshold:
I would't put the target on a point but on a range; that is, the range from 1500 to 2500, or 1400 to 2600. Autoland aims at 2500, and that's not a fault. It's fine.

Pitch:
-4 is a crash landing, and 12 is a tailstrike. I would put the pitch minimum for flaps 30 on 2, max on 6. For flaps 25 min on 3, max on 7.

Bank:
Max 4, no matter what wind condition. Otherwise: Pod strike.

Speed:
Min VREF. Max VREF + 5 + wind corr.
The target is never VREF. It's always +5 minimum. VREF+30 for the max may be too high in most cases as it may activate the flap relief.


Cheers,

|-|ardy

[alcannata]

PSX Landing Championship inbound!

Aldo

[Jeroen Hoppenbrouwers]

No vol. %?

[G-CIVA]

Agree with Hardy.

You must take into consideration a Flap 25 or 30 landing.

In very gusty or windshear conditions you might opt for 25 rather than 30.

That changes the pitch dynamics.

Same with landing rate. No point greasing the thing on in a crosswind with a wet runway .... much better to plant it firmly in the TDZ.

As for the VREF ... you have to take into consideration the VREF + 5 additive. But what about +10, 15 or up to the maximum of 20 in conditions as I mentioned above?

[Britjet]

It is allowed (and assumed) to bleed off a +5 increment on VRef in the flare. Notwithstanding, not many pilots would do that.
Peter

[Hardy Heinlin]

That's why the airspeed you see on the Analysis page is recorded at 25 feet AGL already, unlike the other variables which are recorded on touchdown. In other words: The airspeed snapshot is taken at the end of the airspeed control phase. Thereafter, as you retard to idle, it's all about attitude, position and sinkrate, and the airspeed is no longer a primary parameter for the analysis.


|-|ardy

[Gary Oliver]

The Simfest boys will install it and take part in the 2020 World Flight Landing challenge too!

[Hardy Heinlin]

I think you guys need to take the wind conditions into your score formula by certain percentages:

• Crosswind component speed
• Crosswind component gust additive (very important)
• Headwind component gust additive (less important)
• Intensity of the turbulence layer (this is independent of gust settings)


|-|

[Steve - Browny]

Hi all,

Just for more background, the Worldflight team came up with the idea of having a board similar to the Top Gear fastest times and we just stick the best main gear touchdown rates on the side of the simulator. (Gotta have some sort of championship for the week ) This ended up being who could level out the most just before hitting the ground and using up the whole runway to flare. This didn't seem to capture the pilotage factor. I then started to come up with various simple fomulas using the touchdown variable that PSX generates and Matt and I just flew a lot of approaches which he would rate his own landings out of 100 subjectively. I would then tweak the formulas and it got pretty close to what he was thinking that landing should be. Currently, the landing scoring is from 0% to 100% with 100% being a perfect landing and was tweaked during WorldFlight (although not rolled out) to rule these out. It is calculated to 2 decimal places and from the 44 legs of WorldFlight gives a 9.4% chance of two pilots scoring the exact same score.

Here's a link of the software provided as is if you wanna give it a go. It is provided as is and was originally created to upload the aircraft's position to the website for our map even if we got disconencted from VATSIM (hence the FTP settings). It even prints out a score sheet for you to take home, frame it on the wall  and boast to your friends about when they come over...
PSXInfo


It currently doesn't take into account wind effects but there are null zones. e.g. touchdown zone or anywhere for the main gear between 0 and 100 fpm gives a 0 deduction from 100% etc. We had some issues with scenery not lining up with PSX so offset from the centreline and threshold wasn't taken into account for WorldFlight. I have turned it on for this download in case others have better scenery. This is currently the scoring system that is being used while David thinks we can improve it. By the way, if you use autoland... you're not gonna win.

Because the values were recorded, we can to tweak it and look at the distribution and compare it.

[James H]

So Browny has both the best and the worst score...interesting 

[Jeroen Hoppenbrouwers]

It's so wonderful seeing professionals trying to come up with stupid bar games, and failing at it   :-)

[alcannata]

Runway length and width should be taken into account.
Eg, a 25 feet lateral offset from centerline is different at Mykonos as compared to KSFO 19L.

Aldo

[DougSnow]

Dammit - so wish I could get down there this year....

[Hardy Heinlin]

I forgot to mention that the parameter "Speed deviation at 25 ft" refers to the MCP bug, not to the VREF setting in the FMC. The analyzer assumes the pilot has set the MCP bug to VREF-plus-whatever and that this is the absolute target. The recommended allowed deviation range of 0 to +5 kt refers to this target.


|-|ardy

[Gary Oliver]

All,

BACARS V5.1.0 now integrates with PSXINFO to do the following :-

-Add the crews details and flight information to the Print Out PSXInfo spits out with the performance data.
-Takes the PSXInfo report and adds it to the PSX Landing Data Leaderboard.

The landing competition board is then available at:-

https://planning.simfest.co.uk/landingboard?username=simfest

replace 'simfest' with 'qfa' or 'gti' for the QFA and GTI world flight teams.

Replace 'simfest' with your BACARS email address for your own landing competition board.

All automatic, no manual work required, just run PSXINFO and BACARS 5.1.0 and above.

Cheers
Gary

[GodAtum]

Thanks Gary, but my score isn't showing up. Do I need to send my completed flight?

[David Palmer]

Thanks to everyone who replied.

Main gear sinkrate:
On wet or short runways a hard landing is better than a soft one. Allow 300 or even 400 fpm.

Reply1: Amended to include MGS of 350 fpm for wet/icy runways. Runway length currently not considered but may be considered in the future.

>Distance from threshold:
>I would't put the target on a point but on a range; that is, the range from 1500 to 2500, or 1400 to 2600.

Reply2: I need to provide the target on a point so I can discriminate scoring either side the 1500 touchdown point.

>Pitch:
>-4 is a crash landing, and 12 is a tailstrike. I would put the pitch minimum for flaps 30 on 2, max on 6. For flaps 25 min on 3, max on 7.

Reply3: The min/max pitch value provides an absolute input either side of the target value represents the worst case metric where exceeding this value just provides a static value. Yes -4 is a crash & 12 is a tailstrike but I need to calculate how 'bad' a crash or tailstrike is to derive the final score.

>Bank:
>Max 4, no matter what wind condition. Otherwise: Pod strike.

Reply4: As with pitch, I'm seeking worst values of already bad situations to asist with score discrimination so amending min/max values to -5/+5.

>Min VREF. Max VREF + 5 + wind corr.
>The target is never VREF. It's always +5 minimum. VREF+30 for the max may be too high in most cases as it may activate the flap relief.

Reply5: I read where smoothly closing the throttles from 30', reaching idle at touchdown, should result in a VREF landing in nil wind conditions so amended to VREF+5 and added a 'Gusty' consideration to the metric with wind correction.

>No vol. %?

Reply6: If I've interpreted what you say correctly, I see that as a 'covert' option. 

>As for the VREF ... you have to take into consideration the VREF + 5 additive. But what about +10, 15 or up to the maximum of 20 in conditions as I mentioned above?

Reply7: See Reply5 above.

>I think you guys need to take the wind conditions into your score formula by certain percentages

Reply8: Done... except for turbulence layer. Will look to implement with a view to a turbulence score multiplier.

>Runway length and width should be taken into account.

Reply9: Width is considered with centreline offset. See Reply1 regarding runway length.

Where to next...

Transparency for the competing pilots is important so here are the target graphs displaying the percentage deductions that are applied to each landing metric.
The 'Landing Score' is expressed as a percentage after the deductions are applied with a perfect score being 100%.

Some proving flights in the sim will be done to allow for some fine tuning of the metric scoring formulas during October.

[Hardy Heinlin]

Hehe, that landing distance penalty looks draconic :-) Any chance to make this curve convex instead of concave? Or at least linear?


|-|

[David Palmer]

Hmmm....
If I do that, I don't have discrimination between very close together landings near the 1500' touchdown POINT. We provided a touchdown ZONE in WF2019 scoring and many skilled pilots were hitting in this zone. No wonder we had a tally at the end of the event with 2 winners, both with scores of 100%.

I'm trying to avoid this at WF2020 by having greater penalty discrimination either side of the touchdown point and lessening penalty for our other pilots who may not have the best touchdown aim.

Making the curve convex then benefits the pilots with good aim as there is less penalty discrimination and a linear relationship doesn't assist those pilots with poor aim skill. I'm happy to accept further counsel on tuning the the graphs keeping in mind the above provides the reasoning behind all the scoring graphs shown.