744 Forum

Apron => Hangar 7 => Topic started by: GodAtum on Sun, 3 Jan 2021 16:01

Title: PSX Server redundancy?
Post by: GodAtum on Sun, 3 Jan 2021 16:01
I was wondering how professional customers make the PSX server redundant, so if it or the PC (by that I mean physical machine, VM or operating system) fails, it will carry on running? At the moment, if the PC running PSX server crashes, it will take the PSX server down with it, which must be unacceptable downtime in a professional environment with customers paying by the hour?

All I can think of is running another PC that takes a copy of the autosave situ file every 7 minutes, so if the 1st PSX server PC fails, one can start a PSX server on the 2nd PC with the latest situ file quickly, thereby minimising downtime.

Title: Re: PSX Server redundancy?
Post by: Gary Oliver on Sun, 3 Jan 2021 16:11
Having run PSX in an exceptionally complicated Sim environment since its release I have had 2 PSX server crashes..... Both of these with badly written addons or doing something entirely stupid when testing out stuff.

Since adding SimStack Router in front of the PSX server on port 10746 so anything 3rd party talks to that and the PSX instances all talk to each other on 10747 I have not had a single PSX server issue.

I think you might be overthinking a problem that doesn't exist.

Regarding OS or hardware failures, not a single one on the PSX server.  Its still running a windows 8.1 install with all updates disabled that was installed when I installed the PC in the sim.

Windows 10 LTSB with patching disabled on all other PC's.


Cheers
G
Title: Re: PSX Server redundancy?
Post by: Jeroen Hoppenbrouwers on Sun, 3 Jan 2021 16:38
I'm not even sure whether professional simulators have any true redundancy. Yes they are expensive to run, and there are most certainly safety features, but nobody gets hurt when the thing freezes in mid-vlight.

Furthermore I don't know whether there are any pro sim setups (commercial) running PSX? That would be interesting to know.

Lastly a lot of pro PSX use is classroom education, and the teacher's PC most certainly won't be redundant.


Hoppie
Title: Re: PSX Server redundancy?
Post by: Hardy Heinlin on Sun, 3 Jan 2021 17:57
Some airlines use PSX also for a multimonitor cockpit mockup.
Title: Re: PSX Server redundancy?
Post by: GodAtum on Sun, 3 Jan 2021 19:10
Thanks for the info all, very interesting.

The reason I ask is that I've finally got round to fixing my setup this holiday. Either, on 50% of flights, the PSX server window goes all brown, and I have to force close it, then restart.

Or the PSX server keeps running, but the clients go out of sync. So they say they are connected but all the data is out of sync. When I try and reconnect, it says no server found on port 10747. When I run a Powershell command "Test-NetConnection <host> -port 10747" it errors. When I try to restart the server in the instructor station window it errors saying port 10747 is not available. The only way to fix this is to restart PSX entirely.

I am using Java 1.8.0_231 on Windows 10.

Hope someone with a lot more knowledge can offer some advice  :D
Title: Re: PSX Server redundancy?
Post by: Hardy Heinlin on Mon, 4 Jan 2021 04:55
GodAtum, for a test, can you make some flights without any add-ons? Just use your computers to run some networked PSX instances. Pure PSX server with pure PSX clients. No routers, no add-ons.

Will that fail as well on 50% of flights? If not, you may continue the test with more and more add-ons and then see at which additional add-on the problem starts.


|-|ardy
Title: Re: PSX Server redundancy?
Post by: GodAtum on Mon, 4 Jan 2021 08:45
Quote from: Hardy Heinlin on Mon,  4 Jan 2021 04:55
GodAtum, for a test, can you make some flights without any add-ons? Just use your computers to run some networked PSX instances. Pure PSX server with pure PSX clients. No routers, no add-ons.

Will that fail as well on 50% of flights? If not, you may continue the test with more and more add-ons and then see at which additional add-on the problem starts.


|-|ardy

Good idea, I'll do that thanks  :D
Title: Re: PSX Server redundancy?
Post by: GodAtum on Mon, 4 Jan 2021 22:12
PSX worked Ok with no addons.

When I added SimStack with the FDS Glare software, the client got out of sync. On the client GUI, when I try to change a dial like the heading or alt, it won't update the value on the client, but will on the server. Same with the CDU, if I press a button, the display updates on the server but not on the client.

On the client, when I stop and start the server connection, it fails with "No server found on port 10747". I then need to restart the Server for it to resync.

Like Gary, I have my PSX client pointed to the server. And the addons like FDS Glare pointed to SimStack.
Title: Re: PSX Server redundancy?
Post by: Jeroen Hoppenbrouwers on Mon, 4 Jan 2021 23:05
We will need a bit more detail on what is connected to what. If you say "client", you probably mean the PSX Client. But where is the SimStack software located? Does that drive the PSX Client from hardware? Does the hardware lose sync, or the client, or both? Is the SimStack software controlling SimStack hardware, or is the FDS Glare something completely independent?

A simple block diagram may help here.

Hoppie
Title: Re: PSX Server redundancy?
Post by: GodAtum on Tue, 5 Jan 2021 09:32
At the moment, I'm just testing psx-server (with only PSX server & SwitchPSX running), psx-boot and avionics-a (with only PSX client & FDS Glare running).

(https://i.imgur.com/tKQaQTr.png)



hoppie added the width=700 configuration
Title: Re: PSX Server redundancy?
Post by: GodAtum on Wed, 6 Jan 2021 16:11
Doing some more testing. When using only the PSX Server and PSX clients, the PSX Server connection dies after an extended period.

As you can see from the screenshot, PSX server is running, but 127.0.0.1 port 10747 refuses connection. When I restart the PSX server, I can then make the connection. Windows firewall is completely off.

(https://i.imgur.com/TITs4Sq.png)
Title: Re: PSX Server redundancy?
Post by: Hardy Heinlin on Wed, 6 Jan 2021 16:30
"127.0.0.1" is "localhost".

PSX clients connect to "localhost" if they are on the same computer as the PSX server.

Are your PSX clients on the same computer as your PSX server?
Title: Re: PSX Server redundancy?
Post by: GodAtum on Wed, 6 Jan 2021 16:35
Quote from: Hardy Heinlin on Wed,  6 Jan 2021 16:30
"127.0.0.1" is "localhost".

PSX clients connect to "localhost" if they are on the same computer as the PSX server.

Are your PSX clients on the same computer as your PSX server?

I'm keeping it simple and just testing the local connection on the same PC in the screenshot. When trying to connect to 127.0.0.1 port 10747, it is refused, even though the PSX server is running.
Title: Re: PSX Server redundancy?
Post by: Hardy Heinlin on Wed, 6 Jan 2021 17:45
What's in Aerowinx/Logs/errors.txt?
Title: Re: PSX Server redundancy?
Post by: GodAtum on Wed, 6 Jan 2021 18:22
It's empty.
Title: Re: PSX Server redundancy?
Post by: Hardy Heinlin on Thu, 7 Jan 2021 02:07
On your screenshot I see 4 client buttons. The 4th client has the number 1000. What kind of client is that? It disconnected and reconnected 1000 times during this test run. For performance reasons, the PSX server doesn't allow more than 1000 connections. It will deny further connection attempts. To reset this limit, you need to cycle the checkbox to "Disconnected" and back to "The main server".

It's not normal when an add-on connects thousands of times within a few hours. Is this add-on on the same computer? If not, are the network wires OK?


|-|ardy
Title: Re: PSX Server redundancy?
Post by: Jeroen Hoppenbrouwers on Thu, 7 Jan 2021 10:21
Extremely useful feature, Hardy. Consider coloring the button differently when this limit is reached, as it is too easy to overlook if you don't know it's there.


Hoppie
Title: Re: PSX Server redundancy?
Post by: Hardy Heinlin on Thu, 7 Jan 2021 11:32
Good idea ...
Title: Re: PSX Server redundancy?
Post by: GodAtum on Sun, 10 Jan 2021 18:16
Thanks so much Hardy, it seems obvious now!
Title: Re: PSX Server redundancy?
Post by: Markus Vitzethum on Tue, 12 Jan 2021 14:51
Hi Hardy,

> The 4th client has the number 1000. What kind of client is that? It disconnected and
> reconnected 1000 times during this test run. For performance reasons, the  PSX server
> doesn't allow more than 1000 connections. It will deny further connection  attempts.
> To reset this limit, you need to cycle the checkbox to "Disconnected" and back to "The
> main server".

interesting - I've experienced this issue before. It is documented in this message which includes a video:

https://aerowinx.com/board/index.php?topic=3629.msg61207#msg61207

In my situation the affected add-on was Externalsim.
However, I don't see Externalsim in Godatum's setup diagram (using WidePSX instead).

What can possibly go wrong that different external programs fall into a reconnect loop?

Markus

Title: Re: PSX Server redundancy?
Post by: Hardy Heinlin on Tue, 12 Jan 2021 15:50
Quote from: Markus Vitzethum on Tue, 12 Jan 2021 14:51
https://aerowinx.com/board/index.php?topic=3629.msg61207#msg61207

In my situation the affected add-on was Externalsim.
However, I don't see Externalsim in Godatum's setup diagram (using WidePSX instead).

What can possibly go wrong that different external programs fall into a reconnect loop?

The question is, which instance causes the disconnection -- the PSX server or the add-on?

I can only tell what the PSX server does. The PSX server disconnects the client in any of the following cases:

• When PSX streams a message (always done by Java.io.PrintWriter.println) to the client and the following Java.io.PrintWriter.checkError (https://docs.oracle.com/javase/6/docs/api/java/io/PrintWriter.html#checkError()) is true. Note that "print" in this context doesn't refer to paper print but to a text in a buffer that streams into the network. I guess the buffer may get full when the client doesn't read the network messages. In that case, PSX assumes the client is dead and thus disconnects.

• When the user clicks the Disconnect button or Stop button etc.

• When the add-on sends "exit".

• When the buffered reader in PSX catches an i/o exception. Such errors are indicated in Aerowinx/Logs/errors.txt


The add-on may reconnect automatically. When the error reoccurs, it will be disconnected. And so on.


Regards,

|-|ardy
Title: Re: PSX Server redundancy?
Post by: Jeroen Hoppenbrouwers on Tue, 12 Jan 2021 18:56
Quote from: Markus Vitzethum on Tue, 12 Jan 2021 14:51
What can possibly go wrong that different external programs fall into a reconnect loop?

Bad Ethernet, bad Wifi, router dropping TCP tables, annoying firewall.

Add-on code that breaks on something that PSX sends always on reconnect. This may be a particular value of some variable that the add-on cannot cope with, but that is rare to occur. Once PSX has this value, the add-on is doomed.

It's good practice for an add-on to have a brief (10 seconds) pause between connection retries. Not only to reduce the load on PSX and prevent running up the connection counter quickly, but also to allow a human to actually see the disconnect in some way. If the add-on goes on behind the scenes, nobody will ever notice what should be a rare event that indicates something went amiss.


Hoppie
Title: Re: PSX Server redundancy?
Post by: Hardy Heinlin on Sun, 14 Feb 2021 01:00
PSX update 10.130 now shows a red message on Instructor > Network > Main when more than 1000 connections have been attempted. See item 130.10:

https://aerowinx.com/board/index.php?topic=4191.0


|-|ardy