The New Netcode -Teatime

Sunday, 13 August 2000 2100Z

Revised: Tuesday, 15 August 2000 2000Z

Intro 

The following is a description of the netcode how I came to understand it. This is bound to be inconclusive (this one took far to long in the making) as well as being partially wrong. So please correct me where I’m wrong and point out what needs to be clarified or laid out more extensively. If possible I will try to continue revising this article. 

One thing I just learned today (hey, I'm an old geezer, things like that don't come easy to me) is that ping is actually the time a packet needs from client to server plus the time the answer of the server needs to reach the client (or the other way around). Latency is the basically the same, but also adds the time the server and client need to process the data (e.g. video and sound). So the player is behind the server roughtly half his ping.

And while we're at numbers: Every usage of ping- or latency times is just approximate. There are way too many factors influencing to traveltime of a packet or the processing of data on client or server to work with exact values. So be indulgent in that matter. 

The Basics

Before I start to discuss the netcode allow me to make some basic remarks on the relationship between server and client.

The center of all is the server. Each client is connected to the server and exchanges data exclusively with the server. At no time two clients communicate directly with each other.

Let us start with establishing a number of rules for the game:

1)      The server is authoritative on the game!

2)      The server runs the game in realtime!

3)      All incoming data from the clients are processed immediately in incoming order!

4)      All objects not controlled by the clients are controlled by the server!

Let’s give these rules a closer look:

1)  The server is authoritative on the game

The server is running the game and receives all data from the clients. The server will determine any kind of interaction between the map and a client as well as between two clients (or more).

So if one player shoots at another player the server will receive the data of both players and determine if one player was hit, what kind and how much damage was done and if the damage resulted in a death.

This has a number of implications, which I will explain later.

2) The server runs the game in realtime

Sounds pretty obvious. However, when dealing with a number of clients connected each with a different ping this is something which has to be clarified: Each process controlled by the server is handled in realtime regardless of the clients.

Example: The player stands in front of a door that is open, but will close automatically after a set time. The door is controlled by the server. So when the time is up the server will send the information of the door closing to the clients. But as the player is always behind the server due to his latency he might decide at that moment to step through the door, because the door still looks open to him. But in the time the information of his movement (him stepping through the door) needs to reach the server the door might have been closed enough so the player would not fit through. So the server would “reject” this movement and correct the client.

As result the player would see the open door, would push the button to move forward, would see the door start closing, would start moving through the doorframe to be stopped and probably moved back abruptly with the door closing in front of him.

So basically the state of the game displayed on the client is actually just a projection of the state on the server, distorted by “predictions” of the client (In the example: Walking through an apparently open door).

As long as latency is not too high and connectivity is good those two states will be close enough for the player not to notice the difference. Set up a server and play on a client in a LAN and likely you won’t even notice a thing.

But in our standard internet game we are not dealing with only one client and the server, we are dealing with a number of clients, each with its own latency and constancy in connection (read: chokes and packet loss).

By what has been stated above it should be obvious that no player has the same perception of the game. Differences might by marginally, but they exist. A closing door will be displayed in a different state of closing on each single client. And when your connection is not steady (like people on cable often experience) the difference between the reality on the server and the perception of this reality on the client might alter a lot.

3) All incoming data from the clients are processed immediately in incoming order

Should be obvious as well, doesn’t it?

But is doesn’t have to, necessarily, and in fact the new netcode is bending the rules in a way.

Let me give you an example how server/client-communication could be handled in a different way:

Imagine server and clients working with a synchronized clock. Now imagine each data created wearing a timestamp, with the server processing all incoming data not in the order of reception, but in chronological order according to the timestamps. In that model the order of reception would be unimportant.

This server would no longer work in realtime either, as he would have to delay all events to create the chronological follow-up of events.

While in theory this might works on LAN where latency is low it would terribly screw up the game on internet with common latencies of 200ms and more.

So what does the processing in incoming order mean for the game?

The lower your latency the earlier your data will arrive at the server. And the earlier you will receive the response from the server. That spawned the nickname Low Ping Bastard (LPB), because the LPB has the advantage of a “faster” communication with the server compared to players with a higher ping (consequently called HPB).

Example: Two snipers - one a LPB, the other one an HPB – aim at each other and shoot at the same time. The data of the LPB will reach the server earlier. The server will determine that the shot killed the HPB and will set the state of HPB to “dead”. So when the data of HPB arrives it will be dismissed, as it doesn’t fit the state HPB currently has (dead players can’t fire a shot).

Depending on the difference in latency the same would apply if HPB had shot before LPB as long as the data of LPB arrives at the server before the data of HPB does.

So if LPB would ping 50 and HPB would ping 300, HPB would have to at least shoot 126 ms (half the ping, remember?) earlier than LPB to get the kill (Just for the record: This is not completely correct, as it ignores the required processing time on the server).  

In the old netcode that required HPB-players to "lead" their shots. They had to take their latency into account and shoot at the point were they target supposedly will be the respective milliseconds later.  

This applied to all hitscan weapons, where a shot results directly in a hit without having a physical projectile. But with the introduction of lag compensation there is no longer the necessity to "lead" shots of hitscan weapons, though the rule of processing data in incoming order remains intact. More to hitscan weapons and lag compensation later.

While the same basics apply to non-hitscan weapons as well things are a bit different due to the nature of nonhitscan weapons. I will cover that also later.

4) All objects not controlled by the clients are controlled by the server

All map objects (lifts, door, etc) as well as all objects spawned by nonhitscan weapons (rockets, grenades, nails, etc) are controlled by the server. While this is pretty clear for map objects it might not be as obvious for the latter. A rocket – once fired – is handled by the server, not by the client who fired it.

Client Interaction

As said in the beginning the clients don’t communicate directly with each other. Each client sends his data to the server, which processes this data (as well as the data of the other clients) and sends this processed data to each client (including the one who sent the original data). I hope by now it’s clear that everything the client does is just a “suggestion” of what he intends to do. Any data reaching the server will be fitted into the state of the game on the server, verified for it’s validity (see the example with the door) and then examined for interaction with other objects of the game. If any interaction took place the server will alter the data accordingly.

Example: The player wants to make a step forward. But at this exact moment he is hit by a rocket. So the player will not step forward, but instead be thrown into the air.

In a low ping area this all happens pretty “naturally”. However, when the player has a high latency his client might display him making a step or two before the information that he was hit by a rocket arrives.

Why allow such a difference between the game on the server and the game on the client?

If each client would exclusively display the game based on the updates from the server the game would be unplayable for people with higher pings. Imagine pressing the forward key as someone pinging 300 only to have to wait more than 600 ms to start to move. 600 milliseconds don’t sound much, but it is well noticeable. To avoid that effect the game predicts the player to move, at the cost of the synchronicity with the game on the server and the displayed state on the client.

This is another reason why the player will have an altered perception of the game as it is factually taking place. The higher the ping the more the perception will differ from the game’s reality.

Another example to clarify this point:

Two HPB players (each with a nasty ping of 500) run into each other. Player A will receive the data of the position of player B the server had 250 ms ago, which – at this time – was already another 250 ms old. But player B has likely continued moving during that half second. Now if player A would want to shoot at player B he would have to consider that the position of B drawn on his client is already 250ms old compared to player B and that his shot-information would take another quarter second to arrive at the server. So basically he would have not to shoot at the position of the player model of player B as it’s displayed in his screen, but to predict where player B would likely be 250 ms later. (And if A would shoot a rocket on B he would even have to lead the shoot to make up for the travelling time of the rocket :-)).

So both players would see a completely different reproduction of the state of the game. A screenshot taken on both clients at the same time would show both player models in a different position.  

I should emphasize that actually the latency of the opponent is unimportant for this example, as the player always deals with the representation of the other players on the server. For that it doesn't matter if your opponent is 40ms or 400ms behind the server, his position on the server will still be the same. It only matters to anticipate the actions of the opponents.

Movement

As established movement is not effected by your latency as long as you don’t interact with other clients (shooting or colliding) or map objects. So when you join an empty server that gives you a high ping you will be able to run around pretty normally, because your client will predict your movement. However, try this on a server that gives you a ping of 1000 or more and going through doors will become an adventure of its own :-).

But in a regular game things get complicated. As the client due to his latency draws the position of every moving object incorrectly it is easy to collide with objects that seem to be in a safe distance. A HPB might collide with a LPB because on his latest update the LPB was running in a different direction, but in the meanwhile the LPB changed course.

For that reason the usage of hitweapons (commonly known as melee weapons) like the medkit, the knife or the wrench is difficult when the “victim” has a lower latency than the attacker.

Maybe that’s the reason why Valve decided to treat melee weapons like hitscan weapons and made them also work with lag compensation. More on that in the next chapter …

Hitscan Weapons

These are all weapons, which will generate an immediate hit without producing a physical projectile. In the same moment the trigger is pulled the hit is generated on the target.

In the old netcode it was handled as follows: The player pressed the attack-button, the data was sent to the server, the server determined if any shootable object was in the line of fire and would than determine the damage on the object and update the clients with the result.

Now imagine a player with a 300-latency shooting his sniper rifle on a 50-latency LPB. While the shot-information still travels to the server the LPB might step out of the line of fire. His movement information reaches the server before the shot-information does. So when finally the shot-information arrives the server determines that the shot has missed because the target was no longer in the line of fire.

This was always an unsatisfactory situation for the HPB’s because this offered the LPB’s an advantage. In the old netcode a LPB could step out behind a wall, see the enemy HPB-sniper aiming at him and retreat before the HPB had a chance to hit him because of his latency.

To make the field a bit more even between LPB’s and HPB’s Valve implemented an option called “lag compensation” to the new netcode, which by default is enabled on the servers.

That causes the server to take latency into account on certain events.

In the previous example it would work as follows:

The LPB steps out behind the wall. This movement information travels to the server and will be sent to the other clients, including our HPB sniper. The LPB sees the sniper and retreats behind the wall. Meanwhile the first movement information arrives at the HPB, his client will show him the LPB stepping out right into his sniper dot. HPB pulls the trigger. The shot information is send to the server. So now the server has basically a shot information on a target that is no longer in position. But lag compensation forces the server to look backwards in time to confirm that the target was in a valid position for the HPB client when the shot was done. As this was the case in our example the server will declare the shot hitting the target as valid.

So our LPB will have the unpleasant experience to have successfully retreated behind the wall just to be headshot a second later (just a number, this could be more or less, depending on a number of factors).

So lag compensation introduces in fact some kind of “timestamp”-operating system for the server/client communication, even though only covering a limited area of data and working backwards not to stall the gameplay. And the processing of data in incoming order remains unaffected. If an HPB is killed after having shot with a hitscan weapon this shot will be dismissed if the data of his kill arrives the server first. So an LPB sniper can still kill an HPB sniper in a face-off even when shooting a few milliseconds later, because his shooting informations reaches the server first. As the server determines the death of the HPB he will not process his shot at all.

Non-hitscan Weapons

But let’s move on to non-hitscan weapons. That’s the part where it gets really interesting.

As already mentioned non-hitscan weapons are all those weapons which actually create a physical projectile in the game. This category includes the rocketlauncher, the two nailguns, all grenades, the IC, the demo’s grenade launchers, the tranquilizer and the railgun.

Nonhitscan weapons - or rather the spawned projectiles - are not controlled by the player, but by the server. So when a player fires the RPG he will submit to the server that he pulled the trigger, accompanied by the information in which direction he looks and what angle the rocketlauncher has (in other words: trajectory). Then the server will create the rocket and will control its route and its impact. After pulling the trigger the shooting player will receive the updates about his rocket just as anybody else.

For starters this means that depending on his ping he will see the rocket launch with a delay. As all firing effects of the weapons itself (like sound, muzzle flash, ejecting shells, etc) are controlled clientside (assuming that cl_lw is set to 1) the player will hear his RPG firing, but he will not see the rocket being launched before the update from the server arrives.

Now if the player shot at is a HPB with a ping of 400 this player will get his first glimpse of the rocket 400 ms late. But during these 400 ms the rocket travelled on the server further in his direction. The player tries to evade and makes step aside. The data of his movement travels another 400 ms to the server while the rockets still flies. Remember that every object on the server moves in realtime! Let’s add the time the player needs to react and you have easily a whole second the rocket might travel before the player’s reaction reaches the server. On shorter distance this is more than enough time for the rocket to hit the player.

To give you something to relate to: A rocket needs approx. 6 seconds to travel from one base to the other in Well. So a firefight in the basement of 2fort is well inside the range for a rocket to hit an HPB-target during his pure latency-based reaction time.

If you take a fast-shooting weapon like the nailguns this also results in multiple projectiles shot before the first information of the shot reaches an HPB-target. An LPB-Medic sneaking up on an HPB-Engy can easily kill him before he has time to react, because the medic can fire several shots before the engy can even hear the SNG firing (which doesn’t mean that he’s been necessarily hit by these projectiles yet, but on the server they will be very close already and a number of them will hit before any movement update of the HPB will reach the server).

Things get very complicated with grenades (and nasty, too).

Grenades are handled by the server as well. When the player primes a grenade the clock doesn’t start ticking until the server received the prime-command. For our 400-ping HPB this means that he presses his grenade key, if he has a wav-based gren-timer it will starting to tick (or count, or beep, or whatever), but it will take 400 ms + before the priming symbol on his HUD will appear. However, at that time the grenade is already primed for 200 ms, because the priming took place when the server received the command. As a result that means that your timer will be off depending on your latency.

The same goes for throwing the grenade. The grenade will not be thrown until the server received the throw-command. In the meantime the timer still continues.

A high-pinger will have to keep this in mind, and release his primed gren early to avoid it exploding in his hand because his throw-command arrived the server late.

Hitweapons

Also referred to as Melee Weapons. These are all weapons that claim to make physical contact with the target, like the crowbar, the knife, the medkit and the wrench. Frankly I doubt that these weapons make indeed any kind of contact with the player models, I think they are more a specialized kind of  hitscan weapons, where a hit is also determined by the distance of the attacker to the target (besides direction and angle).

Hitweapons do consequently also use lag compensation. But the effects of lag compensation with hitweapons are even more irritating than the effects of normal hitscan weapons. When a HPB spy stabs his LPB victim he might as well evade before dying of the hit. Also a Medic might infect an HPB-player though for the HPB the medic seems to be in safe distance. But that is only because on the server the Medic is already close enough, but this update hasn’t reached the HPB yet.

Pros and Cons

The recent netcode is actually a pretty wild mix of different methods in handling client/server communication.

Movement and shooting-effects (like muzzle flash or sounds) are done immediately on the client, regardless of latency. Nonhitscan weapons are done rather “traditionally”, with the unavoidable side effects due to latency. Hitscan weapons and Hitweapons on the other hand are handled in some kind of 0-ping environment, achieved by backdating the respective events.

I guess it is this inconclusiveness of the mixture that causes so much reservation against the netcode. Most complaints however come from the LPB’s, many HPB’s seem to like the new netcode. But this is not surprising; as for an HPB the new netcode is in large parts an enhancement. The movement is much smoother now and the usage of hitscan weapons is more successful than before, when lag made things a lot more difficult (All the praise on pre-TF1.5-HPB-snipers :-)).

The major thing which changed for the LPB are the effects created by lag compensation, which become more striking the more the ping of both players involved differ.

There are more side effects of the netcode:

- People report of players moving with abnormal speed. I assume that – as movement is handled client side – when a player is lagged due to packet loss that once his updates arrive again at the server the server tries to correct his position based on the client’s data instead of the server’s – outdated – data. And instead of “teleporting” him to the corrected position it just speeds his movement up until the position on client and server are synchronized again.

- There are also reports of rockets speeding up or even changing course to hit a target. I have no idea what could cause this. Either it’s a bug in the netcode or another “feature” which I just haven’t understood yet.

Comments

What do I think about the netcode?

The netcode is far from being perfect. Most of its problems are originated in the attempt to compensate for some of the effects created by the differing latencies of players, which is done by different methods and succeed to a varying degree.

I think we can agree that it’s evident that Valve tried to create a more common ground between LPB and HPB (If the methods used are appropriate is another story). Some of the advantages brought to the HPB’s were achieved on cost of some aspects of the former gameplay of the LPB.

Was it worth it? I think yes, because IMO the disadvantages brought to the LPB’s are far smaller than what was gained for the HPB’s.

The major complaint of LPB’s are the effects caused by lag compensation. While I agree that it is very irritating to be hit when you assumed being covered, or killed by a grenade that appeared to be in a safe distance it doesn’t contradict the validity of those hits. Those complaining LPB’s should keep one thing in mind: If the person who fired the shot (or throw the grenade) had been a LPB as well he would have hit anyway. The major difference is that you will notice the damage a bit later. It’s not elegant, but it is definitely not unfair.

When I read some of the whining about lag compensation I can’t help but wonder what those people would do on a LAN-party, where they’d have no ping advantage at all …

People should keep in mind that the game is pretty complicated for an HPB. The behaviour of the game (and I hope I succeeded in describing some of the effects) forces some strange rules on the high pinging player. I don’t say this to make you LPB’s feel sorry for them. But maybe you should think of those disadvantages brought to you by the netcode as a small sacrifice to enhance the gameplay a bit for other players who probably cannot afford broadband connection or to whom it might not even be available. Your gameplay is still much better than the one they experience.

In conclusion: You can get used to the new netcode, but it doesn’t create a homogenous feeling for the events taking place. It’ not convincing as a whole, and it is noticeably a piece of work still under development. I doubt that this is the future of netcodes either, but it seems to be an acceptable compromise until other alternatives like Powerplay come up, especially assuming that Valve will hopefully smooth out some of the side-effects of the netcode.

Comments?

Email me or use the PlanetFortress Forums for discussion.

Once more a big “THANKS” to PainKilleR and Yahn for all the help  

Enough? Back to the Fort ...

Glossary

Chokes  

When a lot of things are happening around you the server will have to send you a lot of updates (imagine an open area with several other players and much action including grenades, rockets and nails). When the amount of data is more than your connection can handle then a few packets won’t arrive and the server will have to resend them.

Netgraph 3 will display choked packets (type “net_graph 3” in the console, without the quotes).

Back to text

Client

In our case the player, or rather the game running on his computer.

Back to text

Hitscan Weapons

All weapons that don’t spawn a projectile, but generate an immediate hit when shooting.

Hitscan weapons are the sniper rifle and the assault rifle, both shotguns, the AC.

Back to text

Lag Compensation

This method is used to make hitweapons and hitscan weapons ping-independent.

This is done by having the server not checking if anyone is in the line of fire when the firing-information reaches the server (which takes some time due to latency, time in which the target likely moved on), but instead checking if anyone was in the line of fire when the shot was fired on the client. To achieve this the server has to look back in time and check the player- and object positions the respective milliseconds ago of the latency of the shooting player.

So when the shooter would have a latency of 300 the server would check the positions all players and objects held 300 ms ago.

There’s a maximum of 500 ms for lag compensation to work, any data coming from someone with a higher latency would not be processed with lag compensation.

On the other hand there’s no minimum value for lag compensation to work. The shots of a 30-ping player will as well work with lag compensation as those of an HPB with a 400-latency. But of course high-pinging player will favour much more from lag compensation than LPB’s.

Back to text

Latency

This term describes the time needed for a data-packet to get from client to server and back (or vice versa) plus the time the recipient needs to process this data (including video and sound), measured in milliseconds.

Back to text

Hitweapons

Also known as melee weapons. These are all weapons that have to make “physical contact” with the victim to create a hit. Hitweapons are: Knife, crowbar, wrench and the medkit.

I’d like to file the flamethrower as well under hitweapons, as this has basically the same behaviour as the other hitweapons, though with a higher range.

Back to text

Non-hitscan Weapons

All weapons which spawn a physical projectile in the game.

Nonhitscan weapons are: Rocketlauncher, the two nailguns, all grenades, the IC, the demo’s grenade launchers, the tranquilizer and the railgun.

Back to text

Packet Loss

This refers to packets that don’t reach its recipient at all (either the packet got lost on its way through the net, or the recipient didn’t accepted the packet because the bandwidth was filled). The server will try to resend lost packets for a certain time until it’s outdated, and then try to send an updated packet.

Massive packet loss will result in a jerky game with players and objects warping to different positions.

Netgraph 3 will display lost packets (type “net_graph 3” in the console, without the quotes).

Back to text

Ping

Ping is the time a data-packet needs to get from one computer to the other in a network plus the time of the answer to get back, measured in milliseconds.

In the case of online gaming ping usually refers to the connection between server and client, though ping is an inaccurate value to measure the connection as it neglects the so-called processing overhead, which means the time each party needs to process the incoming data into a useful form. See also:  latency

Back to text

Server

The machine hosting the game. The server requires the necessary resources and bandwidth to run the game smoothly for the players. Insufficient bandwidth will just as well lead to chokes and packet loss in sending and receiving the data from and to the clients.

Since patch 1.1.0.0 server admins have reported a higher usage of resources (processing power and RAM). One reason could be lag compensation, as it likely forces the server to store all the player data for of the last 500 ms (which is the maximum value for lag compensation to work) for every client.

Back to text