Netcode in Rust with ECS.
Introduction
Online multiplayer games require accurate, precise and sophisticated code libraries that enable a medium to high amount of physics based entities and/or players in the game world without running into the infamous network bottleneck. The video-game has to be developed as such that physics gets calculated authoritatively on the server while clients aim to replicate and locally simulate the same world with precision, accuracy and with the ability to support a variety of latency scenarios while latency is ever-changing and dynamic. The latency of a connection is usually decided by geographic locations, the closer a client is to the server it connects to, the lower the latency. A problem with sending and receiving data over the network is that there is always some latency between when a message was sent and when it was received on the other end. When the server sends positional data of an entity the clients will all receive it several milliseconds later. When you aim to replicate physics simulations in real-time across the net you need to take into account all these problems.
Thankfully there have been useful resources on these issues. A very helpful YouTube playlist explaining the netcode model I have implemented is found here.
Netcode showcase videos
Media I: Mid-high latency test, ping differs between 20ms-70ms~ involves a server hosted in a datacentre located in different country and a VPN connection to intentionally worsen net conditions
Media II: Low latency test.
The entire netcode framework is open source and can be found on Github.
Plugins
At the time of writing Space Frontiers uses a fork of the renet plugin which provides a UDP-based standardized netcode game library that would offer things like DDOS protection with game server providers.
For physics there is also a native plugin called bevy_xpbd. The fact that this plugin is native and integrated with the game engine's ECS is great because it allows for some epic multi-world and multi-schedule processing.
Tickrates and syncing
The default tickrate of Space Frontiers is 60hz. This means that every second 60 game loop steps get made at fixed intervals that step and progress the delta time of the world and physics simulation. One tick per 12 milliseconds. Each tick that gets processed has a unique integer ID/stamp. When server and client send messages to one another they also send the stamp of the tick they're at. This way for each message that is received we can determine how much latency (in ticks) there is.
For each new connection the server sends the current tick ID its at and the client copies that integer and also steps the game loop from that stamp on.
We put the client a few number of ticks ahead of the server. The amount clients are ahead depends on the message latency the client has with the server. This way when a client sends a message for tick x while the server is at tick x-latency. So when the server receives and processes the message its at tick x and the client input can be accurately inserted into the server game loop with precise timing and the client will not experience any jitter or teleporting.
The network is ever-changing so latency and increase and decrease per connection over time. This means that the amount of ticks at which the clients should be ahead of the server is subject to change. When the server detects a latency change it will make sync adjustment requests to the client which will then either freeze for a specified amount of ticks or speed up and go ahead in time.
Latency
Clients obtain and send user input such as movement keys and mouse input forward to the server and the server then forwards it to all the connected peers that can see the player that sent the input. Since the client is put ahead of the server equal to the amount of latency and sending a message to the client also takes that amount of latency to arrive: we receive those peer inputs at double the latency. This means when the server forwarded peer input at tick X, the client will receive it when it is at tick X + latency in ticks * 2. This means that the peer input message the client receives is several ticks old. If we were to just take that data and apply it without taking this fact into account we would be randomly applying data in each tick and the physics simulation would not go accurately and it wouldn't look smooth on the users screen.
Client-side input & physics caching, rollback and prediction
Clients cache the physics data of each entity for every tick and store it in a Resource so it can spin up a new physics simulation, clone this data and even add a set changes (ie position or input updates from the server). So the client always receives authoritative data from the server a few ticks behind the tick it is currently at. When we receive new data from the server we can go back in time to the tick contained by the message header and step the simulation to the tick the client is currently at and then apply those to the main world.
Since the core principles of ECS and Rust are data-driven it is actually relatively easy to cache everything and to restore previous physics states. It is simply a matter of querying physics components from physics entities and storing them in a map linked by entity and tick ID as keys.
Essentially we can now predict the future and go back in the past. And they said magic wasn't possible..
![[Image: cache_copying.png]](https://gitlab.starwolves.io/starwolf/forum-images/-/raw/master/space_frontiers/journal/netcode_and_ecs/cache_copying.png)
![[Image: input_debug.png]](https://gitlab.starwolves.io/starwolf/forum-images/-/raw/master/space_frontiers/journal/netcode_and_ecs/input_debug.png)
Media III: physics cache copying. Media IV: Fun times debugging peer input.
Consistent processing
To ensure consistent processing the client and server send messages in batches. Both the client and server process one message batch per tick (per client).
This way the event loops stay consistent and no unexpected results happen when multiple batch messages have received for a single tick to process, the software will simply split those messages up and call them in the upcoming ticks from a queue.
SubApps and multiple Worlds
The physics rollback and prediction happens inside a SubApp. Which means that the systems and data are separated from the main world that gets rendered. This way we have two ECS Worlds. In a single step of the main world we can step this SubApp correction world for any amount we desire. A correction is triggered by providing a start and end tick id and the correction SubApp will step this range of physics calculations and return the results. All we do is pass the caches of physics entities and player input to the correction and let it run the physics steps as fast as possible. There are some sophisticated systems written to do this properly. Because entity spawning and despawning has to be emulated in the correction world too. We use the same worker threads as the main app, reducing thread overhead. Both worlds execute asynchronously but as is standard in any Bevy application the inner logic of each world is entirely parallelized through systems. There is also a synchronous implementation of physics correction, but it is unfinished and causes jitter. However in theory it is possible to run correction physics steps completely synchronously to the main game loop for great performance gains, the down-side is that you would get one tick of extra latency.
Optimizations
The gridmap is split up into chunks in terms of data and collision. There is a compound gridmap collider for each populated gridmap chunk.
The correction SubApp is lean and runs a skeleton Plugin selection. This is important because with another player in the scene the client could easily be tasked to rollback and step several ticks worth of physics per client tick. At the default 60hz tickrate this means clients are expected to rollback hundreds of physics steps per second to achieve smooth results with a narrow execution time of 12 milliseconds per frame.
There are still several big steps that will eventually be made to optimize physics rollback, such as finishing the synchronous execution of the rollback World and reducing the amount of cache data that is cloned and sent between Worlds.
Afterword
Up next is integrating more 3D assets, like 3D character models, animations and hopefully to expand the gridmap with higher fidelity assets and to make the current default map larger.
A pioneering first-person real-time low-bandwidth netcode replication framework with Bevy Engine.
By RamsesIntroduction
Online multiplayer games require accurate, precise and sophisticated code libraries that enable a medium to high amount of physics based entities and/or players in the game world without running into the infamous network bottleneck. The video-game has to be developed as such that physics gets calculated authoritatively on the server while clients aim to replicate and locally simulate the same world with precision, accuracy and with the ability to support a variety of latency scenarios while latency is ever-changing and dynamic. The latency of a connection is usually decided by geographic locations, the closer a client is to the server it connects to, the lower the latency. A problem with sending and receiving data over the network is that there is always some latency between when a message was sent and when it was received on the other end. When the server sends positional data of an entity the clients will all receive it several milliseconds later. When you aim to replicate physics simulations in real-time across the net you need to take into account all these problems.
Thankfully there have been useful resources on these issues. A very helpful YouTube playlist explaining the netcode model I have implemented is found here.
Netcode showcase videos
Media I: Mid-high latency test, ping differs between 20ms-70ms~ involves a server hosted in a datacentre located in different country and a VPN connection to intentionally worsen net conditions
Media II: Low latency test.
The entire netcode framework is open source and can be found on Github.
Plugins
At the time of writing Space Frontiers uses a fork of the renet plugin which provides a UDP-based standardized netcode game library that would offer things like DDOS protection with game server providers.
For physics there is also a native plugin called bevy_xpbd. The fact that this plugin is native and integrated with the game engine's ECS is great because it allows for some epic multi-world and multi-schedule processing.
Tickrates and syncing
The default tickrate of Space Frontiers is 60hz. This means that every second 60 game loop steps get made at fixed intervals that step and progress the delta time of the world and physics simulation. One tick per 12 milliseconds. Each tick that gets processed has a unique integer ID/stamp. When server and client send messages to one another they also send the stamp of the tick they're at. This way for each message that is received we can determine how much latency (in ticks) there is.
For each new connection the server sends the current tick ID its at and the client copies that integer and also steps the game loop from that stamp on.
We put the client a few number of ticks ahead of the server. The amount clients are ahead depends on the message latency the client has with the server. This way when a client sends a message for tick x while the server is at tick x-latency. So when the server receives and processes the message its at tick x and the client input can be accurately inserted into the server game loop with precise timing and the client will not experience any jitter or teleporting.
The network is ever-changing so latency and increase and decrease per connection over time. This means that the amount of ticks at which the clients should be ahead of the server is subject to change. When the server detects a latency change it will make sync adjustment requests to the client which will then either freeze for a specified amount of ticks or speed up and go ahead in time.
Latency
Clients obtain and send user input such as movement keys and mouse input forward to the server and the server then forwards it to all the connected peers that can see the player that sent the input. Since the client is put ahead of the server equal to the amount of latency and sending a message to the client also takes that amount of latency to arrive: we receive those peer inputs at double the latency. This means when the server forwarded peer input at tick X, the client will receive it when it is at tick X + latency in ticks * 2. This means that the peer input message the client receives is several ticks old. If we were to just take that data and apply it without taking this fact into account we would be randomly applying data in each tick and the physics simulation would not go accurately and it wouldn't look smooth on the users screen.
Client-side input & physics caching, rollback and prediction
Clients cache the physics data of each entity for every tick and store it in a Resource so it can spin up a new physics simulation, clone this data and even add a set changes (ie position or input updates from the server). So the client always receives authoritative data from the server a few ticks behind the tick it is currently at. When we receive new data from the server we can go back in time to the tick contained by the message header and step the simulation to the tick the client is currently at and then apply those to the main world.
Since the core principles of ECS and Rust are data-driven it is actually relatively easy to cache everything and to restore previous physics states. It is simply a matter of querying physics components from physics entities and storing them in a map linked by entity and tick ID as keys.
Essentially we can now predict the future and go back in the past. And they said magic wasn't possible..
Media III: physics cache copying. Media IV: Fun times debugging peer input.
Consistent processing
To ensure consistent processing the client and server send messages in batches. Both the client and server process one message batch per tick (per client).
This way the event loops stay consistent and no unexpected results happen when multiple batch messages have received for a single tick to process, the software will simply split those messages up and call them in the upcoming ticks from a queue.
SubApps and multiple Worlds
The physics rollback and prediction happens inside a SubApp. Which means that the systems and data are separated from the main world that gets rendered. This way we have two ECS Worlds. In a single step of the main world we can step this SubApp correction world for any amount we desire. A correction is triggered by providing a start and end tick id and the correction SubApp will step this range of physics calculations and return the results. All we do is pass the caches of physics entities and player input to the correction and let it run the physics steps as fast as possible. There are some sophisticated systems written to do this properly. Because entity spawning and despawning has to be emulated in the correction world too. We use the same worker threads as the main app, reducing thread overhead. Both worlds execute asynchronously but as is standard in any Bevy application the inner logic of each world is entirely parallelized through systems. There is also a synchronous implementation of physics correction, but it is unfinished and causes jitter. However in theory it is possible to run correction physics steps completely synchronously to the main game loop for great performance gains, the down-side is that you would get one tick of extra latency.
Optimizations
The gridmap is split up into chunks in terms of data and collision. There is a compound gridmap collider for each populated gridmap chunk.
The correction SubApp is lean and runs a skeleton Plugin selection. This is important because with another player in the scene the client could easily be tasked to rollback and step several ticks worth of physics per client tick. At the default 60hz tickrate this means clients are expected to rollback hundreds of physics steps per second to achieve smooth results with a narrow execution time of 12 milliseconds per frame.
There are still several big steps that will eventually be made to optimize physics rollback, such as finishing the synchronous execution of the rollback World and reducing the amount of cache data that is cloned and sent between Worlds.
Afterword
Up next is integrating more 3D assets, like 3D character models, animations and hopefully to expand the gridmap with higher fidelity assets and to make the current default map larger.